Updates from: 08/06/2022 01:10:15
Service Microsoft Docs article Related commit history on GitHub Change details
active-directory-b2c Contentdefinitions https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory-b2c/contentdefinitions.md
The following example shows the content definition identifiers and the correspon
<DataUri>urn:com:microsoft:aad:b2c:elements:contract:providerselection:1.2.1</DataUri> </ContentDefinition> <ContentDefinition Id="api.signuporsignin">
- <DataUri>urn:com:microsoft:aad:b2c:elements:contract:unifiedssp:2.1.5</DataUri>
+ <DataUri>urn:com:microsoft:aad:b2c:elements:contract:unifiedssp:2.1.7</DataUri>
</ContentDefinition> <ContentDefinition Id="api.selfasserted">
- <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.7</DataUri>
+ <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.14</DataUri>
</ContentDefinition> <ContentDefinition Id="api.selfasserted.profileupdate">
- <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.7</DataUri>
+ <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.14</DataUri>
</ContentDefinition> <ContentDefinition Id="api.localaccountsignup">
- <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.7</DataUri>
+ <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.14</DataUri>
</ContentDefinition> <ContentDefinition Id="api.localaccountpasswordreset">
- <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.7</DataUri>
+ <DataUri>urn:com:microsoft:aad:b2c:elements:contract:selfasserted:2.1.14</DataUri>
</ContentDefinition> <ContentDefinition Id="api.phonefactor"> <RecoveryUri>~/common/default_page_error.html</RecoveryUri>
active-directory-b2c Extensions App https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory-b2c/extensions-app.md
To verify that the b2c-extensions-app is present:
## Recover the extensions app
-If you accidentally deleted the b2c-extensions-app, you have 30 days to recover it. You can restore the app using the Graph API:
+If you accidentally deleted the `b2c-extensions-app`, you have 30 days to recover it.
+> [!NOTE]
+> An application can only be restored if it has been deleted within the last 30 days. If it has been more than 30 days, data will be permanently lost. For more assistance, file a support ticket.
+
+### Recover the extensions app using the Azure portal
+
+1. Sign in to your Azure AD B2C tenant.
+2. Search for and open **App registrations**.
+1. Select the **Deleted applications** tab and identify the `b2c-extensions-app` from the list of recently deleted applications.
+1. Select **Restore app registration**.
+
+You should now be able to [see the restored app](#verifying-that-the-extensions-app-is-present) in the Azure portal.
+
+### Recover the extensions app using Microsoft Graph
+To restore the app using Microsoft Graph, you must restore both the application and the service principal.
+
+To restore the application:
1. Browse to [https://developer.microsoft.com/en-us/graph/graph-explorer](https://developer.microsoft.com/en-us/graph/graph-explorer). 1. Log in to the site as a global administrator for the Azure AD B2C directory that you want to restore the deleted app for. This global administrator must have an email address similar to the following: `username@{yourTenant}.onmicrosoft.com`.
-1. Issue an HTTP GET against the URL `https://graph.microsoft.com/beta/directory/deleteditems/microsoft.graph.application`. This operation will list all of the applications that have been deleted within the past 30 days.
-1. Find the application in the list where the name begins with 'b2c-extensions-appΓÇÖ and copy its `objectid` property value.
-1. Issue an HTTP POST against the URL `https://graph.microsoft.com/beta/directory/deleteditems/{id}/restore`. Replace the `{id}` portion of the URL with the `objectid` from the previous step.
+1. Issue an HTTP GET against the URL `https://graph.microsoft.com/v1.0/directory/deleteditems/microsoft.graph.application`. This operation will list all of the applications that have been deleted within the past 30 days. You can also use the URL `https://graph.microsoft.com/v1.0/directory/deletedItems/microsoft.graph.application?$filter=displayName eq 'b2c-extensions-app. Do not modify. Used by AADB2C for storing user data.'` to filter by the app's **displayName** property.
+1. Find the application in the list where the name begins with `b2c-extensions-app` and copy its `id` property value.
+1. Issue an HTTP POST against the URL `https://graph.microsoft.com/v1.0/directory/deleteditems/{id}/restore`. Replace the `{id}` portion of the URL with the `id` from the previous step.]
-You should now be able to [see the restored app](#verifying-that-the-extensions-app-is-present) in the Azure portal.
+To restore the service principal:
+1. Issue an HTTP GET against the URL `https://graph.microsoft.com/v1.0/directory/deleteditems/microsoft.graph.servicePrincipal`. This operation will list all of the service principals that have been deleted within the past 30 days. You can also use the URL `https://graph.microsoft.com/v1.0/directory/deletedItems/microsoft.graph.servicePrincipal?$filter=displayName eq 'b2c-extensions-app. Do not modify. Used by AADB2C for storing user data.'` to filter by the app's **displayName** property.
+1. Find the service principal in the list where the name begins with `b2c-extensions-app` and copy its `id` property value.
+1. Issue an HTTP POST against the URL `https://graph.microsoft.com/v1.0/directory/deleteditems/{id}/restore`. Replace the `{id}` portion of the URL with the `id` from the previous step.
-> [!NOTE]
-> An application can only be restored if it has been deleted within the last 30 days. If it has been more than 30 days, data will be permanently lost. For more assistance, file a support ticket.
+You should now be able to [see the restored app](#verifying-that-the-extensions-app-is-present) in the Azure portal.
active-directory-b2c Microsoft Graph Operations https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory-b2c/microsoft-graph-operations.md
The top-level resource for policy keys in the Microsoft Graph API is the [Truste
- [Create oauth2Permission Grant](/graph/api/resources/oauth2permissiongrant) - [Delete application](/graph/api/application-delete)
-## Application extension properties
+## Application extension (directory extension) properties
-- [Create extension properties](/graph/api/application-post-extensionproperty)-- [List extension properties](/graph/api/application-list-extensionproperty)-- [Get an extension property](/graph/api/extensionproperty-get)-- [Delete extension property](/graph/api/extensionproperty-delete)-- [Get available extension properties](/graph/api/directoryobject-getavailableextensionproperties)
+Application extension properties are also known as directory or Azure AD extensions. To manage them in Azure AD B2C, use the [identityUserFlowAttribute resource type](/graph/api/resources/identityuserflowattribute) and its associated methods.
-<!--
-#Hiding this note because user flows and extension attributes are different things in Microsoft Graph.
+- [Create user flow attribute](/graph/api/identityuserflowattribute-post)
+- [List user flow attributes](/graph/api/identityuserflowattribute-list)
+- [Get a user flow attribute](/graph/api/identityuserflowattribute-get)
+- [Update a user flow attribute](/graph/api/identityuserflowattribute-update)
+- [Delete a user flow attribute](/graph/api/identityuserflowattribute-delete)
-Azure AD B2C provides a directory that can hold 100 custom attributes per user. For user flows, these extension properties are [managed by using the Azure portal](user-flow-custom-attributes.md). For custom policies, Azure AD B2C creates the property for you, the first time the policy writes a value to the extension property.
>
+You can store up to 100 directory extension values per user. To manage the directory extension properties for a user, use the following [User APIs](/graph/api/resources/user) in Microsoft Graph.
-Azure AD B2C provides a directory that can hold 100 extension values per user. To manage the extension values for a user, use the following [User APIs](/graph/api/resources/user) in Microsoft Graph.
+- [Update user](/graph/api/user-update): To write or remove the value of the directory extension property from the user object.
+- [Get a user](/graph/api/user-get): To retrieve the value of the directory extension for the user. The property will be returned by default through the `beta` endpoint, but only on `$select` through the `v1.0` endpoint.
-- [Update user](/graph/api/user-update): To write or remove the extension property value from the user.-- [Get a user](/graph/api/user-get): To retrieve the extension property value for the user. The extension property will be returned by default through the `beta` endpoint, but only on `$select` through the `v1.0` endpoint.
+For user flows, these extension properties are [managed by using the Azure portal](user-flow-custom-attributes.md). For custom policies, Azure AD B2C creates the property for you, the first time the policy writes a value to the extension property.
+
+> [!NOTE]
+> In Azure AD, directory extensions are managed through the [extensionProperty resource type](/graph/api/resources/extensionproperty) and its associated methods. However, because they are used in B2C through the `b2c-extensions-app` app which should not be updated, they are managed in Azure AD B2C using the [identityUserFlowAttribute resource type](/graph/api/resources/identityuserflowattribute) and its associated methods.
## Audit logs
For more information about accessing Azure AD B2C audit logs, see [Accessing Azu
- [Update a Conditional Access policy](/graph/api/conditionalaccesspolicy-update) - [Delete a Conditional Access policy](/graph/api/conditionalaccesspolicy-delete)
+## Retrieve or restore deleted users and applications
+
+Deleted items can only be restored if they were deleted within the last 30 days.
+
+- [List deleted items](/graph/api/directory-deleteditems-list)
+- [Get a deleted item](/graph/api/directory-deleteditems-get)
+- [Restore a deleted item](/graph/api/directory-deleteditems-restore)
+- [Permanently delete a deleted item](/graph/api/directory-deleteditems-delete)
+ ## How to programmatically manage Microsoft Graph When you want to manage Microsoft Graph, you can either do it as the application using the application permissions, or you can use delegated permissions. For delegated permissions, either the user or an administrator consents to the permissions that the app requests. The app is delegated with the permission to act as a signed-in user when it makes calls to the target resource. Application permissions are used by apps that do not require a signed in user present and thus require application permissions. Because of this, only administrators can consent to application permissions.
active-directory-b2c User Flow Custom Attributes https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory-b2c/user-flow-custom-attributes.md
Your Azure AD B2C directory comes with a [built-in set of attributes](user-profi
The terms *extension property*, *custom attribute*, and *custom claim* refer to the same thing in the context of this article. The name varies depending on the context, such as application, object, or policy.
-Azure AD B2C allows you to extend the set of attributes stored on each user account. You can also read and write these attributes by using the [Microsoft Graph API](microsoft-graph-operations.md).
+Azure AD B2C allows you to extend the set of attributes stored on each user account. You can also read and write these attributes by using the [Microsoft Graph API](microsoft-graph-operations.md#application-extension-directory-extension-properties).
## Prerequisites
The custom attribute is now available in the list of **User attributes** and for
1. Select **Application claims** and then select the custom attribute. 1. Select **Save**.
-Once you've created a new user using a user flow, which uses the newly created custom attribute, the object can be queried in [Microsoft Graph Explorer](https://developer.microsoft.com/graph/graph-explorer). Alternatively you can use the [Run user flow](./tutorial-create-user-flows.md) feature on the user flow to verify the customer experience. You should now see **ShoeSize** in the list of attributes collected during the sign-up journey, and see it in the token sent back to your application.
+Once you've created a new user using the user flow, you can use the [Run user flow](./tutorial-create-user-flows.md) feature on the user flow to verify the customer experience. You should now see **ShoeSize** in the list of attributes collected during the sign-up journey, and see it in the token sent back to your application.
::: zone-end
You can create these attributes by using the portal UI before or after you use t
|Name |Used in | ||| |`extension_loyaltyId` | Custom policy|
-|`extension_<b2c-extensions-app-guid>_loyaltyId` | [Microsoft Graph API](microsoft-graph-operations.md)|
+|`extension_<b2c-extensions-app-guid>_loyaltyId` | [Microsoft Graph API](microsoft-graph-operations.md#application-extension-directory-extension-properties)|
The following example demonstrates the use of custom attributes in an Azure AD B2C custom policy claim definition.
The following example demonstrates the use of a custom attribute in Azure AD B2C
## Manage extension attributes through Microsoft Graph
-You can use the Microsoft Graph API to create and manage extension attributes then set the values for a user.
+You can use Microsoft Graph to create and manage the custom attributes then set the values for a user. Extension attributes are also called directory or Azure AD extensions.
-Extension attributes in the Microsoft Graph API are named by using the convention `extension_ApplicationClientID_attributename`, where the `ApplicationClientID` is equivalent to the **appId** but without the hyphens. For example, if the **appId** of the `b2c-extensions-app` application is `25883231-668a-43a7-80b2-5685c3f874bc` and the **attributename** is `loyaltyId`, then the extension attribute will be named `extension_25883231668a43a780b25685c3f874bc_loyaltyId`.
+Custom attributes (directory extensions) in the Microsoft Graph API are named by using the convention `extension_{appId-without-hyphens}_{extensionProperty-name}` where `{appId-without-hyphens}` is the stripped version of the **appId** (called Client ID on the Azure AD B2C portal) for the `b2c-extensions-app` with only characters 0-9 and A-Z. For example, if the **appId** of the `b2c-extensions-app` application is `25883231-668a-43a7-80b2-5685c3f874bc` and the attribute name is `loyaltyId`, then the custom attribute will be named `extension_25883231668a43a780b25685c3f874bc_loyaltyId`.
-Learn how to [manage extension attributes in your Azure AD B2C tenant](microsoft-graph-operations.md#application-extension-properties) using the Microsoft Graph API.
+Learn how to [manage extension attributes in your Azure AD B2C tenant](microsoft-graph-operations.md#application-extension-directory-extension-properties) using the Microsoft Graph API.
## Remove extension attribute
-Unlike built-in attributes, extension/custom attributes can be removed. The extension attributes' values can also be removed.
+Unlike built-in attributes, custom attributes can be removed. The extension attributes' values can also be removed.
> [!Important]
-> Before you remove the extension/custom attribute, for each account in the directory, set the extension attribute value to `null`. In this way you explicitly remove the extension attributesΓÇÖs values. Then continue to remove the extension attribute itself. Extension/custom attribute is queryable using MS Graph API.
+> Before you remove the custom attribute, for each account in the directory, set the extension attribute value to `null`. In this way you explicitly remove the extension attributesΓÇÖs values. Then continue to remove the extension attribute itself. Custom attributes can be queries using Microsoft Graph API.
::: zone pivot="b2c-user-flow"
-Use the following steps to remove extension/custom attribute from a user flow in your:
+Use the following steps to remove a custom attribute from a user flow in your:
1. Sign in to the [Azure portal](https://portal.azure.com/) as the global administrator of your Azure AD B2C tenant. 2. Make sure you're using the directory that contains your Azure AD B2C tenant:
Use the following steps to remove extension/custom attribute from a user flow in
::: zone pivot="b2c-custom-policy"
-Use the [Microsoft Graph API](microsoft-graph-operations.md#application-extension-properties) to delete the extension attribute from the application or to delete the extension attribute from the user.
+Use the [Microsoft Graph API](microsoft-graph-operations.md#application-extension-directory-extension-properties) to manage the custom attributes.
::: zone-end
Follow the guidance for how to [add claims and customize user input using custom
<!-- LINKS --> [ms-graph]: /graph/
-[ms-graph-api]: /graph/api/overview
+[ms-graph-api]: /graph/api/overview
active-directory Concept Sspr Howitworks https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/authentication/concept-sspr-howitworks.md
Consider the following example scenario:
* Administrator *A* resets their password by using SSPR. * Administrators *B*, *C*, and *D* receive an email alerting them of the password reset.
+> [!NOTE]
+> Email notifications from the SSPR service will be sent from the following addresses based on the Azure cloud you are working with:
+> - Public: msonlineservicesteam@microsoft.com
+> - China: msonlineservicesteam@oe.21vianet.com
+> - Government: msonlineservicesteam@azureadnotifications.us
+
+> If you observe issues in receiving notifications, please check your spam settings.
## On-premises integration If you have a hybrid environment, you can configure Azure AD Connect to write password change events back from Azure AD to an on-premises directory.
active-directory Howto Sspr Deployment https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/authentication/howto-sspr-deployment.md
Set **Number of days before users is asked to reconfirm their authentication inf
Configure both the **Notify users on password resets** and the **Notify all admins when other admins reset their password** to **Yes**. Selecting **Yes** on both increases security by ensuring that users are aware when their password is reset. It also ensures that all admins are aware when an admin changes a password. If users or admins receive a notification and they haven't initiated the change, they can immediately report a potential security issue.
+> [!NOTE]
+> Email notifications from the SSPR service will be sent from the following addresses based on the Azure cloud you are working with:
+> - Public: msonlineservicesteam@microsoft.com
+> - China: msonlineservicesteam@oe.21vianet.com
+> - Government: msonlineservicesteam@azureadnotifications.us
+> If you observe issues in receiving notifications, please check your spam settings.
+ ### Customization settings It's critical to customize the helpdesk email or URL to ensure users who experience problems can get help immediately. Set this option to a common helpdesk email address or web page that your users are familiar with.
active-directory Concept Conditional Access Grant https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/conditional-access/concept-conditional-access-grant.md
Previously updated : 06/29/2022 Last updated : 08/05/2022
Restrictions when you configure a policy using the password change control.
If your organization has created terms of use, other options may be visible under grant controls. These options allow administrators to require acknowledgment of terms of use as a condition of accessing the resources protected by the policy. More information about terms of use can be found in the article, [Azure Active Directory terms of use](terms-of-use.md).
+### Custom controls (preview)
+
+Custom controls is a preview capability of the Azure Active Directory. When using custom controls, your users are redirected to a compatible service to satisfy authentication requirements outside of Azure Active Directory. For more information, check out the [Custom controls](controls.md) article.
+ ## Next steps - [Conditional Access: Session controls](concept-conditional-access-session.md)
active-directory Concept Conditional Access Policies https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/conditional-access/concept-conditional-access-policies.md
Previously updated : 01/11/2022 Last updated : 08/05/2022
If a policy where "Require one of the selected controls" is selected, we prompt
All policies are enforced in two phases: -- Phase 1: Collect session details
+- **Phase 1**: Collect session details
- Gather session details, like network location and device identity that will be necessary for policy evaluation. - Phase 1 of policy evaluation occurs for enabled policies and policies in [report-only mode](concept-conditional-access-report-only.md).-- Phase 2: Enforcement
+- **Phase 2**: Enforcement
- Use the session details gathered in phase 1 to identify any requirements that haven't been met. - If there's a policy that is configured to block access, with the block grant control, enforcement will stop here and the user will be blocked. - The user will be prompted to complete more grant control requirements that weren't satisfied during phase 1 in the following order, until policy is satisfied:
- - Multi-factor authenticationΓÇï
- - Approved client app/app protection policyΓÇï
- - Managed device (compliant or hybrid Azure AD join)ΓÇï
- - Terms of use
- - Custom controls
+ - [Multi-factor authenticationΓÇï](concept-conditional-access-grant.md#require-multifactor-authentication)
+ - [Device to be marked as compliant](./concept-conditional-access-grant.md#require-device-to-be-marked-as-compliant)
+ - [Hybrid Azure AD joined device](./concept-conditional-access-grant.md#require-hybrid-azure-ad-joined-device)
+ - [Approved client app](./concept-conditional-access-grant.md#require-approved-client-app)
+ - [App protection policy](./concept-conditional-access-grant.md#require-app-protection-policy)
+ - [Password change](./concept-conditional-access-grant.md#require-password-change)
+ - [Terms of use](concept-conditional-access-grant.md#terms-of-use)
+ - [Custom controls](./concept-conditional-access-grant.md#custom-controls-preview)
- Once all grant controls have been satisfied, apply session controls (App Enforced, Microsoft Defender for Cloud Apps, and token Lifetime) - Phase 2 of policy evaluation occurs for all enabled policies.
Location data is provided by IP geolocation data. Administrators can choose to d
#### Client apps
-By default, all newly created Conditional Access policies will apply to all client app types even if the client apps condition isn't configured.
+The software the user is employing to access the cloud app. For example, 'Browser', and 'Mobile apps and desktop clients'. By default, all newly created Conditional Access policies will apply to all client app types even if the client apps condition isn't configured.
The behavior of the client apps condition was updated in August 2020. If you have existing Conditional Access policies, they'll remain unchanged. However, if you select on an existing policy, the configure toggle has been removed and the client apps the policy applies to are selected.
Block access does just that, it will block access under the specified assignment
The grant control can trigger enforcement of one or more controls. -- Require multi-factor authentication (Azure AD Multi-Factor Authentication)
+- Require multi-factor authentication
- Require device to be marked as compliant (Intune) - Require Hybrid Azure AD joined device - Require approved client app
Administrators can choose to require one of the previous controls or all selecte
- Use app enforced restrictions - Currently works with Exchange Online and SharePoint Online only.
- - Passes device information to allow control of experience granting full or limited access.
+ - Passes device information to allow control of experience granting full or limited access.
- Use Conditional Access App Control - Uses signals from Microsoft Defender for Cloud Apps to do things like: - Block download, cut, copy, and print of sensitive documents.
Administrators can choose to require one of the previous controls or all selecte
- Ability to change the default sign in frequency for modern authentication. - Persistent browser session - Allows users to remain signed in after closing and reopening their browser window.
+- Customize continuous access evaluation
+- Disable resilience defaults
## Simple policies
active-directory Concept Conditional Access Users Groups https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/conditional-access/concept-conditional-access-users-groups.md
Previously updated : 06/01/2022 Last updated : 08/05/2022
The following options are available to include when creating a Conditional Acces
- All users that exist in the directory including B2B guests. - Select users and groups - All guest and external users
- - This selection includes any B2B guests and external users including any user with the `user type` attribute set to `guest`. This selection also applies to any external user signed-in from a different organization like a Cloud Solution Provider (CSP).
+ - This selection includes any [B2B guests and external users](../external-identities/external-identities-overview.md) including any user with the `user type` attribute set to `guest`. This selection also applies to any external user signed-in from a different organization like a Cloud Solution Provider (CSP).
- Directory roles
- - Allows administrators to select specific built-in Azure AD directory roles used to determine policy assignment. For example, organizations may create a more restrictive policy on users assigned the global administrator role. Other role types aren't supported, including administrative unit-scoped roles and custom roles.
+ - Allows administrators to select specific [built-in Azure AD directory roles](../roles/permissions-reference.md) used to determine policy assignment. For example, organizations may create a more restrictive policy on users assigned the global administrator role. Other role types aren't supported, including administrative unit-scoped roles and custom roles.
- Users and groups - Allows targeting of specific sets of users. For example, organizations can select a group that contains all members of the HR department when an HR app is selected as the cloud app. A group can be any type of user group in Azure AD, including dynamic or assigned security and distribution groups. Policy will be applied to nested users and groups.
active-directory Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/conditional-access/overview.md
Previously updated : 04/15/2022 Last updated : 08/05/2022
Common signals that Conditional Access can take in to account when making a poli
- Application - Users attempting to access specific applications can trigger different Conditional Access policies. - Real-time and calculated risk detection
- - Signals integration with Azure AD Identity Protection allows Conditional Access policies to identify risky sign-in behavior. Policies can then force users to change their password, do multi-factor authentication to reduce their risk level, or block access until an administrator takes manual action.
-- Microsoft Defender for Cloud Apps
+ - Signals integration with [Azure AD Identity Protection](../identity-protection/overview-identity-protection.md) allows Conditional Access policies to identify risky sign-in behavior. Policies can then force users to change their password, do multi-factor authentication to reduce their risk level, or block access until an administrator takes manual action.
+- [Microsoft Defender for Cloud Apps](/defender-cloud-apps/what-is-defender-for-cloud-apps)
- Enables user application access and sessions to be monitored and controlled in real time, increasing visibility and control over access to and activities done within your cloud environment. ## Common decisions
active-directory Require Tou https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/conditional-access/require-tou.md
Previously updated : 11/21/2019 Last updated : 08/05/2022
This section provides you with the steps to create a sample ToU. When you create
1. In Microsoft Word, create a new document. 1. Type **My terms of use**, and then save the document on your computer as **mytou.pdf**. 1. Sign in to your [Azure portal](https://portal.azure.com) as global administrator, security administrator, or a Conditional Access administrator.
-1. In the Azure portal, on the left navbar, click **Azure Active Directory**.
+1. Search for and select **Azure Active Directory**. From the menu on the left-hand side select **Security**.
![Azure Active Directory](./media/require-tou/02.png)
-1. On the **Azure Active Directory** page, in the **Security** section, click **Conditional Access**.
+1. Select **Conditional Access**.
![Conditional Access](./media/require-tou/03.png)
active-directory Single Sign On Saml Protocol https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/develop/single-sign-on-saml-protocol.md
A `Signature` element in `AuthnRequest` elements is optional. Azure AD does not
### Subject
-Don't include a `Subject` element. Azure AD doesn't support specifying a subject for a request and will return an error if one is provided.
+Don't include a `Subject` element. Azure AD doesn't support specifying a subject in `AuthnRequest` and will return an error if one is provided.
+
+A subject can instead be provided by adding a `login_hint` parameter to the HTTP request to the single sign-on URL, with the subject's NameID as the parameter value.
## Response
active-directory Cross Tenant Access Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/external-identities/cross-tenant-access-overview.md
Previously updated : 06/30/2022 Last updated : 08/05/2022
Azure AD organizations can use External Identities cross-tenant access settings to manage how they collaborate with other Azure AD organizations and other Microsoft Azure clouds through B2B collaboration and [B2B direct connect](cross-tenant-access-settings-b2b-direct-connect.md). [Cross-tenant access settings](cross-tenant-access-settings-b2b-collaboration.md) give you granular control over how external Azure AD organizations collaborate with you (inbound access) and how your users collaborate with external Azure AD organizations (outbound access). These settings also let you trust multi-factor authentication (MFA) and device claims ([compliant claims and hybrid Azure AD joined claims](../conditional-access/howto-conditional-access-policy-compliant-device.md)) from other Azure AD organizations.
-This article describes cross-tenant access settings, which are used to manage B2B collaboration and B2B direct connect with external Azure AD organizations, including across Microsoft clouds. Additional settings are available for B2B collaboration with non-Azure AD identities (for example, social identities or non-IT managed external accounts). These [external collaboration settings](external-collaboration-settings-configure.md) include options for restricting guest user access, specifying who can invite guests, and allowing or blocking domains.
-
-![Overview diagram of cross-tenant access settings.](media/cross-tenant-access-overview/cross-tenant-access-settings-overview.png)
+This article describes cross-tenant access settings, which are used to manage B2B collaboration and B2B direct connect with external Azure AD organizations, including across Microsoft clouds. More settings are available for B2B collaboration with non-Azure AD identities (for example, social identities or non-IT managed external accounts). These [external collaboration settings](external-collaboration-settings-configure.md) include options for restricting guest user access, specifying who can invite guests, and allowing or blocking domains.
## Manage external access with inbound and outbound settings
+The external identities cross-tenant access settings manage how you collaborate with other Azure AD organizations. These settings determine both the level of inbound access users in external Azure AD organizations have to your resources, and the level of outbound access your users have to external organizations.
+
+The following diagram shows the cross-tenant access inbound and outbound settings. The **Resource Azure AD tenant** is the tenant containing the resources to be shared. In the case of B2B collaboration, the resource tenant is the inviting tenant (for example, your corporate tenant, where you want to invite the external users to). The **User's home Azure AD tenant** is the tenant where the external users are managed.
+
+![Overview diagram of cross-tenant access settings.](media/cross-tenant-access-overview/cross-tenant-access-settings-overview.png)
+ By default, B2B collaboration with other Azure AD organizations is enabled, and B2B direct connect is blocked. But the following comprehensive admin settings let you manage both of these features. - **Outbound access settings** control whether your users can access resources in an external organization. You can apply these settings to everyone, or specify individual users, groups, and applications.
By default, B2B collaboration with other Azure AD organizations is enabled, and
The default cross-tenant access settings apply to all Azure AD organizations external to your tenant, except those for which you've configured organizational settings. You can change your default settings, but the initial default settings for B2B collaboration and B2B direct connect are as follows: -- **B2B collaboration**: All your internal users are enabled for B2B collaboration by default. This means your users can invite external guests to access your resources and they can be invited to external organizations as guests. MFA and device claims from other Azure AD organizations aren't trusted.
+- **B2B collaboration**: All your internal users are enabled for B2B collaboration by default. This setting means your users can invite external guests to access your resources and they can be invited to external organizations as guests. MFA and device claims from other Azure AD organizations aren't trusted.
- **B2B direct connect**: No B2B direct connect trust relationships are established by default. Azure AD blocks all inbound and outbound B2B direct connect capabilities for all external Azure AD tenants.
Microsoft cloud settings let you collaborate with organizations from different M
To set up B2B collaboration, both organizations configure their Microsoft cloud settings to enable the partner's cloud. Then each organization uses the partner's tenant ID to find and add the partner to their organizational settings. From there, each organization can allow their default cross-tenant access settings apply to the partner, or they can configure partner-specific inbound and outbound settings. After you establish B2B collaboration with a partner in another cloud, you'll be able to: - Use B2B collaboration to invite a user in the partner tenant to access resources in your organization, including web line-of-business apps, SaaS apps, and SharePoint Online sites, documents, and files.-- Use B2B collaboration to [share Power BI content to a user in the partner tenant](https://docs.microsoft.com/power-bi/enterprise/service-admin-azure-ad-b2b#cross-cloud-b2b).
+- Use B2B collaboration to [share Power BI content to a user in the partner tenant](/power-bi/enterprise/service-admin-azure-ad-b2b#cross-cloud-b2b).
- Apply Conditional Access policies to the B2B collaboration user and opt to trust device claims (compliant claims and hybrid Azure AD joined claims) from the userΓÇÖs home tenant. > [!NOTE]
To collaborate with a partner tenant in a different Microsoft Azure cloud, both
- To configure cross-tenant access settings in the Azure portal, you'll need an account with a Global administrator or Security administrator role. -- To configure trust settings or apply access settings to specific users, groups, or applications, you'll need an Azure AD Premium P1 license.
+- To configure trust settings or apply access settings to specific users, groups, or applications, you'll need an Azure AD Premium P1 license. The license is required on the tenant that you configure. For B2B direct connect, where mutual trust relationship with another Azure AD organization is required, you'll need an Azure AD Premium P1 license in both tenants.
- Cross-tenant access settings are used to manage B2B collaboration and B2B direct connect with other Azure AD organizations. For B2B collaboration with non-Azure AD identities (for example, social identities or non-IT managed external accounts), use [external collaboration settings](external-collaboration-settings-configure.md). External collaboration settings include B2B collaboration options for restricting guest user access, specifying who can invite guests, and allowing or blocking domains.
If your organization exports sign-in logs to a Security Information and Event Ma
The Azure AD audit logs capture all activity around cross-tenant access setting changes and activity. To audit changes to your cross-tenant access settings, use the **category** of ***CrossTenantAccessSettings*** to filter all activity to show changes to cross-tenant access settings.
-![Audit logs for cross-tenant access settings.](media/cross-tenant-access-overview/cross-tenant-access-settings-audit-logs.png)
## Next steps
active-directory Cross Tenant Access Settings B2b Direct Connect https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/external-identities/cross-tenant-access-settings-b2b-direct-connect.md
Previously updated : 06/30/2022 Last updated : 08/05/2022
With inbound settings, you select which external users and groups will be able t
1. Under **Applies to**, select one of the following: - **All external users and groups**: Applies the action you chose under **Access status** to all users and groups from external Azure AD organizations.
- - **Select external users and groups** (requires an Azure AD premium subscription): Lets you apply the action you chose under **Access status** to specific users and groups within the external organization.
+ - **Select external users and groups**: Lets you apply the action you chose under **Access status** to specific users and groups within the external organization. An Azure AD Premium P1 license is required on the tenant that you configure.
![Screenshot showing selecting the target users for b2b direct connect](media/cross-tenant-access-settings-b2b-direct-connect/generic-inbound-external-users-groups-target.png)
With outbound settings, you select which of your users and groups will be able t
- When you're done selecting users and groups, choose **Select**. > [!NOTE]
- > When targeting your users and groups, you won't be able to select users who have configured [SMS-based authentication](https://docs.microsoft.com/azure/active-directory/authentication/howto-authentication-sms-signin). This is because users who have a "federated credential" on their user object are blocked to prevent external users from being added to outbound access settings. As a workaround, you can use the [Microsoft Graph API](https://docs.microsoft.com/graph/api/resources/crosstenantaccesspolicy-overview?view=graph-rest-1.0) to add the user's object ID directly or target a group the user belongs to.
+ > When targeting your users and groups, you won't be able to select users who have configured [SMS-based authentication](../authentication/howto-authentication-sms-signin.md). This is because users who have a "federated credential" on their user object are blocked to prevent external users from being added to outbound access settings. As a workaround, you can use the [Microsoft Graph API](/graph/api/resources/crosstenantaccesspolicy-overview) to add the user's object ID directly or target a group the user belongs to.
1. Select **Save**. 1. Select the **External applications** tab.
active-directory User Properties https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/external-identities/user-properties.md
Previously updated : 03/31/2022 Last updated : 08/05/2022
For external users who are using internal credentials, the **Issuer** property i
### User Principal Name The user principal name for a B2B collaboration user object contains an #EXT# identifier.+ ### User type This property indicates the relationship of the user to the host tenancy. This property can have two values:
Microsoft account | This user is homed in a Microsoft account and authenticates
{hostΓÇÖs domain} | This user authenticates by using an Azure AD account that belongs to this organization. google.com | This user has a Gmail account and has signed up by using self-service to the other organization. facebook.com | This user has a Facebook account and has signed up by using self-service to the other organization.
-mail | This user has an email address that doesn't match with verified Azure AD or SAML/WS-Fed domains, and is not a Gmail address or a Microsoft account.
-phone | This user has an email address that doesn't match a verified Azure AD domain or a SAML/WS-Fed domain, and is not a Gmail address or Microsoft account.
+mail | This user has an email address that doesn't match with verified Azure AD or SAML/WS-Fed domains, and isn't a Gmail address or a Microsoft account.
+phone | This user has an email address that doesn't match a verified Azure AD domain or a SAML/WS-Fed domain, and isn't a Gmail address or Microsoft account.
{issuer URI} | This user is homed in an external organization that doesn't use Azure Active Directory as their identity provider, but instead uses a SAML/WS-Fed-based identity provider. The issuer URI is shown when the issuer field is clicked. ### Directory synced
Typically, an Azure AD B2B user and guest user are synonymous. Therefore, an Azu
It's possible to convert UserType from Member to Guest and vice-versa by editing the user's profile in the Azure portal or by using PowerShell. However, the UserType property represents the user's relationship to the organization. Therefore, you should change this property only if the relationship of the user to the organization changes. If the relationship of the user changes, should the user principal name (UPN) change? Should the user continue to have access to the same resources? Should a mailbox be assigned?
-## Remove guest user limitations
+## Guest user permissions
-There may be cases where you want to give your guest users higher privileges. You can add a guest user to any role and even remove the default guest user restrictions in the directory to give a user the same privileges as members.
+Guest users have [default restricted directory permissions](../fundamentals/users-default-permissions.md). They can manage their own profile, change their own password, and retrieve some information about other users, groups, and apps. However, they can't read all directory information.
-It's possible to turn off the default limitations so that a guest user in the company directory has the same permissions as a member user.
+There may be cases where you want to give your guest users higher privileges. You can add a guest user to any role and even remove the default guest user restrictions in the directory to give a user the same privileges as members. It's possible to turn off the default limitations so that a guest user in the company directory has the same permissions as a member user. For more information, check out the [Restrict guest access permissions in Azure Active Directory](../enterprise-users/users-restrict-guest-permissions.md) article.
![Screenshot showing the External users option in the user settings](media/user-properties/remove-guest-limitations.png) ## Can I make guest users visible in the Exchange Global Address List?+ Yes. By default, guest objects aren't visible in your organization's global address list, but you can use Azure Active Directory PowerShell to make them visible. For details, see "Add guests to the global address list" in the [Microsoft 365 per-group guest access article](/microsoft-365/solutions/per-group-guest-access). ## Can I update a guest user's email address?
active-directory Multi Tenant Common Considerations https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/fundamentals/multi-tenant-common-considerations.md
There are many considerations that are relevant to more than one collaboration p
You can use the console to manually create an invitation for a guest user account. When you do, the user object is created with a user type of *Guest*. Using other techniques to create invitations enable you to set the user type to something other than a Guest account. For example, when using the API you can configure whether the account is a member account or a guest account.
-* Some of the [limits on Guest functionality can be removed](../external-identities/user-properties.md#remove-guest-user-limitations).
+* Some of the [limits on Guest functionality can be removed](../external-identities/user-properties.md#guest-user-permissions).
* [You can convert Guest accounts to a user type of Member.](../external-identities/user-properties.md#can-azure-ad-b2b-users-be-added-as-members-instead-of-guests)
You can represent users from another tenant using a traditional GAL synchronizat
* A mail-contact object and a mail-enabled guest user (member or guest) can't coexist in the same tenant with the same email address at the same time.
-* If a mail-contact object exists for the same mail address as the invited guest user, the guest user will be created but is NOT mail enabled.
+* If a mail-contact object exists for the same mail address as the invited guest user, the guest user will be created, but is NOT mail enabled.
* If the mail-enabled guest user exists with the same mail, an attempt to create a mail-contact object will throw an exception at creation time.
The following are the results of various mail-contact objects and guest user sta
* Create external member users, which show in the GAL by default, but aren't mail-enabled.
-Some organizations use the mail-contact object to show users in the GAL. This approach integrates a GAL without providing other permissions as mail-contacts are not security principals.
+Some organizations use the mail-contact object to show users in the GAL. This approach integrates a GAL without providing other permissions as mail-contacts aren't security principals.
A better approach to achieve this goal is to: * Invite guest users * Unhide them from the GAL * Disable them by [blocking them from sign in](/powershell/module/azuread/set-azureaduser).
-A mail-contact object cannot be converted to a user object. Therefore, any properties associated with a mail-contact object cannot be transferred. For example, group memberships and other resource access aren't transferred.
+A mail-contact object can't be converted to a user object. Therefore, any properties associated with a mail-contact object can't be transferred. For example, group memberships and other resource access aren't transferred.
Using a mail-contact object to represent a user presents the following challenges. * **Office 365 Groups** ΓÇô Office 365 groups support policies governing the types of users allowed to be members of groups and interact with content associated with groups. For example, a group may not allow guest accounts to join. These policies can't govern mail-contact objects.
-* **Azure AD Self-service group management (SSGM)** ΓÇô Mail-contact objects aren't eligible to be members in groups using the SSGM feature. Additional tools may be needed to manage groups with recipients represented as contacts instead of user objects.
+* **Azure AD Self-service group management (SSGM)** ΓÇô Mail-contact objects aren't eligible to be members in groups using the SSGM feature. More tools may be needed to manage groups with recipients represented as contacts instead of user objects.
* **Azure AD Identity Governance - Access Reviews** ΓÇô The access reviews feature can be used to review and attest to membership of Office 365 group. Access reviews are based on user objects. Members represented by mail-contact objects are out of scope of access reviews.
Additionally, while the following CA conditions can be used, be aware of the pos
## Other access control considerations
-Some additional considerations when configuring access control.
+More considerations when configuring access control.
+ * Define [access control policies](../external-identities/authentication-conditional-access.md) to control access to resources. * Design CA policies with guest users in mind. * Create policies specifically for guest users.
For information on hardening dynamic groups that utilize the [All Users] express
### Require User Assignment
-If an application has the [User assignment required?] property set to [No], guest users can access the application. Application admins must understand access control impacts, especially if the application contains sensitive information. For more information, see [How to restrict you Azure AD app to a set of users](../develop/howto-restrict-your-app-to-a-set-of-users.md).
+If an application has the [User assignment required?] property set to [No], guest users can access the application. Application admins must understand access control impacts, especially if the application contains sensitive information. For more information, see [How to restrict your Azure AD app to a set of users](../develop/howto-restrict-your-app-to-a-set-of-users.md).
### Terms and Conditions
If an application has the [User assignment required?] property set to [No], gues
### Licensing considerations for guest users with Azure AD Premium features
-Azure AD External Identities (guest user) pricing is based on monthly active users (MAU). The active users is the count of unique users with authentication activity within a calendar month. MAU billing helps you reduce costs by offering a free tier and flexible, predictable pricing. In addition, the first 50,000 MAUs per month are free for both Premium P1 and Premium P2 features. Premium features include Conditional Access Policies and Azure MFA for guest users.
+Azure AD External Identities (guest user) pricing is based on monthly active users (MAU). The number of active users is the count of unique users with authentication activity within a calendar month. MAU billing helps you reduce costs by offering a free tier and flexible, predictable pricing. In addition, the first 50,000 MAUs per month are free for both Premium P1 and Premium P2 features. Premium features include Conditional Access Policies and Azure AD Multi-Factor Authentication for guest users.
For more information, see [MAU billing model for Azure AD External Identities](../external-identities/external-identities-pricing.md).
The following information addresses Office 365 in the context of this paperΓÇÖs
Exchange online limits certain functionality for guest users. The limits may be lessened by creating external members instead of external guests. However, none of the following are supported for external users at this time.
-* A guest user can be assigned an Exchange Online license. However, they are prevented from being issued a token for Exchange Online. The results are that they are not able to access the resource.
+* A guest user can be assigned an Exchange Online license. However, they're prevented from being issued a token for Exchange Online. The results are that they aren't able to access the resource.
* Guest users can't use shared or delegated Exchange Online mailboxes in the resource tenant. * A guest user can be assigned to a shared mailbox, but can't access it.
-* Guest users need to be unhidden in order to be included in the GAL. By default, they are hidden.
+* Guest users need to be unhidden in order to be included in the GAL. By default, they're hidden.
* Hidden guest users are created at invite time. The creation is independent of whether the user has redeemed their invitation. So, if all guest users are unhidden, the list includes user objects of guest users who haven't redeemed an invitation. Based on your scenario, you may or may not want the objects listed.
Exchange online limits certain functionality for guest users. The limits may be
* Updates to Exchange-specific properties, such as the PrimarySmtpAddress, ExternalEmailAddress, EmailAddresses, and MailTip, can only be set using [Exchange Online PowerShell](/powershell/exchange/exchange-online-powershell-v2?view=exchange-ps&preserve-view=true). The Exchange Online Admin Center doesn't allow you to modify the attributes using the GUI.
-As shown above, you can use the [Set-MailUser](/powershell/module/exchange/set-mailuser?view=exchange-ps&preserve-view=true) PowerShell cmdlet for mail-specific properties. Many additional user properties you can modify with the [Set-User](/powershell/module/exchange/set-user?view=exchange-ps&preserve-view=true) PowerShell cmdlet. Most of the properties can also be modified using the Azure AD Graph APIs.
+As shown above, you can use the [Set-MailUser](/powershell/module/exchange/set-mailuser?view=exchange-ps&preserve-view=true) PowerShell cmdlet for mail-specific properties. More user properties you can modify with the [Set-User](/powershell/module/exchange/set-user?view=exchange-ps&preserve-view=true) PowerShell cmdlet. Most of the properties can also be modified using the Azure AD Graph APIs.
### Microsoft SharePoint Online
Microsoft Teams has features to limit access and based on user type. Changes to
### Licensing considerations for guest users in Teams
-When using Azure B2B with Office 365 workloads,, there are some key considerations. There are instances in which guest accounts do not have the same experience as a member account.
+When using Azure B2B with Office 365 workloads, there are some key considerations. There are instances in which guest accounts don't have the same experience as a member account.
**Microsoft groups**. See [Adding guests to office 365 Groups](https://support.office.com/article/adding-guests-to-office-365-groups-bfc7a840-868f-4fd6-a390-f347bf51aff6) to better understand the guest account experience in Microsoft Groups.
When using Azure B2B with Office 365 workloads,, there are some key consideratio
You can enable a full fidelity experience in Teams by using B2B External Members. Office 365 recently clarified its licensing policy for Multi-tenant organizations.
-* Users that are licensed in their home tenant may access resources in another tenant within the same legal entity. The access is granted using **External Members** setting with no additional licensing fees. The setting applies for SharePoint, OneDrive for Business, Teams, and Groups.
+* Users that are licensed in their home tenant may access resources in another tenant within the same legal entity. The access is granted using **External Members** setting with no extra licensing fees. The setting applies for SharePoint, OneDrive for Business, Teams, and Groups.
- * Engineering work is underway to automatically check the license status of a user in their home tenant and enable them to participate as a Member with no extra license assignment or configuration. However, for customers who wish to use External Members now, there is a licensing workaround that requires the Account Executive to work with the Microsoft Business Desk.
+ * Engineering work is underway to automatically check the license status of a user in their home tenant and enable them to participate as a Member with no extra license assignment or configuration. However, for customers who wish to use External Members now, there's a licensing workaround that requires the Account Executive to work with the Microsoft Business Desk.
* From now until the engineered licensing solution is enabled, customers can utilize a *Teams Trial license*. The license can be assigned to each user in their foreign tenant. The license has a one-year duration and enables all of the workloads listed above.
active-directory Create Access Review https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/governance/create-access-review.md
If you are reviewing access to an application, then before creating the review,
1. In the **Enable review decision helpers** section choose whether you want your reviewer to receive recommendations during the review process: 1. If you select **No sign-in within 30 days**, users who have signed in during the previous 30-day period are recommended for approval. Users who haven't signed in during the past 30 days are recommended for denial. This 30-day interval is irrespective of whether the sign-ins were interactive or not. The last sign-in date for the specified user will also display along with the recommendation.
- 1. If you select **Peer outlier**, approvers will be recommended to keep or deny access to users based on the access the users' peers have. If a user doesn't have the same access as their peers, the system will recommend that the reviewer deny them access.
> [!NOTE] > If you create an access review based on applications, your recommendations are based on the 30-day interval period depending on when the user last signed in to the application rather than the tenant.
active-directory Review Recommendations Access Reviews https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/active-directory/governance/review-recommendations-access-reviews.md
Title: Review recommendations for Access reviews - Azure AD description: Learn how to review access of group members with review recommendations in Azure Active Directory access reviews. -+ editor: markwahl-msft
na Previously updated : 7/18/2022- Last updated : 8/5/2022+
For more information, see [License requirements](access-reviews-overview.md#lice
## Inactive user recommendations A user is considered 'inactive' if they have not signed into the tenant within the last 30 days. This behavior is adjusted for reviews of application assignments, which checks each user's last activity in the app as opposed to the entire tenant. When inactive user recommendations are enabled for an access review, the last sign-in date for each user will be evaluated once the review starts, and any user that has not signed-in within 30 days will be given a recommended action of Deny. Additionally, when these decision helpers are enabled, reviewers will be able to see the last sign-in date for all users being reviewed. This sign-in date (as well as the resulting recommendation) is determined when the review begins and will not get updated while the review is in-progress.
-## Peer outlier recommendations
-If review decision helpers are enabled by the creator of the access review, reviewers can receive peer outlier recommendations for reviews of group access reviews.
-
-Peer analysis recommendation detects users with outlier access to a group, based on reporting-structure similarity with other group members. The outlier recommendation relies on a scoring mechanism which is calculated by computing the userΓÇÖs average distance to the remaining users in the group.
-
-A *peer* in an organizationΓÇÖs chart is defined as two or more users who share similar characteristics in the organization's reporting structure. Users who are very distant from all the other group members based on their organization's chart, are considered a ΓÇ£peer outlierΓÇ¥ in a group.
-
-> [!NOTE]
-> Currently, this feature is only available for uses in your directory. Use of the peer outlier recommendations is not supported for guest users.
--
-The following image has an example of an organization's reporting structure in a cosmetics company:
-
-![Example hierarchial organization chart for a cosmetics company](./media/review-recommendations-group-access-reviews/org-chart-example.png)
-
-Based on the reporting structure in the example image, members outside of a division that is under a group review, would be denied access by the system if the peer outlier recommendation was taken by the reviewer.
-
-For example, Phil who works within the Personal care division is in a group with Debby, Irwin, and Emily who all work within the Cosmetics division. The group is called *Fresh Skin*. If an Access Review for the group Fresh Skin is performed, based on the reporting structure and distance away from the other group members, Phil would be considered an outlier. The system will create a **Deny** recommendation in the group access review.
- ## Next Steps - [Create an access review](create-access-review.md) - [Review access to groups or applications](perform-access-review.md)
aks Cluster Configuration https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/aks/cluster-configuration.md
Title: Cluster configuration in Azure Kubernetes Services (AKS)
description: Learn how to configure a cluster in Azure Kubernetes Service (AKS) Previously updated : 02/09/2020 Last updated : 08/05/2022
This enables an OIDC Issuer URL of the provider which allows the API server to d
[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
+### Limitations
+
+OIDC issuer is only supported in Azure Public regions now.
+ > [!WARNING] > Enable/disable OIDC Issuer will change the current service account token issuer to a new value, which causes some down time and make API server restart. If the application pods based on service account token keep in failed status after enable/disable OIDC Issuer, it's recommended to restart the pods manually.
az aks show -n aks -g myResourceGroup --query "oidcIssuerProfile.issuerUrl" -ots
[az-feature-register]: /cli/azure/feature#az_feature_register [az-feature-list]: /cli/azure/feature#az_feature_list [az-provider-register]: /cli/azure/provider#az_provider_register
-[aks-add-np-containerd]: ./learn/quick-windows-container-deploy-cli.md#add-a-windows-server-node-pool-with-containerd
+[aks-add-np-containerd]: ./learn/quick-windows-container-deploy-cli.md#add-a-windows-server-node-pool-with-containerd
api-management Virtual Network Reference https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/api-management/virtual-network-reference.md
When an API Management service instance is hosted in a VNet, the ports in the fo
| * / 443, 12000 | Outbound | TCP | VirtualNetwork / AzureCloud | Health and Monitoring Extension (optional) | External & Internal | | * / 1886, 443 | Outbound | TCP | VirtualNetwork / AzureMonitor | Publish [Diagnostics Logs and Metrics](api-management-howto-use-azure-monitor.md), [Resource Health](../service-health/resource-health-overview.md), and [Application Insights](api-management-howto-app-insights.md) (optional) | External & Internal | | * / 25, 587, 25028 | Outbound | TCP | VirtualNetwork / Internet | Connect to SMTP Relay for sending e-mail (optional) | External & Internal |
-| * / 6381 - 6383 | Inbound & Outbound | TCP | VirtualNetwork / VirtualNetwork | Access Redis Service for [Cache](api-management-caching-policies.md) policies between machines (optional) | External & Internal |
+| * / 6380 | Inbound & Outbound | TCP | VirtualNetwork / VirtualNetwork | Access external Azure Cache for Redis service for [caching](api-management-caching-policies.md) policies between machines (optional) | External & Internal |
+| * / 6381 - 6383 | Inbound & Outbound | TCP | VirtualNetwork / VirtualNetwork | Access internal Azure Cache for Redis service for [caching](api-management-caching-policies.md) policies between machines (optional) | External & Internal |
| * / 4290 | Inbound & Outbound | UDP | VirtualNetwork / VirtualNetwork | Sync Counters for [Rate Limit](api-management-access-restriction-policies.md#LimitCallRateByKey) policies between machines (optional) | External & Internal | | * / 6390 | Inbound | TCP | AzureLoadBalancer / VirtualNetwork | **Azure Infrastructure Load Balancer** | External & Internal |
app-service Provision Resource Terraform https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/app-service/provision-resource-terraform.md
description: Create your first app to Azure App Service in seconds using a Terra
Previously updated : 8/26/2021 Last updated : 8/5/2022 ms.tool: terraform
terraform {
required_providers { azurerm = { source = "hashicorp/azurerm"
- version = "~> 2.65"
+ version = "~> 3.0.0"
} } required_version = ">= 0.14.9"
terraform {
provider "azurerm" { features {} }+ # Generate a random integer to create a globally unique name resource "random_integer" "ri" { min = 10000 max = 99999 }+ # Create the resource group resource "azurerm_resource_group" "rg" { name = "myResourceGroup-${random_integer.ri.result}" location = "eastus" }+ # Create the Linux App Service Plan
-resource "azurerm_app_service_plan" "appserviceplan" {
+resource "azurerm_service_plan" "appserviceplan" {
name = "webapp-asp-${random_integer.ri.result}" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name
- sku {
- tier = "Free"
- size = "F1"
- }
+ os_type = "Linux"
+ sku_name = "F1"
}
-# Create the web app, pass in the App Service Plan ID, and deploy code from a public GitHub repo
-resource "azurerm_app_service" "webapp" {
- name = "webapp-${random_integer.ri.result}"
- location = azurerm_resource_group.rg.location
- resource_group_name = azurerm_resource_group.rg.name
- app_service_plan_id = azurerm_app_service_plan.appserviceplan.id
- source_control {
- repo_url = "https://github.com/Azure-Samples/nodejs-docs-hello-world"
- branch = "master"
- manual_integration = true
- use_mercurial = false
+
+# Create the web app, pass in the App Service Plan ID
+resource "azurerm_linux_web_app" "webapp" {
+ name = "webapp-${random_integer.ri.result}"
+ location = azurerm_resource_group.rg.location
+ resource_group_name = azurerm_resource_group.rg.name
+ service_plan_id = azurerm_service_plan.appserviceplan.id
+ https_only = true
+ site_config {
+ minimum_tls_version = "1.2"
} }+
+# Deploy code from a public GitHub repo
+resource "azurerm_app_service_source_control" "sourcecontrol" {
+ app_id = azurerm_linux_web_app.webapp.id
+ repo_url = "https://github.com/Azure-Samples/nodejs-docs-hello-world"
+ branch = "master"
+ use_manual_integration = true
+ use_mercurial = false
+}
```
-Three Azure resources and one subresource are defined in the template. Links to the [Azure Provider Terraform Registry](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) are given below for further details and usage information:
+Four Azure resources are defined in the template. Links to the [Azure Provider Terraform Registry](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) are given below for further details and usage information:
* [**Microsoft.Resources/resourcegroups**](/azure/templates/microsoft.resources/resourcegroups?tabs=json): create a Resource Group if one doesn't already exist. * [azurerm_resource_group](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group) * [**Microsoft.Web/serverfarms**](/azure/templates/microsoft.web/serverfarms): create an App Service plan.
- * [azurerm_app_service_plan](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_plan)
-* [**Microsoft.Web/sites**](/azure/templates/microsoft.web/sites): create an App Service app.
- * [azurerm_app_service](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service)
+ * [azurerm_service_plan](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/service_plan)
+* [**Microsoft.Web/sites**](/azure/templates/microsoft.web/sites): create a Linux App Service app.
+ * [azurerm_linux_web_app](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/linux_web_app)
* [**Microsoft.Web/sites/sourcecontrols**](/azure/templates/microsoft.web/sites/sourcecontrols): create an external git deployment configuration.
- * [source_control](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service#source_control)
+ * [azurerm_app_service_source_control](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/app_service_source_control)
For further information on how to construct Terraform templates, have a look at the [Terraform Learn documentation](https://learn.hashicorp.com/collections/terraform/azure-get-started?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS).
When no longer needed, either [delete the resource group](../azure-resource-mana
> [!div class="nextstepaction"] > [Learn more about using Terraform in Azure](/azure/terraform) > [!div class="nextstepaction"]
-> [Terraform samples for Azure App Service](./samples-terraform.md)
+> [Terraform samples for Azure App Service](./samples-terraform.md)
app-service Quickstart Wordpress https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/app-service/quickstart-wordpress.md
Other WP options on Azure:
[WordPress](https://www.wordpress.org) is an open source content management system (CMS) that can be used to create websites, blogs, and other applications. Over 40% of the web uses WordPress from blogs to major news websites.
-In this quickstart, you'll learn how to create and deploy your first [WordPress](https://www.wordpress.org) site to [Azure App Service on Linux](overview.md#app-service-on-linux) using [Azure portal](https://portal.azure.com). It uses the **Basic** tier and [**incurs a cost**](https://azure.microsoft.com/pricing/details/app-service/linux/) for your Azure subscription. The WordPress installation comes with pre-installed plugins for performance improvements, [W3TC](https://wordpress.org/plugins/w3-total-cache/) for caching and [Smush](https://wordpress.org/plugins/wp-smushit/) for image compression.
+In this quickstart, you'll learn how to create and deploy your first [WordPress](https://www.wordpress.org) site to [Azure App Service on Linux](overview.md#app-service-on-linux) using the [WordPress on the Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/WordPress.WordPress?tab=Overview). It uses the **Basic** tier and [**incurs a cost**](https://azure.microsoft.com/pricing/details/app-service/linux/) for your Azure subscription. The WordPress installation comes with pre-installed plugins for performance improvements, [W3TC](https://wordpress.org/plugins/w3-total-cache/) for caching and [Smush](https://wordpress.org/plugins/wp-smushit/) for image compression.
+
+To complete this quickstart, you need an Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?ref=microsoft.com&utm_source=microsoft.com&utm_medium=docs).
> [!IMPORTANT]
-> [After November 28, 2022, PHP will only be supported on App Service on Linux.](https://github.com/Azure/app-service-linux-docs/blob/master/Runtime_Support/php_support.md#end-of-life-for-php-74) Due to this reason, this article deploys WordPress to App Service on Linux.
+> - [After November 28, 2022, PHP will only be supported on App Service on Linux.](https://github.com/Azure/app-service-linux-docs/blob/master/Runtime_Support/php_support.md#end-of-life-for-php-74).
+> - The MySQL Flexible Server is created behind a private [Virtual Network](/virtual-network/virtual-networks-overview) and can't be accessed directly. To access the database, use phpMyAdmin that's deployed with the WordPress site. It can be found at the URL : https://`<sitename>`.azurewebsites.net/phpmyadmin
>
+> If you have feedback to improve this WordPress offering on App Service, submit your ideas at [Web Apps Community](https://feedback.azure.com/d365community/forum/b09330d1-c625-ec11-b6e6-000d3a4f0f1c).
## Create WordPress site using Azure portal
-1. Sign in to the Azure portal at https://portal.azure.com.
-1. In the Azure portal, click **Create a resource**.
-
- :::image type="content" source="./media/quickstart-wordpress/01-portal-create-resource.png?text=Azure portal create a resource" alt-text="Screenshot of Azure portal create resource":::
-
-1. In **Create a resource**, type **WordPress** in the search and press **enter**.
+1. To start creating the WordPress site, browse to [https://ms.portal.azure.com/#create/WordPress.WordPress](https://ms.portal.azure.com/#create/WordPress.WordPress).
- :::image type="content" source="./media/quickstart-wordpress/02-portal-create-resource-search-wordpress.png?text=Azure portal Create Resource WordPress Details" alt-text="Screenshot of WordPress in Create Resource search":::
-
-1. Select the **WordPress** product for **App Service**.
-
- :::image type="content" source="./media/quickstart-wordpress/03-wordpress-marketplace.png?text=WordPress in Azure Marketplace" alt-text="Screenshot of WordPress in Azure Marketplace":::
+ :::image type="content" source="./media/quickstart-wordpress/01-portal-create-wordpress-on-app-service.png?text=WordPress from Azure Marketplace" alt-text="Screenshot of Create a WordPress site.":::
1. In the **Basics** tab, under **Project details**, make sure the correct subscription is selected and then choose to **Create new** resource group. Type **`myResourceGroup`** for the name and select a **Region** you want to serve your app from.
- :::image type="content" source="./media/quickstart-wordpress/04-wordpress-basics-project-details.png?text=Azure portal WordPress Project Details" alt-text="Screenshot of WordPress project details":::
+ :::image type="content" source="./media/quickstart-wordpress/04-wordpress-basics-project-details.png?text=Azure portal WordPress Project Details" alt-text="Screenshot of WordPress project details.":::
1. Under **Instance details**, type a globally unique name for your web app and choose **Linux** for **Operating System**. Select **Basic** for **Hosting plan**. See the table below for app and database SKUs for given hosting plans. You can view [hosting plans details in the announcement](https://techcommunity.microsoft.com/t5/apps-on-azure-blog/the-new-and-better-wordpress-on-app-service/ba-p/3202594). For pricing, visit [App Service pricing](https://azure.microsoft.com/pricing/details/app-service/linux/) and [Azure Database for MySQL pricing](https://azure.microsoft.com/pricing/details/mysql/flexible-server/).
- :::image type="content" source="./media/quickstart-wordpress/05-wordpress-basics-instance-details.png?text=WordPress basics instance details" alt-text="Screenshot of WordPress instance details":::
+ :::image type="content" source="./media/quickstart-wordpress/05-wordpress-basics-instance-details.png?text=WordPress basics instance details" alt-text="Screenshot of WordPress instance details.":::
1. <a name="wordpress-settings"></a>Under **WordPress Settings**, type an **Admin Email**, **Admin Username**, and **Admin Password**. The **Admin Email** here is used for WordPress administrative sign-in only.
- :::image type="content" source="./media/quickstart-wordpress/06-wordpress-basics-wordpress-settings.png?text=Azure portal WordPress settings" alt-text="Screenshot of WordPress settings":::
+ :::image type="content" source="./media/quickstart-wordpress/06-wordpress-basics-wordpress-settings.png?text=Azure portal WordPress settings" alt-text="Screenshot of WordPress settings.":::
1. Select the **Advanced** tab. Under **Additional Settings** choose your preferred **Site Language** and **Content Distribution**. If you're unfamiliar with a [Content Delivery Network](../cdn/cdn-overview.md) or [Blob Storage](../storage/blobs/storage-blobs-overview.md), select **Disabled**. For more details on the Content Distribution options, see [WordPress on App Service](https://azure.github.io/AppService/2022/02/23/WordPress-on-App-Service-Public-Preview.html).
- :::image type="content" source="./media/quickstart-wordpress/08-wordpress-advanced-settings.png" alt-text="Screenshot of WordPress Advanced Settings":::
+ :::image type="content" source="./media/quickstart-wordpress/08-wordpress-advanced-settings.png" alt-text="Screenshot of WordPress Advanced Settings.":::
1. Select the **Review + create** tab. After validation runs, select the **Create** button at the bottom of the page to create the WordPress site.
- :::image type="content" source="./media/quickstart-wordpress/09-wordpress-create.png?text=WordPress create button" alt-text="Screenshot of WordPress create button":::
+ :::image type="content" source="./media/quickstart-wordpress/09-wordpress-create.png?text=WordPress create button" alt-text="Screenshot of WordPress create button.":::
1. Browse to your site URL and verify the app is running properly. The site may take a few minutes to load. If you receive an error, allow a few more minutes then refresh the browser.
- :::image type="content" source="./media/quickstart-wordpress/wordpress-sample-site.png?text=WordPress sample site" alt-text="Screenshot of WordPress site":::
+ :::image type="content" source="./media/quickstart-wordpress/wordpress-sample-site.png?text=WordPress sample site" alt-text="Screenshot of WordPress site.":::
1. To access the WordPress Admin page, browse to `/wp-admin` and use the credentials you created in the [WordPress settings step](#wordpress-settings).
- :::image type="content" source="./media/quickstart-wordpress/wordpress-admin-login.png?text=WordPress admin login" alt-text="Screenshot of WordPress admin login":::
-
+ :::image type="content" source="./media/quickstart-wordpress/wordpress-admin-login.png?text=WordPress admin login" alt-text="Screenshot of WordPress admin login.":::
## Clean up resources When no longer needed, you can delete the resource group, App service, and all related resources. 1. From your App Service *overview* page, click the *resource group* you created in the [Create WordPress site using Azure portal](#create-wordpress-site-using-azure-portal) step.
- :::image type="content" source="./media/quickstart-wordpress/resource-group.png" alt-text="Resource group in App Service overview page":::
+ :::image type="content" source="./media/quickstart-wordpress/resource-group.png" alt-text="Resource group in App Service overview page.":::
1. From the *resource group* page, select **Delete resource group**. Confirm the name of the resource group to finish deleting the resources.
- :::image type="content" source="./media/quickstart-wordpress/delete-resource-group.png" alt-text="Delete resource group":::
+ :::image type="content" source="./media/quickstart-wordpress/delete-resource-group.png" alt-text="Delete resource group.":::
## MySQL password The [Application Settings](reference-app-settings.md#wordpress) for MySQL database credentials are used by WordPress to connect to the MySQL database. To change the MySQL database password, see [update admin password](/azure/mysql/single-server/how-to-create-manage-server-portal#update-admin-password). Whenever the MySQL database credentials are changed, the [Application Settings](reference-app-settings.md#wordpress) also need to be updated. The [Application Settings for MySQL database](reference-app-settings.md#wordpress) begin with the **`DATABASE_`** prefix. For more information on updating MySQL passwords, see [WordPress on App Service](https://azure.github.io/AppService/2022/02/23/WordPress-on-App-Service-Public-Preview.html#known-limitations).
automation Automation Linux Hrw Install https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/automation/automation-linux-hrw-install.md
Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookW
Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookWorkers/delete| Deletes a Hybrid Runbook Worker.
+## Check version of Hybrid Worker
+To check the version of agent-based Linux Hybrid Runbook Worker, go to the following path:
+
+`vi/opt/microsoft/omsconfig/modules/nxOMSAutomationWorker/VERSION`
+
+The file *VERSION* has the version number of Hybrid Runbook Worker.
+ ## Next steps * To learn how to configure your runbooks to automate processes in your on-premises datacenter or other cloud environment, see [Run runbooks on a Hybrid Runbook Worker](automation-hrw-run-runbooks.md).
automation Automation Windows Hrw Install https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/automation/automation-windows-hrw-install.md
Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/delete | Delet
Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookWorkers/read | Reads a Hybrid Runbook Worker. Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookWorkers/delete | Deletes a Hybrid Runbook Worker. +
+## Check version of Hybrid Worker
+
+To check version of agent-based Windows Hybrid Runbook Worker, go to the following path:
+
+`C:\ProgramFiles\Microsoft Monitoring Agent\Agent\AzureAutomation\`
+
+The *AzureAutomation* folder has a sub-folder with the version number as the name of the sub-folder.
## Next steps
automation Extension Based Hybrid Runbook Worker Install https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/automation/extension-based-hybrid-runbook-worker-install.md
The extension-based onboarding is only for **User** Hybrid Runbook Workers. This
For **System** Hybrid Runbook Worker onboarding, see [Deploy an agent-based Windows Hybrid Runbook Worker in Automation](./automation-windows-hrw-install.md) or [Deploy an agent-based Linux Hybrid Runbook Worker in Automation](./automation-linux-hrw-install.md).
-You can use the user Hybrid Runbook Worker feature of Azure Automation to run runbooks directly on an Azure or non-Azure machine, including servers registered with [Azure Arc-enabled servers](../azure-arc/servers/overview.md). From the machine or server that's hosting the role, you can run runbooks directly against it and against resources in the environment to manage those local resources.
+You can use the user Hybrid Runbook Worker feature of Azure Automation to run runbooks directly on an Azure machine or a non-Azure machine through servers registered with [Azure Arc-enabled servers](../azure-arc/servers/overview.md). From the machine or server that's hosting the role, you can run runbooks directly against it and against resources in the environment to manage those local resources.
Azure Automation stores and manages runbooks and then delivers them to one or more chosen machines. After you successfully deploy a runbook worker, review [Run runbooks on a Hybrid Runbook Worker](automation-hrw-run-runbooks.md) to learn how to configure your runbooks to automate processes in your on-premises datacenter or other cloud environment.
Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookW
Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookWorkers/move/action | Moves Hybrid Runbook Worker from one Worker Group to another. Microsoft.Automation/automationAccounts/hybridRunbookWorkerGroups/hybridRunbookWorkers/delete | Deletes a Hybrid Runbook Worker.
+## Check version of Hybrid Worker
+To check the version of the extension-based Hybrid Runbook Worker:
+
+|OS types | Paths | Description|
+| | | |
+|**Windows** |`C:\Packages\Plugins\Microsoft.Azure.Automation.HybridWorker.HybridWorkerForWindows\`| The path has *version* folder that has the version information. |
+|**Linux** | `/var/lib/waagent/Microsoft.Azure.Automation.HybridWorker.HybridWorkerForLinux-<version>` | The folder name ends with *version* information. |
## Next steps
azure-arc Conceptual Agent Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-arc/kubernetes/conceptual-agent-overview.md
This article provides an overview of the Azure Arc agents deployed on the Kubern
Most on-premises datacenters enforce strict network rules that prevent inbound communication on the network boundary firewall. Azure Arc-enabled Kubernetes works with these restrictions by not requiring inbound ports on the firewall. Azure Arc agents only require outbound communication to a [set list of network endpoints](quickstart-connect-cluster.md#meet-network-requirements).
-![Diagram showing an architectural overview of the Azure Arc-enabled Kubernetes agents](./media/architectural-overview.png) ](./media/architectural-overview.png#lightbox)
-The following high-level steps are involved in [connecting a Kubernetes cluster to Azure Ar](quickstart-connect-cluster.md)c:
+The following high-level steps are involved in [connecting a Kubernetes cluster to Azure Arc](quickstart-connect-cluster.md):
1. Create a Kubernetes cluster on your choice of infrastructure (VMware vSphere, Amazon Web Services, Google Cloud Platform, etc.).
azure-arc Network Requirements https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-arc/servers/network-requirements.md
The table below lists the URLs that must be available in order to install and us
|`*.blob.core.windows.net`|Download source for Azure Arc-enabled servers extensions|Always, except when using private endpoints| Not used when private link is configured | |`dc.services.visualstudio.com`|Agent telemetry|Optional| Public |
+> [!NOTE]
+> To translate the `*.servicebus.windows.net` wildcard into specific endpoints, use the command `\GET /urls/allowlist?api-version=2020-01-01&location=<location>`. Within this command, the region must be specified for the <location> placeholder.
+ ### [Azure Government](#tab/azure-government) | Agent resource | Description | When required| Endpoint used with private link |
azure-cache-for-redis Cache Best Practices Connection https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-cache-for-redis/cache-best-practices-connection.md
Title: Best practices for connection resilience
description: Learn how to make your Azure Cache for Redis connections resilient. + Previously updated : 11/3/2021 Last updated : 08/03/2022
Test your system's resiliency to connection breaks using a [reboot](cache-admini
## TCP settings for Linux-hosted client applications
-Some Linux versions use optimistic TCP settings by default. The TCP settings can create a situation where a client connection to a cache cannot be reestablished for a long time when a Redis server stops responding before closing the connection gracefully. The failure to reestablish a connection can happen if the primary node of your Azure Cache For Redis becomes unavailable, for example, for unplanned maintenance.
+Some Linux versions use TCP settings that are too high by default. The higher TCP settings can create a situation where a client connection to a cache cannot be reestablished for a long time when a Redis server stops responding. The client waits too long before closing the connection gracefully.
+
+The failure to reestablish a connection can happen if the primary node of your Azure Cache For Redis becomes unavailable, for example, for unplanned maintenance.
We recommend these TCP settings:
azure-cache-for-redis Cache High Availability https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-cache-for-redis/cache-high-availability.md
Various high availability options are available in the Standard, Premium, and En
| [Zone redundancy](#zone-redundancy) | Multi-node replicated configuration across Availability Zones, with automatic failover | 99.9% in Premium; 99.99% in Enterprise (see [details](https://azure.microsoft.com/support/legal/sla/cache/v1_1/)) |-|Γ£ö|Γ£ö| | [Geo-replication](#geo-replication) | Linked cache instances in two regions, with user-controlled failover | Premium; Enterprise (see [details](https://azure.microsoft.com/support/legal/sla/cache/v1_1/)) |-|Passive|Active| | [Import/Export](#importexport) | Point-in-time snapshot of data in cache. | 99.9% (see [details](https://azure.microsoft.com/support/legal/sla/cache/v1_1/)) |-|Γ£ö|Γ£ö|
-| [Persistence](#persistence) | Periodic data saving to storage account. | 99.9% (see [details](https://azure.microsoft.com/support/legal/sla/cache/v1_1/)) |-|Γ£ö|-|
+| [Persistence](#persistence) | Periodic data saving to storage account. | 99.9% (see [details](https://azure.microsoft.com/support/legal/sla/cache/v1_1/)) |-|Γ£ö|Preview|
## Standard replication for high availability
When a data node becomes unavailable or a network split happens, a failover simi
## Persistence
-Applicable tiers: **Premium**
+Applicable tiers: **Premium**, **Enterprise (preview)**, **Enterprise Flash (preview)**
Because your cache data is stored in memory, a rare and unplanned failure of multiple nodes can cause all the data to be dropped. To avoid losing data completely, [Redis persistence](https://redis.io/topics/persistence) allows you to take periodic snapshots of in-memory data, and store it to your storage account. If you experience a failure across multiple nodes causing data loss, your cache loads the snapshot from storage account. For more information, see [Configure data persistence for a Premium Azure Cache for Redis instance](cache-how-to-premium-persistence.md).
azure-functions Functions Reference Python https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-functions/functions-reference-python.md
The *host.json* file must also be updated to include an HTTP `routePrefix` value
} ```
-Update the Python code file *init.py*, based on the interface that your framework uses. The following example shows either an ASGI handler approach or a WSGI wrapper approach for Flask:
+Update the Python code file *__init__.py*, based on the interface that your framework uses. The following example shows either an ASGI handler approach or a WSGI wrapper approach for Flask:
# [ASGI](#tab/asgi)
azure-monitor Sdk Support Guidance https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-monitor/app/sdk-support-guidance.md
Support engineers are expected to provide SDK update guidance according to the f
|Current SDK version in use |Alternative version available |Update policy for support | ||||
-|Latest stable minor version of a GA SDK | Newer supported stable version | **UPDATE REQUIRED** |
-|Unsupported ([support policy](/lifecycle/faq/azure)) | Any supported version | **UPDATE REQUIRED** |
-|Preview | Stable version | **UPDATE REQUIRED** |
-|Preview | Older stable version | **UPDATE RECOMMENDED** |
-|Preview | Newer preview version, no older stable version | **UPDATE RECOMMENDED** |
+|Latest GA SDK | No newer supported stable version | **NO UPDATE NECESSARY** |
+|Stable minor version of a GA SDK | Newer supported stable version | **UPDATE RECOMMENDED** |
+|Unsupported ([support policy](/lifecycle/faq/azure)) | Any supported version | **UPDATE REQUIRED** |
+|Preview | Stable version | **UPDATE REQUIRED** |
+|Preview | Older stable version | **UPDATE RECOMMENDED** |
+|Preview | Newer preview version, no older stable version | **UPDATE RECOMMENDED** |
> [!TIP] > Switching to [auto-instrumentation](codeless-overview.md) eliminates the need for manual SDK updates.
Support engineers are expected to provide SDK update guidance according to the f
> [!WARNING] > Only commercially reasonable support is provided for Preview versions of the SDK. If a support incident requires escalation to development for further guidance, customers will be asked to use a fully supported SDK version to continue support. Commercially reasonable support does not include an option to engage Microsoft product development resources; technical workarounds may be limited or not possible.
-To see the current version of Application Insights SDKs and previous versions release dates, reference the [release notes](release-notes.md).
+To see the current version of Application Insights SDKs and previous versions release dates, reference the [release notes](release-notes.md).
azure-monitor Container Insights Cost https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-monitor/containers/container-insights-cost.md
The following is a summary of what types of data are collected from a Kubernetes
## What is collected from Kubernetes clusters
-Container insights includes a predefined set of metrics and inventory items collected that are written as log data in your Log Analytics workspace. All metrics listed below are collected by default every one minute.
+Container insights includes a predefined set of metrics and inventory items collected that are written as log data in your Log Analytics workspace. All metrics listed below are collected every one minute.
### Node metrics collected
If you are utilizing [Prometheus metric scraping](container-insights-prometheus-
## Next steps
-For more information about how to understand what the costs are likely to be based on recent usage patterns from data collected with Container insights, see [Analyze usage in Log Analytics workspace](../logs/analyze-usage.md).
+For more information about how to understand what the costs are likely to be based on recent usage patterns from data collected with Container insights, see [Analyze usage in Log Analytics workspace](../logs/analyze-usage.md).
azure-monitor Data Collection Transformations Structure https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-monitor/essentials/data-collection-transformations-structure.md
Title: KQL limitations in data collection transformations
+ Title: Structure of transformation in Azure Monitor (preview)
description: Structure of transformation in Azure Monitor including limitations of KQL allowed in a transformation. Last updated 06/29/2022
azure-netapp-files Azacsnap Troubleshoot https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-netapp-files/azacsnap-troubleshoot.md
na Previously updated : 06/13/2022 Last updated : 08/05/2022
To troubleshoot this error:
[19/Nov/2020:18:41:10 +13:00] DEBUG: [PID:0020257:StorageANF:659] [1] Innerexception: Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials ```
+> [!TIP]
+> For more information on generating a new Service Principal, refer to the section [Enable communication with Storage](azacsnap-installation.md?tabs=azure-netapp-files%2Csap-hana#enable-communication-with-storage) in the [Install Azure Application Consistent Snapshot tool](azacsnap-installation.md) guide.
+ ## Troubleshoot failed 'test hana' command The command `azacsnap -c test --test hana` might not complete successfully.
azure-netapp-files Create Active Directory Connections https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-netapp-files/create-active-directory-connections.md
na Previously updated : 07/29/2022 Last updated : 08/05/2022 # Create and manage Active Directory connections for Azure NetApp Files
-Several features of Azure NetApp Files require that you have an Active Directory connection. For example, you need to have an Active Directory connection before you can create an [SMB volume](azure-netapp-files-create-volumes-smb.md), a [NFSv4.1 Kerberos volume](configure-kerberos-encryption.md), or a [dual-protocol volume](create-volumes-dual-protocol.md). This article shows you how to create and manage Active Directory connections for Azure NetApp Files.
+Several features of Azure NetApp Files require that you have an Active Directory connection. For example, you need to have an Active Directory connection before you can create an [SMB volume](azure-netapp-files-create-volumes-smb.md), a [NFSv4.1 Kerberos volume](configure-kerberos-encryption.md), or a [dual-protocol volume](create-volumes-dual-protocol.md). This article shows you how to create and manage Active Directory connections for Azure NetApp Files.
## <a name="requirements-for-active-directory-connections"></a>Requirements and considerations for Active Directory connections
If you accidentally reset the password of the AD computer account on the AD serv
### Register the feature
-The reset Active Directory computer account password feature is currently in public preview. If you are using this feature for the first time, you need to register the feature first.
+The reset Active Directory computer account password feature is currently in public preview. If you're using this feature for the first time, you need to register the feature first.
1. Register the **reset Active Directory computer account password** feature: ```azurepowershell-interactive
You can also use [Azure CLI commands](/cli/azure/feature) `az feature register`
1. Navigate to the volume **Overview** menu. Select **Reset Active Directory Account**. :::image type="content" source="../media/azure-netapp-files/active-directory-reset-overview.png" alt-text="Azure Volume Overview interface with the Reset Active Directory Account button highlighted." lightbox="../media/azure-netapp-files/active-directory-reset-overview.png":::
-Alternately, navigate to the **Volumes** menu. Identify the volume for which you want to reset the Active Directory account and select the meatballs menu at the end of the row. Select **Reset Active Directory Account**.
+Alternately, navigate to the **Volumes** menu. Identify the volume for which you want to reset the Active Directory account and select the three dots (`...`) at the end of the row. Select **Reset Active Directory Account**.
:::image type="content" source="../media/azure-netapp-files/active-directory-reset-list.png" alt-text="Azure volume list with the Reset Active Directory Account button highlighted." lightbox="../media/azure-netapp-files/active-directory-reset-list.png"::: 2. A warning message that explains the implications of this action will pop up. Type **yes** in the text box to proceed. ## Next steps
azure-netapp-files Troubleshoot Volumes https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-netapp-files/troubleshoot-volumes.md
na Previously updated : 07/28/2022 Last updated : 08/05/2022 # Troubleshoot volume errors for Azure NetApp Files
This article describes error messages and resolutions that can help you troubles
| Error conditions | Resolutions | |--|-|
-| The SMB or dual-protocol volume creation fails with the following error: <br> `{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"InternalServerError", "message":"Error when creating - Could not query DNS server. Verify that the network configuration is correct and that DNS servers are available."}]}` | This error indicates that the DNS is not reachable. <br> Consider the following solutions: <ul><li>Check if AD DS and the volume are being deployed in same region.</li> <li>Check if AD DS and the volume are using the same VNet. If they are using different VNETs, make sure that the VNets are peered with each other. See [Guidelines for Azure NetApp Files network planning](azure-netapp-files-network-topologies.md). </li> <li>The DNS server might have network security groups (NSGs) applied. As such, it does not allow the traffic to flow. In this case, open the NSGs to the DNS or AD to connect to various ports. For port requirements, see [Requirements for Active Directory connections](create-active-directory-connections.md#requirements-for-active-directory-connections). </li></ul> <br>The same solutions apply for Azure AD DS. Azure AD DS should be deployed in the same region. The VNet should be in the same region or peered with the VNet used by the volume. |
+| The SMB or dual-protocol volume creation fails with the following error: <br> `{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"InternalServerError", "message":"Error when creating - Could not query DNS server. Verify that the network configuration is correct and that DNS servers are available."}]}` | This error indicates that the DNS is not reachable. <br> Consider the following solutions: <ul><li>Check if AD DS and the volume are being deployed in same region.</li> <li>Check if AD DS and the volume are using the same VNet. If they're using different VNETs, make sure that the VNets are peered with each other. See [Guidelines for Azure NetApp Files network planning](azure-netapp-files-network-topologies.md). </li> <li>The DNS server might have network security groups (NSGs) applied. As such, it does not allow the traffic to flow. In this case, open the NSGs to the DNS or AD to connect to various ports. For port requirements, see [Requirements for Active Directory connections](create-active-directory-connections.md#requirements-for-active-directory-connections). </li></ul> <br>The same solutions apply for Azure AD DS. Azure AD DS should be deployed in the same region. The VNet should be in the same region or peered with the VNet used by the volume. |
| The SMB or dual-protocol volume creation fails with the following error: <br> `{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"InternalServerError", "message":"Error when creating - Failed to create the Active Directory machine account \"SMBTESTAD-C1C8\". Reason: Kerberos Error: Invalid credentials were given Details: Error: Machine account creation procedure failed\n [ 563] Loaded the preliminary configuration.\n**[ 670] FAILURE: Could not authenticate as 'test@contoso.com':\n** Unknown user (KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN)\n. "}]}` | <ul><li>Make sure that the username entered is correct. </li> <li>Make sure that the user is part of the Administrator group that has the privilege to create machine accounts. </li> <li> If you use Azure AD DS, make sure that the user is part of the Azure AD group `Azure AD DC Administrators`. </li></ul> | | The SMB or dual-protocol volume creation fails with the following error: <br> `{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"InternalServerError", "message":"Error when creating - Failed to create the Active Directory machine account \"SMBTESTAD-A452\". Reason: Kerberos Error: Pre-authentication information was invalid Details: Error: Machine account creation procedure failed\n [ 567] Loaded the preliminary configuration.\n [ 671] Successfully connected to ip 10.x.x.x, port 88 using TCP\n**[ 1099] FAILURE: Could not authenticate as\n** 'user@contoso.com': CIFS server account password does\n** not match password stored in Active Directory\n** (KRB5KDC_ERR_PREAUTH_FAILED)\n. "}]}` | Make sure that the password entered for joining the AD connection is correct. | | The SMB or dual-protocol volume creation fails with the following error: <br> `{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"InternalServerError","message":"Error when creating - Failed to create the Active Directory machine account \"SMBTESTAD-D9A2\". Reason: SecD Error: ou not found Details: Error: Machine account creation procedure failed\n [ 561] Loaded the preliminary configuration.\n [ 665] Successfully connected to ip 10.x.x.x, port 88 using TCP\n [ 1039] Successfully connected to ip 10.x.x.x, port 389 using TCP\n**[ 1147] FAILURE: Specifed OU 'OU=AADDC Com' does not exist in\n** contoso.com\n. "}]}` | Make sure that the OU path specified for joining the AD connection is correct. If you use Azure AD DS, make sure that the organizational unit path is `OU=AADDC Computers`. |
-| The SMB or dual-protocol volume creation fails with the following error: <br> `Failed to create the Active Directory machine account \"SMB-ANF-VOL. Reason: LDAP Error: Local error occurred Details: Error: Machine account creation procedure failed. [nnn] Loaded the preliminary configuration. [nnn] Successfully connected to ip 10.x.x.x, port 88 using TCP [nnn] Successfully connected to ip 10.x.x.x, port 389 using [nnn] Entry for host-address: 10.x.x.x not found in the current source: FILES. Ignoring and trying next available source [nnn] Source: DNS unavailable. Entry for host-address:10.x.x.x found in any of the available sources\n*[nnn] FAILURE: Unable to SASL bind to LDAP server using GSSAPI: local error [nnn] Additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot determine realm for numeric host address) [nnn] Unable to connect to LDAP (Active Directory) service on contoso.com (Error: Local error) [nnn] Unable to make a connection (LDAP (Active Directory):contosa.com, result: 7643. ` | The pointer (PTR) record of the AD host machine might be missing on the DNS server. You need to create a reverse lookup zone on the DNS server, and then add a PTR record of the AD host machine in that reverse lookup zone. <br> For example, assume that the IP address of the AD machine is `10.x.x.x`, the hostname of the AD machine (as found by using the `hostname` command) is `AD1`, and the domain name is `contoso.com`. The PTR record added to the reverse lookup zone should be `10.x.x.x` -> `contoso.com`. |
+| The SMB or dual-protocol volume creation fails with the following error: <br> `Failed to create the Active Directory machine account \"SMB-ANF-VOL. Reason: LDAP Error: Local error occurred Details: Error: Machine account creation procedure failed. [nnn] Loaded the preliminary configuration. [nnn] Successfully connected to ip 10.x.x.x, port 88 using TCP [nnn] Successfully connected to ip 10.x.x.x, port 389 using [nnn] Entry for host-address: 10.x.x.x not found in the current source: FILES. Ignoring and trying next available source [nnn] Source: DNS unavailable. Entry for host-address:10.x.x.x found in any of the available sources\n*[nnn] FAILURE: Unable to SASL bind to LDAP server using GSSAPI: local error [nnn] Additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Cannot determine realm for numeric host address) [nnn] Unable to connect to LDAP (Active Directory) service on contoso.com (Error: Local error) [nnn] Unable to make a connection (LDAP (Active Directory):contosa.com, result: 7643. ` | The pointer (PTR) record of the AD host machine might be missing on the DNS server. You need to create a reverse lookup zone on the DNS server, and then add a PTR record of the AD host machine in that reverse lookup zone. |
| The SMB or dual-protocol volume creation fails with the following error: <br> `Failed to create the Active Directory machine account \"SMB-ANF-VOL\". Reason: Kerberos Error: KDC has no support for encryption type Details: Error: Machine account creation procedure failed [nnn]Loaded the preliminary configuration. [nnn]Successfully connected to ip 10.x.x.x, port 88 using TCP [nnn]FAILURE: Could not authenticate as 'contosa.com': KDC has no support for encryption type (KRB5KDC_ERR_ETYPE_NOSUPP) ` | Make sure that [AES Encryption](./create-active-directory-connections.md#create-an-active-directory-connection) is enabled both in the Active Directory connection and for the service account. | | The SMB or dual-protocol volume creation fails with the following error: <br> `Failed to create the Active Directory machine account \"SMB-NTAP-VOL\". Reason: LDAP Error: Strong authentication is required Details: Error: Machine account creation procedure failed\n [ 338] Loaded the preliminary configuration.\n [ nnn] Successfully connected to ip 10.x.x.x, port 88 using TCP\n [ nnn ] Successfully connected to ip 10.x.x.x, port 389 using TCP\n [ 765] Unable to connect to LDAP (Active Directory) service on\n dc51.area51.com (Error: Strong(er) authentication\n required)\n*[ nnn] FAILURE: Unable to make a connection (LDAP (Active\n* Directory):contoso.com), result: 7609\n. "` | The LDAP Signing option is not selected, but the AD client has LDAP signing. [Enable LDAP Signing](create-active-directory-connections.md#create-an-active-directory-connection) and retry. |
-| SMB volume creation fails with the following error: <br> `Failed to create the Active Directory machine account. Reason: LDAP Error: Intialization of LDAP library failed Details: Error: Machine account creation procedure failed` | This error occurs because the service or user account used in the Azure NetApp Files Active Directory connections does not have sufficient privilege to create computer objects or make modifications to the newly created computer object. <br> To solve the issue, you should grant the account being used greater privilege. You can apply a default role with sufficient privilege. You can also delegate additional privilege to the user or service account or to a group it is part of. |
+| SMB volume creation fails with the following error: <br> `Failed to create the Active Directory machine account. Reason: LDAP Error: Intialization of LDAP library failed Details: Error: Machine account creation procedure failed` | This error occurs because the service or user account used in the Azure NetApp Files Active Directory connections does not have sufficient privilege to create computer objects or make modifications to the newly created computer object. <br> To solve the issue, you should grant the account being used greater privilege. You can apply a default role with sufficient privilege. You can also delegate additional privilege to the user or service account or to a group it's part of. |
## Errors for dual-protocol volumes
This article describes error messages and resolutions that can help you troubles
| LDAP over TLS is enabled, and dual-protocol volume creation fails with the error `This Active Directory has no Server root CA Certificate`. | If this error occurs when you are creating a dual-protocol volume, make sure that the root CA certificate is uploaded in your NetApp account. | | Dual-protocol volume creation fails with the error `Failed to validate LDAP configuration, try again after correcting LDAP configuration`. | The pointer (PTR) record of the AD host machine might be missing on the DNS server. You need to create a reverse lookup zone on the DNS server, and then add a PTR record of the AD host machine in that reverse lookup zone. <br> For example, assume that the IP address of the AD machine is `10.x.x.x`, the hostname of the AD machine (as found by using the `hostname` command) is `AD1`, and the domain name is `contoso.com`. The PTR record added to the reverse lookup zone should be `10.x.x.x` -> `contoso.com`. | | Dual-protocol volume creation fails with the error `Failed to create the Active Directory machine account \\\"TESTAD-C8DD\\\". Reason: Kerberos Error: Pre-authentication information was invalid Details: Error: Machine account creation procedure failed\\n [ 434] Loaded the preliminary configuration.\\n [ 537] Successfully connected to ip 10.x.x.x, port 88 using TCP\\n**[ 950] FAILURE`. | This error indicates that the AD password is incorrect when Active Directory is joined to the NetApp account. Update the AD connection with the correct password and try again. |
-| Dual-protocol volume creation fails with the error `Could not query DNS server. Verify that the network configuration is correct and that DNS servers are available`. | This error indicates that DNS is not reachable. The reason might be because DNS IP is incorrect, or there is a networking issue. Check the DNS IP entered in AD connection and make sure that the IP is correct. <br> Also, make sure that the AD and the volume are in same region and in same VNet. If they are in different VNETs, ensure that VNet peering is established between the two VNets. <br> See [Guidelines for Azure NetApp Files network planning](azure-netapp-files-network-topologies.md#azure-native-environments) for details. |
+| Dual-protocol volume creation fails with the error `Could not query DNS server. Verify that the network configuration is correct and that DNS servers are available`. | This error indicates that DNS is not reachable. The reason might be because DNS IP is incorrect, or there's a networking issue. Check the DNS IP entered in AD connection and make sure that the IP is correct. <br> Also, make sure that the AD and the volume are in same region and in same VNet. If they are in different VNETs, ensure that VNet peering is established between the two VNets. <br> See [Guidelines for Azure NetApp Files network planning](azure-netapp-files-network-topologies.md#azure-native-environments) for details. |
| Permission is denied error when mounting a dual-protocol volume. | A dual-protocol volume supports both the NFS and SMB protocols. When you try to access the mounted volume on the UNIX system, the system attempts to map the UNIX user you use to a Windows user. <br> Ensure that the `POSIX` attributes are properly set on the AD DS User object. | ## Errors for NFSv4.1 Kerberos volumes
This article describes error messages and resolutions that can help you troubles
|`Error allocating volume - Export policy rules does not match kerberosEnabled flag` | Azure NetApp Files does not support Kerberos for NFSv3 volumes. Kerberos is supported only for the NFSv4.1 protocol. | |`This NetApp account has no configured Active Directory connections` | Configure Active Directory for the NetApp account with fields **KDC IP** and **AD Server Name**. See [Configure the Azure portal](configure-kerberos-encryption.md#configure-the-azure-portal) for instructions. | |`Mismatch between KerberosEnabled flag value and ExportPolicyRule's access type parameter values.` | Azure NetApp Files does not support converting a plain NFSv4.1 volume to Kerberos NFSv4.1 volume, and vice-versa. |
-|`mount.nfs: access denied by server when mounting volume <SMB_SERVER_NAME-XXX.DOMAIN_NAME>/<VOLUME_NAME>` <br> Example: `smb-test-64d9.contoso.com:/nfs41-vol101` | <ol><li> Ensure that the A/PTR records are properly set up and exist in the Active Directory for the server name `smb-test-64d9.contoso.com`. <br> In the NFS client, if `nslookup` of `smb-test-64d9.contoso.com` resolves to IP address IP1 (that is, `10.1.1.68`), then `nslookup` of IP1 must resolve to only one record (that is, `smb-test-64d9.contoso.com`). `nslookup` of IP1 *must* not resolve to multiple names. </li> <li>Set AES-256 for the NFS machine account of type `NFS-<Smb NETBIOS NAME>-<few random characters>` on AD using either PowerShell or the UI. <br> Example commands: <ul><li>`Set-ADComputer <NFS_MACHINE_ACCOUNT_NAME> -KerberosEncryptionType AES256` </li><li>`Set-ADComputer NFS-SMB-TEST-64 -KerberosEncryptionType AES256` </li></ul> </li> <li>Ensure that the time of the NFS client, AD, and Azure NetApp Files storage software is synchronized with each other and is within a five-minute skew range. </li> <li>Get the Kerberos ticket on the NFS client using the command `kinit <administrator>`.</li> <li>Reduce the NFS client hostname to less than 15 characters and perform the realm join again. </li><li>Restart the NFS client and the `rpcgssd` service as follows. The command might vary depending on the OS.<br> RHEL 7: <br> `service nfs restart` <br> `service rpcgssd restart` <br> CentOS 8: <br> `systemctl enable nfs-client.target && systemctl start nfs-client.target` <br> Ubuntu: <br> (Restart the `rpc-gssd` service.) <br> `sudo systemctl start rpc-gssd.service` </ul>|
+|`mount.nfs: access denied by server when mounting volume <SMB_SERVER_NAME-XXX.DOMAIN_NAME>/<VOLUME_NAME>` <br> Example: `smb-test-64d9.contoso.com:/nfs41-vol101` | <ol><li> Ensure that the A/PTR records are properly set up and exist in the Active Directory for the server name `smb-test-64d9.contoso.com`. <br> In the NFS client, if `nslookup` of `smb-test-64d9.contoso.com` resolves to IP address IP1 (that is, `10.1.1.68`), then `nslookup` of IP1 must resolve to only one record (that is, `smb-test-64d9.contoso.com`). `nslookup` of IP1 *must* not resolve to multiple names. </li> <li>Set AES-256 for the NFS machine account of type `NFS-<Smb NETBIOS NAME>-<few random characters>` on AD using either PowerShell or the UI. <br> Example commands: <ul><li>`Set-ADComputer <NFS_MACHINE_ACCOUNT_NAME> -KerberosEncryptionType AES256` </li><li>`Set-ADComputer NFS-SMB-TEST-64 -KerberosEncryptionType AES256` </li></ul> </li> <li>Ensure that the time of the NFS client, AD, and Azure NetApp Files storage software is synchronized with each other and is within a five-minute skew range. </li> <li>Get the Kerberos ticket on the NFS client using the command `kinit <administrator>`.</li> <li>Reduce the NFS client hostname to fewer than 15 characters and perform the realm join again. </li><li>Restart the NFS client and the `rpcgssd` service as follows. The command might vary depending on the OS.<br> RHEL 7: <br> `service nfs restart` <br> `service rpcgssd restart` <br> CentOS 8: <br> `systemctl enable nfs-client.target && systemctl start nfs-client.target` <br> Ubuntu: <br> (Restart the `rpc-gssd` service.) <br> `sudo systemctl start rpc-gssd.service` </ul>|
|`mount.nfs: an incorrect mount option was specified` | The issue might be related to the NFS client issue. Reboot the NFS client. | |`Hostname lookup failed` | You need to create a reverse lookup zone on the DNS server, and then add a PTR record of the AD host machine in that reverse lookup zone. <br> For example, assume that the IP address of the AD machine is `10.1.1.4`, the hostname of the AD machine (as found by using the hostname command) is `AD1`, and the domain name is `contoso.com`. The PTR record added to the reverse lookup zone should be `10.1.1.4 -> AD1.contoso.com`. | |`Volume creation fails due to unreachable DNS server` | Two possible solutions are available: <br> <ul><li> This error indicates that DNS is not reachable. The reason might be an incorrect DNS IP or a networking issue. Check the DNS IP entered in AD connection and make sure that the IP is correct. </li> <li> Make sure that the AD and the volume are in same region and in same VNet. If they are in different VNets, ensure that VNet peering is established between the two VNets. </li></ul> |
This article describes error messages and resolutions that can help you troubles
|-|-| | Error when creating an SMB volume with ldapEnabled as true: <br> `Error Message: ldapEnabled option is only supported with NFS protocol volume. ` | You cannot create an SMB volume with LDAP enabled. <br> Create SMB volumes with LDAP disabled. | | Error when updating the ldapEnabled parameter value for an existing volume: <br> `Error Message: ldapEnabled parameter is not allowed to update` | You cannot modify the LDAP option setting after creating a volume. <br> Do not update the LDAP option setting on a created volume. See [Configure AD DS LDAP with extended groups for NFS volume access](configure-ldap-extended-groups.md) for details. |
-| Error when creating an LDAP-enabled NFS volume: <br> `Could not query DNS server` <br> `Sample error message:` <br> `"log": time="2020-10-21 05:04:04.300" level=info msg=Res method=GET url=/v2/Volumes/070d0d72-d82c-c893-8ce3-17894e56cea3 x-correlation-id=9bb9e9fe-abb6-4eb5-a1e4-9e5fbb838813 x-request-id=c8032cb4-2453-05a9-6d61-31ca4a922d85 xresp="200: {\"created\":\"2020-10-21T05:02:55.000Z\",\"lifeCycleState\":\"error\",\"lifeCycleStateDetails\":\"Error when creating - Could not query DNS server. Verify that the network configuration is correct and that DNS servers are available.\",\"name\":\"smb1\",\"ownerId\ \":\"8c925a51-b913-11e9-b0de-9af5941b8ed0\",\"region\":\"westus2stage\",\"volumeId\":\"070d0d72-d82c-c893-8ce3-` | This error occurs because DNS is unreachable. <br> <ul><li> Check if you have configured the correct site (site scoping) for Azure NetApp Files. </li><li> The reason that DNS is unreachable might be an incorrect DNS IP address or networking issues. Check the DNS IP address entered in the AD connection to make sure that it is correct. </li><li> Make sure that the AD and the volume are in the same region and the same VNet. If they are in different VNets, ensure that VNet peering is established between the two VNets.</li></ul> |
+| Error when creating an LDAP-enabled NFS volume: <br> `Could not query DNS server` <br> `Sample error message:` <br> `"log": time="2020-10-21 05:04:04.300" level=info msg=Res method=GET url=/v2/Volumes/070d0d72-d82c-c893-8ce3-17894e56cea3 x-correlation-id=9bb9e9fe-abb6-4eb5-a1e4-9e5fbb838813 x-request-id=c8032cb4-2453-05a9-6d61-31ca4a922d85 xresp="200: {\"created\":\"2020-10-21T05:02:55.000Z\",\"lifeCycleState\":\"error\",\"lifeCycleStateDetails\":\"Error when creating - Could not query DNS server. Verify that the network configuration is correct and that DNS servers are available.\",\"name\":\"smb1\",\"ownerId\ \":\"8c925a51-b913-11e9-b0de-9af5941b8ed0\",\"region\":\"westus2stage\",\"volumeId\":\"070d0d72-d82c-c893-8ce3-` | This error occurs because DNS is unreachable. <br> <ul><li> Check if you've configured the correct site (site scoping) for Azure NetApp Files. </li><li> The reason that DNS is unreachable might be an incorrect DNS IP address or networking issues. Check the DNS IP address entered in the AD connection to make sure that it is correct. </li><li> Make sure that the AD and the volume are in the same region and the same VNet. If they are in different VNets, ensure that VNet peering is established between the two VNets.</li></ul> |
| Error when creating volume from a snapshot: <br> `Aggregate does not exist` | Azure NetApp Files doesnΓÇÖt support provisioning a new, LDAP-enabled volume from a snapshot that belongs to an LDAP-disabled volume. <br> Try creating new an LDAP-disabled volume from the given snapshot. | ## Errors for volume allocation
azure-resource-manager Bicep Config Linter https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/bicep/bicep-config-linter.md
Title: Linter settings for Bicep config description: Describes how to customize configuration values for the Bicep linter Previously updated : 07/29/2022 Last updated : 08/01/2022 # Add linter settings in the Bicep config file
The following example shows the rules that are available for configuration.
"adminusername-should-not-be-literal": { "level": "warning" },
+ "artifacts-parameters": {
+ "level": "warning"
+ },
"no-hardcoded-env-urls": { "level": "warning" },
azure-resource-manager Deploy Powershell https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/bicep/deploy-powershell.md
description: Use Azure Resource Manager and Azure PowerShell to deploy resources
Previously updated : 10/01/2021 Last updated : 08/05/2022 # Deploy resources with Bicep and Azure PowerShell
azure-resource-manager Deploy To Resource Group https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/bicep/deploy-to-resource-group.md
Title: Use Bicep to deploy resources to resource groups description: Describes how to deploy resources in a Bicep file. It shows how to target more than one resource group. Previously updated : 09/02/2021 Last updated : 08/05/2022 # Resource group deployments with Bicep files
To deploy to a resource group, use the resource group deployment commands.
# [Azure CLI](#tab/azure-cli)
-For Azure CLI, use [az deployment group create](/cli/azure/deployment/group#az-deployment-group-create). The following example deploys a template to create a resource group:
+For Azure CLI, use [az deployment group create](/cli/azure/deployment/group#az-deployment-group-create). The following example deploys a template to create a resource group. The resource group you specify in the `--resource-group` parameter is the **target resource group**.
```azurecli-interactive az deployment group create \
az deployment group create \
# [PowerShell](#tab/azure-powershell)
-For the PowerShell deployment command, use [New-AzResourceGroupDeployment](/powershell/module/az.resources/new-azresourcegroupdeployment). The following example deploys a template to create a resource group:
+For the PowerShell deployment command, use [New-AzResourceGroupDeployment](/powershell/module/az.resources/new-azresourcegroupdeployment). The following example deploys a template to create a resource group. The resource group you specify in the `-ResourceGroupName` parameter is the **target resource group**.
```azurepowershell-interactive New-AzResourceGroupDeployment `
azure-resource-manager Linter Rule Artifacts Parameters https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/bicep/linter-rule-artifacts-parameters.md
+
+ Title: Linter rule - artifacts parameters
+description: Linter rule - artifacts parameters
+ Last updated : 08/05/2022++
+# Linter rule - artifacts parameters
+
+This rule verifies whether the artifacts parameters are defined correctly. The following conditions must be met to pass the test:
+
+- If you provide one parameter (either `_artifactsLocation` or `_artifactsLocationSasToken`), you must provide the other.
+- `_artifactsLocation` must be a string.
+- If `_artifactsLocation` has a default value, it must be either `deployment().properties.templateLink.uri` or a raw URL for its default value.
+- `_artifactsLocationSasToken` must be a secure string.
+- If `_artifactsLocationSasToken` has a default value, it must be an empty string.
+- If a referenced module has an `_artifactsLocation` or `_artifactsLocationSasToken` parameter, a value must be passed in for those parameters, even if they have default values in the module.
+
+## Linter rule code
+
+Use the following value in the [Bicep configuration file](bicep-config-linter.md) to customize rule settings:
+
+`artifacts-parameters`
+
+## Solution
+
+The following example fails this test because `_artifactsLocationSasToken` is missing:
+
+```bicep
+@description('The base URI where artifacts required by this template are located including a trailing \'/\'')
+param _artifactsLocation string = deployment().properties.templateLink.uri
+
+...
+```
+
+The next example fails this test because `_artifactsLocation` must be either `deployment().properties.templateLink.uri` or a raw URL when the default value is provided, and the default value of `_artifactsLocationSasToken` is not an empty string.
+
+```bicep
+@description('The base URI where artifacts required by this template are located including a trailing \'/\'')
+param _artifactsLocation string = 'something'
+
+@description('SAS Token for accessing script path')
+@secure()
+param _artifactsLocationSasToken string = 'something'
+
+...
+````
+
+This example passes this test.
+
+```bicep
+@description('The base URI where artifacts required by this template are located including a trailing \'/\'')
+param _artifactsLocation string = deployment().properties.templateLink.uri
+
+@description('SAS Token for accessing script path')
+@secure()
+param _artifactsLocationSasToken string = ''
+
+...
+```
+
+## Next steps
+
+For more information about the linter, see [Use Bicep linter](./linter.md).
azure-resource-manager Linter https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/bicep/linter.md
The linter is integrated into the Bicep CLI and the Bicep extension for Visual S
The default set of linter rules is minimal and taken from [arm-ttk test cases](../templates/template-test-cases.md). The extension and Bicep CLI check the following rules, which are set to the warning level. - [adminusername-should-not-be-literal](./linter-rule-admin-username-should-not-be-literal.md)
+- [artifacts-parameters](./linter-rule-artifacts-parameters.md)
- [max-outputs](./linter-rule-max-outputs.md) - [max-params](./linter-rule-max-parameters.md) - [max-resources](./linter-rule-max-resources.md)
azure-resource-manager Best Practices https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/templates/best-practices.md
The following information can be helpful when you work with [resources](./syntax
] ```
+ If your ARM template is stored in a `.jsonc` file, comments using the `//` syntax are supported, as shown here.
+
+ ```javascript
+ "resources": [
+ {
+ // This storage account is used to store the VM disks.
+ "name": "[variables('storageAccountName')]",
+ "type": "Microsoft.Storage/storageAccounts",
+ "apiVersion": "2019-06-01",
+ "location": "[resourceGroup().location]",
+ ...
+ }
+ ]
+ ```
+
+ For more details about comments and metadata see [Understand the structure and syntax of ARM templates](/azure/azure-resource-manager/templates/syntax#comments-and-metadata).
+ * If you use a *public endpoint* in your template (such as an Azure Blob storage public endpoint), *don't hard-code* the namespace. Use the `reference` function to dynamically retrieve the namespace. You can use this approach to deploy the template to different public namespace environments without manually changing the endpoint in the template. Set the API version to the same version that you're using for the storage account in your template. ```json
The following information can be helpful when you work with [resources](./syntax
* Specify explicit values for properties that have default values that could change over time. For example, if you're deploying an AKS cluster, you can either specify or omit the `kubernetesVersion` property. If you don't specify it, then [the cluster is defaulted to the N-1 minor version and latest patch](../../aks/supported-kubernetes-versions.md#azure-portal-and-cli-versions). When you deploy the cluster using an ARM template, this default behavior might not be what you expect. Redeploying your template may result in the cluster being upgraded to a new Kubernetes version unexpectedly. Instead, consider specifying an explicit version number and then manually changing it when you're ready to upgrade your cluster.
+## Comments
+
+In addition to the `comments` property, comments using the `//` syntax are supported. For more details about comments and metadata see [Understand the structure and syntax of ARM templates](/azure/azure-resource-manager/templates/syntax#comments-and-metadata). You may choose to save JSON files that contain `//` comments using the `.jsonc` file extension, to indicate the JSON file contains comments. The ARM service will also accept comments in any JSON file including parameters files.
+
+## Visual Studio Code ARM Tools
+
+Working with ARM templates is much easier with the Azure Resource Manager (ARM) Tools for Visual Studio Code. This extension provides language support, resource snippets, and resource auto-completion to help you create and validate Azure Resource Manager templates. To learn more and install the extension, see [Azure Resource Manager (ARM) Tools](https://marketplace.visualstudio.com/items?itemName=msazurermtools.azurerm-vscode-tools).
+ ## Use test toolkit The ARM template test toolkit is a script that checks whether your template uses recommended practices. When your template isn't compliant with recommended practices, it returns a list of warnings with suggested changes. The test toolkit can help you learn how to implement best practices in your template.
azure-resource-manager Deploy Cli https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/templates/deploy-cli.md
az deployment group create \
--parameters storageAccountType=Standard_GRS ```
+The value of the `--template-file` parameter must be a Bicep file or a `.json` or `.jsonc` file. The `.jsonc` file extension indicates the file can contain `//` style comments. The ARM system accepts `//` comments in `.json` files. It does not care about the file extension. For more details about comments and metadata see [Understand the structure and syntax of ARM templates](/azure/azure-resource-manager/templates/syntax#comments-and-metadata).
+ The Azure deployment template can take a few minutes to complete. When it finishes, you see a message that includes the result: ```output
az deployment group create \
--parameters '@storage.parameters.json' ```
-## Handle extended JSON format
+## Comments and the extended JSON format
+
+You can include `//` style comments in your parameter file, but you must name the file with a `.jsonc` extension.
+
+```azurecli-interactive
+az deployment group create \
+ --name ExampleDeployment \
+ --resource-group ExampleGroup \
+ --template-file storage.json \
+ --parameters '@storage.parameters.jsonc'
+```
+For more details about comments and metadata see [Understand the structure and syntax of ARM templates](/azure/azure-resource-manager/templates/syntax#comments-and-metadata).
-To deploy a template with multi-line strings or comments using Azure CLI with version 2.3.0 or older, you must use the `--handle-extended-json-format` switch. For example:
+If you are using Azure CLI with version 2.3.0 or older, you can deploy a template with multi-line strings or comments using the `--handle-extended-json-format` switch. For example:
```json {
azure-resource-manager Deploy To Resource Group https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-resource-manager/templates/deploy-to-resource-group.md
Title: Deploy resources to resource groups description: Describes how to deploy resources in an Azure Resource Manager template. It shows how to target more than one resource group. Previously updated : 01/19/2022 Last updated : 08/05/2022
To deploy to a resource group, use the resource group deployment commands.
# [Azure CLI](#tab/azure-cli)
-For Azure CLI, use [az deployment group create](/cli/azure/deployment/group#az-deployment-group-create). The following example deploys a template to create a resource group:
+For Azure CLI, use [az deployment group create](/cli/azure/deployment/group#az-deployment-group-create). The following example deploys a template to create a resource group. The resource group you specify in the `--resource-group` parameter is the **target resource group**.
```azurecli-interactive az deployment group create \
az deployment group create \
# [PowerShell](#tab/azure-powershell)
-For the PowerShell deployment command, use [New-AzResourceGroupDeployment](/powershell/module/az.resources/new-azresourcegroupdeployment). The following example deploys a template to create a resource group:
+For the PowerShell deployment command, use [New-AzResourceGroupDeployment](/powershell/module/az.resources/new-azresourcegroupdeployment). The following example deploys a template to create a resource group. The resource group you specify in the `-ResourceGroupName` parameter is the **target resource group**.
```azurepowershell-interactive New-AzResourceGroupDeployment `
azure-video-indexer Connect To Azure https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/connect-to-azure.md
# Create an Azure Video Indexer account + When creating an Azure Video Indexer account, you can choose a free trial account (where you get a certain number of free indexing minutes) or a paid option (where you're not limited by the quota). With a free trial, Azure Video Indexer provides up to 600 minutes of free indexing to users and up to 2400 minutes of free indexing to users that subscribe to the Azure Video Indexer API on the [developer portal](https://aka.ms/avam-dev-portal). With the paid options, Azure Video Indexer offers two types of accounts: classic accounts(General Availability), and ARM-based accounts(Public Preview). Main difference between the two is account management platform. While classic accounts are built on the API Management, ARM-based accounts management is built on Azure, which enables apply access control to all services with role-based access control (Azure RBAC) natively. > [!NOTE]
azure-video-indexer Create Account Portal https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/create-account-portal.md
Last updated 06/10/2022
# Get started with Azure Video Indexer in Azure portal + This Quickstart walks you through the steps to get started with Azure Video Indexer. You'll create an Azure Video Indexer account and its accompanying resources by using the Azure portal. To start using Azure Video Indexer, you'll need to create an Azure Video Indexer account. The account needs to be associated with a [Media Services][docs-ms] resource and a [managed identity][docs-uami]. The managed identity will need to have Contributor permissions role on the Media Services.
azure-video-indexer Customize Content Models Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/customize-content-models-overview.md
# Customizing content models in Azure Video Indexer + Azure Video Indexer allows you to customize some of its models to be adapted to your specific use case. These models include [brands](customize-brands-model-overview.md), [language](customize-language-model-overview.md), and [person](customize-person-model-overview.md). You can easily customize these models using the Azure Video Indexer website or API. This article gives links to articles that explain the benefits of each type of customization. The article also links to how-to guides that show how you can implement the customization of each model.
azure-video-indexer Customize Person Model Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/customize-person-model-overview.md
# Customize a Person model in Azure Video Indexer + Azure Video Indexer supports celebrity recognition in your videos. The celebrity recognition feature covers approximately one million faces based on commonly requested data source such as IMDB, Wikipedia, and top LinkedIn influencers. Faces that are not recognized by Azure Video Indexer are still detected but are left unnamed. Customers can build custom Person models and enable Azure Video Indexer to recognize faces that are not recognized by default. Customers can build these Person models by pairing a person's name with image files of the person's face. If your account caters to different use-cases, you can benefit from being able to create multiple Person models per account. For example, if the content in your account is meant to be sorted into different channels, you might want to create a separate Person model for each channel.
azure-video-indexer Customize Person Model With Api https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/customize-person-model-with-api.md
# Customize a Person model with the Azure Video Indexer API + Azure Video Indexer supports face detection and celebrity recognition for video content. The celebrity recognition feature covers about one million faces based on commonly requested data source such as IMDB, Wikipedia, and top LinkedIn influencers. Faces that aren't recognized by the celebrity recognition feature are detected but left unnamed. After you upload your video to Azure Video Indexer and get results back, you can go back and name the faces that weren't recognized. Once you label a face with a name, the face and name get added to your account's Person model. Azure Video Indexer will then recognize this face in your future videos and past videos. You can use the Azure Video Indexer API to edit faces that were detected in a video, as described in this topic. You can also use the Azure Video Indexer website, as described in [Customize Person model using the Azure Video Indexer website](customize-person-model-with-api.md).
azure-video-indexer Customize Person Model With Website https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/customize-person-model-with-website.md
# Customize a Person model with the Azure Video Indexer website + Azure Video Indexer supports celebrity recognition for video content. The celebrity recognition feature covers approximately one million faces based on commonly requested data source such as IMDB, Wikipedia, and top LinkedIn influencers. For a detailed overview, see [Customize a Person model in Azure Video Indexer](customize-person-model-overview.md). You can use the Azure Video Indexer website to edit faces that were detected in a video, as described in this article. You can also use the API, as described in [Customize a Person model using APIs](customize-person-model-with-api.md).
azure-video-indexer Deploy With Arm Template https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/deploy-with-arm-template.md
# Tutorial: Deploy Azure Video Indexer by using an ARM template + In this tutorial, you'll create an Azure Video Indexer account by using the Azure Resource Manager template (ARM template, which is in preview). The resource will be deployed to your subscription and will create the Azure Video Indexer resource based on parameters defined in the *avam.template* file. > [!NOTE]
azure-video-indexer Limited Access Features https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/limited-access-features.md
Customers and partners who wish to use Limited Access features of Azure Video In
The Azure Video Indexer service is made available to customers and partners under the terms governing their subscription to Microsoft Azure Services (including the [Service Specific Terms](https://www.microsoft.com/licensing/terms/productoffering/MicrosoftAzure/MCA#ServiceSpecificTerms)). Please review these terms carefully as they contain important conditions and obligations governing your use of Azure Video Indexer.
+## Limited access features
+
+This section talks about limited access features in Azure Video Indexer.
+
+|When did I create the account?|Trial Account (Free)| Paid Account <br/>(classic or ARM-based)|
+||||
+|Existing VI accounts <br/><br/>created before June 21, 2022|Able to access face identification, customization and celebrities recognition till June 2023. <br/><br/>**Recommended**: Move to a paid account and afterward fill in the [intake form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUMkZIOUE1R0YwMkU0M1NMUTA0QVNXVDlKNiQlQCN0PWcu) and based on the eligibility criteria we will enable the features also after the grace period. |Able to access face identification, customization and celebrities recognition till June 2023\*.<br/><br/>**Recommended**: fill in the [intake form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUMkZIOUE1R0YwMkU0M1NMUTA0QVNXVDlKNiQlQCN0PWcu) and based on the eligibility criteria we will enable the features also after the grace period. <br/><br/>We proactively sent emails to these customers + AEs.|
+|New VI accounts <br/><br/>created after June 21, 2022 |Not able the access face identification, customization and celebrities recognition as of today. <br/><br/>**Recommended**: Move to a paid account and afterward fill in the [intake form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUMkZIOUE1R0YwMkU0M1NMUTA0QVNXVDlKNiQlQCN0PWcu). Based on the eligibility criteria we will enable the features (after max 10 days).|Azure Video Indexer disables the access face identification, customization and celebrities recognition as of today by default, but gives the option to enable it. <br/><br/>**Recommended**: Fill in the [intake form](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUMkZIOUE1R0YwMkU0M1NMUTA0QVNXVDlKNiQlQCN0PWcu) and based on the eligibility criteria we will enable the features (after max 10 days).|
+
+\*In Brazil South we also disabled the face detection.
+ ## Help and support FAQ about Limited Access can be found [here](https://aka.ms/limitedaccesscogservices).
azure-video-indexer Matched Person https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/matched-person.md
# Matched person (preview) + Azure Video Indexer matches observed people that were detected in the video with the corresponding faces ("People" insight). To produce the matching algorithm, the bounding boxes for both the faces and the observed people are assigned spatially along the video. The API returns the confidence level of each matching. The following are some scenarios that benefit from this feature:
The following JSON response illustrates what Azure Video Indexer returns when tr
It's important to note the limitations of Mapped person, to avoid or mitigate the effects of miss matches between people or people who have no matches.
-**Precondition** for the matching is that the person that showing in the Observed person was detected and can be found in the People insight.
+**Precondition** for the matching is that the person that showing in the observed faces was detected and can be found in the People insight.
**Pose**: The tracks are optimized to handle observed people who most often appear on the front.
-**Obstructions**: There may be miss matching between persons and observed people where there are obstruction (people or faces overlapping each other).
-**Spatial allocation per frame**: There may be miss matching where different people appear in the same spatial position relatively to the frame in a short time.
+**Obstructions**: There is no match between faces and observed people where there are obstruction (people or faces overlapping each other).
+**Spatial allocation per frame**: There is no match where different people appear in the same spatial position relatively to the frame in a short time.
See the limitations of Observed people: [Trace observed people in a video](observed-people-tracing.md)
azure-video-indexer Video Indexer Get Started https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/video-indexer-get-started.md
# Quickstart: How to sign up and upload your first video + This getting started quickstart shows how to sign in to the Azure Video Indexer website and how to upload your first video. When creating an Azure Video Indexer account, you can choose a free trial account (where you get a certain number of free indexing minutes) or a paid option (where you aren't limited by the quota). With free trial, Azure Video Indexer provides up to 600 minutes of free indexing to website users and up to 2400 minutes of free indexing to API users. With paid option, you create an Azure Video Indexer account that is [connected to your Azure subscription and an Azure Media Services account](connect-to-azure.md). You pay for minutes indexed, for more information, see [Media Services pricing](https://azure.microsoft.com/pricing/details/media-services/).
azure-video-indexer Video Indexer Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-video-indexer/video-indexer-overview.md
[!INCLUDE [regulation](./includes/regulation.md)] Azure Video Indexer is a cloud application, part of Azure Applied AI Services, built on Azure Media Services and Azure Cognitive Services (such as the Face, Translator, Computer Vision, and Speech). It enables you to extract the insights from your videos using Azure Video Indexer video and audio models.
azure-vmware Enable Public Ip Nsx Edge https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-vmware/enable-public-ip-nsx-edge.md
A Sourced Network Translation Service (SNAT) with Port Address Translation (PAT)
Logging can be enabled by way of the logging slider. For more information on NSX-T NAT configuration and options, see the [NSX-T NAT Administration Guide](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-7AD2C384-4303-4D6C-A44A-DEF45AA18A92.html)
-## No NAT rule for specific address ranges
+### No NAT rule for specific address ranges
A No NAT rule can be used to exclude certain matches from performing Network Address Translation. This policy can be used to allow private IP traffic to bypass the NAT rule. 1. From your Azure VMware Solution private cloud, select **vCenter Credentials**. 2. Locate your NSX-T URL and credentials.
-3. Log in to **VMWare NSX-T** and then select **3 NAT Rules**.
-1. Select the T1 Router and then select **ADD NAT RULE**.
+3. Log in to **VMWare NSX-T** and then select **NAT Rules**.
+1. Select the T1 Router and then select **ADD NAT RULE**.
+1. The **Source IP** is **Any** and **Destination IP** is the Azure VMware Solution reserved Public IP.
1. Select **SAVE**. ### Inbound Internet Access for VMs
azure-web-pubsub Concept Billing Model https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/concept-billing-model.md
Title: Billing model of Azure Web PubSub service description: An overview of key concepts about billing model of Azure Web PubSub service.--++ Previously updated : 11/08/2021 Last updated : 07/29/2022
-# Billing model of Azure Web PubSub service
+# Billing model for Azure Web PubSub service
-The billing model for Azure Web PubSub service is based on the number of units and the size of outbound traffic (message count). This article explains how units and outbound traffic (message count) are defined and counted for billing.
+The billing model for Azure Web PubSub service is based on the number of units allocated and the message count of outbound traffic. This article explains how units and outbound traffic (message count) are defined and counted for billing.
-## How units are counted with billing model
+## Terms used in billing
-### What is the connection
+### Connection
-The connection is between client and service. You could monitor the connection count in Azure portal. The view of "Connection (Max)" shows the maximum number of connections in a specific period.
+A *connection*, also known as a client or a client connection, represents an individual WebSocket connection connected to the Web PubSub service.
-### What is the unit
+### Unit
-The unit is an abstract concept of the capability of Azure Web PubSub service. Each unit supports 1,000 concurrent connections at most. Each Azure Web PubSub service instance could be 1, 2, 5, 10, 20, 50 or 100 units. So the unit count specifies how many connections your Web PubSub service instance can accept.
+A *unit* is an abstract concept of the capability of Web PubSub service. Each unit supports up to 1,000 concurrent connections. Each Web PubSub service instance can have 1, 2, 5, 10, 20, 50 or 100 units. The unit count * 1000 equals the maximum number of connections your Web PubSub service instance can accept.
-### How units are counted with billing model
+In production, it's recommended to plan for no more than 80% unit utilization before scaling up to more units to maintain acceptable system performance. For more information, see [Performance guide for Azure Web PubSub service](concept-performance.md).
-The units are counted based on the number of units and the usage time (seconds) of the units, and billed daily.
+### Message count
-For example, imagine you have one Azure Web PubSub service instance with five units, scale up to 10 units from 10:00 AM to 16:00 PM and then scale back to five units after 16:00 PM. Total usage for the day is 5 units for 18 hours and 10 units for 6 hours.
+The *message count* is an abstract concept for billing purposes. It's defined as the size of outbound traffic (bytes) in 2-KB increments, with each increment counting as one message for billing. For example, 100 KB of traffic is counted as 50 messages.
-> Total units are used for billing = (5 units * 18 hours + 10 units * 6 hours) / 24 hours = 6.25 Unit/Day
-
-## How outbound traffic is counted with billing model
-
-### What is inbound/outbound traffic
+### Outbound traffic
-The outbound traffic is the messages sent out of Azure Web PubSub service. You could monitor the outbound traffic by Azure portal. The view of "Outbound Traffic (total)" shows the aggregated outbound messages size (bytes) in a specific period.
+The *outbound traffic* is the messages sent out of Web PubSub service.
- The messages broadcasted from service to receivers. - The messages sent from the service to the upstream webhooks.-- The resource logs with [live trace tool](./howto-troubleshoot-resource-logs.md#capture-resource-logs-by-using-the-live-trace-tool).
+- The resource logs with live trace tool.
-The inbound traffic is the messages sent to the Azure Web PubSub service.
+### Inbound traffic
+
+The *inbound traffic* is the messages sent to the Azure Web PubSub service.
- The messages sent from clients to service. - The messages sent from server or functions to service.
-### What is message count
-
-The message count for billing purpose is an abstract concept and defined as the size of outbound traffic (bytes) divided in 2 KB. For example, 100-KB traffic is counted as 50 messages.
+For more information, see [Metrics in Azure Web PubSub service](concept-metrics.md).
-### How traffic is counted with billing model
+## How units are counted for billing
-Only the outbound traffic is counted for billing.
+The units are counted based on the number of units and the usage time in seconds, and billed daily.
-For example, imagine you have an application with Azure Web PubSub service and Azure Functions. One user broadcast 4 KB of data to 10 connections in a group. Total data is 4 KB for upstream from service to function and 40 KB from the service broadcast to 10 connections * 4 KB each.
+For example, imagine you have one Web PubSub Enterprise tier instance with five units allocated. You've added a custom scale condition to scale up to 10 units from 10:00 AM to 16:00 PM and then scale back to five units after 16:00 PM. Total usage for the day is 5 units for 18 hours and 10 units for 6 hours.
-> Outbound traffic for billing = 4 KB (upstream traffic) + 4 KB * 10 (service broadcasting to clients traffic) = 44 KB
+> Total units are used for billing = (5 units * 18 hours + 10 units * 6 hours) / 24 hours = 6.25 Unit/Day
-> Equivalent message count = 44 KB / 2 KB = 22
+## How outbound traffic is counted for billing
-The Azure Web PubSub service also offers a daily free quota of outbound traffic (message count) based on the usage of the units. The outbound traffic (message count) beyond the free quota is the extra outbound traffic not included in the base quota. Consider standard tier as example, the free quota is 2,000,000-KB outbound traffic (1,000,000 messages) per unit/day.
+Only the outbound traffic is counted for billing.
-In the previous unit usage example, the application uses 6.25 units per day that ensures the daily free quota as 12,500,000-KB outbound traffic (6.25 million messages). Assuming the daily outbound traffic is 30,000,000 KB (15 million messages), the extra messages will be 17,500,000-KB outbound traffic (8.75 million messages). As a result, you'll be billed with 6.25 standard units and 8.75 additional message units for the day.
+For example, imagine you have an application with Web PubSub service and Azure Functions. One user broadcast 4 KB of data to 10 connections in a group. Total data is 4 KB upstream from service to function, and 40 KB from the service broadcast to 10 connections * 4 KB each.
-## Pricing
+> Outbound traffic for billing = 4 KB (upstream traffic to Azure Functions) + 4 KB * 10 (from service broadcasting to clients) = 44 KB
-The Azure Web PubSub service offers multiple tiers with different pricing. For more information about Web PubSub pricing, see [Azure Web PubSub service pricing](https://azure.microsoft.com/pricing/details/web-pubsub).
+> Equivalent message count = 44 KB / 2 KB = 22
+The Web PubSub service also offers a daily free quota of outbound traffic (message count) based on the usage of the units. The outbound traffic beyond the free quota is the outbound traffic not included in the base quota. Consider standard tier as example: the free quota is 2,000,000-KB outbound traffic (1,000,000 messages) per unit per day.
+For example, an application that uses 6.25 units per day has a daily free quota of 12,500,000-KB outbound traffic or 6.25 million messages. Assuming that the actual daily outbound traffic is 30,000,000 KB (15 million messages), the extra messages above the free quota is 17,500,000-KB outbound traffic, which counts as 8.75 million messages for billing.
+As a result, you'll be billed with 6.25 standard units and 8.75 additional message units for the day.
+## Pricing
+The Web PubSub service offers multiple tiers with different pricing. For more information about Web PubSub pricing, see [Azure Web PubSub service pricing](https://azure.microsoft.com/pricing/details/web-pubsub).
azure-web-pubsub Concept Metrics https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/concept-metrics.md
Title: Metrics in Azure Web PubSub Service
-description: Metrics in Azure Web PubSub Service.
+ Title: Metrics in Azure Web PubSub service
+description: Metrics in Azure Web PubSub service.
# Metrics in Azure Web PubSub Service
-Azure Web PubSub Service has some built-in metrics and you and sets up [alerts](../azure-monitor/alerts/alerts-overview.md) base on metrics.
+Azure Web PubSub service has some built-in metrics and you and sets up [alerts](../azure-monitor/alerts/alerts-overview.md) base on metrics.
## Understand metrics
Metrics provide the running info of the service. The available metrics are:
|Metric|Unit|Recommended Aggregation Type|Description|Dimensions| |||||| |Connection Close Count|Count|Sum|The count of connections closed by various reasons.|ConnectionCloseCategory|
-|Connection Count|Count|Max / Avg|The amount of connection.|No Dimensions|
+|Connection Count|Count|Max / Avg|The number of connections to the service.|No Dimensions|
|Connection Open Count|Count|Sum|The count of new connections opened.|No Dimensions|
-|Connection Quota Utilization|Percent|Max / Avg|The percentage of connection connected relative to connection quota.|No Dimensions|
-|Inbound Traffic|Bytes|Sum|The inbound traffic of service|No Dimensions|
-|Outbound Traffic|Bytes|Sum|The outbound traffic of service|No Dimensions|
-|Server Load|Percent|Max / Avg|The percentage of server load|No Dimensions|
+|Connection Quota Utilization|Percent|Max / Avg|The percentage of connections relative to connection quota.|No Dimensions|
+|Inbound Traffic|Bytes|Sum|The inbound traffic to the service.|No Dimensions|
+|Outbound Traffic|Bytes|Sum|The outbound traffic from the service.|No Dimensions|
+|Server Load|Percent|Max / Avg|The percentage of server load.|No Dimensions|
### Understand Dimensions
The dimension available in some metrics:
- SendEventFailed: Event handler invokes failed. - EventHandlerNotFound: Event handler not found. - SlowClient: Too many messages queued up at service side, which needed to be sent.
- - ServiceTransientError: Internal server error
- - BadRequest: This caused by invalid hub name, wrong payload, etc.
- - ServiceReload: This is triggered when a connection is dropped due to an internal service component reload. This event doesn't indicate a malfunction and is part of normal service operation.
- - Unauthorized: The connection is unauthorized
+ - ServiceTransientError: Internal server error.
+ - BadRequest: Caused by an invalid hub name, wrong payload, etc.
+ - ServiceReload: Triggered when a connection is dropped due to an internal service component reload. This event doesn't indicate a malfunction and is part of normal service operation.
+ - Unauthorized: The connection is unauthorized.
Learn more about [multi-dimensional metrics](../azure-monitor/essentials/data-platform-metrics.md#multi-dimensional-metrics)
azure-web-pubsub Concept Service Internals https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/concept-service-internals.md
Previously updated : 11/08/2021 Last updated : 07/27/2022 # Azure Web PubSub service internals Azure Web PubSub Service provides an easy way to publish/subscribe messages using simple [WebSocket](https://tools.ietf.org/html/rfc6455) connections. -- Client can be written in any language having WebSocket support-- Both text and binary messages are supported within one connection-- A simple protocol for clients to do direct client-client message publish-- The service manages the WebSocket connections for you
+- Clients can be written in any language that has Websocket support.
+- Both text and binary messages are supported within one connection.
+- There's a simple protocol for clients to do direct client-to-client message publishing.
+- The service manages the WebSocket connections for you.
## Terms * **Service**: Azure Web PubSub Service. [!INCLUDE [Terms](includes/terms.md)]
-* **Client Connection** and **ConnectionId**: A client connects to the `/client` endpoint, when connected, a unique `connectionId` is generated by the service as the unique identity of the client connection. Users can then manage the client connection using this `connectionId`. Details are described in [Client Protocol](#client_protocol) section.
-
-* **Client Events**: Events are created during the lifecycle of a client connection. For example, a simple WebSocket client connection creates a `connect` event when it tries to connect to the service, a `connected` event when it successfully connected to the service, a `message` event when it sends messages to the service and a `disconnected` event when it disconnects from the service. Details about *client events* are illustrated in [Client Protocol](#client_protocol) section.
-
-* **Event Handler**: The event handler contains the logic to handle the client events. Register and configure event handlers in the service through the portal or Azure CLI beforehand. Details are described in [Event Handler](#event_handler) section. The place to host the event handler logic is considered as the server-side.
-
-* **Server**: The server can handle client events, manage client connections, and publish messages to groups. The server, comparing to the client, is trustworthy. Details about **server** are described in [Server Protocol](#server_protocol) section.
-
-<a name="workflow"></a>
- ## Workflow ![Diagram showing the Web PubSub service workflow.](./media/concept-service-internals/workflow.png)
-As illustrated by the above workflow graph:
-1. A *client* connects to the service `/client` endpoint using WebSocket transport. Service forward every WebSocket frame to the configured upstream(server). The WebSocket connection can connect with any custom subprotocol for the server to handle, or it can connect with the service-supported subprotocol `json.webpubsub.azure.v1`, which empowers the clients to do pub/sub directly. Details are described in [client protocol](#client_protocol).
-2. The service invokes the server using **CloudEvents HTTP protocol** on different client events. [**CloudEvents**](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) is a standardized and protocol-agnostic definition of the structure and metadata description of events hosted by the Cloud Native Computing Foundation (CNCF). Details are described in [server protocol](#server_protocol).
-3. Server can invoke the service using REST API to send messages to clients or to manage the connected clients. Details are described in [server protocol](#server_protocol)
-
-<a name="client_protocol"></a>
+Workflow as shown in the above graph:
+1. A *client* connects to the service `/client` endpoint using WebSocket transport. Service forward every WebSocket frame to the configured upstream(server). The WebSocket connection can connect with any custom subprotocol for the server to handle, or it can connect with the service-supported subprotocol `json.webpubsub.azure.v1`, which empowers the clients to do pub/sub directly. Details are described in [client protocol](#client-protocol).
+1. The service invokes the server using **CloudEvents HTTP protocol** on different client events. [**CloudEvents**](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) is a standardized and protocol-agnostic definition of the structure and metadata description of events hosted by the Cloud Native Computing Foundation (CNCF). Details are described in [server protocol](#server-protocol).
+1. The Web PubSub server can invoke the service using the REST API to send messages to clients or to manage the connected clients. Details are described in [server protocol](#server-protocol)
## Client protocol A client connection connects to the `/client` endpoint of the service using [WebSocket protocol](https://tools.ietf.org/html/rfc6455). The WebSocket protocol provides full-duplex communication channels over a single TCP connection and was standardized by the IETF as RFC 6455 in 2011. Most languages have native support to start WebSocket connections. Our service supports two kinds of clients:-- One is called [the simple WebSocket client](#simple_client)-- The other is called [the PubSub WebSocket client](#pubsub_client)-
-<a name="simple_client"></a>
+- One is called [the simple WebSocket client](#the-simple-websocket-client)
+- The other is called [the PubSub WebSocket client](#the-pubsub-websocket-client)
### The simple WebSocket client A simple WebSocket client, as the naming indicates, is a simple WebSocket connection. It can also have its custom subprotocol.
-For example, in JS, a simple WebSocket client can be created using:
+For example, in JS, a simple WebSocket client can be created using the following code.
```js // simple WebSocket client1 var client1 = new WebSocket('wss://test.webpubsub.azure.com/client/hubs/hub1');
A simple WebSocket client follows a client<->server architecture, as the below s
![Diagram showing the sequence for a client connection.](./media/concept-service-internals/simple-client-sequence.png)
-1. When the client starts WebSocket handshake, the service tries to invoke the `connect` event handler (the server) for WebSocket handshake. Users can use this handler to handle the WebSocket handshake, determine the subprotocol to use, auth the client, and join the client to some groups.
-2. When the client is successfully connected, the service invokes a `connected` event handler. It works as some notification and doesn't block the client from sending messages. Users can use this handler to do some data storage and can respond with messages sending to the client.
-2. When the client sends messages, the services trigger the `message` event to the event handler (the server) to handle the messages sent. This event is a general event containing the messages sent in a WebSocket frame. User needs to dispatch the messages on their own inside this event handler.
-3. When the client disconnects, the service tries to trigger the `disconnected` event to the event handler (the server) once it detects the disconnect.
+1. When the client starts a WebSocket handshake, the service tries to invoke the `connect` event handler (the server) for WebSocket handshake. Developers can use this handler to handle the WebSocket handshake, determine the subprotocol to use, authenticate the client, and join the client to groups.
+1. When the client is successfully connected, the service invokes a `connected` event handler. It works as a notification and doesn't block the client from sending messages. Developers can use this handler to do data storage and can respond with messages to the client.
+1. When the client sends messages, the service triggers the `message` event to the event handler (the server) to handle the messages sent. This event is a general event containing the messages sent in a WebSocket frame. Your code needs to dispatch the messages inside this event handler.
+1. When the client disconnects, the service tries to trigger the `disconnected` event to the event handler (the server) once it detects the disconnect.
The events fall into two categories:
-* synchronous events (blocking)
+* Synchronous events (blocking)
Synchronous events block the client workflow. When such an event trigger fails, the service drops the client connection. * `connect` * `message`
-* asynchronous events (non-blocking)
+* Asynchronous events (non-blocking)
Asynchronous events don't block the client workflow, it acts as some notification to the upstream event handler. When such an event trigger fails, the service logs the error detail. * `connected` * `disconnected`
-#### Scenarios:
-Such connection can be used in a typical client-server architecture, that the client sends messages to the server, and the server handles incoming messages using [Event Handlers](#event_handler). It can also be used when customers apply existing [subprotocols](https://www.iana.org/assignments/websocket/websocket.xml) in their application logic.
-
-<a name="pubsub_client"></a>
+#### Scenarios
+These connections can be used in a typical client-server architecture where the client sends messages to the server and the server handles incoming messages using [Event Handlers](#event-handler). It can also be used when customers apply existing [subprotocols](https://www.iana.org/assignments/websocket/websocket.xml) in their application logic.
### The PubSub WebSocket client
-The service also supports a specific subprotocol called `json.webpubsub.azure.v1`, which empowers the clients to do publish/subscribe directly instead of a round trip to the upstream server. We call the WebSocket connection with `json.webpubsub.azure.v1` subprotocol a PubSub WebSocket client.
+The service also supports a specific subprotocol called `json.webpubsub.azure.v1`, which empowers the clients to do publish/subscribe directly instead of a round trip to the upstream server. We call the WebSocket connection with `json.webpubsub.azure.v1` subprotocol a PubSub WebSocket client. For more information, see the [Web PubSub client specification](https://github.com/Azure/azure-webpubsub/blob/main/protocols/client/client-spec.md) on GitHub.
-For example, in JS, a PubSub WebSocket client can be created using:
+For example, in JS, a PubSub WebSocket client can be created using the following code.
```js // PubSub WebSocket client var pubsub = new WebSocket('wss://test.webpubsub.azure.com/client/hubs/hub1', 'json.webpubsub.azure.v1');
A PubSub WebSocket client can:
[PubSub WebSocket Subprotocol](./reference-json-webpubsub-subprotocol.md) contains the details of the `json.webpubsub.azure.v1` subprotocol.
-You may have noticed that for a [simple WebSocket client](#simple_client), the *server* is a MUST HAVE role to handle the events from clients. A simple WebSocket connection always triggers a `message` event when it sends messages, and always relies on the server-side to process messages and do other operations. With the help of the `json.webpubsub.azure.v1` subprotocol, an authorized client can join a group and publish messages to a group directly. It can also route messages to different upstream (event handlers) by customizing the *event* the message belongs.
+You may have noticed that for a [simple WebSocket client](#the-simple-websocket-client), the *server* is a **must have** role to handle the events from clients. A simple WebSocket connection always triggers a `message` event when it sends messages, and always relies on the server-side to process messages and do other operations. With the help of the `json.webpubsub.azure.v1` subprotocol, an authorized client can join a group and publish messages to a group directly. It can also route messages to different upstream (event handlers) by customizing the *event* the message belongs.
#### Scenarios: Such clients can be used when clients want to talk to each other. Messages are sent from `client2` to the service and the service delivers the message directly to `client1` if the clients are authorized to do so.
client2.onopen = e => {
As the above example shows, `client2` sends data directly to `client1` by publishing messages to `Group1` which `client1` is in.
-<a name="client_message_limit"></a>
- ### Client message limit The maximum allowed message size for one WebSocket frame is **1MB**.
-<a name="client_auth"></a>
+### Client authentication
-### Client Auth
+#### Authentication workflow
-#### Auth workflow
+Client uses a signed JWT token to connect to the service. The upstream can also reject the client when it's `connect` event handler of the incoming client. The event handler authenticates the client by specifying the `userId` and the `role`s the client has in the webhook response, or decline the client with 401. [Event handler](#event-handler) section describes it in detail.
-Client uses a signed JWT token to connect to the service. The upstream can also reject the client when it is `connect` event handler of the incoming client. The event handler auth the client by specifying the `userId` and the `role`s the client has in the webhook response, or decline the client with 401. [Event handler](#event_handler) section describes it in detail.
-
-The below graph describes the workflow:
+The following graph describes the workflow.
![Diagram showing the client authentication workflow.](./media/concept-service-internals/client-connect-workflow.png)
-As you may have noticed when we describe the PubSub WebSocket clients, that a client can publish to other clients only when it is *authorized* to. The `role`s of the client determines the *initial* permissions the client have:
+As you may have noticed when we describe the PubSub WebSocket clients, that a client can publish to other clients only when it's *authorized* to. The `role`s of the client determines the *initial* permissions the client have:
| Role | Permission | |||
As you may have noticed when we describe the PubSub WebSocket clients, that a cl
| `webpubsub.joinLeaveGroup.<group>` | The client can join/leave group `<group>`. | `webpubsub.sendToGroup.<group>` | The client can publish messages to group `<group>`.
-The server-side can also grant or revoke permissions of the client dynamically through [server protocol](#connection_manager) as to be illustrated in a later section.
-
-<a name="server_protocol"></a>
+The server-side can also grant or revoke permissions of the client dynamically through [server protocol](#connection-manager) as to be illustrated in a later section.
-## Server Protocol
+## Server protocol
Server protocol provides the functionality for the server to manage the client connections and the groups. In general, server protocol contains two roles:
-1. [Event handler](#event_handler)
-2. [Connection manager](#connection_manager)
-
-<a name="event_handler"></a>
+1. [Event handler](#event-handler)
+1. [Connection manager](#connection-manager)
### Event handler
-The event handler handles the incoming client events. Event handlers are registered and configured in the service through portal or Azure CLI beforehand so that when a client event is triggered, the service can identify if the event is expected to be handled or not. Now we use `PUSH` mode to invoke the event handler: that the event handler as the server side, exposes public accessible endpoint for the service to invoke when the event is triggered. It acts as a **webhook**.
+The event handler handles the incoming client events. Event handlers are registered and configured in the service through the portal or Azure CLI. When a client event is triggered, the service can identify if the event is to be handled or not. Now we use `PUSH` mode to invoke the event handler. The event handler on the server side exposes a publicly accessible endpoint for the service to invoke when the event is triggered. It acts as a **webhook**.
-Service delivers client events to the upstream webhook using the [CloudEvents HTTP protocol](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md).
+Web PubSub service delivers client events to the upstream webhook using the [CloudEvents HTTP protocol](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md).
-For every event, it formulates an HTTP POST request to the registered upstream and expects an HTTP response.
+For every event, the service formulates an HTTP POST request to the registered upstream and expects an HTTP response.
-The data sending from the service to the server is always in CloudEvents `binary` format.
+The data sent from the service to the server is always in CloudEvents `binary` format.
![Diagram showing the Web PubSub service event push mode.](./media/concept-service-internals/event-push.png) #### Upstream and Validation
-Event handlers need to be registered and configured in the service through portal or Azure CLI beforehand so that when a client event is triggered, the service can identify if the event is expected to be handled or not. For public preview, we use `PUSH` mode to invoke the event handler: that the event handler as the server side, exposes public accessible endpoint for the service to invoke when the event is triggered. It acts as a **webhook** **upstream**.
+Event handlers need to be registered and configured in the service through the portal or Azure CLI before first use. When a client event is triggered, the service can identify if the event must be handled or not. For public preview, we use `PUSH` mode to invoke the event handler. The event handler on the server side exposes publicly accessible endpoint for the service to invoke when the event is triggered. It acts as a **webhook** **upstream**.
-When configuring the webhook endpoint, the URL can use `{event}` parameter to define a URL template. The service calculates the value of the webhook URL dynamically when the client request comes in. For example, when a request `/client/hubs/chat` comes in, with a configured event handler URL pattern `http://host.com/api/{event}` for hub `chat`, when the client connects, it will first POST to this URL: `http://host.com/api/connect`. This can be useful when a PubSub WebSocket client sends custom events, that the event handler helps dispatch different events to different upstream. Note that the `{event}` parameter is not allowed in the URL domain name.
+The URL can use `{event}` parameter to define a URL template for the webhook handler. The service calculates the value of the webhook URL dynamically when the client request comes in. For example, when a request `/client/hubs/chat` comes in, with a configured event handler URL pattern `http://host.com/api/{event}` for hub `chat`, when the client connects, it will first POST to this URL: `http://host.com/api/connect`. This behavior can be useful when a PubSub WebSocket client sends custom events, that the event handler helps dispatch different events to different upstream. The `{event}` parameter isn't allowed in the URL domain name.
When setting up the event handler upstream through Azure portal or CLI, the service follows the [CloudEvents abuse protection](https://github.com/cloudevents/spec/blob/v1.0/http-webhook.md#4-abuse-protection) to validate the upstream webhook. The `WebHook-Request-Origin` request header is set to the service domain name `xxx.webpubsub.azure.com`, and it expects the response having header `WebHook-Allowed-Origin` to contain this domain name.
-When doing the validation, the `{event}` parameter is resolved to `validate`. For example, when trying to set the URL to `http://host.com/api/{event}`, the service tries to **OPTIONS** a request to `http://host.com/api/validate` and only when the response is valid the configure can be set successfully.
+When doing the validation, the `{event}` parameter is resolved to `validate`. For example, when trying to set the URL to `http://host.com/api/{event}`, the service tries to **OPTIONS** a request to `http://host.com/api/validate` and only when the response is valid the configuration can be set successfully.
-For now, we do not support [WebHook-Request-Rate](https://github.com/cloudevents/spec/blob/v1.0/http-webhook.md#414-webhook-request-rate) and [WebHook-Request-Callback](https://github.com/cloudevents/spec/blob/v1.0/http-webhook.md#413-webhook-request-callback).
+For now, we don't support [WebHook-Request-Rate](https://github.com/cloudevents/spec/blob/v1.0/http-webhook.md#414-webhook-request-rate) and [WebHook-Request-Callback](https://github.com/cloudevents/spec/blob/v1.0/http-webhook.md#413-webhook-request-callback).
#### Authentication between service and webhook - Anonymous mode-- Simple Auth that `code` is provided through the configured Webhook URL.-- Use AAD Auth, check [here](howto-use-managed-identity.md) for details.
+- Simple authentication that `code` is provided through the configured Webhook URL.
+- Use Azure Active Directory (Azure AD) authentication. For more information, see [how to use managed identity](howto-use-managed-identity.md) for details.
- Step1: Enable Identity for the Web PubSub service
- - Step2: Select from existing AAD application that stands for your webhook web app
+ - Step2: Select from existing Azure AD application that stands for your webhook web app
-<a name="connection_manager"></a>
### Connection manager
The server is by nature an authorized user. With the help of the *event handler
- Send messages to a client - Send messages to clients that belong to the same user - Add a client to a group
- - Add clients authed as the same user to a group
+ - Add clients authenticated as the same user to a group
- Remove a client from a group
- - Remove clients authed as the same user from a group
+ - Remove clients authenticated as the same user from a group
- Publish messages to a group It can also grant or revoke publish/join permissions for a PubSub client:
- - Grant Join/Publish permissions to some specific group or to all groups
- - Revoke Join/Publish permissions for some specific group or for all groups
- - Check if the client has permission to Join/Publish to some specific group or to all groups
+ - Grant publish/join permissions to some specific group or to all groups
+ - Revoke publish/joinh permissions for some specific group or for all groups
+ - Check if the client has permission to join or publish to some specific group or to all groups
-The service provides REST APIs for the server to do connection management:
+The service provides REST APIs for the server to do connection management.
![Diagram showing the Web PubSub service connection manager workflow.](./media/concept-service-internals/manager-rest.png) The detailed REST API protocol is defined [here][rest]. ### Summary
-You may have noticed that the *event handler role* handles communication from the service to the server while *the manager role* handles communication from the server to the service. So combing the two roles, the data flow between service and server looks as similar to below, using HTTP protocol:
+You may have noticed that the *event handler role* handles communication from the service to the server while *the manager role* handles communication from the server to the service. After combining the two roles, the data flow between service and server looks similar to the following diagram using HTTP protocol.
![Diagram showing the Web PubSub service bi-directional workflow.](./media/concept-service-internals/http-service-server.png)
azure-web-pubsub Howto Secure Shared Private Endpoints https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/howto-secure-shared-private-endpoints.md
# Secure Azure Web PubSub outbound traffic through Shared Private Endpoints
-If you're using [event handler](concept-service-internals.md#event_handler) in Azure Web PubSub Service, you might have outbound traffic to upstream. Upstream such as
+If you're using an [event handler](concept-service-internals.md#event-handler) in Azure Web PubSub Service, you might have outbound traffic to an upstream. Upstream such as
Azure Web App and Azure Functions, can be configured to accept connections from a list of virtual networks and refuse outside connections that originate from a public network. You can create an outbound [private endpoint connection](../private-link/private-endpoint-overview.md) to reach these endpoints. :::image type="content" alt-text="Diagram showing architecture of shared private endpoint." source="media\howto-secure-shared-private-endpoints\shared-private-endpoint-overview.png" border="false" :::
This outbound method is subject to the following requirements:
## Shared Private Link Resources Management
-Private endpoints of secured resources that are created through Azure Web PubSub Service APIs are referred to as *shared private link resources*. This is because you're "sharing" access to a resource, such as an Azure Function, that has been integrated with the [Azure Private Link service](https://azure.microsoft.com/services/private-link/). These private endpoints are created inside Azure Web PubSub Service execution environment and aren't directly visible to you.
+Private endpoints of secured resources that are created through Azure Web PubSub Service APIs are referred to as *shared private link resources*. This term is used because you're "sharing" access to a resource, such as an Azure Function that has been integrated with the [Azure Private Link service](https://azure.microsoft.com/services/private-link/). These private endpoints are created inside Azure Web PubSub Service execution environment and aren't directly visible to you.
> [!NOTE] > The examples in this article are based on the following assumptions:
The rest of the examples show how the _contoso-webpubsub_ service can be configu
1. In the Azure portal, go to your Azure Web PubSub Service resource. 1. In the menu pane, select **Networking**. Switch to **Private access** tab.
-1. Click **Add shared private endpoint**.
+1. Select **Add shared private endpoint**.
:::image type="content" alt-text="Screenshot of shared private endpoints management." source="media\howto-secure-shared-private-endpoints\portal-shared-private-endpoints-management.png" lightbox="media\howto-secure-shared-private-endpoints\portal-shared-private-endpoints-management.png" ::: 1. Fill in a name for the shared private endpoint. 1. Select the target linked resource either by selecting from your owned resources or by filling a resource ID.
-1. Click **Add**.
+1. Select **Add**.
:::image type="content" alt-text="Screenshot of adding a shared private endpoint." source="media\howto-secure-shared-private-endpoints\portal-shared-private-endpoints-add.png" :::
The contents of the *create-pe.json* file, which represent the request body to t
} ```
-The process of creating an outbound private endpoint is a long-running (asynchronous) operation. As in all asynchronous Azure operations, the `PUT` call returns an `Azure-AsyncOperation` header value that looks like the following:
+The process of creating an outbound private endpoint is a long-running (asynchronous) operation. As in all asynchronous Azure operations, the `PUT` call returns an `Azure-AsyncOperation` header value that looks like the following example.
```plaintext "Azure-AsyncOperation": "https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/contoso/providers/Microsoft.SignalRService/webPubSub/contoso-webpubsub/operationStatuses/c0786383-8d5f-4554-8d17-f16fcf482fb2?api-version=2021-06-01-preview"
Wait until the status changes to "Succeeded" before proceeding to the next steps
#### [Azure portal](#tab/azure-portal)
-1. In the Azure portal, select the **Networking** tab of your Function App and navigate to **Private endpoint connections**. Click **Configure your private endpoint connections**. After the asynchronous operation has succeeded, there should be a request for a private endpoint connection with the request message from the previous API call.
+1. In the Azure portal, select the **Networking** tab of your Function App and navigate to **Private endpoint connections**. Select **Configure your private endpoint connections**. After the asynchronous operation has succeeded, there should be a request for a private endpoint connection with the request message from the previous API call.
:::image type="content" alt-text="Screenshot of the Azure portal, showing the Private endpoint connections pane." source="media\howto-secure-shared-private-endpoints\portal-function-approve-private-endpoint.png" lightbox="media\howto-secure-shared-private-endpoints\portal-function-approve-private-endpoint.png" :::
It takes minutes for the approval to be propagated to Azure Web PubSub Service.
az rest --method get --uri https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/contoso/providers/Microsoft.SignalRService/webPubSub/contoso-webpubsub/sharedPrivateLinkResources/func-pe?api-version=2021-06-01-preview ```
-This would return a JSON, where the connection state would show up as "status" under the "properties" section.
+This command would return JSON, where the connection state would show up as "status" under the "properties" section.
```json {
This would return a JSON, where the connection state would show up as "status" u
```
-If the "Provisioning State" (`properties.provisioningState`) of the resource is `Succeeded` and "Connection State" (`properties.status`) is `Approved`, it means that the shared private link resource is functional and Azure Web PubSub Service can communicate over the private endpoint.
+If the "Provisioning State" (`properties.provisioningState`) of the resource is `Succeeded` and "Connection State" (`properties.status`) is `Approved`, it means that the shared private link resource is functional, and Azure Web PubSub Service can communicate over the private endpoint.
--
azure-web-pubsub Howto Service Tags https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/howto-service-tags.md
Title: Use service tags
-description: Use service tags to allow outbound traffic to your Azure Web PubSub Service
+description: Use service tags to allow outbound traffic to your Azure Web PubSub service.
- -+ Last updated 11/08/2021 # Use service tags for Azure Web PubSub Service
-You can use [Service Tags](../virtual-network/network-security-groups-overview.md#service-tags) for Azure Web PubSub Service when configuring [Network Security Group](../virtual-network/network-security-groups-overview.md#network-security-groups). It allows you to define inbound/outbound network security rule for Azure Web PubSub Service endpoints without need to hardcode IP addresses.
+You can use [Service Tags](../virtual-network/network-security-groups-overview.md#service-tags) for Azure Web PubSub service when configuring [Network Security Group](../virtual-network/network-security-groups-overview.md#network-security-groups). It allows you to define inbound/outbound network security rule for Azure Web PubSub Service endpoints without need to hardcode IP addresses.
-Azure Web PubSub Service manages these service tags. You can't create your own service tag or modify an existing one. Microsoft manages these address prefixes that match to the service tag and automatically updates the service tag as addresses change.
+Azure Web PubSub service manages these service tags. You can't create your own service tag or modify an existing tag. Microsoft manages the address prefixes that match to the service tag and automatically updates the service tag as addresses change.
> [!Note]
-> Starting from 15 August 2021, Azure Web PubSub Service supports bidirectional Service Tag for both inbound and outbound traffic.
+> Starting from 15 August 2021, Azure Web PubSub service supports bidirectional Service Tag for both inbound and outbound traffic.
## Use service tag via Azure CLI ### Configure outbound traffic
-You can allow outbound traffic to Azure Web PubSub Service by adding a new outbound network security rule:
+You can allow outbound traffic to Azure Web PubSub service by adding a new outbound network security rule:
```azurecli-interactive az network nsg rule create -n <rule-name> --nsg-name <nsg-name> -g <resource-group> --priority 100 --direction Outbound --destination-address-prefixes AzureWebPubSub
az network nsg rule create -n <rule-name> --nsg-name <nsg-name> -g <resource-gro
### Configure inbound traffic
-If you're using [event handler](concept-service-internals.md#event_handler), you can also allow inbound traffic from Azure Web PubSub Service by adding a new inbound network security rule:
+If you're using [event handler](concept-service-internals.md#event-handler), you can also allow inbound traffic from Azure Web PubSub service by adding a new inbound network security rule:
```azurecli-interactive az network nsg rule create -n <rule-name> --nsg-name <nsg-name> -g <resource-group> --priority 100 --direction Inbound --source-address-prefixes AzureWebPubSub
azure-web-pubsub Key Concepts https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/key-concepts.md
Title: Azure Web PubSub basic concepts about hubs, groups, and connections
-description: Better understand the terms used in Azure Web PubSub.
-
+description: Understand the basic concepts and terms used in Azure Web PubSub.
+ - Previously updated : 11/08/2021+ Last updated : 07/28/2022 # Azure Web PubSub basic concepts
-Azure Web PubSub Service helps you build real-time messaging web applications. The clients connect to the service using the [standard WebSocket protocol](https://datatracker.ietf.org/doc/html/rfc6455), and the service exposes [REST APIs](/rest/api/webpubsub) and SDKs for you to manage these clients.
+Azure Web PubSub service helps you build real-time messaging web applications. The clients connect to the service using the [standard WebSocket protocol](https://datatracker.ietf.org/doc/html/rfc6455), and the service exposes [REST APIs](/rest/api/webpubsub) and SDKs for you to manage these clients.
## Terms
A typical workflow using the service is shown as below:
As illustrated by the above workflow graph:
-1. A *client* connects to the service `/client` endpoint using WebSocket transport. Service forward every WebSocket frame to the configured upstream(server). The WebSocket connection can connect with any custom subprotocol for the server to handle, or it can connect with the service-supported subprotocol `json.webpubsub.azure.v1`, which empowers the clients to do pub/sub directly. Details are described in [client protocol](concept-service-internals.md#client_protocol).
+1. A *client* connects to the service `/client` endpoint using WebSocket transport. Service forward every WebSocket frame to the configured upstream(server). The WebSocket connection can connect with any custom subprotocol for the server to handle, or it can connect with the service-supported subprotocol `json.webpubsub.azure.v1`, which empowers the clients to do pub/sub directly. Details are described in [client protocol](concept-service-internals.md#client-protocol).
-2. The service invokes the server using **CloudEvents HTTP protocol** on different client events. [**CloudEvents**](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) is a standardized and protocol-agnostic definition of the structure and metadata description of events hosted by the Cloud Native Computing Foundation (CNCF). Details are described in [server protocol](concept-service-internals.md#server_protocol).
+2. The service invokes the server using **CloudEvents HTTP protocol** on different client events. [**CloudEvents**](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) is a standardized and protocol-agnostic definition of the structure and metadata description of events hosted by the Cloud Native Computing Foundation (CNCF). Details are described in [server protocol](concept-service-internals.md#server-protocol).
-3. Server can invoke the service using REST API to send messages to clients or to manage the connected clients. Details are described in [server protocol](concept-service-internals.md#server_protocol)
+3. Server can invoke the service using REST API to send messages to clients or to manage the connected clients. Details are described in [server protocol](concept-service-internals.md#server-protocol)
azure-web-pubsub Reference Client Specification https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/azure-web-pubsub/reference-client-specification.md
+
+ Title: Reference - Azure Web PubSub Client Specification
+description: This article provides a summary of the Web PubSub client specification.
++++ Last updated : 07/26/2022++
+# Azure Web PubSub client specification
+
+## Summary
+
+The client specification outlines the complete feature set available to an Azure Web PubSub client. It's highly recommended that client library developers familiarize themselves with the client specification. The specification defines the behaviors and features a client library **must** implement to work properly with Web PubSub service.
+
+For more information, see the [entire client specification](https://github.com/Azure/azure-webpubsub/blob/main/protocols/client/client-spec.md), located on GitHub.
+
+## Next steps
+
+> [!div class="nextstepaction"]
+> [Learn about Web PubSub service internals](concept-service-internals.md)
+
+> [!div class="nextstepaction"]
+> [Explore the REST API](reference-rest-api-data-plane.md)
+
+> [!div class="nextstepaction"]
+> [Explore Azure Web PubSub code samples](https://aka.ms/awps/samples)
bastion Bastion Faq https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/bastion/bastion-faq.md
# Azure Bastion FAQ
-## <a name="host"></a>Bastion
+## <a name="host"></a>Bastion FAQs
### <a name="browsers"></a>Which browsers are supported?
Review any error messages and [raise a support request in the Azure portal](../a
Azure Bastion is deployed within VNets or peered VNets, and is associated to an Azure region. You're responsible for deploying Azure Bastion to a Disaster Recovery (DR) site VNet. In the event of an Azure region failure, perform a failover operation for your VMs to the DR region. Then, use the Azure Bastion host that's deployed in the DR region to connect to the VMs that are now deployed there.
-## <a name="vm"></a>VM features and connections
+## <a name="vm"></a>VM features and connection FAQs
### <a name="roles"></a>Are any roles required to access a virtual machine?
Currently, 1920x1080 (1080p) is the maximum supported resolution.
Azure Bastion currently doesn't support timezone redirection and isn't timezone configurable.
-## <a name="peering"></a>VNet peering
+## <a name="peering"></a>VNet peering FAQs
### Can I still deploy multiple Bastion hosts across peered virtual networks?
Make sure the user has **read** access to both the VM, and the peered VNet. Addi
|Microsoft.Network/virtualNetworks/read|Get the virtual network definition|Action| |Microsoft.Network/virtualNetworks/subnets/virtualMachines/read|Gets references to all the virtual machines in a virtual network subnet|Action| |Microsoft.Network/virtualNetworks/virtualMachines/read|Gets references to all the virtual machines in a virtual network|Action|+
+## Next steps
+
+For more information, see [What is Azure Bastion](bastion-overview.md).
bastion Bastion Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/bastion/bastion-overview.md
Title: 'Azure Bastion | Microsoft Docs'
+ Title: 'About Azure Bastion'
description: Learn about Azure Bastion, which provides secure and seamless RDP/SSH connectivity to your virtual machines without exposing RDP/SSH ports externally.- # Customer intent: As someone with a basic network background, but is new to Azure, I want to understand the capabilities of Azure Bastion so that I can securely connect to my Azure virtual machines. Previously updated : 03/17/2022 Last updated : 08/05/2022
For frequently asked questions, see the Bastion [FAQ](bastion-faq.md).
## Next steps
-* [Tutorial: Create an Azure Bastion host and connect to a Windows VM](tutorial-create-host-portal.md).
+* [Quickstart: Deploy Bastion using default settings](quickstart-host-portal.md).
+* [Tutorial: Deploy Bastion using specified settings](tutorial-create-host-portal.md).
* [Learn module: Introduction to Azure Bastion](/learn/modules/intro-to-azure-bastion/). * Learn about some of the other key [networking capabilities](../networking/fundamentals/networking-overview.md) of Azure.
cognitive-services Text Moderation Api https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Content-Moderator/text-moderation-api.md
The following example shows a sample response:
## Auto-correction
-Suppose the input text is (the "qu!ck," "f0x," and "lzay" are intentional):
-
-> The qu!ck brown f0x jumps over the lzay dog.
-
-If you ask for auto-correction, the response contains the corrected version of the text:
-
-> The quick brown fox jumps over the lazy dog.
+The text moderation response can optionally return the text with basic auto-correction applied. It can fix some spelling errors, for example.
## Creating and managing your custom lists of terms
cognitive-services Custom Speech Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/custom-speech-overview.md
With Custom Speech, you can upload your own data, test and train a custom model,
Here's more information about the sequence of steps shown in the previous diagram:
-1. [Create a project](how-to-custom-speech-create-project.md) and choose a model. Use a <a href="https://portal.azure.com/#create/Microsoft.CognitiveServicesSpeechServices" title="Create a Speech resource" target="_blank">Speech resource</a> that you create in the Azure portal.
+1. [Create a project](how-to-custom-speech-create-project.md) and choose a model. Use a <a href="https://portal.azure.com/#create/Microsoft.CognitiveServicesSpeechServices" title="Create a Speech resource" target="_blank">Speech resource</a> that you create in the Azure portal. If you will train a custom model with audio data, choose a Speech resource region with dedicated hardware for training audio data. See footnotes in the [regions](regions.md#speech-service) table for more information.
1. [Upload test data](./how-to-custom-speech-upload-data.md). Upload test data to evaluate the Microsoft speech-to-text offering for your applications, tools, and products. 1. [Test recognition quality](how-to-custom-speech-inspect-data.md). Use the [Speech Studio](https://aka.ms/speechstudio/customspeech) to play back uploaded audio and inspect the speech recognition quality of your test data. 1. [Test model quantitatively](how-to-custom-speech-evaluate-data.md). Evaluate and improve the accuracy of the speech-to-text model. The Speech service provides a quantitative word error rate (WER), which you can use to determine if additional training is required. 1. [Train a model](how-to-custom-speech-train-model.md). Provide written transcripts and related text, along with the corresponding audio data. Testing a model before and after training is optional but recommended. 1. [Deploy a model](how-to-custom-speech-deploy-model.md). Once you're satisfied with the test results, deploy the model to a custom endpoint. With the exception of [batch transcription](batch-transcription.md), you must deploy a custom endpoint to use a Custom Speech model.
-If you will train a custom model with audio data, choose a Speech resource [region](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation) with dedicated hardware for training audio data. In regions with dedicated hardware for Custom Speech training, the Speech service will use up to 20 hours of your audio training data, and can process about 10 hours of data per day. In other regions, the Speech service uses up to 8 hours of your audio data, and can process about 1 hour of data per day. After a model is trained, you can copy it to a Speech resource in another region as needed.
- ## Next steps * [Create a project](how-to-custom-speech-create-project.md)
cognitive-services How To Audio Content Creation https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-audio-content-creation.md
After you sign up for the Azure account, you need to create a Speech resource in
It takes a few moments to deploy your new Speech resource. After the deployment is complete, you can start using the Audio Content Creation tool. > [!NOTE]
- > If you plan to use neural voices, make sure that you create your resource in [a region that supports neural voices](regions.md#prebuilt-neural-voices).
+ > If you plan to use neural voices, make sure that you create your resource in [a region that supports neural voices](regions.md#speech-service).
### Step 3: Sign in to Audio Content Creation with your Azure account and Speech resource
cognitive-services How To Custom Speech Create Project https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-custom-speech-create-project.md
To create a Custom Speech project, follow these steps:
1. Sign in to the [Speech Studio](https://aka.ms/speechstudio/customspeech). 1. Select the subscription and Speech resource to work with. +
+ > [!IMPORTANT]
+ > If you will train a custom model with audio data, choose a Speech resource region with dedicated hardware for training audio data. See footnotes in the [regions](regions.md#speech-service) table for more information.
+ 1. Select **Custom speech** > **Create a new project**. 1. Follow the instructions provided by the wizard to create your project.
cognitive-services How To Custom Speech Test And Train https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-custom-speech-test-and-train.md
Training with plain text or structured text usually finishes within a few minute
> > Start with small sets of sample data that match the language, acoustics, and hardware where your model will be used. Small datasets of representative data can expose problems before you invest in gathering larger datasets for training. For sample Custom Speech data, see <a href="https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/sampledata/customspeech" target="_target">this GitHub repository</a>.
-If you will train a custom model with audio data, choose a Speech resource [region](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation) with dedicated hardware available for training audio data. In regions with dedicated hardware for Custom Speech training, the Speech service will use up to 20 hours of your audio training data, and can process about 10 hours of data per day. In other regions, the Speech service uses up to 8 hours of your audio data, and can process about 1 hour of data per day. After the model is trained, you can copy the model to another region as needed with the [CopyModelToSubscription](https://eastus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0/operations/CopyModelToSubscription) REST API.
+If you will train a custom model with audio data, choose a Speech resource region with dedicated hardware for training audio data. See footnotes in the [regions](regions.md#speech-service) table for more information. In regions with dedicated hardware for Custom Speech training, the Speech service will use up to 20 hours of your audio training data, and can process about 10 hours of data per day. In other regions, the Speech service uses up to 8 hours of your audio data, and can process about 1 hour of data per day. After the model is trained, you can copy the model to another region as needed with the [CopyModelToSubscription](https://eastus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0/operations/CopyModelToSubscription) REST API.
## Consider datasets by scenario
You can use audio + human-labeled transcript data for both [training](how-to-cus
For a list of base models that support training with audio data, see [Language support](language-support.md#speech-to-text). Even if a base model does support training with audio data, the service might use only part of the audio. And it will still use all the transcripts. > [!IMPORTANT]
-> If a base model doesn't support customization with audio data, only the transcription text will be used for training. If you switch to a base model that supports customization with audio data, the training time may increase from several hours to several days. The change in training time would be most noticeable when you switch to a base model in a [region](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation) without dedicated hardware for training. If the audio data is not required, you should remove it to decrease the training time.
+> If a base model doesn't support customization with audio data, only the transcription text will be used for training. If you switch to a base model that supports customization with audio data, the training time may increase from several hours to several days. The change in training time would be most noticeable when you switch to a base model in a [region](regions.md#speech-service) without dedicated hardware for training. If the audio data is not required, you should remove it to decrease the training time.
Audio with human-labeled transcripts offers the greatest accuracy improvements if the audio comes from the target use case. Samples must cover the full scope of speech. For example, a call center for a retail store would get the most calls about swimwear and sunglasses during summer months. Ensure that your sample includes the full scope of speech that you want to detect.
Consider these details:
* The Speech service automatically uses the transcripts to improve the recognition of domain-specific words and phrases, as though they were added as related text. * It can take several days for a training operation to finish. To improve the speed of training, be sure to create your Speech service subscription in a region that has dedicated hardware for training.
-A large training dataset is required to improve recognition. Generally, we recommend that you provide word-by-word transcriptions for 1 to 20 hours of audio. However, even as little as 30 minutes can help improve recognition results. Although creating human-labeled transcription can take time, improvements in recognition will only be as good as the data that you provide. You should only upload only high-quality transcripts.
+A large training dataset is required to improve recognition. Generally, we recommend that you provide word-by-word transcriptions for 1 to 20 hours of audio. However, even as little as 30 minutes can help improve recognition results. Although creating human-labeled transcription can take time, improvements in recognition will only be as good as the data that you provide. You should upload only high-quality transcripts.
Audio files can have silence at the beginning and end of the recording. If possible, include at least a half-second of silence before and after speech in each sample file. Although audio with low recording volume or disruptive background noise is not helpful, it shouldn't limit or degrade your custom model. Always consider upgrading your microphones and signal processing hardware before gathering audio samples.
Here are key details about the supported Markdown format:
|`@list`|A list of items that can be referenced in an example sentence.|Maximum of 10 lists. Maximum of 4,000 items per list.| |`speech:phoneticlexicon`|A list of phonetic pronunciations according to the [Universal Phone Set](customize-pronunciation.md). Pronunciation is adjusted for each instance where the word appears in a list or training sentence. For example, if you have a word that sounds like "cat" and you want to adjust the pronunciation to "k ae t", you would add `- cat/k ae t` to the `speech:phoneticlexicon` list.|Maximum of 15,000 entries. Maximum of 2 pronunciations per word.| |`#ExampleSentences`|A pound symbol (`#`) delimits a section of example sentences. The section heading can only contain letters, digits, and underscores. Example sentences should reflect the range of speech that your model should expect. A training sentence can refer to items under a `@list` by using surrounding left and right curly braces (`{@list name}`). You can refer to multiple lists in the same training sentence, or none at all.|Maximum of 50,000 example sentences|
-|`//`|Comments follow a double slash (`//`) .|Not applicable|
+|`//`|Comments follow a double slash (`//`).|Not applicable|
Here's an example structured text file:
Use <a href="http://sox.sourceforge.net" target="_blank" rel="noopener">SoX</a>
Not all base models support [training with audio data](language-support.md#speech-to-text). For a list of base models that support training with audio data, see [Language support](language-support.md#speech-to-text).
-Even if a base model supports training with audio data, the service might use only part of the audio. In [regions](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation) with dedicated hardware available for training audio data, the Speech service will use up to 20 hours of your audio training data. In other regions, the Speech service uses up to 8 hours of your audio data.
+Even if a base model supports training with audio data, the service might use only part of the audio. In [regions](regions.md#speech-service) with dedicated hardware available for training audio data, the Speech service will use up to 20 hours of your audio training data. In other regions, the Speech service uses up to 8 hours of your audio data.
## Next steps
cognitive-services How To Custom Speech Train Model https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-custom-speech-train-model.md
zone_pivot_groups: speech-studio-cli-rest
In this article, you'll learn how to train a custom model to improve recognition accuracy from the Microsoft base model. The speech recognition accuracy and quality of a Custom Speech model will remain consistent, even when a new base model is released.
+> [!NOTE]
+> You pay to use Custom Speech models, but you are not charged for training a model.
+ Training a model is typically an iterative process. You will first select a base model that is the starting point for a new model. You train a model with [datasets](./how-to-custom-speech-test-and-train.md) that can include text and audio, and then you test. If the recognition quality or accuracy doesn't meet your requirements, you can create a new model with additional or modified training data, and then test again. You can use a custom model for a limited time after it's trained. You must periodically recreate and adapt your custom model from the latest base model to take advantage of the improved accuracy and quality. For more information, see [Model and endpoint lifecycle](./how-to-custom-speech-model-and-endpoint-lifecycle.md).
-> [!NOTE]
-> You pay to use Custom Speech models, but you are not charged for training a model.
-
-If you plan to train a model with audio data, use a Speech resource in a [region](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation) with dedicated hardware for training. After a model is trained, you can [copy it to a Speech resource](#copy-a-model) in another region as needed.
+> [!IMPORTANT]
+> If you will train a custom model with audio data, choose a Speech resource region with dedicated hardware for training audio data. After a model is trained, you can [copy it to a Speech resource](#copy-a-model) in another region as needed.
+>
+> In regions with dedicated hardware for Custom Speech training, the Speech service will use up to 20 hours of your audio training data, and can process about 10 hours of data per day. In other regions, the Speech service uses up to 8 hours of your audio data, and can process about 1 hour of data per day. See footnotes in the [regions](regions.md#speech-service) table for more information.
## Create a model
The top-level `self` property in the response body is the model's URI. Use this
## Copy a model
-You can copy a model to another project that uses the same locale. For example, after a model is trained with audio data in a [region](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation) with dedicated hardware for training, you can copy it to a Speech resource in another region as needed.
+You can copy a model to another project that uses the same locale. For example, after a model is trained with audio data in a [region](regions.md#speech-service) with dedicated hardware for training, you can copy it to a Speech resource in another region as needed.
::: zone pivot="speech-studio"
cognitive-services How To Custom Voice Create Voice https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-custom-voice-create-voice.md
After you've updated the engine version for your voice model, you need to [redep
For more information, [learn more about the capabilities and limits of this feature, and the best practice to improve your model quality](/legal/cognitive-services/speech-service/custom-neural-voice/characteristics-and-limitations-custom-neural-voice?context=%2fazure%2fcognitive-services%2fspeech-service%2fcontext%2fcontext). > [!NOTE]
-> Custom Neural Voice training is only available in some regions. But you can easily copy a neural voice model from these regions to other regions. For more information, see the [regions for Custom Neural Voice](regions.md#text-to-speech).
+> Custom Neural Voice training is only available in some regions. But you can easily copy a neural voice model from these regions to other regions. For more information, see the [regions for Custom Neural Voice](regions.md#speech-service).
## Next steps
cognitive-services How To Custom Voice https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-custom-voice.md
# Create a Project
-[Custom Neural Voice](https://aka.ms/customvoice) is a set of online tools that you use to create a recognizable, one-of-a-kind voice for your brand. All it takes to get started are a handful of audio files and the associated transcriptions. See if Custom Neural Voice supports your [language](language-support.md#custom-neural-voice) and [region](regions.md#custom-neural-voices).
+[Custom Neural Voice](https://aka.ms/customvoice) is a set of online tools that you use to create a recognizable, one-of-a-kind voice for your brand. All it takes to get started are a handful of audio files and the associated transcriptions. See if Custom Neural Voice supports your [language](language-support.md#custom-neural-voice) and [region](regions.md#speech-service).
> [!IMPORTANT] > Custom Neural Voice Pro can be used to create higher-quality models that are indistinguishable from human recordings. For access you must commit to using it in alignment with our responsible AI principles. Learn more about our [policy on limited access](/legal/cognitive-services/speech-service/custom-neural-voice/limited-access-custom-neural-voice?context=%2fazure%2fcognitive-services%2fspeech-service%2fcontext%2fcontext) and [apply here](https://aka.ms/customneural).
cognitive-services How To Deploy And Use Endpoint https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-deploy-and-use-endpoint.md
The custom endpoint is functionally identical to the standard endpoint that's us
You can copy your voice model to another project for the same region or another region. For example, you can copy a neural voice model that was trained in one region, to a project for another region. > [!NOTE]
-> Custom neural voice training is only available in the these regions: East US, Southeast Asia, and UK South. But you can copy a neural voice model from those regions to other regions. For more information, see the [regions for custom neural voice](regions.md#text-to-speech).
+> Custom neural voice training is only available in the these regions: East US, Southeast Asia, and UK South. But you can copy a neural voice model from those regions to other regions. For more information, see the [regions for custom neural voice](regions.md#speech-service).
To copy your custom neural voice model to another project:
cognitive-services How To Develop Custom Commands Application https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-develop-custom-commands-application.md
Another way to customize Custom Commands responses is to select an output voice.
> ![Screenshot showing sample sentences and parameters.](media/custom-commands/select-custom-voice.png) > [!NOTE]
-> For public voices, neural types are available only for specific regions. For more information, see [Speech service supported regions](./regions.md#prebuilt-neural-voices).
+> For public voices, neural types are available only for specific regions. For more information, see [Speech service supported regions](./regions.md#speech-service).
> > You can create custom voices on the **Custom Voice** project page. For more information, see [Get started with Custom Voice](./how-to-custom-voice.md).
cognitive-services How To Pronunciation Assessment https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/how-to-pronunciation-assessment.md
You can get pronunciation assessment scores for:
- Phonemes in SAPI or IPA format > [!NOTE]
-> For information about availability of pronunciation assessment, see [supported languages](language-support.md#pronunciation-assessment) and [available regions](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation).
+> For information about availability of pronunciation assessment, see [supported languages](language-support.md#pronunciation-assessment) and [available regions](regions.md#speech-service).
> > The syllable groups, IPA phonemes, and spoken phoneme features of pronunciation assessment are currently only available for the en-US locale.
cognitive-services Keyword Recognition Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/keyword-recognition-overview.md
You can use custom keyword to generate two types of on-device models for any key
| Advanced | Best suited for product integration purposes. Models are generated with adaptation of a common base model by using simulated training data to improve accuracy characteristics. It can take up to 48 hours for models to be ready. | > [!NOTE]
-> You can view a list of regions that support the **Advanced** model type in the [keyword recognition region support](regions.md#keyword-recognition) documentation.
+> You can view a list of regions that support the **Advanced** model type in the [keyword recognition region support](regions.md#speech-service) documentation.
Neither model type requires you to upload training data. Custom keyword fully handles data generation and model training.
cognitive-services Language Support https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/language-support.md
The following table lists the prebuilt neural voices supported in each language.
The following neural voices are in public preview. > [!NOTE]
-> Voices and styles in public preview are only available in three service [regions](regions.md#prebuilt-neural-voices): East US, West Europe, and Southeast Asia.
+> Voices and styles in public preview are only available in three service [regions](regions.md): East US, West Europe, and Southeast Asia.
| Language | Locale | Gender | Voice name | Style support | |-||--|-||
The following table lists the released languages and public preview languages.
|Spanish (Spain)|`es-ES`<sup>Public preview</sup> | > [!NOTE]
-> If you want to use languages that aren't listed here, please contact us by email at [mspafeedback@microsoft.com](mailto:mspafeedback@microsoft.com).
->
-> For pronunciation assessment supported regions, see [available regions](regions.md#speech-to-text-pronunciation-assessment-text-to-speech-and-translation).
+> For pronunciation assessment, `en-US` and `en-GB` are available inΓÇ»[all regions](regions.md#speech-service), `zh-CN` is available in East Asia and Southeast Asia regions, `de-DE`, `es-ES`, and `fr-FR` are available in West Europe region, and `en-AU` is available in Australia East region.
## Speech translation
cognitive-services Regions https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/regions.md
Previously updated : 06/13/2022 Last updated : 07/27/2022
The Speech service allows your application to convert audio to text, perform spe
Keep in mind the following points:
-* If your application uses a [Speech SDK](speech-sdk.md), you provide the region identifier, such as `westus`, when you create a speech configuration. Make sure the region matches the region of your subscription.
+* If your application uses a [Speech SDK](speech-sdk.md), you provide the region identifier, such as `westus`, when you create a `SpeechConfig`. Make sure the region matches the region of your subscription.
* If your application uses one of the Speech service REST APIs, the region is part of the endpoint URI you use when making requests. * Keys created for a region are valid only in that region. If you attempt to use them with other regions, you get authentication errors. > [!NOTE] > Speech service doesn't store or process customer data outside the region the customer deploys the service instance in.
-## Speech SDK
-
-In the [Speech SDK](speech-sdk.md), you specify the region as a parameter (for example, in the Speech SDK for C#, you specify the region as a parameter to `SpeechConfig.FromSubscription`).
-
-### Speech-to-text, pronunciation assessment, text-to-speech, and translation
-
-The Speech service is available in these regions for speech-to-text, pronunciation assessment, text-to-speech, and translation:
--
-If you plan to train a custom model with audio data, use one of the regions with dedicated hardware for faster training. Then you can use the [Speech-to-text REST API v3.0](rest-speech-to-text.md) to [copy the trained model](how-to-custom-speech-train-model.md#copy-a-model) to another region.
-
-> [!TIP]
-> For pronunciation assessment, `en-US` and `en-GB` are available in all regions listed above, `zh-CN` is available in East Asia and Southeast Asia regions, `de-DE`, `es-ES`, and `fr-FR` are available in West Europe region, and `en-AU` is available in Australia East region.
-
-### Intent recognition
+## Speech service
+
+The following regions are supported for Speech service features such as speech-to-text, text-to-speech, pronunciation assessment, and translation. The geographies are listed in alphabetical order.
+
+| Geography | Region | Region identifier |
+| -- | -- | -- |
+| Africa | South Africa North | `southafricanorth` <sup>6</sup>|
+| Asia Pacific | East Asia | `eastasia` <sup>5</sup>|
+| Asia Pacific | Southeast Asia | `southeastasia` <sup>1,2,3,4,5</sup>|
+| Asia Pacific | Australia East | `australiaeast` <sup>1,2,3,4</sup>|
+| Asia Pacific | Central India | `centralindia` <sup>1,2,3,4,5</sup>|
+| Asia Pacific | Japan East | `japaneast` <sup>2,5</sup>|
+| Asia Pacific | Japan West | `japanwest` |
+| Asia Pacific | Korea Central | `koreacentral` <sup>2</sup>|
+| Canada | Canada Central | `canadacentral` <sup>1</sup>|
+| Europe | North Europe | `northeurope` <sup>1,2,4,5</sup>|
+| Europe | West Europe | `westeurope` <sup>1,2,3,4,5</sup>|
+| Europe | France Central | `francecentral` |
+| Europe | Germany West Central | `germanywestcentral` |
+| Europe | Norway East | `norwayeast` |
+| Europe | Switzerland North | `switzerlandnorth` <sup>6</sup>|
+| Europe | Switzerland West | `switzerlandwest` |
+| Europe | UK South | `uksouth` <sup>1,2,3,4</sup>|
+| Middle East | UAE North | `uaenorth` <sup>6</sup>|
+| South America | Brazil South | `brazilsouth` <sup>6</sup>|
+| US | Central US | `centralus` |
+| US | East US | `eastus` <sup>1,2,3,4,5</sup>|
+| US | East US 2 | `eastus2` <sup>1,2,4,5</sup>|
+| US | North Central US | `northcentralus` <sup>1,4,6</sup>|
+| US | South Central US | `southcentralus` <sup>1,2,3,4,5,6</sup>|
+| US | West Central US | `westcentralus` <sup>5</sup>|
+| US | West US | `westus` <sup>2,5</sup>|
+| US | West US 2 | `westus2` <sup>1,2,4,5</sup>|
+| US | West US 3 | `westus3` |
+
+<sup>1</sup> The region has dedicated hardware for Custom Speech training. If you plan to train a custom model with audio data, use one of the regions with dedicated hardware for faster training. Then you can [copy the trained model](how-to-custom-speech-train-model.md#copy-a-model) to another region.
+
+<sup>2</sup> The region is available for Custom Neural Voice training. You can copy a trained neural voice model to other regions for deployment.
+
+<sup>3</sup> The Long Audio API is available in the region.
+
+<sup>4</sup> The region supports custom keyword advanced models.
+
+<sup>5</sup> The region supports keyword verification.
+
+<sup>6</sup> The region does not support Speaker Recognition.
+
+## Intent recognition
Available regions for intent recognition via the Speech SDK are in the following table.
Available regions for intent recognition via the Speech SDK are in the following
This is a subset of the publishing regions supported by the [Language Understanding service (LUIS)](../luis/luis-reference-regions.md).
-### Voice assistants
+## Voice assistants
The [Speech SDK](speech-sdk.md) supports voice assistant capabilities through [Direct Line Speech](./direct-line-speech.md) for regions in the following table.
The [Speech SDK](speech-sdk.md) supports voice assistant capabilities through [D
| Asia | East Asia | `eastasia` | | Asia | Southeast Asia | `southeastasia` | | India | Central India | `centralindia` |-
-### Speaker recognition
-
-Available regions for speaker recognition are in the following table.
-
-| Geography | Region | Region identifier |
-| - | - | -- |
-| Americas | Central US | `centralus` |
-| Americas | East US | `eastus` |
-| Americas | East US 2 | `eastus2` |
-| Americas | West Central US | `westcentralus` |
-| Americas | West US | `westus` |
-| Americas | West US 2 | `westus2` |
-| Americas | West US 3 | `westus3` |
-| Asia Pacific | East Asia | `eastasia` |
-| Asia Pacific | Southeast Asia | `southeastasia` |
-| Asia Pacific | Central India | `centralindia` |
-| Asia Pacific | Japan East | `japaneast` |
-| Asia Pacific | Japan West | `japanwest` |
-| Asia Pacific | Korea Central | `koreacentral` |
-| Australia | Australia East | `australiaeast` |
-| Canada | Canada Central | `canadacentral` |
-| Europe | North Europe | `northeurope` |
-| Europe | West Europe | `westeurope` |
-| Europe | France Central | `francecentral` |
-| Europe | Germany West Central | `germanywestcentral` |
-| Europe | Norway East | `norwayeast` |
-| Europe | Switzerland West | `switzerlandwest` |
-| Europe | UK South | `uksouth` |
-
-### Keyword recognition
-
-Available regions for keyword recognition are in the following table.
-
-| Region | Custom keyword (basic models) | Custom keyword (advanced models) | Keyword verification |
-| | -- | -- | -- |
-| West US | Yes | No | Yes |
-| West US 2 | Yes | Yes | Yes |
-| East US | Yes | Yes | Yes |
-| East US 2 | Yes | Yes | Yes |
-| West Central US | Yes | No | Yes |
-| South Central US | Yes | Yes | Yes |
-| West Europe | Yes | Yes | Yes |
-| North Europe | Yes | Yes | Yes |
-| UK South | Yes | Yes | No |
-| East Asia | Yes | No | Yes |
-| Southeast Asia | Yes | Yes | Yes |
-| Central India | Yes | Yes | Yes |
-| Japan East | Yes | No | Yes |
-| Japan West | Yes | No | No |
-| Australia East | Yes | Yes | No |
-| Brazil South | Yes | No | No |
-| Canada Central | Yes | No | No |
-| Korea Central | Yes | No | No |
-| France Central | Yes | No | No |
-| North Central US | Yes | Yes | No |
-| Central US | Yes | No | No |
-| South Africa North | Yes | No | No |
-
-## REST APIs
-
-The Speech service also exposes REST endpoints for speech-to-text, text-to-speech, and speaker recognition requests.
-
-### Speech-to-text
-
-The endpoint for the REST API has this format:
-
-```
-https://<REGION_IDENTIFIER>.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1
-```
-
-Replace `<REGION_IDENTIFIER>` with the identifier matching the region of your subscription from this table:
--
-> [!NOTE]
-> The language parameter must be appended to the URL to avoid receiving an HTTP error. For example, the language set to `US English` by using the `West US` endpoint is: `https://westus.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US`.
-
-For more information, see the [speech-to-text REST API](rest-speech-to-text.md).
-
-### Text-to-speech
-
-For more information, see the [text-to-speech REST API](rest-text-to-speech.md).
--
-### Speaker recognition
-
-For more information, see the [speaker recognition REST API](/rest/api/speakerrecognition/). The regions available are the same as those for the speaker recognition SDK.
cognitive-services Rest Speech To Text Short https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/rest-speech-to-text-short.md
The endpoint for the REST API for short audio has this format:
https://<REGION_IDENTIFIER>.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1 ```
-Replace `<REGION_IDENTIFIER>` with the identifier that matches the region of your subscription from this table:
-
+Replace `<REGION_IDENTIFIER>` with the identifier that matches the [region](regions.md) of your subscription.
> [!NOTE] > You must append the language parameter to the URL to avoid receiving a 4xx HTTP error. For example, the language set to US English via the West US endpoint is: `https://westus.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US`.
cognitive-services Rest Text To Speech https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/rest-text-to-speech.md
The Speech service allows you to [convert text into synthesized speech](#convert
The text-to-speech REST API supports neural text-to-speech voices, which support specific languages and dialects that are identified by locale. Each available endpoint is associated with a region. A subscription key for the endpoint or region that you plan to use is required. Here are links to more information: - For a complete list of voices, see [Language and voice support for the Speech service](language-support.md#text-to-speech).-- For information about regional availability, see [Speech service supported regions](regions.md#text-to-speech).
+- For information about regional availability, see [Speech service supported regions](regions.md#speech-service).
- For Azure Government and Azure China endpoints, see [this article about sovereign clouds](sovereign-clouds.md). > [!IMPORTANT]
cognitive-services Speaker Recognition Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/speaker-recognition-overview.md
As with all of the Cognitive Services resources, developers who use the speaker
| What situations am I most likely to use speaker recognition? | Good examples include call center customer verification, voice-based patient check-in, meeting transcription, and multi-user device personalization.| | What's the difference between identification and verification? | Identification is the process of detecting which member from a group of speakers is speaking. Verification is the act of confirming that a speaker matches a known, *enrolled* voice.| | What languages are supported? | See [Speaker recognition language support](language-support.md#speaker-recognition). |
-| What Azure regions are supported? | See [Speaker recognition region support](regions.md#speaker-recognition).|
+| What Azure regions are supported? | See [Speaker recognition region support](regions.md#speech-service).|
| What audio formats are supported? | Mono 16 bit, 16 kHz PCM-encoded WAV. | | Can you enroll one speaker multiple times? | Yes, for text-dependent verification, you can enroll a speaker up to 50 times. For text-independent verification or speaker identification, you can enroll with up to 300 seconds of audio. | | What data is stored in Azure? | Enrollment audio is stored in the service until the voice profile is [deleted](./get-started-speaker-recognition.md#delete-voice-profile-enrollments). Recognition audio samples aren't retained or stored. |
cognitive-services Swagger Documentation https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/swagger-documentation.md
Speech service offers a Swagger specification to interact with a handful of REST
The [Swagger specification](https://westus.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0) has options that allow you to quickly test for various paths. However, sometimes it's desirable to generate code for all paths, creating a single library of calls that you can base future solutions on. Let's take a look at the process to generate a Python library.
-You'll need to set Swagger to the region of your Speech resource. You can confirm the region in the **Overview** part of your Speech resource settings in Azure portal. The complete list of supported regions is available [here](regions.md#speech-to-text).
+You'll need to set Swagger to the region of your Speech resource. You can confirm the region in the **Overview** part of your Speech resource settings in Azure portal. The complete list of supported regions is available [here](regions.md#speech-service).
-1. In a browser, go to the Swagger specification for your [region](regions.md#speech-to-text):
+1. In a browser, go to the Swagger specification for your [region](regions.md#speech-service):
`https://<your-region>.dev.cognitive.microsoft.com/docs/services/speech-to-text-api-v3-0` 1. On that page, click **API definition**, and click **Swagger**. Copy the URL of the page that appears. 1. In a new browser, go to [https://editor.swagger.io](https://editor.swagger.io)
cognitive-services Tutorial Voice Enable Your Bot Speech Sdk https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/Speech-Service/tutorial-voice-enable-your-bot-speech-sdk.md
This section walks you through creating a resource group in the West US region.
### Choose an Azure region
-Ensure that you use a [supported Azure region](regions.md#voice-assistants). The Direct Line Speech channel uses the text-to-speech service, which has neural and standard voices. Neural voices are used at [these Azure regions](regions.md#prebuilt-neural-voices), and standard voices (retiring) are used at [these Azure regions](how-to-migrate-to-prebuilt-neural-voice.md).
+Ensure that you use a [supported Azure region](regions.md#voice-assistants). The Direct Line Speech channel uses the text-to-speech service, which has neural and standard voices. Neural voices are used at [these Azure regions](regions.md#speech-service), and standard voices (retiring) are used at [these Azure regions](how-to-migrate-to-prebuilt-neural-voice.md).
For more information about regions, see [Azure locations](https://azure.microsoft.com/global-infrastructure/locations/).
If you get an error message in your main app window, use this table to identify
|Error (AuthenticationFailure) : WebSocket Upgrade failed with an authentication error (401). Check for correct subscription key (or authorization token) and region name| On the **Settings** page of the app, make sure that you entered the subscription key and its region correctly. | |Error (ConnectionFailure) : Connection was closed by the remote host. Error code: 1011. Error details: We could not connect to the bot before sending a message | Make sure that you [selected the Enable Streaming Endpoint checkbox](#register-the-direct-line-speech-channel) and/or [turned on web sockets](#enable-web-sockets).<br>Make sure that Azure App Service is running. If it is, try restarting it.| |Error (ConnectionFailure) : Connection was closed by the remote host. Error code: 1002. Error details: The server returned status code '503' when status code '101' was expected | Make sure that you [selected the Enable Streaming Endpoint checkbox](#register-the-direct-line-speech-channel) box and/or [turned on web sockets](#enable-web-sockets).<br>Make sure that Azure App Service is running. If it is, try restarting it.|
-|Error (ConnectionFailure) : Connection was closed by the remote host. Error code: 1011. Error details: Response status code does not indicate success: 500 (InternalServerError)| Your bot specified a neural voice in the [speak](https://github.com/microsoft/botframework-sdk/blob/master/specs/botframework-activity/botframework-activity.md#speak) field of its output activity, but the Azure region associated with your subscription key doesn't support neural voices. See [neural voices](./regions.md#prebuilt-neural-voices) and [standard voices](how-to-migrate-to-prebuilt-neural-voice.md).|
+|Error (ConnectionFailure) : Connection was closed by the remote host. Error code: 1011. Error details: Response status code does not indicate success: 500 (InternalServerError)| Your bot specified a neural voice in the [speak](https://github.com/microsoft/botframework-sdk/blob/master/specs/botframework-activity/botframework-activity.md#speak) field of its output activity, but the Azure region associated with your subscription key doesn't support neural voices. See [neural voices](./regions.md#speech-service) and [standard voices](how-to-migrate-to-prebuilt-neural-voice.md).|
If the actions in the table don't address your problem, see [Voice assistants: Frequently asked questions](faq-voice-assistants.yml). If you still can't resolve your problem after following all the steps in this tutorial, please enter a new issue on the [Voice Assistant GitHub page](https://github.com/Azure-Samples/Cognitive-Services-Voice-Assistant/issues).
If you're not going to continue using the echo bot deployed in this tutorial, yo
## Explore documentation * [Deploy to an Azure region near you](https://azure.microsoft.com/global-infrastructure/locations/) to see the improvement in bot response time.
-* [Deploy to an Azure region that supports high-quality neural text-to-speech voices](./regions.md#prebuilt-neural-voices).
+* [Deploy to an Azure region that supports high-quality neural text-to-speech voices](./regions.md#speech-service).
* Get pricing associated with the Direct Line Speech channel: * [Bot Service pricing](https://azure.microsoft.com/pricing/details/bot-service/) * [Speech service](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/)
cognitive-services Call Api https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/language-service/custom-named-entity-recognition/how-to/call-api.md
First you will need to get your resource key and endpoint:
4. After you've installed the client library, use the following samples on GitHub to start calling the API.
- * [C#](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample9_RecognizeCustomEntities.md)
+ * [C#](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample8_RecognizeCustomEntities.md)
* [Java](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/textanalytics/azure-ai-textanalytics/src/samples/java/com/azure/ai/textanalytics/lro/RecognizeCustomEntities.java)
- * [JavaScript](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js)
+ * [JavaScript](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/ai-text-analytics_6.0.0-beta.1/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js)
* [Python](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_recognize_custom_entities.py) 5. See the following reference documentation for more information on the client, and return object:
First you will need to get your resource key and endpoint:
## Next steps * [Enrich a Cognitive Search index tutorial](../tutorials/cognitive-search.md)---
cognitive-services Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/language-service/custom-named-entity-recognition/overview.md
As you use custom NER, see the following reference documentation and samples for
|||| |REST APIs (Authoring) | [REST API documentation](https://aka.ms/ct-authoring-swagger) | | |REST APIs (Runtime) | [REST API documentation](https://aka.ms/ct-runtime-swagger) | |
-|C# (Runtime) | [C# documentation](/dotnet/api/azure.ai.textanalytics?view=azure-dotnet-preview&preserve-view=true) | [C# samples](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample9_RecognizeCustomEntities.md) |
+|C# (Runtime) | [C# documentation](/dotnet/api/azure.ai.textanalytics?view=azure-dotnet-preview&preserve-view=true) | [C# samples](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample8_RecognizeCustomEntities.md) |
| Java (Runtime) | [Java documentation](/java/api/overview/azure/ai-textanalytics-readme?view=azure-java-preview&preserve-view=true) | [Java Samples](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/textanalytics/azure-ai-textanalytics/src/samples/java/com/azure/ai/textanalytics/lro/RecognizeCustomEntities.java) |
-|JavaScript (Runtime) | [JavaScript documentation](/javascript/api/overview/azure/ai-text-analytics-readme?view=azure-node-preview&preserve-view=true) | [JavaScript samples](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js) |
+|JavaScript (Runtime) | [JavaScript documentation](/javascript/api/overview/azure/ai-text-analytics-readme?view=azure-node-preview&preserve-view=true) | [JavaScript samples](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/ai-text-analytics_6.0.0-beta.1/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js) |
|Python (Runtime) | [Python documentation](/python/api/azure-ai-textanalytics/azure.ai.textanalytics?view=azure-python-preview&preserve-view=true) | [Python samples](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_recognize_custom_entities.py) | ## Responsible AI
cognitive-services Call Api https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/language-service/custom-text-classification/how-to/call-api.md
First you will need to get your resource key and endpoint:
4. After you've installed the client library, use the following samples on GitHub to start calling the API. Single label classification:
- * [C#](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample10_SingleCategoryClassify.md)
+ * [C#](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample9_SingleLabelClassify.md)
* [Java](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/textanalytics/azure-ai-textanalytics/src/samples/java/com/azure/ai/textanalytics/lro/ClassifyDocumentSingleCategory.java)
- * [JavaScript](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js)
- * [Python](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_single_category_classify.py)
+ * [JavaScript](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/ai-text-analytics_6.0.0-beta.1/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js)
+ * [Python](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_single_label_classify.py)
Multi label classification:
- * [C#](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample11_MultiCategoryClassify.md)
+ * [C#](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample10_MultiLabelClassify.md)
* [Java](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/textanalytics/azure-ai-textanalytics/src/samples/java/com/azure/ai/textanalytics/lro/ClassifyDocumentMultiCategory.java)
- * [JavaScript](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js)
- * [Python](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_multi_category_classify.py)
+ * [JavaScript](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/ai-text-analytics_6.0.0-beta.1/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js)
+ * [Python](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_multi_label_classify.py)
5. See the following reference documentation for more information on the client, and return object:
cognitive-services Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cognitive-services/language-service/custom-text-classification/overview.md
As you use custom text classification, see the following reference documentation
|||| |REST APIs (Authoring) | [REST API documentation](https://aka.ms/ct-authoring-swagger) | | |REST APIs (Runtime) | [REST API documentation](https://aka.ms/ct-runtime-swagger) | |
-|C# (Runtime) | [C# documentation](/dotnet/api/azure.ai.textanalytics?view=azure-dotnet-preview&preserve-view=true) | [C# samples - Single label classification](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample10_SingleCategoryClassify.md) [C# samples - Multi label classification](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample11_MultiCategoryClassify.md) |
+|C# (Runtime) | [C# documentation](/dotnet/api/azure.ai.textanalytics?view=azure-dotnet-preview&preserve-view=true) | [C# samples - Single label classification](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample9_SingleLabelClassify.md) [C# samples - Multi label classification](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/textanalytics/Azure.AI.TextAnalytics/samples/Sample10_MultiLabelClassify.md) |
| Java (Runtime) | [Java documentation](/java/api/overview/azure/ai-textanalytics-readme?view=azure-java-preview&preserve-view=true) | [Java Samples - Single label classification](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/textanalytics/azure-ai-textanalytics/src/samples/java/com/azure/ai/textanalytics/lro/ClassifyDocumentSingleCategory.java) [Java Samples - Multi label classification](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/textanalytics/azure-ai-textanalytics/src/samples/java/com/azure/ai/textanalytics/lro/ClassifyDocumentMultiCategory.java) |
-|JavaScript (Runtime) | [JavaScript documentation](/javascript/api/overview/azure/ai-text-analytics-readme?view=azure-node-preview&preserve-view=true) | [JavaScript samples - Single label classification](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js) [JavaScript samples - Multi label classification](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js) |
-|Python (Runtime)| [Python documentation](/python/api/azure-ai-textanalytics/azure.ai.textanalytics?view=azure-python-preview&preserve-view=true) | [Python samples - Single label classification](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_single_category_classify.py) [Python samples - Multi label classification](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_multi_category_classify.py) |
+|JavaScript (Runtime) | [JavaScript documentation](/javascript/api/overview/azure/ai-text-analytics-readme?view=azure-node-preview&preserve-view=true) | [JavaScript samples - Single label classification](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/ai-text-analytics_6.0.0-beta.1/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js) [JavaScript samples - Multi label classification](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/ai-text-analytics_6.0.0-beta.1/sdk/textanalytics/ai-text-analytics/samples/v5/javascript/customText.js) |
+|Python (Runtime)| [Python documentation](/python/api/azure-ai-textanalytics/azure.ai.textanalytics?view=azure-python-preview&preserve-view=true) | [Python samples - Single label classification](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_single_label_classify.py) [Python samples - Multi label classification](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/textanalytics/azure-ai-textanalytics/samples/sample_multi_label_classify.py) |
## Responsible AI
communication-services Authentication https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/communication-services/concepts/authentication.md
The following table shows the Azure Communication Services SDKs and their authen
| Identity | Access Key or Azure AD authentication | | SMS | Access Key or Azure AD authentication | | Phone Numbers | Access Key or Azure AD authentication |
+| Email | Access Key or Azure AD authentication |
| Calling | User Access Token | | Chat | User Access Token |
cosmos-db Analytical Store Introduction https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cosmos-db/analytical-store-introduction.md
Analytical store partitioning is completely independent of partitioning in
* **Network isolation using private endpoints** - You can control network access to the data in the transactional and analytical stores independently. Network isolation is done using separate managed private endpoints for each store, within managed virtual networks in Azure Synapse workspaces. To learn more, see how to [Configure private endpoints for analytical store](analytical-store-private-endpoints.md) article.
+* **Data encryption at rest** - Your analytical store encryption is enabled by default.
+ * **Data encryption with customer-managed keys** - You can seamlessly encrypt the data across transactional and analytical stores using the same customer-managed keys in an automatic and transparent manner. Azure Synapse Link only supports configuring customer-managed keys using your Azure Cosmos DB account's managed identity. You must configure your account's managed identity in your Azure Key Vault access policy before [enabling Azure Synapse Link](configure-synapse-link.md#enable-synapse-link) on your account. To learn more, see how to [Configure customer-managed keys using Azure Cosmos DB accounts' managed identities](how-to-setup-cmk.md#using-managed-identity) article. ## Support for multiple Azure Synapse Analytics runtimes
cosmos-db How To Setup Rbac https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cosmos-db/mongodb/how-to-setup-rbac.md
These roles already exist on every database and do not need to be created.
### read Has the following privileges: changeStream, collStats, find, killCursors, listIndexes, listCollections
-### readwrite
+### readWrite
Has the following privileges: collStats, createCollection, dropCollection, createIndex, dropIndex, find, insert, killCursors, listIndexes, listCollections, remove, update ### dbAdmin
cost-management-billing Ea Portal Enrollment Invoices https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/cost-management-billing/manage/ea-portal-enrollment-invoices.md
Title: Azure Enterprise enrollment invoices
description: This article explains how to manage and act on your Azure Enterprise invoice. Previously updated : 07/20/2022 Last updated : 08/05/2022
The Azure Enterprise portal filters inactive enrollments out of view. You'll nee
![Unchecking the active box allows user to see inactive enrollments](./media/ea-portal-enrollment-invoices/unchecked-active-box.png)
-## Change a PO number for an upcoming overage invoice
-
-The Azure Enterprise portal automatically generates a default purchase order (PO) number unless the enterprise administrator sets one before the invoice date. An enterprise administrator can update the PO number up to seven days after receiving an automated invoice notification email.
-
-To prevent automatic generation of PO numbers every month, you can lock the PO number. See [Lock PO number](#lock-po-number-to-prevent-automatic-update-in-upcoming-billing-cycles).
-
-To update the PO number, the **Lock PO number** must not be selected.
-
-### Update the Azure services purchase order number
-
-1. From the Azure Enterprise portal, select **Report** > **Usage Summary**.
-1. Select **Edit PO Numbers** in the upper-right corner.
-1. Select the **Azure Services** radio button.
-1. Select an **Invoice Period** from the date ranges drop-down menu.
- You can edit a PO number during a seven-day period after you get an invoice notification, but before you've paid the invoice.
-1. Enter a new PO number in the **PO Number** field.
-1. Select **Save** to submit your change.
-
-### Update the Azure Marketplace purchase order number
-
-1. From the Azure Enterprise portal, select **Report** > **Usage Summary**.
-1. Select **Edit PO Numbers** in the upper-right corner.
-1. Select the **Marketplace** radio button.
-1. Select an **Invoice Period** from the date ranges drop-down menu.
- You can edit a PO number during a seven-day period after you get an invoice notification, but before you've paid the invoice.
-1. Enter a new PO number in the **PO Number** field.
-1. Select **Save** to submit your change.
-
-### Lock PO number to prevent automatic update in upcoming billing cycles
-
-After you lock the PO number, it remains locked for all new invoices and you don't have to update the PO number.
-
-1. In the Azure Enterprise portal, select **Report** > **Usage Summary**.
-2. Select **Edit PO Numbers** in the upper-right corner.
-3. Type a new PO number in the **PO Number** field.
-4. Select the **Lock PO number** box.
-5. Select **Save** to submit your change.
- :::image type="content" source="./media/ea-portal-enrollment-invoices/lock-po.png" alt-text="Screenshot showing the View/Edit PO Numbers box." lightbox="./media/ea-portal-enrollment-invoices/lock-po.png" :::
+## PO number management
+PO number management functionality in the EA portal is getting deprecated. It is currently read-only in the EA portal. Instead, an EA administrator can use the Azure portal to manage PO numbers. For more information, see [Update a PO number](direct-ea-azure-usage-charges-invoices.md#update-a-po-number-for-an-upcoming-overage-invoice).
## Azure enterprise billing frequency
data-factory Concepts Data Flow Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/concepts-data-flow-overview.md
Each transformation contains at least four configuration tabs.
The first tab in each transformation's configuration pane contains the settings specific to that transformation. For more information, see that transformation's documentation page. #### Optimize
data-factory Data Factory Service Identity https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-factory-service-identity.md
Managed identities eliminate the need to manage credentials. Managed identities
There are two types of supported managed identities: -- **System-assigned:** You can enable a managed identity directly on a service instance. When you allow a system-assigned managed identity during the creation of the service, an identity is created in Azure AD tied to that service instance's lifecycle. By design, only that Azure resource can use this identity to request tokens from Azure AD. So when the resource is deleted, Azure automatically deletes the identity for you. --
+- **System-assigned:** You can enable a managed identity directly on a service instance. When you allow a system-assigned managed identity during the creation of the service, an identity is created in Azure AD tied to that service instance's lifecycle. By design, only that Azure resource can use this identity to request tokens from Azure AD. So when the resource is deleted, Azure automatically deletes the identity for you.
- **User-assigned:** You may also create a managed identity as a standalone Azure resource. You can [create a user-assigned managed identity](../active-directory/managed-identities-azure-resources/how-to-manage-ua-identity-portal.md) and assign it to one or more instances of a data factory. In user-assigned managed identities, the identity is managed separately from the resources that use it. Managed identity provides the below benefits:
data-factory Data Flow Rank https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-rank.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Rank transformation in mapping data flow
data-factory Data Flow Reserved Capacity Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-reserved-capacity-overview.md
Previously updated : 02/05/2021 Last updated : 08/04/2022 # Save costs for resources with reserved capacity - Azure Data Factory data flows
data-factory Data Flow Script https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-script.md
Previously updated : 09/22/2021 Last updated : 08/04/2022 # Data flow script (DFS)
data-factory Data Flow Select https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-select.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Select transformation in mapping data flow
data-factory Data Flow Sink https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-sink.md
Previously updated : 06/23/2022 Last updated : 08/04/2022 # Sink transformation in mapping data flow
data-factory Data Flow Sort https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-sort.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Sort transformation in mapping data flow
data-factory Data Flow Source https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-source.md
Previously updated : 05/31/2022 Last updated : 08/04/2022 # Source transformation in mapping data flow
After you've added a source, configure via the **Source settings** tab. Here you
Development values for dataset parameters can be configured in [debug settings](concepts-data-flow-debug-mode.md). (Debug mode must be turned on.) **Output stream name**: The name of the source transformation.
The **Source options** tab contains settings specific to the connector and forma
Like schemas in datasets, the projection in a source defines the data columns, types, and formats from the source data. For most dataset types, such as SQL and Parquet, the projection in a source is fixed to reflect the schema defined in a dataset. When your source files aren't strongly typed (for example, flat .csv files rather than Parquet files), you can define the data types for each field in the source transformation. If your text file has no defined schema, select **Detect data type** so that the service will sample and infer the data types. Select **Define default format** to autodetect the default data formats.
data-factory Data Flow Stringify https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-stringify.md
Previously updated : 10/06/2021 Last updated : 08/04/2022 # Stringify transformation in mapping data flow
data-factory Data Flow Surrogate Key https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-surrogate-key.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Surrogate key transformation in mapping data flow
data-factory Data Flow Transformation Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-transformation-overview.md
Previously updated : 07/14/2022 Last updated : 08/04/2022 # Mapping data flow transformation overview
data-factory Data Flow Troubleshoot Connector Format https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-troubleshoot-connector-format.md
Previously updated : 01/21/2022 Last updated : 08/04/2022
data-factory Data Flow Troubleshoot Errors https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-troubleshoot-errors.md
Previously updated : 07/25/2022 Last updated : 08/04/2022 # Common error codes and messages
data-factory Data Flow Troubleshoot Guide https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-troubleshoot-guide.md
Previously updated : 10/01/2021 Last updated : 08/04/2022 # Troubleshoot mapping data flows in Azure Data Factory
data-factory Data Flow Tutorials https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-tutorials.md
Previously updated : 12/13/2021 Last updated : 08/04/2022 # Mapping data flow video tutorials
data-factory Data Flow Understand Reservation Charges https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-understand-reservation-charges.md
Previously updated : 02/05/2021 Last updated : 08/04/2022
data-factory Data Flow Union https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-union.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Union transformation in mapping data flow
data-factory Data Flow Unpivot https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-unpivot.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Unpivot transformation in mapping data flow
data-factory Data Flow Window Functions https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-window-functions.md
Previously updated : 02/02/2022 Last updated : 08/04/2022 # Window functions in mapping data flow
data-factory Data Flow Window https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-flow-window.md
Previously updated : 09/09/2021 Last updated : 08/04/2022 # Window transformation in mapping data flow
The Window transformation is where you will define window-based aggregations of
## Over Set the partitioning of column data for your window transformation. The SQL equivalent is the ```Partition By``` in the Over clause in SQL. If you wish to create a calculation or create an expression to use for the partitioning, you can do that by hovering over the column name and select "computed column". ## Sort Another part of the Over clause is setting the ```Order By```. This will set the data sort ordering. You can also create an expression for a calculate value in this column field for sorting. ## Range By Next, set the window frame as Unbounded or Bounded. To set an unbounded window frame, set the slider to Unbounded on both ends. If you choose a setting between Unbounded and Current Row, then you must set the Offset start and end values. Both values will be positive integers. You can use either relative numbers or values from your data.
data-factory Data Migration Guidance Hdfs Azure Storage https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-migration-guidance-hdfs-azure-storage.md
Previously updated : 01/27/2022 Last updated : 08/05/2022 # Use Azure Data Factory to migrate data from an on-premises Hadoop cluster to Azure Storage
data-factory Data Migration Guidance Netezza Azure Sqldw https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-migration-guidance-netezza-azure-sqldw.md
Previously updated : 12/09/2020 Last updated : 08/05/2022 # Use Azure Data Factory to migrate data from an on-premises Netezza server to Azure
data-factory Data Migration Guidance Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-migration-guidance-overview.md
Previously updated : 01/27/2022 Last updated : 08/05/2022 # Use Azure Data Factory to migrate data from your data lake or data warehouse to Azure
data-factory Data Migration Guidance S3 Azure Storage https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-migration-guidance-s3-azure-storage.md
Previously updated : 01/27/2022 Last updated : 08/05/2022 # Use Azure Data Factory to migrate data from Amazon S3 to Azure Storage
data-factory Data Movement Security Considerations https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-movement-security-considerations.md
Previously updated : 10/22/2021 Last updated : 08/05/2022 # Security considerations for data movement in Azure Data Factory
data-factory Data Transformation Functions https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/data-transformation-functions.md
Previously updated : 02/02/2022 Last updated : 08/05/2022 # Data transformation expressions in mapping data flow
data-factory Delete Activity https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/delete-activity.md
Previously updated : 09/09/2021 Last updated : 08/05/2022 # Delete Activity in Azure Data Factory and Azure Synapse Analytics
data-factory Deploy Linked Arm Templates With Vsts https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/deploy-linked-arm-templates-with-vsts.md
Previously updated : 12/13/2021 Last updated : 08/05/2022 # Deploy linked ARM templates with VSTS
data-factory Enable Aad Authentication Azure Ssis Ir https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/enable-aad-authentication-azure-ssis-ir.md
Previously updated : 02/15/2022 Last updated : 08/05/2022 # Enable Azure Active Directory authentication for Azure-SSIS integration runtime
data-factory Enable Customer Managed Key https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/enable-customer-managed-key.md
Previously updated : 10/22/2021 Last updated : 08/05/2022
data-factory Encrypt Credentials Self Hosted Integration Runtime https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/encrypt-credentials-self-hosted-integration-runtime.md
Previously updated : 01/27/2022 Last updated : 08/05/2022
data-factory Format Avro https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-avro.md
Previously updated : 03/25/2022 Last updated : 08/05/2022
data-factory Format Binary https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-binary.md
Previously updated : 09/09/2021 Last updated : 08/05/2022
data-factory Format Common Data Model https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-common-data-model.md
Previously updated : 09/09/2021 Last updated : 08/05/2022
data-factory Format Delimited Text https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-delimited-text.md
Previously updated : 03/25/2022 Last updated : 08/05/2022
data-factory Format Delta https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-delta.md
Previously updated : 06/13/2022 Last updated : 08/05/2022
data-factory Format Excel https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-excel.md
Previously updated : 03/25/2022 Last updated : 08/05/2022
data-factory Format Json https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-json.md
Previously updated : 03/25/2022 Last updated : 08/05/2022
data-factory Format Orc https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-orc.md
Previously updated : 03/25/2022 Last updated : 08/05/2022
data-factory Format Parquet https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-parquet.md
Previously updated : 06/22/2022 Last updated : 08/05/2022
data-factory Format Xml https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/format-xml.md
Previously updated : 03/25/2022 Last updated : 08/05/2022
data-factory Whats New https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-factory/whats-new.md
Azure Data Factory is improved on an ongoing basis. To stay up to date with the
This page is updated monthly, so revisit it regularly.
+## July 2022
+<br>
+<table>
+<tr><td><b>Service category</b></td><td><b>Service improvements</b></td><td><b>Details</b></td></tr>
+<tr><td rowspan=5><b>Data flow</b></td><td>Asana connector added as source</td><td>WeΓÇÖve added a new REST-based connector to mapping data flows! Users can now read their tables from Asana. Note: This connector is only available when using inline datasets.<br><a href="connector-asana.md">Learn more</a></td></tr>
+<tr><td>3 new data transformation functions are supported</td><td>3 new data transformation functions have been added to mapping data flows in Azure Data Factory and Azure Synapse Analytics. Now, users are able to use collectUnique(), to create a new collection of unique values in an array, substringIndex(), to extract the substring before n occurrences of a delimiter, and topN(), to return the top n results after sorting your data.<br><a href=https://techcommunity.microsoft.com/t5/azure-data-factory-blog/3-new-data-transformation-functions-in-adf/ba-p/3582738>Learn more</a></td></tr>
+<tr><td>Refetch from source available in Refresh for data source change scenarios</td><td>When building and debugging a data flow, your source data can change. There is now a new easy way to refetch the latest updated source data from the refresh button in the data preview pane.<br><a href="concepts-data-flow-debug-mode.md#data-preview">Learn more</a></td></tr>
+<tr><td>User defined functions (GA) </td><td>Create reusable and customized expressions to avoid building complex logic over and over<br><a href="concepts-data-flow-debug-mode.md#data-preview">Learn more</a></td></tr>
+<tr><td>Easier configuration on data flow runtime ΓÇô choose compute size among Small, Medium, and Large to pre-configure all integration runtime settings</td><td>Azure Data Factory has made it easier for users to configure Azure Integration Runtime for mapping data flows by choosing compute size among Small, Medium, and Large to pre-configure all integration runtime settings. You can still set your own custom configurations.<br><a href=https://techcommunity.microsoft.com/t5/azure-data-factory-blog/adf-makes-it-easy-to-select-azure-ir-size-for-data-flows/ba-p/3578033>Learn more</a></td></tr>
+<tr><td rowspan=1><b>Continuous integration and continuous delivery (CI/CD)</b></td><td>Include Global parameters supported in ARM template</td><td>WeΓÇÖve added a new mechanism to include Global Parameters in the ARM templates. This helps to solve an earlier issue, which overrode some configurations during deployments when users included global parameters in ARM templates.<br><a href=https://techcommunity.microsoft.com/t5/azure-data-factory-blog/ci-cd-improvement-using-global-parameters-in-azure-data-factory/ba-p/3557265#M665>Learn more</a></td></tr>
+<tr><td><b>Developer productivity</b></td><td>Azure Data Factory studio preview experience</td><td>Be a part of Azure Data Factory Preview features to experience latest Azure Data Factory capabilities, and be the first to share your feedback<br><a href=https://techcommunity.microsoft.com/t5/azure-data-factory-blog/introducing-the-azure-data-factory-studio-preview-experience/ba-p/3563880>Learn more</a></td></tr>
+</table>
++ ## June 2022 <br> <table>
data-lake-store Data Lake Store Data Scenarios https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-lake-store/data-lake-store-data-scenarios.md
description: Understand the different scenarios and tools using which data can i
Previously updated : 06/27/2018 Last updated : 08/05/2022
This represents data that can be generated by various sources such as applicatio
Following are tools that you can use: * [Azure Stream Analytics](../stream-analytics/stream-analytics-define-outputs.md) - Events ingested into Event Hubs can be written to Azure Data Lake Storage Gen1 using an Azure Data Lake Storage Gen1 output.
-* [Azure HDInsight Storm](../hdinsight/storm/apache-storm-write-data-lake-store.md) - You can write data directly to Data Lake Storage Gen1 from the Storm cluster.
* [EventProcessorHost](../event-hubs/event-hubs-dotnet-standard-getstarted-send.md) ΓÇô You can receive events from Event Hubs and then write it to Data Lake Storage Gen1 using the [Data Lake Storage Gen1 .NET SDK](data-lake-store-get-started-net-sdk.md). ### Relational data
You can use a mix of services to create visual representations of data stored in
![Visualize data in Data Lake Storage Gen1](./media/data-lake-store-data-scenarios/visualize-data.png "Visualize data in Data Lake Storage Gen1") * You can start by using [Azure Data Factory to move data from Data Lake Storage Gen1 to Azure Synapse Analytics](../data-factory/copy-activity-overview.md)
-* After that, you can [integrate Power BI with Azure Synapse Analytics](/power-bi/connect-data/service-azure-sql-data-warehouse-with-direct-connect) to create visual representation of the data.
+* After that, you can [integrate Power BI with Azure Synapse Analytics](/power-bi/connect-data/service-azure-sql-data-warehouse-with-direct-connect) to create visual representation of the data.
data-lake-store Data Lake Store Hdinsight Hadoop Use Portal https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-lake-store/data-lake-store-hdinsight-hadoop-use-portal.md
You can use a Spark cluster to run Spark jobs on data that is stored in a Data L
### Use Data Lake Storage Gen1 in a Storm topology
-You can use the storage account with Data Lake Storage Gen1 to write data from a Storm topology. For instructions on how to achieve this scenario, see [Use Azure Data Lake Storage Gen1 with Apache Storm with HDInsight](../hdinsight/storm/apache-storm-write-data-lake-store.md).
- ## See also * [Use Data Lake Storage Gen1 with Azure HDInsight clusters](../hdinsight/hdinsight-hadoop-use-data-lake-storage-gen1.md) * [PowerShell: Create an HDInsight cluster to use Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-powershell.md) [makecert]: /windows-hardware/drivers/devtest/makecert
-[pvk2pfx]: /windows-hardware/drivers/devtest/pvk2pfx
+[pvk2pfx]: /windows-hardware/drivers/devtest/pvk2pfx
data-lake-store Data Lake Store Performance Tuning Storm https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/data-lake-store/data-lake-store-performance-tuning-storm.md
Understand the factors that should be considered when you tune the performance o
* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/). * **An Azure Data Lake Storage Gen1 account**. For instructions on how to create one, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md). * **An Azure HDInsight cluster** with access to a Data Lake Storage Gen1 account. See [Create an HDInsight cluster with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md). Make sure you enable Remote Desktop for the cluster.
-* **Running a Storm cluster on Data Lake Storage Gen1**. For more information, see [Storm on HDInsight](../hdinsight/storm/apache-storm-overview.md).
* **Performance tuning guidelines on Data Lake Storage Gen1**. For general performance concepts, see [Data Lake Storage Gen1 Performance Tuning Guidance](./data-lake-store-performance-tuning-guidance.md). ## Tune the parallelism of the topology
To check if you are getting throttled, enable the debug logging on the client si
## Next steps Additional performance tuning for Storm can be referenced in [this blog](/archive/blogs/shanyu/performance-tuning-for-hdinsight-storm-and-microsoft-azure-eventhubs).
-For an additional example to run, see [this one on GitHub](https://github.com/hdinsight/storm-performance-automation).
+For an additional example to run, see [this one on GitHub](https://github.com/hdinsight/storm-performance-automation).
databox-online Azure Stack Edge Gpu 2207 Release Notes https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/databox-online/azure-stack-edge-gpu-2207-release-notes.md
Previously updated : 07/22/2022 Last updated : 08/04/2022
The following release notes identify the critical open issues and the resolved i
The release notes are continuously updated, and as critical issues requiring a workaround are discovered, they're added. Before you deploy your device, carefully review the information contained in the release notes.
-This article applies to the **Azure Stack Edge 2207** release, which maps to software version number **2.2.2026.5318**. This software can be applied to your device if you're running at least Azure Stack Edge 2106 (2.2.1636.3457) software.
+This article applies to the **Azure Stack Edge 2207** release, which maps to software version number **2.2.2037.5375**. This software can be applied to your device if you're running at least Azure Stack Edge 2106 (2.2.1636.3457) software.
## What's new
databox-online Azure Stack Edge Gpu Install Update https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/databox-online/azure-stack-edge-gpu-install-update.md
Previously updated : 07/20/2022 Last updated : 08/04/2022 # Update your Azure Stack Edge Pro GPU
The procedure described in this article was performed using a different version
The current update is Update 2207. This update installs two updates, the device update followed by Kubernetes updates. The associated versions for this update are: -- Device software version - **2.2.2026.5318**
+- Device software version - **2.2.2037.5375**
+- Device Kubernetes version - **2.2.2037.5375**
- Kubernetes server version - **v1.22.6** - IoT Edge version: **0.1.0-beta15** - Azure Arc version: **1.6.6**
Depending on the software version that you are running, install process may diff
Go to the local web UI and then go to **Software update** page. Verify that the Kubernetes update has successfully installed and the software version reflects that.
- ![Software version after update 17](./media/azure-stack-edge-gpu-install-update/portal-update-16-m.png)
+ ![Screenshot of software version after update 17.](./media/azure-stack-edge-gpu-install-update/portal-update-17.png)
Once the device software and Kubernetes updates are successfully installed, the banner notification disappears.
This procedure takes around 20 minutes to complete. Perform the following steps
10. After the Kubernetes update is successfully installed, there is no change to the displayed software in **Maintenance** > **Software update**.
- ![update device 6](./media/azure-stack-edge-gpu-install-update/portal-update-16-m.png)
-
+ ![Screenshot of update device 6.](./media/azure-stack-edge-gpu-install-update/portal-update-17.png)
## Next steps
defender-for-cloud Defender For Storage Exclude https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/defender-for-cloud/defender-for-storage-exclude.md
Title: Microsoft Defender for Storage - excluding a storage account
-description: Excluding a specific storage account from a subscription with Microsoft Defender for Storage enabled.
Previously updated : 02/06/2022
+ Title: Exclude storage accounts from Microsoft Defender for Storage
+description: Learn how to exclude specific Azure Storage accounts from Microsoft Defender for Storage protections.
Last updated : 08/04/2022
-# Exclude a storage account from Microsoft Defender for Storage protections
-
-> [!CAUTION]
-> Excluding resources from advanced threat protection is not recommended and leaves your cloud workload exposed.
-
-When you [enable Microsoft Defender for Storage](../storage/common/azure-defender-storage-configure.md#set-up-microsoft-defender-for-cloud) on a subscription, all existing Azure Storage accounts will be protected and any storage resources added to that subscription in the future will also be automatically protected.
-If you need to exempt a specific Azure Storage account from this Defender plan, use the instructions on this page.
+# Exclude a storage account from Microsoft Defender for Storage protections
-> [!TIP]
-> We recommend enabling [Microsoft Defender for Resource Manager](defender-for-resource-manager-introduction.md) for any accounts with unprotected Azure Storage resources. Defender for Resource Manager automatically monitors your organization's resource management operations, whether they're performed through the Azure portal, Azure REST APIs, Azure CLI, or other Azure programmatic clients.
+When you [enable Microsoft Defender for Storage](../storage/common/azure-defender-storage-configure.md#set-up-microsoft-defender-for-cloud) on a subscription, all current and future Azure Storage accounts in that subscription are protected. If you have specific accounts that you want to exclude from the Defender for Storage protections, you can exclude them using the Azure portal, PowerShell, or the Azure CLI.
+We don't recommend that you exclude storage accounts from Defender for Storage because attackers can use any opening in order to compromise your environment. If you want to optimize your Azure costs and remove storage accounts that you feel are low risk from Defender for Storage, you can use the [Price Estimation Workbook](https://ms.portal.azure.com/#view/Microsoft_Azure_Security/SecurityMenuBlade/~/28) in the Azure portal to evaluate the cost savings.
-## Exclude a specific storage account
+## Exclude an Azure Storage account
-To exclude specific storage accounts from Microsoft Defender for Storage when the plan is enabled on a subscription:
+To exclude an Azure Storage account from Microsoft Defender for Storage:
### [**PowerShell**](#tab/enable-storage-protection-ps)
To exclude specific storage accounts from Microsoft Defender for Storage when th
Microsoft Defender for Storage can exclude specific active Databricks workspace storage accounts, when the plan is already enabled on a subscription.
-**To exclude an active Databricks workspace**:
+**To exclude an active Databricks workspace**:
1. Sign in to the [Azure portal](https://portal.azure.com). - 1. Navigate to **Azure Databricks** > **`Your Databricks workspace`** > **Tags**.- 1. In the Name field, enter `AzDefenderPlanAutoEnable`.- 1. In the Value field, enter `off`.- 1. Select **Apply**. :::image type="content" source="media/defender-for-storage-exclude/workspace-exclude.png" alt-text="Screenshot showing the location, and how to apply the tag to your Azure Databricks account."::: 1. Navigate to **Microsoft Defender for Cloud** > **Environment settings** > **`Your subscription`**.- 1. Toggle the Defender for Storage plan to **Off**. :::image type="content" source="media/defender-for-storage-exclude/storage-off.png" alt-text="Screenshot showing how to switch the Defender for Storage plan to off."::: 1. Select **Save**.- 1. Toggle the Defender for Storage plan to **On**.- 1. Select **Save**. The tags will be inherited by the Storage account of the Databricks workspace and prevent Defender for Storage from turning on.
-> [!Note]
+> [!Note]
> Tags can't be added directly to the Databricks Storage account, or its Managed Resource Group. ### Prevent auto-enabling on a new Databricks workspace storage account
When you create a new Databricks workspace, you have the ability to add a tag th
**To prevent auto-enabling on a new Databricks workspace storage account**:
- 1. Follow [these steps](/azure/databricks/scenarios/quickstart-create-Databricks-workspace-portal?tabs=azure-portal) to create a new Azure Databricks workspace.
+1. Follow [these steps](/azure/databricks/scenarios/quickstart-create-Databricks-workspace-portal?tabs=azure-portal) to create a new Azure Databricks workspace.
- 1. In the Tags tab, enter a tag named `AzDefenderPlanAutoEnable`.
+1. In the Tags tab, enter a tag named `AzDefenderPlanAutoEnable`.
- 1. Enter the value `off`.
+1. Enter the value `off`.
:::image type="content" source="media/defender-for-storage-exclude/tag-off.png" alt-text="Screenshot that shows how to create a tag in the Databricks workspace.":::
event-hubs Event Hubs Storm Getstarted Receive https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/event-hubs/event-hubs-storm-getstarted-receive.md
Before you start with the quickstart, **create an Event Hubs namespace and an ev
## Create project and add code
-This tutorial uses an [HDInsight Storm][HDInsight Storm] installation, which comes with the Event Hubs spout already available.
-
-1. Follow the [HDInsight Storm - Get Started](../hdinsight/storm/apache-storm-overview.md) procedure to create a new HDInsight cluster, and connect to it via Remote Desktop.
-2. Copy the `%STORM_HOME%\examples\eventhubspout\eventhubs-storm-spout-0.9-jar-with-dependencies.jar` file to your local development environment. This contains the events-storm-spout.
-3. Use the following command to install the package into the local Maven store. This enables you to add it as a reference in the Storm project in a later step.
+1. Use the following command to install the package into the local Maven store. This enables you to add it as a reference in the Storm project in a later step.
```shell mvn install:install-file -Dfile=target\eventhubs-storm-spout-0.9-jar-with-dependencies.jar -DgroupId=com.microsoft.eventhubs -DartifactId=eventhubs-storm-spout -Dversion=0.9 -Dpackaging=jar ```
-4. In Eclipse, create a new Maven project (click **File**, then **New**, then **Project**).
+1. In Eclipse, create a new Maven project (click **File**, then **New**, then **Project**).
![File -> New -> Project][12]
-5. Select **Use default Workspace location**, then click **Next**
-6. Select the **maven-archetype-quickstart** archetype, then click **Next**
-7. Insert a **GroupId** and **ArtifactId**, then click **Finish**
-8. In **pom.xml**, add the following dependencies in the `<dependency>` node.
+1. Select **Use default Workspace location**, then click **Next**
+1. Select the **maven-archetype-quickstart** archetype, then click **Next**
+1. Insert a **GroupId** and **ArtifactId**, then click **Finish**
+1. In **pom.xml**, add the following dependencies in the `<dependency>` node.
```xml <dependency>
This tutorial uses an [HDInsight Storm][HDInsight Storm] installation, which com
</dependency> ```
-9. In the **src** folder, create a file called **Config.properties** and copy the following content, substituting the `receive rule key` and `event hub name` values:
+1. In the **src** folder, create a file called **Config.properties** and copy the following content, substituting the `receive rule key` and `event hub name` values:
```java eventhubspout.username = ReceiveRule
This tutorial uses an [HDInsight Storm][HDInsight Storm] installation, which com
eventhub.receiver.credits = 10 ``` The value for **eventhub.receiver.credits** determines how many events are batched before releasing them to the Storm pipeline. For the sake of simplicity, this example sets this value to 10. In production, it should usually be set to higher values; for example, 1024.
-10. Create a new class called **LoggerBolt** with the following code:
+1 . Create a new class called **LoggerBolt** with the following code:
```java import java.util.Map;
governance Guest Configuration Baseline Windows https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/governance/policy/samples/guest-configuration-baseline-windows.md
For more information, see [Azure Policy guest configuration](../../machine-confi
|Name<br /><sub>(ID)</sub> |Details |Expected value<br /><sub>(Type)</sub> |Severity | ||||| |Audit PNP Activity<br /><sub>(AZ-WIN-00182)</sub> |**Description**: This policy setting allows you to audit when plug and play detects an external device. The recommended state for this setting is: `Success`. **Note:** A Windows 10, Server 2016 or higher OS is required to access and set this value in Group Policy.<br />**Key Path**: {0CCE9248-69AE-11D9-BED3-505054503030}<br />**OS**: WS2016, WS2019, WS2022<br />**Server Type**: Domain Controller, Domain Member, Workgroup Member |\>\= Success<br /><sub>(Audit)</sub> |Critical |
-|Audit Process Creation<br /><sub>(CCE-36059-4)</sub> |**Description**: This subcategory reports the creation of a process and the name of the program or user that created it. Events for this subcategory include: - 4688: A new process has been created. - 4696: A primary token was assigned to process. Refer to Microsoft Knowledge Base article 947226: [Description of security events in Windows Vista and in Windows Server 2008](https://support.microsoft.com/en-us/kb/947226) for the most recent information about this setting. The recommended state for this setting is: `Success`.<br />**Key Path**: {0CCE922B-69AE-11D9-BED3-505054503030}<br />**OS**: WS2008, WS2008R2, WS2012, WS2012R2, WS2016, WS2019, WS2022<br />**Server Type**: Domain Controller, Domain Member, Workgroup Member |\>\= Success<br /><sub>(Audit)</sub> |Critical |
+|Audit Process Creation<br /><sub>(CCE-36059-4)</sub> |**Description**: This subcategory reports the creation of a process and the name of the program or user that created it. Events for this subcategory include: - 4688: A new process has been created. - 4696: A primary token was assigned to process. Refer to Microsoft Knowledge Base article 947226: Description of security events in Windows Vista and in Windows Server 2008 for the most recent information about this setting. The recommended state for this setting is: `Success`.<br />**Key Path**: {0CCE922B-69AE-11D9-BED3-505054503030}<br />**OS**: WS2008, WS2008R2, WS2012, WS2012R2, WS2016, WS2019, WS2022<br />**Server Type**: Domain Controller, Domain Member, Workgroup Member |\>\= Success<br /><sub>(Audit)</sub> |Critical |
## System Audit Policies - Logon-Logoff
hdinsight Apache Hadoop Visual Studio Tools Get Started https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hadoop/apache-hadoop-visual-studio-tools-get-started.md
keywords: hadoop tools,hive query,visual studio,visual studio hadoop
Previously updated : 04/07/2022 Last updated : 08/05/2022 # Use Data Lake Tools for Visual Studio to connect to Azure HDInsight and run Apache Hive queries
Learn how to use Microsoft Azure Data Lake and Stream Analytics Tools for Visual
For more information about using HDInsight, see [Get started with HDInsight](apache-hadoop-linux-tutorial-get-started.md).
-For more information on connecting to Apache Storm, see [Develop C# topologies for Apache Storm by using the Data Lake tools](../storm/apache-storm-develop-csharp-visual-studio-topology.md).
- You can use Data Lake Tools for Visual Studio to access Azure Data Lake Analytics and HDInsight. For information about Data Lake Tools, see [Develop U-SQL scripts by using Data Lake Tools for Visual Studio](../../data-lake-analytics/data-lake-analytics-data-lake-tools-get-started.md). ## Prerequisites
hdinsight Hdinsight 36 Component Versioning https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-36-component-versioning.md
Title: Apache Hadoop components and versions - Azure HDInsight 3.6
description: Learn about the Apache Hadoop components and versions in Azure HDInsight 3.6. Previously updated : 02/15/2022 Last updated : 08/05/2022 # HDInsight 3.6 component versions
The table below lists the support timeframe for HDInsight 3.6 cluster types.
| HDInsight 3.6 Kafka | 1.1 | June 30, 2021 | September 30, 2022 | October 1, 2022 | | HDInsight 3.6 HBase | 1.1 | June 30, 2021 | September 30, 2022 | October 1, 2022 | | HDInsight 3.6 Interactive Query | 2.1 | June 30, 2021 | September 30, 2022 | October 1, 2022 |
-| HDInsight 3.6 Storm | 1.1 | June 30, 2021 | September 30, 2022 | October 1, 2022 |
| HDInsight 3.6 ML Services | 9.3 | - | - | December 31, 2020 | | HDInsight 3.6 Spark | 2.2 | - | - | June 30, 2020 | | HDInsight 3.6 Spark | 2.1 | - | - | June 30, 2020 |
The OSS component versions associated with HDInsight 3.6 are listed in the follo
| Apache Sqoop | 1.4.6 | | Apache Oozie | 4.2.0 | | Apache Zookeeper | 3.4.6 |
-| Apache Storm | 1.1.0 |
| Apache Mahout | 0.9.0+ | | Apache Phoenix | 4.7.0 | | Apache Spark | 2.3.2. |
The OSS component versions associated with HDInsight 3.6 are listed in the follo
- [Migrate Azure HDInsight 3.6 Hive workloads to HDInsight 4.0](interactive-query/apache-hive-migrate-workloads.md). - [Migrate Apache Kafka workloads to Azure HDInsight 4.0](kafk). - [Migrate an Apache HBase cluster to a new version](hbase/apache-hbase-migrate-new-version.md).-- [Migrate Azure HDInsight 3.6 Apache Storm to HDInsight 4.0 Apache Spark](storm/migrate-storm-to-spark.md). ## Next steps
hdinsight Hdinsight Apache Storm With Kafka https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-apache-storm-with-kafka.md
description: Learn how to create a streaming pipeline using Apache Storm and Apa
Previously updated : 06/25/2019 Last updated : 08/05/2022 #Customer intent: As a developer, I want to learn how to build a streaming pipeline that uses Storm and Kafka to process streaming data.
In this tutorial, you learn how to:
* Familiarity with creating Kafka topics. For more information, see the [Kafka on HDInsight quickstart](./kafk) document.
-* Familiarity with building and deploying Storm solutions (topologies). Specifically, topologies that use [Apache Storm Flux](https://storm.apache.org/releases/current/flux.html) framework. For more information, see the [Create an Apache Storm topology in Java](./storm/apache-storm-develop-java-topology.md) document.
- * [Java JDK 1.8](https://www.oracle.com/technetwork/pt/java/javase/downloads/jdk8-downloads-2133151.html) or higher. HDInsight 3.5 or higher require Java 8. * [Maven 3.x](https://maven.apache.org/download.cgi)
hdinsight Hdinsight Capacity Planning https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-capacity-planning.md
For better performance, use only one container per storage account.
## Choose a cluster type
-The cluster type determines the workload your HDInsight cluster is configured to run. Types include [Apache Hadoop](./hadoop/apache-hadoop-introduction.md), [Apache Storm](./storm/apache-storm-overview.md), [Apache Kafka](./kafk#cluster-types-in-hdinsight). Each cluster type has a specific deployment topology that includes requirements for the size and number of nodes.
+The cluster type determines the workload your HDInsight cluster is configured to run. Types include [Apache Hadoop](./hadoop/apache-hadoop-introduction.md), [Apache Kafka](./kafk#cluster-types-in-hdinsight). Each cluster type has a specific deployment topology that includes requirements for the size and number of nodes.
## Choose the VM size and type
hdinsight Hdinsight Component Versioning https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-component-versioning.md
Title: Apache Hadoop components and versions - Azure HDInsight
description: Learn about the Apache Hadoop components and versions in Azure HDInsight. Previously updated : 02/15/2022 Last updated : 08/05/2022 # Azure HDInsight versions
Microsoft does not encourage creating analytics pipelines or solutions on cluste
- [Migrate Azure HDInsight 3.6 Hive workloads to HDInsight 4.0](interactive-query/apache-hive-migrate-workloads.md). - [Migrate Apache Kafka workloads to Azure HDInsight 4.0](kafk). - [Migrate an Apache HBase cluster to a new version](hbase/apache-hbase-migrate-new-version.md).-- [Migrate Azure HDInsight 3.6 Apache Storm to HDInsight 4.0 Apache Spark](storm/migrate-storm-to-spark.md). ## Release notes
hdinsight Hdinsight Hadoop Create Linux Clusters Azure Cli https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-cli.md
description: Learn how to create Azure HDInsight clusters using the cross-platfo
Previously updated : 02/03/2020 Last updated : 08/05/2022 # Create HDInsight clusters using the Azure CLI
The steps in this document walk-through creating a HDInsight 3.6 cluster using t
||| |`--workernode-count`| The number of worker nodes in the cluster. This article uses the variable `clusterSizeInNodes` as the value passed to `--workernode-count`. | |`--version`| The HDInsight cluster version. This article uses the variable `clusterVersion` as the value passed to `--version`. See also: [Supported HDInsight versions](./hdinsight-component-versioning.md#supported-hdinsight-versions).|
- |`--type`| Type of HDInsight cluster, like: hadoop, interactivehive, hbase, kafka, storm, spark, rserver, mlservices. This article uses the variable `clusterType` as the value passed to `--type`. See also: [Cluster types and configuration](./hdinsight-hadoop-provision-linux-clusters.md#cluster-type).|
+ |`--type`| Type of HDInsight cluster, like: hadoop, interactivehive, hbase, kafka, spark, rserver, mlservices. This article uses the variable `clusterType` as the value passed to `--type`. See also: [Cluster types and configuration](./hdinsight-hadoop-provision-linux-clusters.md#cluster-type).|
|`--component-version`|The versions of various Hadoop components, in space-separated versions in 'component=version' format. This article uses the variable `componentVersion` as the value passed to `--component-version`. See also: [Hadoop components](./hdinsight-component-versioning.md).| Replace `RESOURCEGROUPNAME`, `LOCATION`, `CLUSTERNAME`, `STORAGEACCOUNTNAME`, and `PASSWORD` with the desired values. Change values for the other variables as desired. Then enter the CLI commands.
The steps in this document walk-through creating a HDInsight 3.6 cluster using t
``` > [!IMPORTANT]
- > HDInsight clusters come in various types, which correspond to the workload or technology that the cluster is tuned for. There is no supported method to create a cluster that combines multiple types, such as Storm and HBase on one cluster.
+ > HDInsight clusters come in various types, which correspond to the workload or technology that the cluster is tuned for. There is no supported method to create a cluster that combines multiple types, such as HBase on one cluster.
It may take several minutes for the cluster creation process to complete. Usually around 15.
Now that you've successfully created an HDInsight cluster using the Azure CLI, u
* [Get started with Apache HBase on HDInsight](hbase/apache-hbase-tutorial-get-started-linux.md) * [Develop Java applications for Apache HBase on HDInsight](hbase/apache-hbase-build-java-maven-linux.md)-
-### Apache Storm clusters
-
-* [Develop Java topologies for Apache Storm on HDInsight](storm/apache-storm-develop-java-topology.md)
-* [Use Python components in Apache Storm on HDInsight](storm/apache-storm-develop-python-topology.md)
-* [Deploy and monitor topologies with Apache Storm on HDInsight](storm/apache-storm-deploy-monitor-topology-linux.md)
hdinsight Hdinsight Hadoop Create Linux Clusters Azure Powershell https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-powershell.md
Title: Create Apache Hadoop clusters using PowerShell - Azure HDInsight
-description: Learn how to create Apache Hadoop, Apache HBase, Apache Storm, or Apache Spark clusters on Linux for HDInsight by using Azure PowerShell.
+description: Learn how to create Apache Hadoop, Apache HBase, or Apache Spark clusters on Linux for HDInsight by using Azure PowerShell.
ms.tool: azure-powershell Previously updated : 12/18/2019 Last updated : 08/05/2022 # Create Linux-based clusters in HDInsight using Azure PowerShell
Now that you've successfully created an HDInsight cluster, use the following res
* [Get started with Apache HBase on HDInsight](hbase/apache-hbase-tutorial-get-started-linux.md) * [Develop Java applications for Apache HBase on HDInsight](hbase/apache-hbase-build-java-maven-linux.md)
-### Storm clusters
-
-* [Develop Java topologies for Storm on HDInsight](storm/apache-storm-develop-java-topology.md)
-* [Use Python components in Storm on HDInsight](storm/apache-storm-develop-python-topology.md)
-* [Deploy and monitor topologies with Storm on HDInsight](storm/apache-storm-deploy-monitor-topology-linux.md)
- ### Apache Spark clusters * [Create a standalone application using Scala](spark/apache-spark-create-standalone-application.md) * [Run jobs remotely on an Apache Spark cluster using Apache Livy](spark/apache-spark-livy-rest-interface.md) * [Apache Spark with BI: Perform interactive data analysis using Spark in HDInsight with BI tools](spark/apache-spark-use-bi-tools.md)
-* [Apache Spark with Machine Learning: Use Spark in HDInsight to predict food inspection results](spark/apache-spark-machine-learning-mllib-ipython.md)
+* [Apache Spark with Machine Learning: Use Spark in HDInsight to predict food inspection results](spark/apache-spark-machine-learning-mllib-ipython.md)
hdinsight Hdinsight Hadoop Create Linux Clusters Curl Rest https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-hadoop-create-linux-clusters-curl-rest.md
description: Learn how to create HDInsight clusters by submitting Azure Resource
Previously updated : 12/10/2019 Last updated : 08/05/2022 # Create Apache Hadoop clusters using the Azure REST API
Now that you've successfully created an HDInsight cluster, use the following to
* [Get started with Apache HBase on HDInsight](hbase/apache-hbase-tutorial-get-started-linux.md) * [Develop Java applications for Apache HBase on HDInsight](hbase/apache-hbase-build-java-maven-linux.md)-
-### Apache Storm clusters
-
-* [Develop Java topologies for Apache Storm on HDInsight](storm/apache-storm-develop-java-topology.md)
-* [Use Python components in Apache Storm on HDInsight](storm/apache-storm-develop-python-topology.md)
-* [Deploy and monitor topologies with Apache Storm on HDInsight](storm/apache-storm-deploy-monitor-topology-linux.md)
hdinsight Hdinsight Hadoop Migrate Dotnet To Linux https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-hadoop-migrate-dotnet-to-linux.md
description: Learn how to use .NET applications for streaming MapReduce on Linux
Previously updated : 02/27/2018 Last updated : 08/05/2022 # Migrate .NET solutions for Windows-based HDInsight to Linux-based HDInsight
Mono version 4.2.1 is included with HDInsight version 3.6. For more information
For more information on compatibility between Mono and .NET, see the [Mono compatibility (https://www.mono-project.com/docs/about-mono/compatibility/)](https://www.mono-project.com/docs/about-mono/compatibility/) document.
-> [!IMPORTANT]
-> The SCP.NET framework is compatible with Mono. For more information on using SCP.NET with Mono, see [Use Visual Studio to develop C# topologies for Apache Storm on HDInsight](storm/apache-storm-develop-csharp-visual-studio-topology.md).
- ## Automated portability analysis The [.NET Portability Analyzer](https://marketplace.visualstudio.com/items?itemName=ConnieYau.NETPortabilityAnalyzer) can be used to generate a report of incompatibilities between your application and Mono. Use the following steps to configure the analyzer to check your application for Mono portability:
For more information on accessing logs, see the following documents:
* [Use C# with MapReduce on HDInsight](hadoop/apache-hadoop-dotnet-csharp-mapreduce-streaming.md) * [Use C# user-defined functions with Apache Hive and Apache Pig](hadoop/apache-hadoop-hive-pig-udf-dotnet-csharp.md)-
-* [Develop C# topologies for Apache Storm on HDInsight](storm/apache-storm-develop-csharp-visual-studio-topology.md)
hdinsight Hdinsight Hadoop Port Settings For Services https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-hadoop-port-settings-for-services.md
description: This article provides a list of ports used by Apache Hadoop service
Previously updated : 04/22/2022 Last updated : 08/05/2022 # Ports used by Apache Hadoop services on HDInsight
The following are available for specific cluster types:
| `Stargate` |443 |HTTPS |HBase |HBase REST API. See [Get started using Apache HBase](hbase/apache-hbase-tutorial-get-started-linux.md) | | Livy |443 |HTTPS |Spark |Spark REST API. See [Submit Apache Spark jobs remotely using Apache Livy](spark/apache-spark-livy-rest-interface.md) | | Spark Thrift server |443 |HTTPS |Spark |Spark Thrift server used to submit Hive queries. See [Use Beeline with Apache Hive on HDInsight](hadoop/apache-hadoop-use-hive-beeline.md) |
-| Storm |443 |HTTPS |Storm |Storm web UI. See [Deploy and manage Apache Storm topologies on HDInsight](storm/apache-storm-deploy-monitor-topology-linux.md) |
| Kafka REST proxy |443 |HTTPS |Kafka |Kafka REST API. See [Interact with Apache Kafka clusters in Azure HDInsight using a REST proxy](kafk) | ### Authentication
hdinsight Hdinsight Hadoop Windows Tools https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-hadoop-windows-tools.md
description: Work from a Windows PC in Hadoop on HDInsight. Manage and query clu
Previously updated : 12/20/2019 Last updated : 08/05/2022 # Work in the Apache Hadoop ecosystem on HDInsight from a Windows PC
The following utilities have a web UI that runs in a browser:
* [Apache Hive View in Apache Ambari](hadoop/apache-hadoop-use-hive-ambari-view.md) * [Apache Tez View in Apache Ambari](./index.yml)
-## Data Lake (Hadoop) Tools for Visual Studio
-
-Use Data Lake Tools for Visual Studio to deploy and manage Storm topologies. Data Lake Tools also installs the SCP.NET SDK, which allows you to develop C# Storm topologies with Visual Studio.
- Before you go to the following examples, [install and try Data Lake Tools for Visual Studio](hadoop/apache-hadoop-visual-studio-tools-get-started.md).
-Examples of tasks you can do with Visual Studio and Data Lake Tools for Visual Studio:
-* [Deploy and manage Storm topologies from Visual Studio](storm/apache-storm-deploy-monitor-topology-linux.md)
-* [Develop C# topologies for Storm using Visual Studio](storm/apache-storm-develop-csharp-visual-studio-topology.md). The bits include example templates for Storm topologies you can connect to databases, such as Azure Cosmos DB and SQL Database.
- ## Visual Studio and the .NET SDK You can use Visual Studio with the .NET SDK to manage clusters and develop big data applications. You can use other IDEs for the following tasks, but examples are shown in Visual Studio.
The Azure command-line interface (CLI) is Microsoft's cross-platform command-lin
If you're new to working in Linux-based clusters, see the follow articles: * [Set up Apache Hadoop, Apache Kafka, Apache Spark, or other clusters](hdinsight-hadoop-provision-linux-clusters.md)
-* [Tips for HDInsight clusters on Linux](hdinsight-hadoop-linux-information.md)
+* [Tips for HDInsight clusters on Linux](hdinsight-hadoop-linux-information.md)
hdinsight Hdinsight Streaming At Scale Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-streaming-at-scale-overview.md
description: How to use data streaming with scalable Apache clusters in Azure HD
Previously updated : 12/17/2019 Last updated : 08/05/2022 # Streaming at scale in HDInsight Real-time big data solutions act on data that is in motion. Typically, this data is most valuable at its time of arrival. If the incoming data stream becomes greater than can be handled at that moment, you may need to throttle down resources. Alternatively, an HDInsight cluster can scale up to meet your streaming solution by adding nodes on demand.
-In a streaming application, one or more data sources are generating events (sometimes in the millions per second) that need to be ingested quickly without dropping any useful information. The incoming events are handled with *stream buffering*, also called *event queuing*, by a service such as [Apache Kafka](kafk). The processed data can be stored in long-term storage systems, like [Azure Data Lake Storage](https://azure.microsoft.com/services/storage/data-lake-storage/), and displayed in real time on a business intelligence dashboard, such as [Power BI](https://powerbi.microsoft.com), Tableau, or a custom web page.
+In a streaming application, one or more data sources are generating events (sometimes in the millions per second) that need to be ingested quickly without dropping any useful information. The incoming events are handled with *stream buffering*, also called *event queuing*, by a service such as [Apache Kafka](kafk) or [Event Hubs](https://azure.microsoft.com/services/event-hubs/). After you collect the events, you can then analyze the data using a real-time analytics system within the *stream processing* layer. The processed data can be stored in long-term storage systems, like [Azure Data Lake Storage](https://azure.microsoft.com/services/storage/data-lake-storage/), and displayed in real time on a business intelligence dashboard, such as [Power BI](https://powerbi.microsoft.com), Tableau, or a custom web page.
:::image type="content" source="./media/hdinsight-streaming-at-scale-overview/HDInsight-streaming-patterns.png" alt-text="Azure HDInsight Streaming Patterns":::
Apache Kafka provides a high-throughput, low-latency message queueing service, a
For more information, see [Introducing Apache Kafka on HDInsight](kafk).
-## Apache Storm
-
-Apache Storm is a distributed, fault-tolerant, open-source computation system that is optimized for processing streams of data in real time with Hadoop. The core unit of data for an incoming event is a Tuple, which is an immutable set of key/value pairs. An unbounded sequence of these Tuples forms a Stream, which is comes from a Spout. The Spout wraps a streaming data source (such as Kafka), and emits Tuples. A storm Topology is a sequence of transformations on these streams.
-
-For more information, see [What is Apache Storm on Azure HDInsight?](storm/apache-storm-overview.md).
- ## Spark Streaming
-Spark Streaming is an extension to Spark, which allows you to reuse the same code that you use for batch processing. You can combine both batch and interactive queries in the same application. Unlike Storm, Spark Streaming provides stateful exactly once processing semantics. When used in combination with the [Kafka Direct API](https://spark.apache.org/docs/latest/streaming-kafka-integration.html), which ensures that all Kafka data is received by Spark Streaming exactly once, it's possible to achieve end-to-end exactly once guarantees. One of Spark Streaming's strengths is its fault-tolerant capabilities, recovering faulted nodes rapidly when multiple nodes are being used within the cluster.
+Spark Streaming is an extension to Spark, which allows you to reuse the same code that you use for batch processing. You can combine both batch and interactive queries in the same application. Unlike Spark Streaming provides stateful exactly once processing semantics. When used in combination with the [Kafka Direct API](https://spark.apache.org/docs/latest/streaming-kafka-integration.html), which ensures that all Kafka data is received by Spark Streaming exactly once, it's possible to achieve end-to-end exactly once guarantees. One of Spark Streaming's strengths is its fault-tolerant capabilities, recovering faulted nodes rapidly when multiple nodes are being used within the cluster.
For more information, see [What is Apache Spark Streaming?](./spark/apache-spark-streaming-overview.md).
The stream buffering technologies Event Hubs and Kafka both use partitions, and
### Scale the stream processing layer
-Both Apache Storm and Spark Streaming support adding worker nodes to their clusters, even while data is being processed.
-
-To take advantage of new nodes added through scaling Storm, you need to rebalance any Storm topologies started before the cluster size was increased. This rebalancing can be done using the Storm web UI or its CLI. For more information, see the [Apache Storm documentation](https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html).
+Apache Spark Streaming support adding worker nodes to their clusters, even while data is being processed.
Apache Spark uses three key parameters for configuring its environment, depending on application requirements: `spark.executor.instances`, `spark.executor.cores`, and `spark.executor.memory`. An *executor* is a process that is launched for a Spark application. An executor runs on the worker node and is responsible for carrying out the application's tasks. The default number of executors and the executor sizes for each cluster are calculated based on the number of worker nodes and the worker node size. These numbers are stored in the `spark-defaults.conf`file on each cluster head node.
These three parameters can be configured at the cluster level, for all applicat
## Next steps
-* [Create and monitor an Apache Storm topology in Azure HDInsight](storm/apache-storm-quickstart.md)
-* [Example topologies for Apache Storm on HDInsight](storm/apache-storm-example-topology.md)
* [Introduction to Apache Spark on HDInsight](spark/apache-spark-overview.md)
-* [Start with Apache Kafka on HDInsight](kafk)
+* [Start with Apache Kafka on HDInsight](kafk)
hdinsight Hdinsight Troubleshoot Guide https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/hdinsight-troubleshoot-guide.md
Title: Azure HDInsight troubleshooting guides
-description: Troubleshoot Azure HDInsight. Step-by-step documentation shows you how to use HDInsight to solve common problems with Apache Hive, Apache Spark, Apache YARN, Apache HBase, HDFS, and Apache Storm.
+description: Troubleshoot Azure HDInsight. Step-by-step documentation shows you how to use HDInsight to solve common problems with Apache Hive, Apache Spark, Apache YARN, Apache HBase, and HDFS.
Previously updated : 06/23/2022 Last updated : 08/05/2022 # Troubleshoot Azure HDInsight
Last updated 06/23/2022
|:::image type="content" source="./medi)| |:::image type="content" source="./medi#how-do-i-download-tez-dag-data-from-a-cluster)| |:::image type="content" source="./medi#scenario-outofmemoryerror-exception-for-apache-spark)|
-|:::image type="content" source="./medi#how-do-i-locate-storm-event-hub-spout-binaries-for-development)|
|:::image type="content" source="./medi#how-do-i-download-yarn-logs-from-a-cluster)| ## HDInsight troubleshooting resources
hdinsight Apache Storm Deploy Monitor Topology Linux https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-deploy-monitor-topology-linux.md
- Title: Deploy and manage Apache Storm topologies on Azure HDInsight
-description: Learn how to deploy, monitor, and manage Apache Storm topologies using the Storm Dashboard on Linux-based HDInsight. Use Hadoop tools for Visual Studio.
--- Previously updated : 07/19/2022--
-# Deploy and manage Apache Storm topologies on Azure HDInsight
-
-In this document, learn the basics of managing and monitoring [Apache Storm](https://storm.apache.org/) topologies running on Storm on HDInsight clusters.
-
-## Prerequisites
-
-* An Apache Storm cluster on HDInsight. See [Create Apache Hadoop clusters using the Azure portal](../hdinsight-hadoop-create-linux-clusters-portal.md) and select **Storm** for **Cluster type**.
-
-* (Optional) Familiarity with Secure Shell (SSH) and Secure Copy (SCP). For more information, see [Connect to HDInsight (Apache Hadoop) using SSH](../hdinsight-hadoop-linux-use-ssh-unix.md).
-
-* (Optional) Visual Studio, Azure SDK 2.5.1 or newer, and the Data Lake Tools for Visual Studio. For more information, see [Apache Hadoop & Visual Studio Data Lake Tools](../hadoop/apache-hadoop-visual-studio-tools-get-started.md).
-
-## Submit a topology using Visual Studio
-
-You can use the Data Lake Tools for Visual Studio to submit C# or hybrid topologies to your Storm cluster. The following steps use a sample application. For information about topology creation using the Data Lake Tools, see [Apache Storm topologies with Visual Studio and C#](apache-storm-develop-csharp-visual-studio-topology.md).
-
-1. If you haven't already installed the latest version of the Data Lake tools for Visual Studio, see [Use Data Lake Tools for Visual Studio](../hadoop/apache-hadoop-visual-studio-tools-get-started.md).
-
- > [!NOTE]
- > The Azure Data Lake and Stream Analytics Tools were formerly called the HDInsight Tools for Visual Studio.
- >
- > Azure Data Lake and Stream Analytics Tools for Visual Studio are included in the **Azure development** workload for Visual Studio 2019.
-
-1. Start Visual Studio.
-
-1. In the **Start** window, select **Create a new project**.
-
-1. In the **Create a new project** window, select the search box, and enter `Storm`. Then choose **Storm Sample** from the result list and select **Next**.
-
-1. In the **Configure your new project** window, enter a **Project name**, and go to or create a **Location** to save the new project in. Then select **Create**.
-
- :::image type="content" source="./media/apache-storm-deploy-monitor-topology-linux/apache-storm-sample1.png" alt-text="Configure your new project window, Visual Studio" border="true":::
-
-1. From **Server Explorer**, right-click **Azure** and select **Connect to Microsoft Azure Subscription...** and complete the sign-in process.
-
-1. From **Solution Explorer**, right-click the project, and choose **Submit to Storm on HDInsight**.
-
- > [!NOTE]
- > If prompted, enter the login credentials for your Azure subscription. If you have more than one subscription, sign in to the one that contains your Storm on HDInsight cluster.
-
-1. In the **Submit Topology** dialog box, under the **Storm Cluster** drop-down list, choose your Storm on HDInsight cluster, and then select **Submit**. You can monitor whether the submission is successful by viewing the **Output** pane.
-
-## Submit a topology using SSH and the Storm command
-
-1. Use [ssh command](../hdinsight-hadoop-linux-use-ssh-unix.md) to connect to your cluster. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
-
- ```cmd
- ssh sshuser@CLUSTERNAME-ssh.azurehdinsight.net
- ```
-
-1. From your ssh session, use the following command to start the **WordCount** example topology:
-
- ```bash
- storm jar /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-*.jar org.apache.storm.starter.WordCountTopology WordCount
- ```
-
- This command starts the example WordCount topology on the cluster. This topology randomly generates sentences, and then counts the occurrence of each word in the sentences.
-
- > [!NOTE]
- > When submitting topology to the cluster, you must first copy the .jar file containing the cluster before using the `storm` command. To copy the file to the cluster, you can use the `scp` command. For example, enter `scp FILENAME.jar USERNAME@CLUSTERNAME-ssh.azurehdinsight.net:FILENAME.jar`.
- >
- > The *WordCount* example, and other storm starter examples, are already included on your cluster at `/usr/hdp/current/storm-client/contrib/storm-starter/`.
-
-## Submit a topology programmatically
-
-You can programmatically deploy a topology using the Nimbus service. [https://github.com/Azure-Samples/hdinsight-java-deploy-storm-topology](https://github.com/Azure-Samples/hdinsight-java-deploy-storm-topology) provides an example Java application that demonstrates how to deploy and start a topology through the Nimbus service.
-
-## Monitor and manage a topology in Visual Studio
-
-When you submit a topology using Visual Studio, the **Storm Topologies View** window appears. Select the topology from the list to view information about the running topology.
--
-> [!NOTE]
-> You can also view **Storm Topologies** from **Server Explorer**. Expand **Azure** > **HDInsight**, right-click a Storm on HDInsight cluster, and then select **View Storm Topologies**.
-
-Select the shape for the spouts or bolts to view information about these components. A tooltip with component information appears for the item selected.
-
-### Deactivate and reactivate a topology
-
-Deactivating a topology pauses it until the topology is killed or reactivated. To do these operations, use the **Deactivate** and **Reactivate** buttons in the **Actions** area at the top of the **Storm Topologies View** window.
-
-### Rebalance a topology
-
-Rebalancing a topology allows the system to revise the parallelism of the topology. For example, if you've resized the cluster to add more notes, rebalancing allows a topology to see the new nodes.
-
-To rebalance a topology, use the **Rebalance** button in the **Actions** area of the **Storm Topologies View** window.
-
-> [!WARNING]
-> Rebalancing a topology deactivates the topology, redistributes workers evenly across the cluster, and then returns the topology to the state it was in before rebalancing occurred. If the topology was active, it becomes active again. If the topology was deactivated, it remains deactivated.
-
-### Kill a running topology
-
-Storm topologies continue running until they're stopped or the cluster is deleted. To stop a topology, use the **Kill** button in the **Actions** area.
-
-## Monitor and manage a topology using SSH and the Storm command
-
-The `storm` utility allows you to work with running topologies from the command line. Use `storm -h` for a full list of commands.
-
-### List topologies
-
-Use the following command to list all running topologies:
-
-```shell
-storm list
-```
-
-This command returns information similar to the following text:
-
-```shell
-Topology_name Status Num_tasks Num_workers Uptime_secs
--
-WordCount ACTIVE 29 2 263
-```
-
-### Deactivate and reactivate a topology
-
-Deactivating a topology pauses it until the topology is killed or reactivated. Use the following commands to deactivate or reactivate:
-
-```shell
-storm Deactivate TOPOLOGYNAME
-```
-
-```shell
-storm Activate TOPOLOGYNAME
-```
-
-### Kill a running topology
-
-Storm topologies, once started, continue running until stopped. To stop a topology, use the following command:
-
-```shell
-storm kill TOPOLOGYNAME
-```
-
-### Rebalance a topology
-
-Rebalancing a topology allows the system to revise the parallelism of the topology. For example, if you've resized the cluster to add more notes, rebalancing allows a topology to see the new nodes.
-
-> [!WARNING]
-> Rebalancing a topology deactivates the topology, redistributes workers evenly across the cluster, and then returns the topology to the state it was in before rebalancing occurred. If the topology was active, it becomes active again. If it was deactivated, it remains deactivated.
-
-```shell
-storm rebalance TOPOLOGYNAME
-```
-
-## Monitor and manage a topology using the Storm UI
-
-The Storm UI provides a web interface for working with running topologies, and it's included on your HDInsight cluster. To view the Storm UI, use a web browser to open `https://CLUSTERNAME.azurehdinsight.net/stormui`, where *CLUSTERNAME* is the name of your cluster.
-
-> [!NOTE]
-> If you're asked to provide a user name and password, enter the cluster administrator username and password that you used when creating the cluster.
-
-### Storm UI main page
-
-The main page of the Storm UI provides the following information:
-
-| Section | Description |
-| | |
-| Cluster summary| Basic information about the Storm cluster. |
-| Nimbus summary | A list of basic Nimbus information. |
-| Topology summary | A list of running topologies. To view more information about a specific topology, select its link in the **Name** column. |
-| Supervisor summary | Information about the Storm supervisor. To see the worker resources associated with a specific supervisor, select its link in the **Host** or **Id** column. |
-| Nimbus configuration | Nimbus configuration for the cluster. |
-
-The Storm UI main page looks similar to this web page:
--
-#### Topology summary
-
-Selecting a link from the **Topology summary** section displays the following information about the topology:
-
-| Section | Description |
-| | |
-| Topology summary | Basic information about the topology. |
-| Topology actions| Management actions that you can do for the topology. The available actions are described later in this section. |
-| Topology stats | Statistics about the topology. To set the time frame for an entry in this section, select its link in the **Window** column. |
-| Spouts *(time frame)* | The spouts used by the topology. To view more information about a specific spout, select its link in the **Id** column. |
-| Bolts *(time frame)* | The bolts used by the topology. To view more information about a specific bolt, select its link in the **Id** column. |
-| Worker resources | A list of worker resources. To view more information about a specific worker resource, select its link in the **Host** column. |
-| Topology visualization | A **Show Visualization** button that displays a visualization of the topology. |
-| Topology configuration | The configuration of the selected topology. |
-
-The Storm topology summary page looks similar to this web page:
--
-In the **Topology actions** section, you can select the following buttons to do an action:
-
-| Button | Description |
-| | |
-| Activate | Resumes processing of a deactivated topology. |
-| Deactivate | Pauses a running topology. |
-| Rebalance | Adjusts the parallelism of the topology. You should rebalance running topologies after you've changed the number of nodes in the cluster. This operation allows the topology to adjust parallelism to compensate for the additional or reduced number of nodes in the cluster.<br/><br/>For more information, see <a href="https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html" target="_blank">Understanding the parallelism of an Apache Storm topology</a>.
-| Kill | Terminates a Storm topology after the specified timeout. |
-| Debug | Begins a debugging session for the running topology. |
-| Stop Debug | Ends the debugging session for the running topology. |
-| Change Log Level | Modifies the debugging log level. |
-
-##### Spout and bolt summary
-
-Selecting a spout from the **Spouts** or **Bolts** sections displays the following information about the selected item:
-
-| Section | Description |
-| | |
-| Component summary | Basic information about the spout or bolt. |
-| Component actions | **Debug** and **Stop Debug** buttons. |
-| Spout stats or bolt stats | Statistics about the spout or bolt. To set the time frame for an entry in this section, select its link in the **Window** column. |
-| (Bolt-only)<br/>Input stats*(time frame)* | Information about the input streams consumed by the bolt. |
-| Output stats *(time frame)* | Information about the streams emitted by the spout or bolt. |
-| Profiling and debugging | Controls for profiling and debugging the components on this page. You can set the **Status / Timeout (Minutes)** value, and you can select buttons for **JStack**, **Restart Worker**, and **Heap**. |
-| Executors *(time frame)* | Information about the instances of the spout or bolt. To view a log of diagnostic information produced for this instance, select the **Port** entry for a specific executor. You can also see the worker resources associated with a specific executor by selecting its link in the **Host** column. |
-| Errors | Any error information for the spout or bolt. |
-
-The Storm bolt summary page looks similar to this web page:
--
-## Monitor and manage the topology using the REST API
-
-The Storm UI is built on top of the REST API, so you can do similar management and monitoring tasks by using the REST API. You can use the REST API to create custom tools for managing and monitoring Storm topologies.
-
-For more information, see [Apache Storm UI REST API](https://storm.apache.org/releases/current/STORM-UI-REST-API.html). The following information is specific to using the REST API with Apache Storm on HDInsight.
-
-> [!IMPORTANT]
-> The Storm REST API is not publicly available over the internet. It must be accessed using an SSH tunnel to the HDInsight cluster head node. For information on creating and using an SSH tunnel, see [Use SSH tunneling to access Azure HDInsight](../hdinsight-linux-ambari-ssh-tunnel.md).
-
-### Base URI
-
-The base URI for the REST API on Linux-based HDInsight clusters is available at URL address `https://HEADNODEFQDN:8744/api/v1/`, where you replace *HEADNODEFQDN* with the head node. The domain name of the head node is generated during cluster creation and isn't static.
-
-You can find the fully qualified domain name (FQDN) for the cluster head node in several ways:
-
-| FQDN discovery method | Description |
-| | |
-| SSH session | Use the command `headnode -f` from an SSH session to the cluster. |
-| Ambari Web | On the Ambari cluster web page (`https://CLUSTERNAME.azurehdinsight.net`), select **Services** from the top of the page, then select **Storm**. From the **Summary** tab, select **Storm UI Server**. The FQDN of the node that hosts the Storm UI and REST API is displayed at the top of the page. |
-| Ambari REST API | Use the command `curl -u admin -G "https://CLUSTERNAME.azurehdinsight.net/api/v1/clusters/CLUSTERNAME/services/STORM/components/STORM_UI_SERVER"` to retrieve information about the node that the Storm UI and REST API are running on. Replace the two instances of *CLUSTERNAME* with the cluster name. When you're prompted, enter the password for the user (admin) account. In the response, the "host_name" entry of the JSON output contains the FQDN of the node. |
-
-### Authentication
-
-Requests to the REST API must use *basic authentication*, so you have to use the administrator name and password for the HDInsight cluster.
-
-> [!NOTE]
-> Because basic authentication is sent by using clear text, you should *always* use HTTPS to secure communications with the cluster.
-
-### Return values
-
-Information that is returned from the REST API may only be usable from within the cluster. For example, the fully qualified domain name (FQDN) returned for [Apache ZooKeeper](https://zookeeper.apache.org/) servers isn't accessible from the internet.
-
-## Next steps
-
-Learn how to [Develop Java-based topologies using Apache Maven](apache-storm-develop-java-topology.md).
-
-For a list of more example topologies, see [Example Apache Storm topologies in Azure HDInsight](apache-storm-example-topology.md).
hdinsight Apache Storm Develop Csharp Event Hub Topology https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-develop-csharp-event-hub-topology.md
- Title: Process events from Event Hubs with Storm - Azure HDInsight
-description: Learn how to process data from Azure Event Hubs with a C# Storm topology created in Visual Studio, by using the HDInsight tools for Visual Studio.
-- Previously updated : 01/14/2020---
-# Process events from Azure Event Hubs with Apache Storm on HDInsight (C#)
-
-Learn how to work with Azure Event Hubs from [Apache Storm](https://storm.apache.org/) on HDInsight. This document uses a C# Storm topology to read and write data from Event Hubs
-
-> [!NOTE]
-> For a Java version of this project, see [Process events from Azure Event Hubs with Apache Storm on HDInsight (Java)](https://github.com/Azure-Samples/hdinsight-java-storm-eventhub).
-
-## SCP.NET
-
-The steps in this document use SCP.NET, a NuGet package that makes it easy to create C# topologies and components for use with Storm on HDInsight.
-
-HDInsight 3.4 and greater use Mono to run C# topologies. The example used in this document works with HDInsight 3.6. If you plan on creating your own .NET solutions for HDInsight, check the [Mono compatibility](https://www.mono-project.com/docs/about-mono/compatibility/) document for potential incompatibilities.
-
-### Cluster versioning
-
-The Microsoft.SCP.Net.SDK NuGet package you use for your project must match the major version of Storm installed on HDInsight. HDInsight versions 3.5 and 3.6 use Storm 1.x, so you must use SCP.NET version 1.0.x.x with these clusters.
-
-C# topologies must also target .NET 4.5.
-
-## How to work with Event Hubs
-
-Microsoft provides a set of Java components that can be used to communicate with Event Hubs from a Storm topology. You can find the Java archive (JAR) file that contains an HDInsight 3.6 compatible version of these components at [https://github.com/hdinsight/mvn-repo/raw/master/org/apache/storm/storm-eventhubs/1.1.0.1/storm-eventhubs-1.1.0.1.jar](https://github.com/hdinsight/mvn-repo/raw/master/org/apache/storm/storm-eventhubs/1.1.0.1/storm-eventhubs-1.1.0.1.jar).
-
-> [!IMPORTANT]
-> While the components are written in Java, you can easily use them from a C# topology.
-
-The following components are used in this example:
-
-* __EventHubSpout__: Reads data from Event Hubs.
-* __EventHubBolt__: Writes data to Event Hubs.
-* __EventHubSpoutConfig__: Used to configure EventHubSpout.
-* __EventHubBoltConfig__: Used to configure EventHubBolt.
-
-### Example spout usage
-
-SCP.NET provides methods for adding an EventHubSpout to your topology. These methods make it easier to add a spout than using the generic methods for adding a Java component. The following example demonstrates how to create a spout by using the __SetEventHubSpout__ and **EventHubSpoutConfig** methods provided by SCP.NET:
-
-```csharp
- topologyBuilder.SetEventHubSpout(
- "EventHubSpout",
- new EventHubSpoutConfig(
- ConfigurationManager.AppSettings["EventHubSharedAccessKeyName"],
- ConfigurationManager.AppSettings["EventHubSharedAccessKey"],
- ConfigurationManager.AppSettings["EventHubNamespace"],
- ConfigurationManager.AppSettings["EventHubEntityPath"],
- eventHubPartitions),
- eventHubPartitions);
-```
-
-The previous example creates a new spout component named __EventHubSpout__, and configures it to communicate with an event hub. The parallelism hint for the component is set to the number of partitions in the event hub. This setting allows Storm to create an instance of the component for each partition.
-
-### Example bolt usage
-
-Use the **JavaComponmentConstructor** method to create an instance of the bolt. The following example demonstrates how to create and configure a new instance of the **EventHubBolt**:
-
-```csharp
-// Java construcvtor for the Event Hub Bolt
-JavaComponentConstructor constructor = JavaComponentConstructor.CreateFromClojureExpr(
- String.Format(@"(org.apache.storm.eventhubs.bolt.EventHubBolt. (org.apache.storm.eventhubs.bolt.EventHubBoltConfig. " +
- @"""{0}"" ""{1}"" ""{2}"" ""{3}"" ""{4}"" {5}))",
- ConfigurationManager.AppSettings["EventHubPolicyName"],
- ConfigurationManager.AppSettings["EventHubPolicyKey"],
- ConfigurationManager.AppSettings["EventHubNamespace"],
- "servicebus.windows.net",
- ConfigurationManager.AppSettings["EventHubName"],
- "true"));
-
-// Set the bolt to subscribe to data from the spout
-topologyBuilder.SetJavaBolt(
- "eventhubbolt",
- constructor,
- partitionCount)
- .shuffleGrouping("Spout");
-```
-
-> [!NOTE]
-> This example uses a Clojure expression passed as a string, instead of using **JavaComponentConstructor** to create an **EventHubBoltConfig**, as the spout example did. Either method works. Use the method that feels best to you.
-
-## Download the completed project
-
-You can download a complete version of the project created in this article from [GitHub](https://github.com/Azure-Samples/hdinsight-dotnet-java-storm-eventhub). However, you still need to provide configuration settings by following the steps in this article.
-
-### Prerequisites
-
-* An Apache Storm cluster on HDInsight. See [Create Apache Hadoop clusters using the Azure portal](../hdinsight-hadoop-create-linux-clusters-portal.md) and select **Storm** for **Cluster type**.
-
-* An [Azure event hub](../../event-hubs/event-hubs-create.md).
-
-* The [Azure .NET SDK](https://azure.microsoft.com/downloads/).
-
-* The [HDInsight tools for Visual Studio](../hadoop/apache-hadoop-visual-studio-tools-get-started.md).
-
-* Java JDK 1.8 or later on your development environment. JDK downloads are available from [Oracle](/azure/developer/java/fundamentals/java-support-on-azure).
-
- * The **JAVA_HOME** environment variable must point to the directory that contains Java.
- * The **%JAVA_HOME%/bin** directory must be in the path.
-
-## Download the Event Hubs components
-
-Download the Event Hubs spout and bolt component from [https://github.com/hdinsight/mvn-repo/raw/master/org/apache/storm/storm-eventhubs/1.1.0.1/storm-eventhubs-1.1.0.1.jar](https://github.com/hdinsight/mvn-repo/raw/master/org/apache/storm/storm-eventhubs/1.1.0.1/storm-eventhubs-1.1.0.1.jar).
-
-Create a directory named `eventhubspout`, and save the file into the directory.
-
-## Configure Event Hubs
-
-Event Hubs is the data source for this example. Use the information in the "Create an event hub" section of [Get started with Event Hubs](../../event-hubs/event-hubs-create.md).
-
-1. After the event hub has been created, view the **EventHub** settings in the Azure portal, and select **Shared access policies**. Select **+ Add** to create the following policies:
-
- | Name | Permissions |
- | | |
- | writer |Send |
- | reader |Listen |
-
- :::image type="content" source="./media/apache-storm-develop-csharp-event-hub-topology/share-access-policies.png" alt-text="Screenshot of Share access policies window" border="true":::
-
-2. Select the **reader** and **writer** policies. Copy and save the primary key value for both policies, as these values are used later.
-
-## Configure the EventHubWriter
-
-1. If you haven't already installed the latest version of the HDInsight tools for Visual Studio, see [Get started using HDInsight tools for Visual Studio](../hadoop/apache-hadoop-visual-studio-tools-get-started.md).
-
-2. Download the solution from [eventhub-storm-hybrid](https://github.com/Azure-Samples/hdinsight-dotnet-java-storm-eventhub).
-
-3. Open **EventHubExample.sln**. In the **EventHubWriter** project, open the **App.config** file. Use the information from the event hub that you configured earlier to fill in the value for the following keys:
-
- | Key | Value |
- | | |
- | EventHubPolicyName |writer (If you used a different name for the policy with *Send* permission, use it instead.) |
- | EventHubPolicyKey |The key for the writer policy. |
- | EventHubNamespace |The namespace that contains your event hub. |
- | EventHubName |Your event hub name. |
- | EventHubPartitionCount |The number of partitions in your event hub. |
-
-4. Save and close the **App.config** file.
-
-## Configure the EventHubReader
-
-1. Open the **EventHubReader** project.
-
-2. Open the **App.config** file for the **EventHubReader**. Use the information from the event hub that you configured earlier to fill in the value for the following keys:
-
- | Key | Value |
- | | |
- | EventHubPolicyName |reader (If you used a different name for the policy with *listen* permission, use it instead.) |
- | EventHubPolicyKey |The key for the reader policy. |
- | EventHubNamespace |The namespace that contains your event hub. |
- | EventHubName |Your event hub name. |
- | EventHubPartitionCount |The number of partitions in your event hub. |
-
-3. Save and close the **App.config** file.
-
-## Deploy the topologies
-
-1. From **Solution Explorer**, right-click the **EventHubReader** project, and select **Submit to Storm on HDInsight**.
-
- :::image type="content" source="./media/apache-storm-develop-csharp-event-hub-topology/submit-to-apache-storm.png" alt-text="Screenshot of Solution Explorer, with Submit to Storm on HDInsight highlighted" border="true":::
-
-2. On the **Submit Topology** dialog box, select your **Storm Cluster**. Expand **Additional Configurations**, select **Java File Paths**, select **...**, and select the directory that contains the JAR file that you downloaded earlier. Finally, click **Submit**.
-
- :::image type="content" source="./media/apache-storm-develop-csharp-event-hub-topology/submit-storm-topology.png" alt-text="Screenshot of Submit Topology dialog box" border="true":::
-
-3. When the topology has been submitted, the **Storm Topologies Viewer** appears. To view information about the topology, select the **EventHubReader** topology in the left pane.
-
- :::image type="content" source="./media/apache-storm-develop-csharp-event-hub-topology/storm-topology-viewer.png" alt-text="Screenshot of Storm Topologies Viewer" border="true":::
-
-4. From **Solution Explorer**, right-click the **EventHubWriter** project, and select **Submit to Storm on HDInsight**.
-
-5. On the **Submit Topology** dialog box, select your **Storm Cluster**. Expand **Additional Configurations**, select **Java File Paths**, select **...**, and select the directory that contains the JAR file you downloaded earlier. Finally, click **Submit**.
-
-6. When the topology has been submitted, refresh the topology list in the **Storm Topologies Viewer** to verify that both topologies are running on the cluster.
-
-7. In **Storm Topologies Viewer**, select the **EventHubReader** topology.
-
-8. To open the component summary for the bolt, double-click the **LogBolt** component in the diagram.
-
-9. In the **Executors** section, select one of the links in the **Port** column. This displays information logged by the component. The logged information is similar to the following text:
-
- ```output
- 2017-03-02 14:51:29.255 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,255 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1830978598,"deviceId":"8566ccbc-034d-45db-883d-d8a31f34068e"}
- 2017-03-02 14:51:29.283 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,283 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1756413275,"deviceId":"647a5eff-823d-482f-a8b4-b95b35ae570b"}
- 2017-03-02 14:51:29.313 m.s.p.TaskHost [INFO] Received C# STDOUT: 2017-03-02 14:51:29,312 [1] INFO EventHubReader_LogBolt [(null)] - Received data: {"deviceValue":1108478910,"deviceId":"206a68fa-8264-4d61-9100-bfdb68ee8f0a"}
- ```
-
-## Stop the topologies
-
-To stop the topologies, select each topology in the **Storm Topology Viewer**, then click **Kill**.
--
-## Delete your cluster
--
-## Next steps
-
-In this document, you've learned how to use the Java Event Hubs spout and bolt from a C# topology to work with data in Azure Event Hubs. To learn more about creating C# topologies, see the following:
-
-* [Develop C# topologies for Apache Storm on HDInsight using Visual Studio](apache-storm-develop-csharp-visual-studio-topology.md)
-* [SCP programming guide](apache-storm-scp-programming-guide.md)
-* [Example topologies for Apache Storm on HDInsight](apache-storm-example-topology.md)
hdinsight Apache Storm Develop Csharp Visual Studio Topology https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-develop-csharp-visual-studio-topology.md
- Title: Apache Storm topologies with Visual Studio and C# - Azure HDInsight
-description: Learn how to create Storm topologies in C#. Create a word count topology in Visual Studio by using the Hadoop tools for Visual Studio.
--- Previously updated : 06/22/2022---
-# Develop C# topologies for Apache Storm by using the Data Lake tools for Visual Studio
-
-Learn how to create a C# Apache Storm topology by using the Azure Data Lake (Apache Hadoop) tools for Visual Studio. This document walks through the process of creating a Storm project in Visual Studio, testing it locally, and deploying it to an Apache Storm on Azure HDInsight cluster.
-
-You also learn how to create hybrid topologies that use C# and Java components.
-
-C# topologies use .NET 4.5, and use Mono to run on the HDInsight cluster. For information about potential incompatibilities, see [Mono compatibility](https://www.mono-project.com/docs/about-mono/compatibility/). To use a C# topology, you must update the `Microsoft.SCP.Net.SDK` NuGet package used by your project to version 0.10.0.6 or later. The version of the package must also match the major version of Storm installed on HDInsight.
-
-| HDInsight version | Apache Storm version | SCP.NET version | Default Mono version |
-|:--:|:-:|::|:--:|
-| 3.4 | 0.10.0.x | 0.10.0.x | 3.2.8 |
-| 3.5 | 1.0.2.x | 1.0.0.x | 4.2.1 |
-| 3.6 | 1.1.0.x | 1.0.0.x | 4.2.8 |
-
-## Prerequisite
-
-An Apache Storm cluster on HDInsight. See [Create Apache Hadoop clusters using the Azure portal](../hdinsight-hadoop-create-linux-clusters-portal.md) and select **Storm** for **Cluster type**.
-
-## Install Visual Studio
-
-You can develop C# topologies with SCP.NET by using [Visual Studio](https://visualstudio.microsoft.com/downloads/). The instructions here use Visual Studio 2019, but you may also use earlier versions of Visual Studio.
-
-## Install Data Lake tools for Visual Studio
-
-To install Data Lake tools for Visual Studio, follow the steps in [Get started using Data Lake tools for Visual Studio](../hadoop/apache-hadoop-visual-studio-tools-get-started.md#install-data-lake-tools-for-visual-studio).
-
-## Install Java
-
-When you submit a Storm topology from Visual Studio, SCP.NET generates a zip file that contains the topology and dependencies. Java is used to create these zip files, because it uses a format that is more compatible with Linux-based clusters.
-
-1. Install the Java Developer Kit (JDK) 7 or later on your development environment. You can get the Oracle JDK from [Oracle](https://openjdk.java.net/). You can also use [other Java distributions](/java/azure/jdk/).
-
-2. Set the `JAVA_HOME` environment variable to the directory that contains Java.
-
-3. Set the `PATH` environment variable to include the `%JAVA_HOME%\bin` directory.
-
-You can build and run the following C# console application to verify that Java and the JDK are correctly installed:
-
-```csharp
-using System;
-using System.IO;
-namespace ConsoleApplication2
-{
-   class Program
-   {
-       static void Main(string[] args)
-       {
-           string javaHome = Environment.GetEnvironmentVariable("JAVA_HOME");
-           if (!string.IsNullOrEmpty(javaHome))
-           {
-               string jarExe = Path.Combine(javaHome + @"\bin", "jar.exe");
-               if (File.Exists(jarExe))
-               {
-                   Console.WriteLine("JAVA Is Installed properly");
-                    return;
-               }
-               else
-               {
-                   Console.WriteLine("A valid JAVA JDK is not found. Looks like JRE is installed instead of JDK.");
-               }
-           }
-           else
-           {
-             Console.WriteLine("A valid JAVA JDK is not found. JAVA_HOME environment variable is not set.");
-           }
-       }  
-   }
-}
-```
-
-## Apache Storm templates
-
-The Data Lake tools for Visual Studio provide the following templates:
-
-| Project type | Demonstrates |
-| | |
-| Storm Application |An empty Storm topology project. |
-| Storm Azure SQL Writer Sample |How to write to Azure SQL Database. |
-| Storm Azure Cosmos DB Reader Sample |How to read from Azure Cosmos DB. |
-| Storm Azure Cosmos DB Writer Sample |How to write to Azure Cosmos DB. |
-| Storm EventHub Reader Sample |How to read from Azure Event Hubs. |
-| Storm EventHub Writer Sample |How to write to Azure Event Hubs. |
-| Storm HBase Reader Sample |How to read from HBase on HDInsight clusters. |
-| Storm HBase Writer Sample |How to write to HBase on HDInsight clusters. |
-| Storm Hybrid Sample |How to use a Java component. |
-| Storm Sample |A basic word count topology. |
-
-> [!WARNING]
-> Not all templates work with Linux-based HDInsight. NuGet packages used by the templates may not be compatible with Mono. To identify potential problems, see [Mono compatibility](https://www.mono-project.com/docs/about-mono/compatibility/) and use the [.NET Portability Analyzer](../hdinsight-hadoop-migrate-dotnet-to-linux.md#automated-portability-analysis).
-
-In the steps in this document, you use the basic Storm Application project type to create a topology.
-
-### Apache HBase templates
-
-The HBase reader and writer templates use the HBase REST API, not the HBase Java API, to communicate with an HBase on HDInsight cluster.
-
-### EventHub templates
-
-> [!IMPORTANT]
-> The Java-based EventHub spout component included with the EventHub Reader template may not work with Storm on HDInsight version 3.5 or later. An updated version of this component is available at [GitHub](https://github.com/hdinsight/hdinsight-storm-examples/tree/master/HDI3.5/lib).
-
-For an example topology that uses this component and works with Storm on HDInsight 3.5, see [GitHub](https://github.com/Azure-Samples/hdinsight-dotnet-java-storm-eventhub).
-
-## Create a C# topology
-
-To create a C# topology project in Visual Studio:
-
-1. Open Visual Studio.
-
-1. In the **Start** window, select **Create a new project**.
-
-1. In the **Create a new project** window, scroll to and pick **Storm Application**, then select **Next**.
-
-1. In the **Configure your new project** window, enter a **Project name** of *WordCount*, go to or create a **Location** directory path for the project, and then select **Create**.
-
- :::image type="content" source="./media/apache-storm-develop-csharp-visual-studio-topology/apache-storm-new-project.png" alt-text="Storm application, Configure your new project dialog box, Visual Studio" border="true":::
-
-After you've created the project, you should have the following files:
-
-* *Program.cs*: The topology definition for your project. A default topology that consists of one spout and one bolt is created by default.
-
-* *Spout.cs*: An example spout that emits random numbers.
-
-* *Bolt.cs*: An example bolt that keeps a count of numbers emitted by the spout.
-
-When you create the project, NuGet downloads the latest [SCP.NET package](https://www.nuget.org/packages/Microsoft.SCP.Net.SDK/).
-
-### Implement the spout
-
-Next, add the code for the spout, which is used to read data in a topology from an external source. This spout randomly emits a sentence into the topology.
-
-1. Open *Spout.cs*. The main components for a spout are:
-
- * `NextTuple`: Called by Storm when the spout is allowed to emit new tuples.
-
- * `Ack` (transactional topology only): Handles acknowledgments started by other components in the topology for tuples sent from the spout. Acknowledging a tuple lets the spout know that it was processed successfully by downstream components.
-
- * `Fail` (transactional topology only): Handles tuples that are fail-processing other components in the topology. Implementing a `Fail` method allows you to re-emit the tuple so that it can be processed again.
-
-2. Replace the contents of the `Spout` class with the following text:
-
- ```csharp
- private Context ctx;
- private Random r = new Random();
- string[] sentences = new string[] {
- "the cow jumped over the moon",
- "an apple a day keeps the doctor away",
- "four score and seven years ago",
- "snow white and the seven dwarfs",
- "i am at two with nature"
- };
-
- public Spout(Context ctx)
- {
- // Set the instance context
- this.ctx = ctx;
-
- Context.Logger.Info("Generator constructor called");
-
- // Declare Output schema
- Dictionary<string, List<Type>> outputSchema = new Dictionary<string, List<Type>>();
- // The schema for the default output stream is
- // a tuple that contains a string field
- outputSchema.Add("default", new List<Type>() { typeof(string) });
- this.ctx.DeclareComponentSchema(new ComponentStreamSchema(null, outputSchema));
- }
-
- // Get an instance of the spout
- public static Spout Get(Context ctx, Dictionary<string, Object> parms)
- {
- return new Spout(ctx);
- }
-
- public void NextTuple(Dictionary<string, Object> parms)
- {
- Context.Logger.Info("NextTuple enter");
- // The sentence to be emitted
- string sentence;
-
- // Get a random sentence
- sentence = sentences[r.Next(0, sentences.Length - 1)];
- Context.Logger.Info("Emit: {0}", sentence);
- // Emit it
- this.ctx.Emit(new Values(sentence));
-
- Context.Logger.Info("NextTuple exit");
- }
-
- public void Ack(long seqId, Dictionary<string, Object> parms)
- {
- // Only used for transactional topologies
- }
-
- public void Fail(long seqId, Dictionary<string, Object> parms)
- {
- // Only used for transactional topologies
- }
- ```
-
-### Implement the bolts
-
-Now create two Storm bolts in this example:
-
-1. Delete the existing *Bolt.cs* file from the project.
-
-2. In **Solution Explorer**, right-click the project, and select **Add** > **New item**. From the list, select **Storm Bolt**, and enter *Splitter.cs* as the name. In the new file's code, change the namespace name to `WordCount`. Then repeat this process to create a second bolt named *Counter.cs*.
-
- * *Splitter.cs*: Implements a bolt that splits sentences into individual words, and emits a new stream of words.
-
- * *Counter.cs*: Implements a bolt that counts each word, and emits a new stream of words and the count for each word.
-
- > [!NOTE]
- > These bolts read and write to streams, but you can also use a bolt to communicate with sources such as a database or service.
-
-3. Open *Splitter.cs*. It has only one method by default: `Execute`. The `Execute` method is called when the bolt receives a tuple for processing. Here, you can read and process incoming tuples, and emit outbound tuples.
-
-4. Replace the contents of the `Splitter` class with the following code:
-
- ```csharp
- private Context ctx;
-
- // Constructor
- public Splitter(Context ctx)
- {
- Context.Logger.Info("Splitter constructor called");
- this.ctx = ctx;
-
- // Declare Input and Output schemas
- Dictionary<string, List<Type>> inputSchema = new Dictionary<string, List<Type>>();
- // Input contains a tuple with a string field (the sentence)
- inputSchema.Add("default", new List<Type>() { typeof(string) });
- Dictionary<string, List<Type>> outputSchema = new Dictionary<string, List<Type>>();
- // Outbound contains a tuple with a string field (the word)
- outputSchema.Add("default", new List<Type>() { typeof(string) });
- this.ctx.DeclareComponentSchema(new ComponentStreamSchema(inputSchema, outputSchema));
- }
-
- // Get a new instance of the bolt
- public static Splitter Get(Context ctx, Dictionary<string, Object> parms)
- {
- return new Splitter(ctx);
- }
-
- // Called when a new tuple is available
- public void Execute(SCPTuple tuple)
- {
- Context.Logger.Info("Execute enter");
-
- // Get the sentence from the tuple
- string sentence = tuple.GetString(0);
- // Split at space characters
- foreach (string word in sentence.Split(' '))
- {
- Context.Logger.Info("Emit: {0}", word);
- //Emit each word
- this.ctx.Emit(new Values(word));
- }
-
- Context.Logger.Info("Execute exit");
- }
- ```
-
-5. Open *Counter.cs*, and replace the class contents with the following code:
-
- ```csharp
- private Context ctx;
-
- // Dictionary for holding words and counts
- private Dictionary<string, int> counts = new Dictionary<string, int>();
-
- // Constructor
- public Counter(Context ctx)
- {
- Context.Logger.Info("Counter constructor called");
- // Set instance context
- this.ctx = ctx;
-
- // Declare Input and Output schemas
- Dictionary<string, List<Type>> inputSchema = new Dictionary<string, List<Type>>();
- // A tuple containing a string field - the word
- inputSchema.Add("default", new List<Type>() { typeof(string) });
-
- Dictionary<string, List<Type>> outputSchema = new Dictionary<string, List<Type>>();
- // A tuple containing a string and integer field - the word and the word count
- outputSchema.Add("default", new List<Type>() { typeof(string), typeof(int) });
- this.ctx.DeclareComponentSchema(new ComponentStreamSchema(inputSchema, outputSchema));
- }
-
- // Get a new instance
- public static Counter Get(Context ctx, Dictionary<string, Object> parms)
- {
- return new Counter(ctx);
- }
-
- // Called when a new tuple is available
- public void Execute(SCPTuple tuple)
- {
- Context.Logger.Info("Execute enter");
-
- // Get the word from the tuple
- string word = tuple.GetString(0);
- // Do we already have an entry for the word in the dictionary?
- // If no, create one with a count of 0
- int count = counts.ContainsKey(word) ? counts[word] : 0;
- // Increment the count
- count++;
- // Update the count in the dictionary
- counts[word] = count;
-
- Context.Logger.Info("Emit: {0}, count: {1}", word, count);
- // Emit the word and count information
- this.ctx.Emit(Constants.DEFAULT_STREAM_ID, new List<SCPTuple> { tuple }, new Values(word, count));
- Context.Logger.Info("Execute exit");
- }
- ```
-
-### Define the topology
-
-Spouts and bolts are arranged in a graph, which defines how the data flows between components. For this topology, the graph is as follows:
--
-The spout emits sentences that are distributed to instances of the Splitter bolt. The Splitter bolt breaks the sentences into words, which are distributed to the Counter bolt.
-
-Because the Counter instance holds the word count locally, you want to make sure that specific words flow to the same Counter bolt instance. Each instance keeps track of specific words. Since the Splitter bolt maintains no state, it really doesn't matter which instance of the splitter receives which sentence.
-
-Open *Program.cs*. The important method is `GetTopologyBuilder`, which is used to define the topology that is submitted to Storm. Replace the contents of `GetTopologyBuilder` with the following code to implement the topology described previously:
-
-```csharp
-// Create a new topology named 'WordCount'
-TopologyBuilder topologyBuilder = new TopologyBuilder(
- "WordCount" + DateTime.Now.ToString("yyyyMMddHHmmss"));
-
-// Add the spout to the topology.
-// Name the component 'sentences'
-// Name the field that is emitted as 'sentence'
-topologyBuilder.SetSpout(
- "sentences",
- Spout.Get,
- new Dictionary<string, List<string>>()
- {
- {Constants.DEFAULT_STREAM_ID, new List<string>(){"sentence"}}
- },
- 1);
-// Add the splitter bolt to the topology.
-// Name the component 'splitter'
-// Name the field that is emitted 'word'
-// Use suffleGrouping to distribute incoming tuples
-// from the 'sentences' spout across instances
-// of the splitter
-topologyBuilder.SetBolt(
- "splitter",
- Splitter.Get,
- new Dictionary<string, List<string>>()
- {
- {Constants.DEFAULT_STREAM_ID, new List<string>(){"word"}}
- },
- 1).shuffleGrouping("sentences");
-
-// Add the counter bolt to the topology.
-// Name the component 'counter'
-// Name the fields that are emitted 'word' and 'count'
-// Use fieldsGrouping to ensure that tuples are routed
-// to counter instances based on the contents of field
-// position 0 (the word). This could also have been
-// List<string>(){"word"}.
-// This ensures that the word 'jumped', for example, will always
-// go to the same instance
-topologyBuilder.SetBolt(
- "counter",
- Counter.Get,
- new Dictionary<string, List<string>>()
- {
- {Constants.DEFAULT_STREAM_ID, new List<string>(){"word", "count"}}
- },
- 1).fieldsGrouping("splitter", new List<int>() { 0 });
-
-// Add topology config
-topologyBuilder.SetTopologyConfig(new Dictionary<string, string>()
-{
- {"topology.kryo.register","[\"[B\"]"}
-});
-
-return topologyBuilder;
-```
-
-## Submit the topology
-
-You're now ready to submit the topology to your HDInsight cluster.
-
-1. Navigate to **View** > **Server Explorer**.
-
-1. Right-click **Azure**, select **Connect to Microsoft Azure Subscription...**, and complete the sign-in process.
-
-1. In **Solution Explorer**, right-click the project, and choose **Submit to Storm on HDInsight**.
-
-1. In the **Submit Topology** dialog box, under the **Storm Cluster** drop-down list, choose your Storm on HDInsight cluster, and then select **Submit**. You can check whether the submission is successful by viewing the **Output** pane.
-
- When the topology has been successfully submitted, the **Storm Topologies View** window for the cluster should appear. Choose the **WordCount** topology from the list to view information about the running topology.
-
- :::image type="content" source="./media/apache-storm-develop-csharp-visual-studio-topology/storm-topology-view.png" alt-text="Storm topology view window, HDInsight cluster, Visual Studio" border="true":::
-
- > [!NOTE]
- > You can also view **Storm Topologies** from **Server Explorer**. Expand **Azure** > **HDInsight**, right-click a Storm on HDInsight cluster, and then choose **View Storm Topologies**.
-
- To view information about the components in the topology, select a component in the diagram.
-
-1. In the **Topology Summary** section, select **Kill** to stop the topology.
-
- > [!NOTE]
- > Storm topologies continue to run until they are deactivated, or the cluster is deleted.
-
-## Transactional topology
-
-The previous topology is non-transactional. The components in the topology don't implement functionality to replaying messages. For an example of a transactional topology, create a project and select **Storm Sample** as the project type.
-
-Transactional topologies implement the following to support replay of data:
-
-* **Metadata caching**: The spout must store metadata about the data emitted, so that the data can be retrieved and emitted again if a failure occurs. Because the data emitted by the sample is small, the raw data for each tuple is stored in a dictionary for replay.
-
-* **Ack**: Each bolt in the topology can call `this.ctx.Ack(tuple)` to acknowledge that it has successfully processed a tuple. When all bolts have acknowledged the tuple, the `Ack` method of the spout is invoked. The `Ack` method allows the spout to remove data that was cached for replay.
-
-* **Fail**: Each bolt can call `this.ctx.Fail(tuple)` to indicate that processing has failed for a tuple. The failure propagates to the `Fail` method of the spout, where the tuple can be replayed by using cached metadata.
-
-* **Sequence ID**: When emitting a tuple, a unique sequence ID can be specified. This value identifies the tuple for replay (`Ack` and `Fail`) processing. For example, the spout in the **Storm Sample** project uses the following method call when emitting data:
-
- `this.ctx.Emit(Constants.DEFAULT_STREAM_ID, new Values(sentence), lastSeqId);`
-
- This code emits a tuple that contains a sentence to the default stream, with the sequence ID value contained in `lastSeqId`. For this example, `lastSeqId` is incremented for every tuple emitted.
-
-As demonstrated in the **Storm Sample** project, whether a component is transactional can be set at runtime, based on configuration.
-
-## Hybrid topology with C# and Java
-
-You can also use Data Lake tools for Visual Studio to create hybrid topologies, where some components are C# and others are Java.
-
-For an example of a hybrid topology, create a project and select **Storm Hybrid Sample**. This sample type demonstrates the following concepts:
-
-* **Java spout** and **C# bolt**: Defined in the `HybridTopology_javaSpout_csharpBolt` class.
-
- A transactional version is defined in the `HybridTopologyTx_javaSpout_csharpBolt` class.
-
-* **C# spout** and **Java bolt**: Defined in the `HybridTopology_csharpSpout_javaBolt` class.
-
- A transactional version is defined in the `HybridTopologyTx_csharpSpout_javaBolt` class.
-
- > [!NOTE]
- > This version also demonstrates how to use Clojure code from a text file as a Java component.
-
-To switch the topology that is used when the project is submitted, move the `[Active(true)]` statement to the topology you want to use, before submitting it to the cluster.
-
-> [!NOTE]
-> All the Java files that are required are provided as part of this project in the *JavaDependency* folder.
-
-Consider the following when you're creating and submitting a hybrid topology:
-
-* Use `JavaComponentConstructor` to create an instance of the Java class for a spout or bolt.
-
-* Use `microsoft.scp.storm.multilang.CustomizedInteropJSONSerializer` to serialize data into or out of Java components from Java objects to JSON.
-
-* When submitting the topology to the server, you must use the **Additional configurations** option to specify the **Java File paths**. The path specified should be the directory that has the JAR files containing your Java classes.
-
-### Azure Event Hubs
-
-SCP.NET version 0.9.4.203 introduces a new class and method specifically for working with the Event Hub spout (a Java spout that reads from Event Hubs). When you create a topology that uses an Event Hub spout (for example, using the **Storm EventHub Reader Sample** template), use the following APIs:
-
-* `EventHubSpoutConfig` class: Creates an object that contains the configuration for the spout component.
-
-* `TopologyBuilder.SetEventHubSpout` method: Adds the Event Hub spout component to the topology.
-
-> [!NOTE]
-> You must still use the `CustomizedInteropJSONSerializer` to serialize data produced by the spout.
-
-## Use ConfigurationManager
-
-Don't use **ConfigurationManager** to retrieve configuration values from bolt and spout components. Doing so can cause a null pointer exception. Instead, pass the configuration for your project into the Storm topology as a key and value pair in the topology context. Each component that relies on configuration values must retrieve them from the context during initialization.
-
-The following code demonstrates how to retrieve these values:
-
-```csharp
-public class MyComponent : ISCPBolt
-{
- // To hold configuration information loaded from context
- Configuration configuration;
- ...
- public MyComponent(Context ctx, Dictionary<string, Object> parms)
- {
- // Save a copy of the context for this component instance
- this.ctx = ctx;
- // If it exists, load the configuration for the component
- if(parms.ContainsKey(Constants.USER_CONFIG))
- {
- this.configuration = parms[Constants.USER_CONFIG]
- as System.Configuration.Configuration;
- }
- // Retrieve the value of "Foo" from configuration
- var foo = this.configuration.AppSettings.Settings["Foo"].Value;
- }
- ...
-}
-```
-
-If you use a `Get` method to return an instance of your component, you must ensure that it passes both the `Context` and `Dictionary<string, Object>` parameters to the constructor. The following example is a basic `Get` method that properly passes these values:
-
-```csharp
-public static MyComponent Get(Context ctx, Dictionary<string, Object> parms)
-{
- return new MyComponent(ctx, parms);
-}
-```
-
-## How to update SCP.NET
-
-Recent releases of SCP.NET support package upgrade through NuGet. When a new update is available, you receive an upgrade notification. To manually check for an upgrade, follow these steps:
-
-1. In **Solution Explorer**, right-click the project, and select **Manage NuGet Packages**.
-
-2. From the package manager, select **Updates**. If an update for the SCP.NET support package is available, it's listed. Select **Update** for the package, and then in the **Preview Changes** dialog box, select **OK** to install it.
-
-> [!IMPORTANT]
-> If your project was created with an earlier version of SCP.NET that did not use NuGet, you must perform the following steps to update to a newer version:
->
-> 1. In **Solution Explorer**, right-click the project, and select **Manage NuGet Packages**.
-> 2. Using the **Search** field, search for, and then add, `Microsoft.SCP.Net.SDK` to the project.
-
-## Troubleshoot common issues with topologies
-
-### Null pointer exceptions
-
-When you're using a C# topology with a Linux-based HDInsight cluster, bolt and spout components that use **ConfigurationManager** to read configuration settings at runtime may return null pointer exceptions.
-
-The configuration for your project is passed into the Storm topology as a key and value pair in the topology context. It can be retrieved from the dictionary object that's passed to your components when they're initialized.
-
-For more information, see the [Use ConfigurationManager](#use-configurationmanager) section of this document.
-
-### System.TypeLoadException
-
-When you're using a C# topology with a Linux-based HDInsight cluster, you may come across the following error:
-
-`System.TypeLoadException: Failure has occurred while loading a type.`
-
-This error occurs when you use a binary that isn't compatible with the version of .NET that Mono supports.
-
-For Linux-based HDInsight clusters, make sure that your project uses binaries compiled for .NET 4.5.
-
-### Test a topology locally
-
-Although it's easy to deploy a topology to a cluster, in some cases, you may need to test a topology locally. Use the following steps to run and test the example topology in this article locally in your development environment.
-
-> [!WARNING]
-> Local testing only works for basic, C#-only topologies. You cannot use local testing for hybrid topologies or topologies that use multiple streams.
-
-1. In **Solution Explorer**, right-click the project, and select **Properties**. In the project properties. Then change the **Output type** to **Console Application**.
-
- :::image type="content" source="./media/apache-storm-develop-csharp-visual-studio-topology/hdi-output-type-window.png" alt-text="HDInsight Storm application, project properties, Output type" border="true":::
-
- > [!NOTE]
- > Remember to change the **Output type** back to **Class Library** before you deploy the topology to a cluster.
-
-1. In **Solution Explorer**, right-click the project, and then select **Add** > **New Item**. Select **Class**, and enter *LocalTest.cs* as the class name. Finally, select **Add**.
-
-1. Open *LocalTest.cs*, and add the following `using` statement at the top:
-
- ```csharp
- using Microsoft.SCP;
- ```
-
-1. Use the following code as the contents of the `LocalTest` class:
-
- ```csharp
- // Drives the topology components
- public void RunTestCase()
- {
- // An empty dictionary for use when creating components
- Dictionary<string, Object> emptyDictionary = new Dictionary<string, object>();
-
- #region Test the spout
- {
- Console.WriteLine("Starting spout");
- // LocalContext is a local-mode context that can be used to initialize
- // components in the development environment.
- LocalContext spoutCtx = LocalContext.Get();
- // Get a new instance of the spout, using the local context
- Spout sentences = Spout.Get(spoutCtx, emptyDictionary);
-
- // Emit 10 tuples
- for (int i = 0; i < 10; i++)
- {
- sentences.NextTuple(emptyDictionary);
- }
- // Use LocalContext to persist the data stream to file
- spoutCtx.WriteMsgQueueToFile("sentences.txt");
- Console.WriteLine("Spout finished");
- }
- #endregion
-
- #region Test the splitter bolt
- {
- Console.WriteLine("Starting splitter bolt");
- // LocalContext is a local-mode context that can be used to initialize
- // components in the development environment.
- LocalContext splitterCtx = LocalContext.Get();
- // Get a new instance of the bolt
- Splitter splitter = Splitter.Get(splitterCtx, emptyDictionary);
-
- // Set the data stream to the data created by the spout
- splitterCtx.ReadFromFileToMsgQueue("sentences.txt");
- // Get a batch of tuples from the stream
- List<SCPTuple> batch = splitterCtx.RecvFromMsgQueue();
- // Process each tuple in the batch
- foreach (SCPTuple tuple in batch)
- {
- splitter.Execute(tuple);
- }
- // Use LocalContext to persist the data stream to file
- splitterCtx.WriteMsgQueueToFile("splitter.txt");
- Console.WriteLine("Splitter bolt finished");
- }
- #endregion
-
- #region Test the counter bolt
- {
- Console.WriteLine("Starting counter bolt");
- // LocalContext is a local-mode context that can be used to initialize
- // components in the development environment.
- LocalContext counterCtx = LocalContext.Get();
- // Get a new instance of the bolt
- Counter counter = Counter.Get(counterCtx, emptyDictionary);
-
- // Set the data stream to the data created by splitter bolt
- counterCtx.ReadFromFileToMsgQueue("splitter.txt");
- // Get a batch of tuples from the stream
- List<SCPTuple> batch = counterCtx.RecvFromMsgQueue();
- // Process each tuple in the batch
- foreach (SCPTuple tuple in batch)
- {
- counter.Execute(tuple);
- }
- // Use LocalContext to persist the data stream to file
- counterCtx.WriteMsgQueueToFile("counter.txt");
- Console.WriteLine("Counter bolt finished");
- }
- #endregion
- }
- ```
-
- Take a moment to read through the code comments. This code uses `LocalContext` to run the components in the development environment. It persists the data stream between components to text files on the local drive.
-
-1. Open *Program.cs*, and add the following code to the `Main` method:
-
- ```csharp
- Console.WriteLine("Starting tests");
- System.Environment.SetEnvironmentVariable("microsoft.scp.logPrefix", "WordCount-LocalTest");
- // Initialize the runtime
- SCPRuntime.Initialize();
-
- //If we are not running under the local context, throw an error
- if (Context.pluginType != SCPPluginType.SCP_NET_LOCAL)
- {
- throw new Exception(string.Format("unexpected pluginType: {0}", Context.pluginType));
- }
- // Create test instance
- LocalTest tests = new LocalTest();
- // Run tests
- tests.RunTestCase();
- Console.WriteLine("Tests finished");
- Console.ReadKey();
- ```
-
-1. Save the changes, and then select **F5** or choose **Debug** > **Start Debugging** to start the project. A console window should appear, and log status as the tests progress. When `Tests finished` appears, select any key to close the window.
-
-1. Use **Windows Explorer** to locate the directory that contains your project. (For example: *C:\\Users\\\<your_user_name>\\source\\repos\\WordCount\\WordCount*.) Then in this directory, open *Bin*, and then select *Debug*. You should see the text files that were produced when the tests ran: *sentences.txt*, *counter.txt*, and *splitter.txt*. Open each text file and inspect the data.
-
- > [!NOTE]
- > String data persists as an array of decimal values in these files. For example, `[[97,103,111]]` in the **splitter.txt** file represents the word *ago*.
-
-> [!NOTE]
-> Be sure to set the **Project type** back to **Class Library** in the project properties before deploying to a Storm on HDInsight cluster.
-
-### Log information
-
-You can easily log information from your topology components by using `Context.Logger`. For example, the following command creates an informational log entry:
-
-`Context.Logger.Info("Component started");`
-
-Logged information can be viewed from the **Hadoop Service Log**, which is found in **Server Explorer**. Expand the entry for your Storm on HDInsight cluster, and then expand **Hadoop Service Log**. Finally, select the log file to view.
-
-> [!NOTE]
-> The logs are stored in the Azure storage account that is used by your cluster. To view the logs in Visual Studio, you must sign in to the Azure subscription that owns the storage account.
-
-### View error information
-
-To view errors that have occurred in a running topology, use the following steps:
-
-1. From **Server Explorer**, right-click the Storm on HDInsight cluster, and select **View Storm Topologies**.
-
- For the **Spout** and **Bolts**, the **Last Error** column contains information on the last error.
-
-2. Select the **Spout ID** or **Bolt ID** for the component that has an error listed. The details page displays additional error information in the **Errors** section at the bottom of the page.
-
-3. To obtain more information, select a **Port** from the **Executors** section of the page, to see the Storm worker log for the last few minutes.
-
-### Errors submitting topologies
-
-If you come across errors submitting a topology to HDInsight, you can find logs for the server-side components that handle topology submission on your HDInsight cluster. To download these logs, use the following command from a command line:
-
-```cmd
-scp sshuser@clustername-ssh.azurehdinsight.net:/var/log/hdinsight-scpwebapi/hdinsight-scpwebapi.out .
-```
-
-Replace *sshuser* with the SSH user account for the cluster. Replace *clustername* with the name of the HDInsight cluster. For more information on using `scp` and `ssh` with HDInsight, see [Use SSH with HDInsight](../hdinsight-hadoop-linux-use-ssh-unix.md).
-
-Submissions can fail for multiple reasons:
-
-* The JDK isn't installed or isn't in the path.
-* Required Java dependencies aren't included in the submission.
-* Dependencies are incompatible.
-* Topology names are duplicated.
-
-If the *hdinsight-scpwebapi.out* log file contains a `FileNotFoundException`, the exception might be caused by the following conditions:
-
-* The JDK isn't in the path on the development environment. Verify that the JDK is installed in the development environment, and that `%JAVA_HOME%/bin` is in the path.
-* You're missing a Java dependency. Make sure you're including any required .jar files as part of the submission.
-
-## Next steps
-
-For an example of processing data from Event Hubs, see [Process events from Azure Event Hubs with Storm on HDInsight](apache-storm-develop-csharp-event-hub-topology.md).
-
-For an example of a C# topology that splits stream data into multiple streams, see [C# Storm example](https://github.com/Blackmist/csharp-storm-example).
-
-To discover more information about creating C# topologies, see [GitHub](https://github.com/hdinsight/hdinsight-storm-examples/blob/master/SCPNet-GettingStarted.md).
-
-For more ways to work with HDInsight and more Storm on HDInsight samples, see the following documents:
-
-**Microsoft SCP.NET**
-
-* [SCP programming guide for Apache Storm in Azure HDInsight](apache-storm-scp-programming-guide.md)
-
-**Apache Storm on HDInsight**
-
-* [Deploy and manage Apache Storm topologies on Azure HDInsight](apache-storm-deploy-monitor-topology-linux.md)
-* [Example Apache Storm topologies in Azure HDInsight](apache-storm-example-topology.md)
-
-**Apache Hadoop on HDInsight**
-
-* [What is Apache Hive and HiveQL on Azure HDInsight?](../hadoop/hdinsight-use-hive.md)
-* [Use MapReduce in Apache Hadoop on HDInsight](../hadoop/hdinsight-use-mapreduce.md)
-
-**Apache HBase on HDInsight**
-
-* [Use Apache HBase in Azure HDInsight](../hbase/apache-hbase-tutorial-get-started-linux.md)
hdinsight Apache Storm Develop Java Topology https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-develop-java-topology.md
- Title: Apache Storm example Java topology - Azure HDInsight
-description: Learn how to create Apache Storm topologies in Java by creating an example word count topology.
--- Previously updated : 04/28/2022--
-# Create an Apache Storm topology in Java
-
-Learn how to create a Java-based topology for Apache Storm. You create a Storm topology that implements a word-count application. You use Apache Maven to build and package the project. Then, you learn how to define the topology using the Apache Storm Flux framework.
-
-After completing the steps in this document, you can deploy the topology to Apache Storm on HDInsight.
-
-> [!NOTE]
-> A completed version of the Storm topology examples created in this document is available at [https://github.com/Azure-Samples/hdinsight-java-storm-wordcount](https://github.com/Azure-Samples/hdinsight-java-storm-wordcount).
-
-## Prerequisites
-
-* [Java Developer Kit (JDK) version 8](/azure/developer/java/fundamentals/java-support-on-azure)
-
-* [Apache Maven](https://maven.apache.org/download.cgi) properly [installed](https://maven.apache.org/install.html) according to Apache. Maven is a project build system for Java projects.
-
-## Test environment
-
-The environment used for this article was a computer running Windows 10. The commands were executed in a command prompt, and the various files were edited with Notepad.
-
-From a command prompt, enter the commands below to create a working environment:
-
-```cmd
-mkdir C:\HDI
-cd C:\HDI
-```
-
-## Create a Maven project
-
-Enter the following command to create a Maven project named **WordCount**:
-
-```cmd
-mvn archetype:generate -DarchetypeArtifactId=maven-archetype-quickstart -DgroupId=com.microsoft.example -DartifactId=WordCount -DinteractiveMode=false
-
-cd WordCount
-mkdir resources
-```
-
-This command creates a directory named `WordCount` at the current location, which contains a basic Maven project. The second command changes the present working directory to `WordCount`. The third command creates a new directory, `resources`, which will be used later. The `WordCount` directory contains the following items:
-
-* `pom.xml`: Contains settings for the Maven project.
-* `src\main\java\com\microsoft\example`: Contains your application code.
-* `src\test\java\com\microsoft\example`: Contains tests for your application.
-
-### Remove the generated example code
-
-Delete the generated test and application files `AppTest.java`, and `App.java` by entering the commands below:
-
-```cmd
-DEL src\main\java\com\microsoft\example\App.java
-DEL src\test\java\com\microsoft\example\AppTest.java
-```
-
-## Add Maven repositories
-
-HDInsight is based on the Hortonworks Data Platform (HDP), so we recommend using the Hortonworks repository to download dependencies for your Apache Storm projects.
-
-Open `pom.xml` by entering the command below:
-
-```cmd
-notepad pom.xml
-```
-
-Then add the following XML after the `<url>https://maven.apache.org</url>` line:
-
-```xml
-<repositories>
- <repository>
- <releases>
- <enabled>true</enabled>
- <updatePolicy>always</updatePolicy>
- <checksumPolicy>warn</checksumPolicy>
- </releases>
- <snapshots>
- <enabled>false</enabled>
- <updatePolicy>never</updatePolicy>
- <checksumPolicy>fail</checksumPolicy>
- </snapshots>
- <id>HDPReleases</id>
- <name>HDP Releases</name>
- <url>https://repo.hortonworks.com/content/repositories/releases/</url>
- <layout>default</layout>
- </repository>
- <repository>
- <releases>
- <enabled>true</enabled>
- <updatePolicy>always</updatePolicy>
- <checksumPolicy>warn</checksumPolicy>
- </releases>
- <snapshots>
- <enabled>false</enabled>
- <updatePolicy>never</updatePolicy>
- <checksumPolicy>fail</checksumPolicy>
- </snapshots>
- <id>HDPJetty</id>
- <name>Hadoop Jetty</name>
- <url>https://repo.hortonworks.com/content/repositories/jetty-hadoop/</url>
- <layout>default</layout>
- </repository>
-</repositories>
-```
-
-## Add properties
-
-Maven allows you to define project-level values called properties. In `pom.xml`, add the following text after the `</repositories>` line:
-
-```xml
-<properties>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <!--
- This is a version of Storm from the Hortonworks repository that is compatible with HDInsight 3.6.
- -->
- <storm.version>1.1.0.2.6.1.9-1</storm.version>
-</properties>
-```
-
-You can now use this value in other sections of the `pom.xml`. For example, when specifying the version of Storm components, you can use `${storm.version}` instead of hard coding a value.
-
-## Add dependencies
-
-Add a dependency for Storm components. In `pom.xml`, add the following text in the `<dependencies>` section:
-
-```xml
-<dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-core</artifactId>
- <version>${storm.version}</version>
- <!-- keep storm out of the jar-with-dependencies -->
- <scope>provided</scope>
-</dependency>
-```
-
-At compile time, Maven uses this information to look up `storm-core` in the Maven repository. It first looks in the repository on your local computer. If the files aren't there, Maven downloads them from the public Maven repository and stores them in the local repository.
-
-> [!NOTE]
-> Notice the `<scope>provided</scope>` line in this section. This setting tells Maven to exclude **storm-core** from any JAR files that are created, because it is provided by the system.
-
-## Build configuration
-
-Maven plug-ins allow you to customize the build stages of the project. For example, how the project is compiled or how to package it into a JAR file. In `pom.xml`, add the following text directly above the `</project>` line.
-
-```xml
-<build>
- <plugins>
- </plugins>
- <resources>
- </resources>
-</build>
-```
-
-This section is used to add plug-ins, resources, and other build configuration options. For a full reference of the `pom.xml` file, see [https://maven.apache.org/pom.html](https://maven.apache.org/pom.html).
-
-### Add plug-ins
-
-* **Exec Maven Plugin**
-
- For Apache Storm topologies implemented in Java, the [Exec Maven Plugin](https://www.mojohaus.org/exec-maven-plugin/) is useful because it allows you to easily run the topology locally in your development environment. Add the following to the `<plugins>` section of the `pom.xml` file to include the Exec Maven plugin:
-
- ```xml
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>exec-maven-plugin</artifactId>
- <version>1.6.0</version>
- <executions>
- <execution>
- <goals>
- <goal>exec</goal>
- </goals>
- </execution>
- </executions>
- <configuration>
- <executable>java</executable>
- <includeProjectDependencies>true</includeProjectDependencies>
- <includePluginDependencies>false</includePluginDependencies>
- <classpathScope>compile</classpathScope>
- <mainClass>${storm.topology}</mainClass>
- <cleanupDaemonThreads>false</cleanupDaemonThreads>
- </configuration>
- </plugin>
- ```
-
-* **Apache Maven Compiler Plugin**
-
- Another useful plug-in is the [`Apache Maven Compiler Plugin`](https://maven.apache.org/plugins/maven-compiler-plugin/), which is used to change compilation options. Change the Java version that Maven uses for the source and target for your application.
-
- * For HDInsight __3.4 or earlier__, set the source and target Java version to __1.7__.
-
- * For HDInsight __3.5__, set the source and target Java version to __1.8__.
-
- Add the following text in the `<plugins>` section of the `pom.xml` file to include the Apache Maven Compiler plugin. This example specifies 1.8, so the target HDInsight version is 3.5.
-
- ```xml
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <version>3.8.1</version>
- <configuration>
- <source>1.8</source>
- <target>1.8</target>
- </configuration>
- </plugin>
- ```
-
-### Configure resources
-
-The resources section allows you to include non-code resources such as configuration files needed by components in the topology. For this example, add the following text in the `<resources>` section of the `pom.xml` file. Then save and close the file.
-
-```xml
-<resource>
- <directory>${basedir}/resources</directory>
- <filtering>false</filtering>
- <includes>
- <include>log4j2.xml</include>
- </includes>
-</resource>
-```
-
-This example adds the resources directory in the root of the project (`${basedir}`) as a location that contains resources, and includes the file named `log4j2.xml`. This file is used to configure what information is logged by the topology.
-
-## Create the topology
-
-A Java-based Apache Storm topology consists of three components that you must author (or reference) as a dependency.
-
-* **Spouts**: Reads data from external sources and emits streams of data into the topology.
-
-* **Bolts**: Does processing on streams emitted by spouts or other bolts, and emits one or more streams.
-
-* **Topology**: Defines how the spouts and bolts are arranged, and provides the entry point for the topology.
-
-### Create the spout
-
-To reduce requirements for setting up external data sources, the following spout simply emits random sentences. It's a modified version of a spout that is provided with the [Storm-Starter examples](https://github.com/apache/storm/blob/0.10.x-branch/examples/storm-starter/src/jvm/storm/starter). Although this topology uses one spout, others may have several that feed data from different sources into the topology`.`
-
-Enter the command below to create and open a new file `RandomSentenceSpout.java`:
-
-```cmd
-notepad src\main\java\com\microsoft\example\RandomSentenceSpout.java
-```
-
-Then copy and paste the Java code below into the new file. Then close the file.
-
-```java
-package com.microsoft.example;
-
-import org.apache.storm.spout.SpoutOutputCollector;
-import org.apache.storm.task.TopologyContext;
-import org.apache.storm.topology.OutputFieldsDeclarer;
-import org.apache.storm.topology.base.BaseRichSpout;
-import org.apache.storm.tuple.Fields;
-import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
-
-import java.util.Map;
-import java.util.Random;
-
-//This spout randomly emits sentences
-public class RandomSentenceSpout extends BaseRichSpout {
- //Collector used to emit output
- SpoutOutputCollector _collector;
- //Used to generate a random number
- Random _rand;
-
- //Open is called when an instance of the class is created
- @Override
- public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
- //Set the instance collector to the one passed in
- _collector = collector;
- //For randomness
- _rand = new Random();
- }
-
- //Emit data to the stream
- @Override
- public void nextTuple() {
- //Sleep for a bit
- Utils.sleep(100);
- //The sentences that are randomly emitted
- String[] sentences = new String[]{ "the cow jumped over the moon", "an apple a day keeps the doctor away",
- "four score and seven years ago", "snow white and the seven dwarfs", "i am at two with nature" };
- //Randomly pick a sentence
- String sentence = sentences[_rand.nextInt(sentences.length)];
- //Emit the sentence
- _collector.emit(new Values(sentence));
- }
-
- //Ack is not implemented since this is a basic example
- @Override
- public void ack(Object id) {
- }
-
- //Fail is not implemented since this is a basic example
- @Override
- public void fail(Object id) {
- }
-
- //Declare the output fields. In this case, an sentence
- @Override
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- declarer.declare(new Fields("sentence"));
- }
-}
-```
-
-> [!NOTE]
-> For an example of a spout that reads from an external data source, see one of the following examples:
->
-> * [TwitterSampleSPout](https://github.com/apache/storm/blob/0.10.x-branch/examples/storm-starter/src/jvm/storm/starter/spout/TwitterSampleSpout.java): An example spout that reads from Twitter.
-> * [Storm-Kafka](https://github.com/apache/storm/tree/0.10.x-branch/external/storm-kafka): A spout that reads from Kafka.
-
-### Create the bolts
-
-Bolts handle the data processing. Bolts can do anything, for example, computation, persistence, or talking to external components. This topology uses two bolts:
-
-* **SplitSentence**: Splits the sentences emitted by **RandomSentenceSpout** into individual words.
-
-* **WordCount**: Counts how many times each word has occurred.
-
-#### SplitSentence
-
-Enter the command below to create and open a new file `SplitSentence.java`:
-
-```cmd
-notepad src\main\java\com\microsoft\example\SplitSentence.java
-```
-
-Then copy and paste the Java code below into the new file. Then close the file.
-
-```java
-package com.microsoft.example;
-
-import java.text.BreakIterator;
-
-import org.apache.storm.topology.BasicOutputCollector;
-import org.apache.storm.topology.OutputFieldsDeclarer;
-import org.apache.storm.topology.base.BaseBasicBolt;
-import org.apache.storm.tuple.Fields;
-import org.apache.storm.tuple.Tuple;
-import org.apache.storm.tuple.Values;
-
-//There are a variety of bolt types. In this case, use BaseBasicBolt
-public class SplitSentence extends BaseBasicBolt {
-
- //Execute is called to process tuples
- @Override
- public void execute(Tuple tuple, BasicOutputCollector collector) {
- //Get the sentence content from the tuple
- String sentence = tuple.getString(0);
- //An iterator to get each word
- BreakIterator boundary=BreakIterator.getWordInstance();
- //Give the iterator the sentence
- boundary.setText(sentence);
- //Find the beginning first word
- int start=boundary.first();
- //Iterate over each word and emit it to the output stream
- for (int end=boundary.next(); end != BreakIterator.DONE; start=end, end=boundary.next()) {
- //get the word
- String word=sentence.substring(start,end);
- //If a word is whitespace characters, replace it with empty
- word=word.replaceAll("\\s+","");
- //if it's an actual word, emit it
- if (!word.equals("")) {
- collector.emit(new Values(word));
- }
- }
- }
-
- //Declare that emitted tuples contain a word field
- @Override
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- declarer.declare(new Fields("word"));
- }
-}
-```
-
-#### WordCount
-
-Enter the command below to create and open a new file `WordCount.java`:
-
-```cmd
-notepad src\main\java\com\microsoft\example\WordCount.java
-```
-
-Then copy and paste the Java code below into the new file. Then close the file.
-
-```java
-package com.microsoft.example;
-
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Iterator;
-
-import org.apache.storm.Constants;
-import org.apache.storm.topology.BasicOutputCollector;
-import org.apache.storm.topology.OutputFieldsDeclarer;
-import org.apache.storm.topology.base.BaseBasicBolt;
-import org.apache.storm.tuple.Fields;
-import org.apache.storm.tuple.Tuple;
-import org.apache.storm.tuple.Values;
-import org.apache.storm.Config;
-
-// For logging
-import org.apache.logging.log4j.Logger;
-import org.apache.logging.log4j.LogManager;
-
-//There are a variety of bolt types. In this case, use BaseBasicBolt
-public class WordCount extends BaseBasicBolt {
- //Create logger for this class
- private static final Logger logger = LogManager.getLogger(WordCount.class);
- //For holding words and counts
- Map<String, Integer> counts = new HashMap<String, Integer>();
- //How often to emit a count of words
- private Integer emitFrequency;
-
- // Default constructor
- public WordCount() {
- emitFrequency=5; // Default to 60 seconds
- }
-
- // Constructor that sets emit frequency
- public WordCount(Integer frequency) {
- emitFrequency=frequency;
- }
-
- //Configure frequency of tick tuples for this bolt
- //This delivers a 'tick' tuple on a specific interval,
- //which is used to trigger certain actions
- @Override
- public Map<String, Object> getComponentConfiguration() {
- Config conf = new Config();
- conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, emitFrequency);
- return conf;
- }
-
- //execute is called to process tuples
- @Override
- public void execute(Tuple tuple, BasicOutputCollector collector) {
- //If it's a tick tuple, emit all words and counts
- if(tuple.getSourceComponent().equals(Constants.SYSTEM_COMPONENT_ID)
- && tuple.getSourceStreamId().equals(Constants.SYSTEM_TICK_STREAM_ID)) {
- for(String word : counts.keySet()) {
- Integer count = counts.get(word);
- collector.emit(new Values(word, count));
- logger.info("Emitting a count of " + count + " for word " + word);
- }
- } else {
- //Get the word contents from the tuple
- String word = tuple.getString(0);
- //Have we counted any already?
- Integer count = counts.get(word);
- if (count == null)
- count = 0;
- //Increment the count and store it
- count++;
- counts.put(word, count);
- }
- }
-
- //Declare that this emits a tuple containing two fields; word and count
- @Override
- public void declareOutputFields(OutputFieldsDeclarer declarer) {
- declarer.declare(new Fields("word", "count"));
- }
-}
-```
-
-### Define the topology
-
-The topology ties the spouts and bolts together into a graph. The graph defines how data flows between the components. It also provides parallelism hints that Storm uses when creating instances of the components within the cluster.
-
-The following image is a basic diagram of the graph of components for this topology.
--
-To implement the topology, enter the command below to create and open a new file `WordCountTopology.java`:
-
-```cmd
-notepad src\main\java\com\microsoft\example\WordCountTopology.java
-```
-
-Then copy and paste the Java code below into the new file. Then close the file.
-
-```java
-package com.microsoft.example;
-
-import org.apache.storm.Config;
-import org.apache.storm.LocalCluster;
-import org.apache.storm.StormSubmitter;
-import org.apache.storm.topology.TopologyBuilder;
-import org.apache.storm.tuple.Fields;
-
-import com.microsoft.example.RandomSentenceSpout;
-
-public class WordCountTopology {
-
- //Entry point for the topology
- public static void main(String[] args) throws Exception {
- //Used to build the topology
- TopologyBuilder builder = new TopologyBuilder();
- //Add the spout, with a name of 'spout'
- //and parallelism hint of 5 executors
- builder.setSpout("spout", new RandomSentenceSpout(), 5);
- //Add the SplitSentence bolt, with a name of 'split'
- //and parallelism hint of 8 executors
- //shufflegrouping subscribes to the spout, and equally distributes
- //tuples (sentences) across instances of the SplitSentence bolt
- builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");
- //Add the counter, with a name of 'count'
- //and parallelism hint of 12 executors
- //fieldsgrouping subscribes to the split bolt, and
- //ensures that the same word is sent to the same instance (group by field 'word')
- builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));
-
- //new configuration
- Config conf = new Config();
- //Set to false to disable debug information when
- // running in production on a cluster
- conf.setDebug(false);
-
- //If there are arguments, we are running on a cluster
- if (args != null && args.length > 0) {
- //parallelism hint to set the number of workers
- conf.setNumWorkers(3);
- //submit the topology
- StormSubmitter.submitTopology(args[0], conf, builder.createTopology());
- }
- //Otherwise, we are running locally
- else {
- //Cap the maximum number of executors that can be spawned
- //for a component to 3
- conf.setMaxTaskParallelism(3);
- //LocalCluster is used to run locally
- LocalCluster cluster = new LocalCluster();
- //submit the topology
- cluster.submitTopology("word-count", conf, builder.createTopology());
- //sleep
- Thread.sleep(10000);
- //shut down the cluster
- cluster.shutdown();
- }
- }
-}
-```
-
-### Configure logging
-
-Storm uses [Apache Log4j 2](https://logging.apache.org/log4j/2.x/) to log information. If you don't configure logging, the topology emits diagnostic information. To control what is logged, create a file named `log4j2.xml` in the `resources` directory by entering the command below:
-
-```cmd
-notepad resources\log4j2.xml
-```
-
-Then copy and paste the XML text below into the new file. Then close the file.
-
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<Configuration>
- <Appenders>
- <Console name="STDOUT" target="SYSTEM_OUT">
- <PatternLayout pattern="%d{HH:mm:ss} [%t] %-5level %logger{36} - %msg%n"/>
- </Console>
- </Appenders>
- <Loggers>
- <Logger name="com.microsoft.example" level="trace" additivity="false">
- <AppenderRef ref="STDOUT"/>
- </Logger>
- <Root level="error">
- <Appender-Ref ref="STDOUT"/>
- </Root>
- </Loggers>
-</Configuration>
-```
-
-This XML configures a new logger for the `com.microsoft.example` class, which includes the components in this example topology. The level is set to trace for this logger, which captures any logging information emitted by components in this topology.
-
-The `<Root level="error">` section configures the root level of logging (everything not in `com.microsoft.example`) to only log error information.
-
-For more information on configuring logging for Log4j 2, see [https://logging.apache.org/log4j/2.x/manual/configuration.html](https://logging.apache.org/log4j/2.x/manual/configuration.html).
-
-> [!NOTE]
-> Storm version 0.10.0 and higher use Log4j 2.x. Older versions of storm used Log4j 1.x, which used a different format for log configuration. For information on the older configuration, see [https://cwiki.apache.org/confluence/display/LOGGINGLOG4J/Log4jXmlFormat](https://cwiki.apache.org/confluence/display/LOGGINGLOG4J/Log4jXmlFormat).
-
-## Test the topology locally
-
-After you save the files, use the following command to test the topology locally.
-
-```cmd
-mvn compile exec:java -Dstorm.topology=com.microsoft.example.WordCountTopology
-```
-
-As it runs, the topology displays startup information. The following text is an example of the word count output:
-
-```output
-17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 56 for word snow
-17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 56 for word white
-17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 112 for word seven
-17:33:27 [Thread-16-count] INFO com.microsoft.example.WordCount - Emitting a count of 195 for word the
-17:33:27 [Thread-30-count] INFO com.microsoft.example.WordCount - Emitting a count of 113 for word and
-17:33:27 [Thread-30-count] INFO com.microsoft.example.WordCount - Emitting a count of 57 for word dwarfs
-17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 57 for word snow
-```
-
-This example log indicates that the word 'and' has been emitted 113 times. The count continues to increase as long as the topology runs. This increase is because the spout continuously emits the same sentences.
-
-There's a 5-second interval between emission of words and counts. The **WordCount** component is configured to only emit information when a tick tuple arrives. It requests that tick tuples are only delivered every five seconds.
-
-## Convert the topology to Flux
-
-[Flux](https://storm.apache.org/releases/2.0.0/flux.html) is a new framework available with Storm 0.10.0 and higher. Flux allows you to separate configuration from implementation. Your components are still defined in Java, but the topology is defined using a YAML file. You can package a default topology definition with your project, or use a standalone file when submitting the topology. When submitting the topology to Storm, use environment variables or configuration files to populate YAML topology definition values.
-
-The YAML file defines the components to use for the topology and the data flow between them. You can include a YAML file as part of the jar file. Or you can use an external YAML file.
-
-For more information on Flux, see [Flux framework (https://storm.apache.org/releases/current/flux.html)](https://storm.apache.org/releases/current/flux.html).
-
-> [!WARNING]
-> Due to a [bug (https://issues.apache.org/jira/browse/STORM-2055)](https://issues.apache.org/jira/browse/STORM-2055) with Storm 1.0.1, you may need to install a [Storm development environment](https://storm.apache.org/releases/current/Setting-up-development-environment.html) to run Flux topologies locally.
-
-1. Previously, `WordCountTopology.java` defined the topology, but isn't needed with Flux. Delete the file with the following command:
-
- ```cmd
- DEL src\main\java\com\microsoft\example\WordCountTopology.java
- ```
-
-1. Enter the command below to create and open a new file `topology.yaml`:
-
- ```cmd
- notepad resources\topology.yaml
- ```
-
- Then copy and paste the text below into the new file. Then close the file.
-
- ```yaml
- name: "wordcount" # friendly name for the topology
-
- config: # Topology configuration
- topology.workers: 1 # Hint for the number of workers to create
-
- spouts: # Spout definitions
- - id: "sentence-spout"
- className: "com.microsoft.example.RandomSentenceSpout"
- parallelism: 1 # parallelism hint
-
- bolts: # Bolt definitions
- - id: "splitter-bolt"
- className: "com.microsoft.example.SplitSentence"
- parallelism: 1
-
- - id: "counter-bolt"
- className: "com.microsoft.example.WordCount"
- constructorArgs:
- - 10
- parallelism: 1
-
- streams: # Stream definitions
- - name: "Spout --> Splitter" # name isn't used (placeholder for logging, UI, etc.)
- from: "sentence-spout" # The stream emitter
- to: "splitter-bolt" # The stream consumer
- grouping: # Grouping type
- type: SHUFFLE
-
- - name: "Splitter -> Counter"
- from: "splitter-bolt"
- to: "counter-bolt"
- grouping:
- type: FIELDS
- args: ["word"] # field(s) to group on
- ```
-
-1. Enter the command below to open `pom.xml` to make the described revisions below:
-
- ```cmd
- notepad pom.xml
- ```
-
- 1. Add the following new dependency in the `<dependencies>` section:
-
- ```xml
- <!-- Add a dependency on the Flux framework -->
- <dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>flux-core</artifactId>
- <version>${storm.version}</version>
- </dependency>
- ```
-
- 1. Add the following plugin to the `<plugins>` section. This plugin handles the creation of a package (jar file) for the project, and applies some transformations specific to Flux when creating the package.
-
- ```xml
- <!-- build an uber jar -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <version>3.2.1</version>
- <configuration>
- <transformers>
- <!-- Keep us from getting a "can't overwrite file error" -->
- <transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer" />
- <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
- <!-- We're using Flux, so refer to it as main -->
- <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <mainClass>org.apache.storm.flux.Flux</mainClass>
- </transformer>
- </transformers>
- <!-- Keep us from getting a bad signature error -->
- <filters>
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- </configuration>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- </execution>
- </executions>
- </plugin>
- ```
-
- 1. For the Exec Maven Plugin section, navigate to `<configuration>` > `<mainClass>` and change `${storm.topology}` to `org.apache.storm.flux.Flux`. This setting allows Flux to handle running the topology locally in development.
-
- 1. In the `<resources>` section, add the following to `<includes>`. This XML includes the YAML file that defines the topology as part of the project.
-
- ```xml
- <include>topology.yaml</include>
- ```
-
-## Test the flux topology locally
-
-1. Enter the following command to compile and execute the Flux topology using Maven:
-
- ```cmd
- mvn compile exec:java -Dexec.args="--local -R /topology.yaml"
- ```
-
- > [!WARNING]
- > If your topology uses Storm 1.0.1 bits, this command fails. This failure is caused by [https://issues.apache.org/jira/browse/STORM-2055](https://issues.apache.org/jira/browse/STORM-2055). Instead, [install Storm in your development environment](https://storm.apache.org/releases/current/Setting-up-development-environment.html) and use the following steps:
- >
- > If you have [installed Storm in your development environment](https://storm.apache.org/releases/current/Setting-up-development-environment.html), you can use the following commands instead:
- >
- > ```cmd
- > mvn compile package
- > storm jar target/WordCount-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local -R /topology.yaml
- > ```
-
- The `--local` parameter runs the topology in local mode on your development environment. The `-R /topology.yaml` parameter uses the `topology.yaml` file resource from the jar file to define the topology.
-
- As it runs, the topology displays startup information. The following text is an example of the output:
-
- ```
- 17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 56 for word snow
- 17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 56 for word white
- 17:33:27 [Thread-12-count] INFO com.microsoft.example.WordCount - Emitting a count of 112 for word seven
- 17:33:27 [Thread-16-count] INFO com.microsoft.example.WordCount - Emitting a count of 195 for word the
- 17:33:27 [Thread-30-count] INFO com.microsoft.example.WordCount - Emitting a count of 113 for word and
- 17:33:27 [Thread-30-count] INFO com.microsoft.example.WordCount - Emitting a count of 57 for word dwarfs
- ```
-
- There's a 10-second delay between batches of logged information.
-
-2. Create a new topology yaml from the project.
-
- 1. Enter the command below to open `topology.xml`:
-
- ```cmd
- notepad resources\topology.yaml
- ```
-
- 1. Find the following section and change the value of `10` to `5`. This modification changes the interval between emitting batches of word counts from 10 seconds to 5.
-
- ```yaml
- - id: "counter-bolt"
- className: "com.microsoft.example.WordCount"
- constructorArgs:
- - 5
- parallelism: 1
- ```
-
- 1. Save file as `newtopology.yaml`.
-
-3. To run the topology, enter the following command:
-
- ```cmd
- mvn exec:java -Dexec.args="--local resources/newtopology.yaml"
- ```
-
- Or, if you have Storm on your development environment:
-
- ```cmd
- storm jar target/WordCount-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local resources/newtopology.yaml
- ```
-
- This command uses the `newtopology.yaml` as the topology definition. Since we didn't include the `compile` parameter, Maven uses the version of the project built in previous steps.
-
- Once the topology starts, you should notice that the time between emitted batches has changed to reflect the value in `newtopology.yaml`. So you can see that you can change your configuration through a YAML file without having to recompile the topology.
-
-For more information on these and other features of the Flux framework, see [Flux (https://storm.apache.org/releases/current/flux.html)](https://storm.apache.org/releases/current/flux.html).
-
-## Trident
-
-[Trident](https://storm.apache.org/releases/current/Trident-API-Overview.html) is a high-level abstraction that is provided by Storm. It supports stateful processing. The primary advantage of Trident is that it guarantees that every message that enters the topology is processed only once. Without using Trident, your topology can only guarantee that messages are processed at least once. There are also other differences, such as built-in components that can be used instead of creating bolts. Bolts are replaced by less-generic components, such as filters, projections, and functions.
-
-Trident applications can be created by using Maven projects. You use the same basic steps as presented earlier in this articleΓÇöonly the code is different. Trident also can't (currently) be used with the Flux framework.
-
-For more information about Trident, see the [Trident API Overview](https://storm.apache.org/releases/current/Trident-API-Overview.html).
-
-## Next Steps
-
-You've learned how to create an Apache Storm topology by using Java. Now learn how to:
-
-* [Deploy and manage Apache Storm topologies on HDInsight](apache-storm-deploy-monitor-topology-linux.md)
-
-* [Develop topologies using Python](apache-storm-develop-python-topology.md)
-
-You can find more example Apache Storm topologies by visiting [Example topologies for Apache Storm on HDInsight](apache-storm-example-topology.md).
hdinsight Apache Storm Develop Python Topology https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-develop-python-topology.md
- Title: Apache Storm with Python components - Azure HDInsight
-description: Learn how to create an Apache Storm topology that uses Python components in Azure HDInsight
--- Previously updated : 06/22/2022--
-# Develop Apache Storm topologies using Python on HDInsight
-
-Learn how to create an [Apache Storm](https://storm.apache.org/) topology that uses Python components. Apache Storm supports multiple languages, even allowing you to combine components from several languages in one topology. The [Flux](https://storm.apache.org/releases/current/flux.html) framework (introduced with Storm 0.10.0) allows you to easily create solutions that use Python components.
-
-> [!IMPORTANT]
-> The information in this document was tested using Storm on HDInsight 3.6.
-
-## Prerequisites
-
-* An Apache Storm cluster on HDInsight. See [Create Apache Hadoop clusters using the Azure portal](../hdinsight-hadoop-create-linux-clusters-portal.md) and select **Storm** for **Cluster type**.
-
-* A local Storm development environment (Optional). A local Storm environment is only needed if you want to run the topology locally. For more information, see [Setting up a development environment](https://storm.apache.org/releases/current/Setting-up-development-environment.html).
-
-* [Python 2.7 or higher](https://www.python.org/downloads/).
-
-* [Java Developer Kit (JDK) version 8](/azure/developer/java/fundamentals/java-support-on-azure).
-
-* [Apache Maven](https://maven.apache.org/download.cgi) properly [installed](https://maven.apache.org/install.html) according to Apache. Maven is a project build system for Java projects.
-
-## Storm multi-language support
-
-Apache Storm was designed to work with components written using any programming language. The components must understand how to work with the Thrift definition for Storm. For Python, a module is provided as part of the Apache Storm project that allows you to easily interface with Storm. You can find this module at [https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py](https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py).
-
-Storm is a Java process that runs on the Java Virtual Machine (JVM). Components written in other languages are executed as subprocesses. The Storm communicates with these subprocesses using JSON messages sent over stdin/stdout. More details on communication between components can be found in the [Multi-lang Protocol](https://storm.apache.org/releases/current/Multilang-protocol.html) documentation.
-
-## Python with the Flux framework
-
-The Flux framework allows you to define Storm topologies separately from the components. The Flux framework uses YAML to define the Storm topology. The following text is an example of how to reference a Python component in the YAML document:
-
-```yaml
-# Spout definitions
-spouts:
- - id: "sentence-spout"
- className: "org.apache.storm.flux.wrappers.spouts.FluxShellSpout"
- constructorArgs:
- # Command line
- - ["python", "sentencespout.py"]
- # Output field(s)
- - ["sentence"]
- # parallelism hint
- parallelism: 1
-```
-
-The class `FluxShellSpout` is used to start the `sentencespout.py` script that implements the spout.
-
-Flux expects the Python scripts to be in the `/resources` directory inside the jar file that contains the topology. So this example stores the Python scripts in the `/multilang/resources` directory. The `pom.xml` includes this file using the following XML:
-
-```xml
-<!-- include the Python components -->
-<resource>
- <directory>${basedir}/multilang</directory>
- <filtering>false</filtering>
-</resource>
-```
-
-As mentioned earlier, there's a `storm.py` file that implements the Thrift definition for Storm. The Flux framework includes `storm.py` automatically when the project is built, so you don't have to worry about including it.
-
-## Build the project
-
-1. Download the project from [https://github.com/Azure-Samples/hdinsight-python-storm-wordcount](https://github.com/Azure-Samples/hdinsight-python-storm-wordcount).
-
-1. Open a command prompt and navigate to the project root: `hdinsight-python-storm-wordcount-master`. Enter the following command:
-
- ```cmd
- mvn clean compile package
- ```
-
- This command creates a `target/WordCount-1.0-SNAPSHOT.jar` file that contains the compiled topology.
-
-## Run the Storm topology on HDInsight
-
-1. Use [ssh command](../hdinsight-hadoop-linux-use-ssh-unix.md) to copy the `WordCount-1.0-SNAPSHOT.jar` file to your Storm on HDInsight cluster. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
-
- ```cmd
- scp target/WordCount-1.0-SNAPSHOT.jar sshuser@CLUSTERNAME-ssh.azurehdinsight.net:
- ```
-
-1. Once the file has been uploaded, connect to the cluster using SSH:
-
- ```cmd
- ssh sshuser@CLUSTERNAME-ssh.azurehdinsight.net
- ```
-
-1. From the SSH session, use the following command to start the topology on the cluster:
-
- ```bash
- storm jar WordCount-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux -r -R /topology.yaml
- ```
-
- Once started, a Storm topology runs until stopped.
-
-1. Use the Storm UI to view the topology on the cluster. The Storm UI is located at `https://CLUSTERNAME.azurehdinsight.net/stormui`. Replace `CLUSTERNAME` with your cluster name.
-
-1. Stop the Storm topology. Use the following command to stop the topology on the cluster:
-
- ```bash
- storm kill wordcount
- ```
-
- Alternatively, you can use the Storm UI. Under **Topology actions** for the topology, select **Kill**.
-
-## Run the topology locally
-
-To run the topology locally, use the following command:
-
-```bash
-storm jar WordCount-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux -l -R /topology.yaml
-```
-
-> [!NOTE]
-> This command requires a local Storm development environment. For more information, see [Setting up a development environment](https://storm.apache.org/releases/current/Setting-up-development-environment.html).
-
-Once the topology starts, it emits information to the local console similar to the following text:
-
-```output
-24302 [Thread-25-sentence-spout-executor[4 4]] INFO o.a.s.s.ShellSpout - ShellLog pid:2436, name:sentence-spout Emiting the cow jumped over the moon
-24302 [Thread-30] INFO o.a.s.t.ShellBolt - ShellLog pid:2438, name:splitter-bolt Emitting the
-24302 [Thread-28] INFO o.a.s.t.ShellBolt - ShellLog pid:2437, name:counter-bolt Emitting years:160
-24302 [Thread-17-log-executor[3 3]] INFO o.a.s.f.w.b.LogInfoBolt - {word=the, count=599}
-24303 [Thread-17-log-executor[3 3]] INFO o.a.s.f.w.b.LogInfoBolt - {word=seven, count=302}
-24303 [Thread-17-log-executor[3 3]] INFO o.a.s.f.w.b.LogInfoBolt - {word=dwarfs, count=143}
-24303 [Thread-25-sentence-spout-executor[4 4]] INFO o.a.s.s.ShellSpout - ShellLog pid:2436, name:sentence-spout Emiting the cow jumped over the moon
-24303 [Thread-30] INFO o.a.s.t.ShellBolt - ShellLog pid:2438, name:splitter-bolt Emitting cow
-24303 [Thread-17-log-executor[3 3]] INFO o.a.s.f.w.b.LogInfoBolt - {word=four, count=160}
-```
-
-To stop the topology, use __Ctrl + C__.
-
-## Next steps
-
-See the following documents for other ways to use Python with HDInsight: [How to use Python User Defined Functions (UDF) in Apache Pig and Apache Hive](../hadoop/python-udf-hdinsight.md).
hdinsight Apache Storm Example Topology https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-example-topology.md
- Title: Example Apache Storm topologies in Azure HDInsight
-description: A list of example Storm topologies created and tested with Apache Storm on HDInsight including basic C# and Java topologies, and working with Event Hubs.
--- Previously updated : 06/29/2022--
-# Example Apache Storm topologies and components for Apache Storm on HDInsight
-
-The following is a list of examples created and maintained by Microsoft for use with [Apache Storm](https://storm.apache.org/) on HDInsight. These examples cover a variety of topics, from creating basic C# and Java topologies to working with Azure services such as Event Hubs, Cosmos DB, SQL Database, [Apache HBase](https://hbase.apache.org/) on HDInsight, and Azure Storage. Some examples also demonstrate how to work with non-Azure, or even non-Microsoft technologies, such as SignalR and Socket.IO.
-
-| Description | Demonstrates | Language/Framework |
-|: |: |: |
-| [Write to Azure Data Lake Storage from Apache Storm](apache-storm-write-data-lake-store.md) |Writing to Azure Data Lake Storage |Java |
-| [Event Hub Spout and Bolt source](https://github.com/apache/storm/tree/master/external/storm-eventhubs) |Source for the Event Hub Spout and Bolt |Java |
-| [Develop Java-based topologies for Apache Storm on HDInsight][5797064f] |Maven |Java |
-| [Develop C# topologies for Apache Storm on HDInsight using Visual Studio][16fce2d1] |HDInsight Tools for Visual Studio |C#, Java |
-| [Process events from Azure Event Hubs with Apache Storm on HDInsight (C#)][844d1d81] |Event Hubs |C# and Java |
-| [Process events from Azure Event Hubs with Storm on HDInsight (Java)](https://github.com/Azure-Samples/hdinsight-java-storm-eventhub) |Event Hubs |Java |
-| [Process vehicle sensor data from Event Hubs using Apache Storm on HDInsight][246ee964] |Event Hubs, Cosmos DB, Azure Storage Blob (WASB) |C#, Java |
-| [Extract, Transform, and Load (ETL) from Azure Event Hubs to Apache HBase, using Apache Storm on HDInsight][b4b68194] |Event Hubs, HBase |C# |
-| [Template C# Storm topology project for working with Azure services from Apache Storm on HDInsight][ce0c02a2] |Event Hubs, Cosmos DB, SQL Database, HBase, SignalR |C#, Java |
-| [Scalability benchmarks for reading from Azure Event Hubs using Apache Storm on HDInsight][d6c540e3] |Message throughput, Event Hubs, SQL Database |C#, Java |
-| [Use Apache Kafka with Apache Storm on HDInsight](../hdinsight-apache-storm-with-kafka.md) | Apache Storm reading and writing to Apache Kafka | Java |
-
-> [!WARNING]
-> The C# examples in this list were originally created and tested with Windows-based HDInsight, and may not work correctly with Linux-based HDInsight clusters. Linux-based clusters use Mono to run .NET code, and may have compatibility problems with the frameworks and packages used in the example.
->
-> Linux is the only operating system used on HDInsight version 3.4 or later.
-
-## Python only
-
-See [Use Python with Apache Storm on HDInsight](apache-storm-develop-python-topology.md) for an example of Python components with a Flux topology.
-
-## Next Steps
-
-* [Create and monitor an Apache Storm topology in Azure HDInsight](./apache-storm-quickstart.md)
-* [Learn how to deploy and manage Apache Storm topologies with Apache Storm on HDInsight][6eb0d3b8]
-
-[6eb0d3b8]:apache-storm-deploy-monitor-topology-linux.md "Learn how to deploy and manage topologies using the web-based Apache Storm Dashboard and Storm UI or the HDInsight Tools for Visual Studio."
-[16fce2d1]:apache-storm-develop-csharp-visual-studio-topology.md "Learn how to create C# Storm topologies by using the HDInsight Tools for Visual Studio."
-[5797064f]:apache-storm-develop-java-topology.md "Learn how to create Storm topologies in Java, using Maven, by creating a basic wordcount topology."
-[844d1d81]:apache-storm-develop-csharp-event-hub-topology.md "Learn how to read and write data from Azure Event Hubs with Storm on HDInsight."
-[246ee964]: https://github.com/hdinsight/hdinsight-storm-examples/blob/master/IotExample/README.md "Learn how to use a Storm topology to read messages from Azure Event Hubs, read documents from Azure Cosmos DB for data referencing and save data to Azure Storage."
-[d6c540e3]: https://github.com/hdinsight/hdinsight-storm-examples/blob/master/EventCountExample "Several topologies to demonstrate throughput when reading from Azure Event Hubs and storing to SQL Database using Apache Storm on HDInsight."
-[b4b68194]: https://github.com/hdinsight/hdinsight-storm-examples/blob/master/RealTimeETLExample "Learn how to read data from Azure Event Hubs, aggregate & transform the data, then store it to HBase on HDInsight."
-[ce0c02a2]: https://github.com/hdinsight/hdinsight-storm-examples/tree/master/templates/HDInsightStormExamples "This project contains templates for spouts, bolts and topologies to interact with various Azure services like Event Hubs, Cosmos DB, and SQL Database."
hdinsight Apache Storm Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-overview.md
- Title: What is Apache Storm - Azure HDInsight
-description: Apache Storm allows you to process streams of data in real time. Azure HDInsight allows you to easily create Storm clusters on the Azure cloud. With Visual Studio, you can create Storm solutions using C#, and then deploy to your HDInsight Storm clusters.
--- Previously updated : 04/20/2020-
-#Customer intent: As a developer, I want to understand how Storm on HDInsight is different from Storm on other platforms.
--
-# What is Apache Storm on Azure HDInsight?
-
-[Apache Storm](https://storm.apache.org/) is a distributed, fault-tolerant, open-source computation system. You can use Storm to process streams of data in real time with [Apache Hadoop](../hadoop/apache-hadoop-introduction.md). Storm solutions can also provide guaranteed processing of data, with the ability to replay data that wasn't successfully processed the first time.
-
-## Why use Apache Storm on HDInsight?
-
-Storm on HDInsight provides the following features:
-
-* __99% Service Level Agreement (SLA) on Storm uptime__: Storm on HDInsight comes with full continuous support. Storm on HDInsight also has an SLA of 99.9 percent. That means Microsoft guarantees that a Storm cluster has external connectivity at least 99.9 percent of the time. For more information, see [Azure support](https://azure.microsoft.com/support/options/). See also, [SLA information for HDInsight](https://azure.microsoft.com/support/legal/sla/hdinsight/v1_0/) document.
-
-* Supports easy customization by running scripts against a Storm cluster during or after creation. For more information, see [Customize HDInsight clusters using script action](../hdinsight-hadoop-customize-cluster-linux.md).
-
-* **Create solutions in multiple languages**: You can write Storm components in the language of your choice, such as Java, C#, and Python.
-
- * Integrates Visual Studio with HDInsight for the development, management, and monitoring of C# topologies. For more information, see [Develop C# Storm topologies with the HDInsight Tools for Visual Studio](apache-storm-develop-csharp-visual-studio-topology.md).
-
- * Supports the Trident Java interface. You can create Storm topologies that support exactly once processing of messages, transactional datastore persistence, and a set of common stream analytics operations.
-
-* **Dynamic scaling**: You can add or remove worker nodes with no impact to running Storm topologies. Deactivate and reactivate running topologies to take advantage of new nodes added through scaling operations.
-
-* **Create streaming pipelines using multiple Azure services**: Storm on HDInsight integrates with other Azure services. Such as Event Hubs, SQL Database, Azure Storage, and Azure Data Lake Storage. For an example solution that integrates with Azure services, see [Process events from Event Hubs with Apache Storm on HDInsight](https://github.com/Azure-Samples/hdinsight-java-storm-eventhub).
-
-For a list of companies that are using Apache Storm for their real-time analytics solutions, see [Companies using Apache Storm](https://storm.apache.org/Powered-By.html).
-
-To get started using Storm, see [Create and monitor an Apache Storm topology in Azure HDInsight](apache-storm-quickstart.md).
-
-## How does Apache Storm work
-
-Storm runs topologies instead of the [Apache Hadoop MapReduce](https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html) jobs that you might be familiar with. Storm topologies are composed of multiple components that are arranged in a directed acyclic graph (DAG). Data flows between the components in the graph. Each component consumes one or more data streams, and can optionally emit one or more streams. The following diagram illustrates how data flows between components in a basic word-count topology:
--
-* Spout components bring data into a topology. They emit one or more streams into the topology.
-
-* Bolt components consume streams emitted from spouts or other bolts. Bolts might optionally emit streams into the topology. Bolts are also responsible for writing data to external services or storage, such as HDFS, Kafka, or HBase.
-
-## Reliability
-
-Apache Storm guarantees that each incoming message is always fully processed, even when the data analysis is spread over hundreds of nodes.
-
-The Nimbus node provides functionality similar to the Apache Hadoop JobTracker. Nimbus assigns tasks to other nodes in a cluster through Apache ZooKeeper. Zookeeper nodes provide coordination for a cluster and assist communication between Nimbus and the Supervisor process on the worker nodes. If one processing node goes down, the Nimbus node is informed, and it assigns the task and associated data to another node.
-
-The default configuration for Apache Storm clusters is to have only one Nimbus node. Storm on HDInsight provides two Nimbus nodes. If the primary node fails, the Storm cluster switches to the secondary node while the primary node is recovered. The following diagram illustrates the task flow configuration for Storm on HDInsight:
--
-## Ease of use
-
-|Use |Description |
-|||
-|Secure Shell (SSH) connectivity|You can access the head nodes of your Storm cluster over the Internet by using SSH. You can run commands directly on your cluster by using SSH. For more information, see [Use SSH with HDInsight](../hdinsight-hadoop-linux-use-ssh-unix.md).|
-|Web connectivity|All HDInsight clusters provide the Ambari web UI. You can easily monitor, configure, and manage services on your cluster by using the Ambari web UI. Storm clusters also provide the Storm UI. You can monitor and manage running Storm topologies from your browser by using the Storm UI. For more information, see the [Manage HDInsight using the Apache Ambari Web UI](../hdinsight-hadoop-manage-ambari.md) and [Monitor and manage using the Apache Storm UI](apache-storm-deploy-monitor-topology-linux.md#monitor-and-manage-a-topology-using-the-storm-ui) documents.|
-|Azure PowerShell and Azure CLI|PowerShell and Azure CLI both provide command-line utilities that you can use from your client system to work with HDInsight and other Azure services.|
-|Visual Studio integration|Azure Data Lake Tools for Visual Studio include project templates for creating C# Storm topologies by using the SCP.NET framework. Data Lake Tools also provide tools to deploy, monitor, and manage solutions with Storm on HDInsight. For more information, see [Develop C# Storm topologies with the HDInsight Tools for Visual Studio](apache-storm-develop-csharp-visual-studio-topology.md).|
-
-## Integration with other Azure services
-
-* __Azure Data Lake Storage__: See [Use Azure Data Lake Storage with Apache Storm on HDInsight](apache-storm-write-data-lake-store.md).
-
-* __Event Hubs__: For an example of using Event Hubs with a Storm cluster, see the following examples:
-
- * [Process events from Azure Event Hubs with Apache Storm on HDInsight (Java)](https://github.com/Azure-Samples/hdinsight-java-storm-eventhub)
-
- * [Process events from Azure Event Hubs with Apache Storm on HDInsight (C#)](apache-storm-develop-csharp-event-hub-topology.md)
-
-* __SQL Database__, __Cosmos DB__, __Event Hubs__, and __HBase__: Template examples are included in the Data Lake Tools for Visual Studio. For more information, see [Develop a C# topology for Apache Storm on HDInsight](apache-storm-develop-csharp-visual-studio-topology.md).
-
-## Apache Storm use cases
-
-The following are some common scenarios for which you might use Storm on HDInsight:
-
-* Internet of Things (IoT)
-* Fraud detection
-* Social analytics
-* Extraction, transformation, and loading (ETL)
-* Network monitoring
-* Search
-* Mobile engagement
-
-For information about real-world scenarios, see the [How companies are using Apache Storm](https://storm.apache.org/Powered-By.html) document.
-
-## Development
-
-.NET developers can design and implement topologies in C# by using Data Lake Tools for Visual Studio. You can also create hybrid topologies that use Java and C# components. For more information, see [Develop C# topologies for Apache Storm on HDInsight using Visual Studio](apache-storm-develop-csharp-visual-studio-topology.md).
-
-You can also develop Java solutions by using the IDE of your choice. For more information, see [Develop Java topologies for Apache Storm on HDInsight](apache-storm-develop-java-topology.md).
-
-Python can also be used to develop Storm components. For more information, see [Develop Apache Storm topologies using Python on HDInsight](apache-storm-develop-python-topology.md).
-
-## Common development patterns
-
-### Guaranteed message processing
-
-Apache Storm can provide different levels of guaranteed message processing. For example, a basic Storm application guarantees at-least-once processing, and Trident can guarantee exactly once processing. See [Guarantees on data processing](https://storm.apache.org/about/guarantees-data-processing.html) at apache.org.
-
-### IBasicBolt
-
-The pattern of reading an input tuple, emitting zero or more tuples, and then confirming the input tuple immediately at the end of the execute method is common. Storm provides the [IBasicBolt](https://storm.apache.org/releases/current/javadocs/org/apache/storm/topology/IBasicBolt.html) interface to automate this pattern.
-
-### Joins
-
-How data streams are joined varies between applications. For example, you can join each tuple from multiple streams into one new stream, or join only batches of tuples for a specific window. Either way, joining can be accomplished by using [fieldsGrouping](https://storm.apache.org/releases/current/javadocs/org/apache/storm/topology/InputDeclarer.html#fieldsGrouping-java.lang.String-org.apache.storm.tuple.Fields-). Field grouping is a way of defining how tuples are routed to bolts.
-
-In the following Java example, fieldsGrouping is used to route tuples that originate from components "1", "2", and "3" to the MyJoiner bolt:
-
-```java
-builder.setBolt("join", new MyJoiner(), parallelism) .fieldsGrouping("1", new Fields("joinfield1", "joinfield2")) .fieldsGrouping("2", new Fields("joinfield1", "joinfield2")) .fieldsGrouping("3", new Fields("joinfield1", "joinfield2"));
-```
-
-### Batches
-
-Apache Storm provides an internal timing mechanism known as a "tick tuple." You can set how often a tick tuple is emitted in your topology.
-
-For an example of using a tick tuple from a C# component, see [PartialBoltCount.cs](https://github.com/hdinsight/hdinsight-storm-examples/blob/3b2c960549cac122e8874931df4801f0934fffa7/EventCountExample/EventCountTopology/src/main/java/com/microsoft/hdinsight/storm/examples/PartialCountBolt.java).
-
-### Caches
-
-In-memory caching is often used as a mechanism for speeding up processing because it keeps frequently used assets in memory. Because a topology is distributed across multiple nodes, and multiple processes within each node, you should consider using [fieldsGrouping](https://storm.apache.org/releases/current/javadocs/org/apache/storm/topology/InputDeclarer.html#fieldsGrouping-java.lang.String-org.apache.storm.tuple.Fields-). Use `fieldsGrouping` to ensure that tuples containing the fields that are used for cache lookup are always routed to the same process. This grouping functionality avoids duplication of cache entries across processes.
-
-### Stream "top N"
-
-When your topology depends on calculating a top N value, calculate the top N value in parallel. Then merge the output from those calculations into a global value. This operation can be done by using [fieldsGrouping](https://storm.apache.org/releases/current/javadocs/org/apache/storm/topology/InputDeclarer.html#fieldsGrouping-java.lang.String-org.apache.storm.tuple.Fields-) to route by field for parallel processing. Then you can route to a bolt that globally determines the top N value.
-
-For an example of calculating a top N value, see the [RollingTopWords](https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/RollingTopWords.java) example.
-
-## Logging
-
-Storm uses Apache Log4j 2 to log information. By default, a large amount of data is logged, and it can be difficult to sort through the information. You can include a logging configuration file as part of your Storm topology to control logging behavior.
-
-For an example topology that demonstrates how to configure logging, see [Java-based WordCount](apache-storm-develop-java-topology.md) example for Storm on HDInsight.
-
-## Next steps
-
-Learn more about real-time analytics solutions with Apache Storm on HDInsight:
-
-* [Create and monitor an Apache Storm topology in Azure HDInsight](apache-storm-quickstart.md)
-* [Example topologies for Apache Storm on HDInsight](apache-storm-example-topology.md)
hdinsight Apache Storm Quickstart https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-quickstart.md
- Title: 'Quickstart: Create/manage Apache Storm topology - Azure HDInsight'
-description: In the quickstart, learn how to create and monitor an Apache Storm topology in Azure HDInsight.
-- Previously updated : 06/14/2019-
-#Customer intent: I want to learn how to create Apache Storm topologies and deploy them to a Storm cluster in Azure HDInsight.
--
-# Quickstart: Create and monitor an Apache Storm topology in Azure HDInsight
-
-Apache Storm is a scalable, fault-tolerant, distributed, real-time computation system for processing streams of data. With Storm on Azure HDInsight, you can create a cloud-based Storm cluster that performs big data analytics in real time.
-
-In this quickstart, you use an example from the Apache [storm-starter](https://github.com/apache/storm/tree/v2.0.0/examples/storm-starter) project to create and monitor an Apache Storm topology to an existing Apache Storm cluster.
-
-## Prerequisites
-
-* An Apache Storm cluster on HDInsight. See [Create Apache Hadoop clusters using the Azure portal](../hdinsight-hadoop-create-linux-clusters-portal.md) and select **Storm** for **Cluster type**.
-
-* An SSH client. For more information, see [Connect to HDInsight (Apache Hadoop) using SSH](../hdinsight-hadoop-linux-use-ssh-unix.md).
-
-## Create the topology
-
-1. Connect to your Storm cluster. Edit the command below by replacing `CLUSTERNAME` with the name of your Storm cluster, and then enter the command:
-
- ```cmd
- ssh sshuser@CLUSTERNAME-ssh.azurehdinsight.net
- ```
-
-2. The **WordCount** example is included on your HDInsight cluster at `/usr/hdp/current/storm-client/contrib/storm-starter/`. The topology generates random sentences and counts how many times words occur. Use the following command to start the **wordcount** topology on the cluster:
-
- ```bash
- storm jar /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-*.jar org.apache.storm.starter.WordCountTopology wordcount
- ```
-
-## Monitor the topology
-
-Storm provides a web interface for working with running topologies, and is included on your HDInsight cluster.
-
-Use the following steps to monitor the topology using the Storm UI:
-
-1. To display the Storm UI, open a web browser to `https://CLUSTERNAME.azurehdinsight.net/stormui`. Replace `CLUSTERNAME` with the name of your cluster.
-
-2. Under **Topology Summary**, select the **wordcount** entry in the **Name** column. Information about the topology is displayed.
-
- :::image type="content" source="./media/apache-storm-quickstart/hdi-topology-summary.png" alt-text="Storm Dashboard with storm-starter WordCount topology information." border="true":::
-
- The new page provides the following information:
-
- |Property | Description |
- |||
- |Topology stats|Basic information on the topology performance, organized into time windows. Selecting a specific time window changes the time window for information displayed in other sections of the page.|
- |Spouts|Basic information about spouts, including the last error returned by each spout.|
- |Bolts|Basic information about bolts.|
- |Topology configuration|Detailed information about the topology configuration.|
- |Activate|Resumes processing of a deactivated topology.|
- |Deactivate|Pauses a running topology.|
- |Rebalance|Adjusts the parallelism of the topology. You should rebalance running topologies after you have changed the number of nodes in the cluster. Rebalancing adjusts parallelism to compensate for the increased/decreased number of nodes in the cluster. For more information, see [Understanding the parallelism of an Apache Storm topology](https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html).|
- |Kill|Terminates a Storm topology after the specified timeout.|
-
-3. From this page, select an entry from the **Spouts** or **Bolts** section. Information about the selected component is displayed.
-
- :::image type="content" source="./media/apache-storm-quickstart/hdi-component-summary.png" alt-text="Storm Dashboard with information about selected components." border="true":::
-
- The new page displays the following information:
-
- |Property | Description |
- |||
- |Spout/Bolt stats|Basic information on the component performance, organized into time windows. Selecting a specific time window changes the time window for information displayed in other sections of the page.|
- |Input stats (bolt only)|Information on components that produce data consumed by the bolt.|
- |Output stats|Information on data emitted by this bolt.|
- |Executors|Information on instances of this component.|
- |Errors|Errors produced by this component.|
-
-4. When viewing the details of a spout or bolt, select an entry from the **Port** column in the **Executors** section to view details for a specific instance of the component.
-
-```output
-2015-01-27 14:18:02 b.s.d.task [INFO] Emitting: split default ["with"]
-2015-01-27 14:18:02 b.s.d.task [INFO] Emitting: split default ["nature"]
-2015-01-27 14:18:02 b.s.d.executor [INFO] Processing received message source: split:21, stream: default, id: {}, [snow]
-2015-01-27 14:18:02 b.s.d.task [INFO] Emitting: count default [snow, 747293]
-2015-01-27 14:18:02 b.s.d.executor [INFO] Processing received message source: split:21, stream: default, id: {}, [white]
-2015-01-27 14:18:02 b.s.d.task [INFO] Emitting: count default [white, 747293]
-2015-01-27 14:18:02 b.s.d.executor [INFO] Processing received message source: split:21, stream: default, id: {}, [seven]
-2015-01-27 14:18:02 b.s.d.task [INFO] Emitting: count default [seven, 1493957]
-```
-
-In this example, the word **seven** has occurred 1493957 times. This count is how many times the word has been encountered since this topology was started.
-
-## Stop the topology
-
-Return to the **Topology summary** page for the word-count topology, and then select the **Kill** button from the **Topology actions** section. When prompted, enter 10 for the seconds to wait before stopping the topology. After the timeout period, the topology no longer appears when you visit the **Storm UI** section of the dashboard.
-
-## Clean up resources
-
-After you complete the quickstart, you may want to delete the cluster. With HDInsight, your data is stored in Azure Storage, so you can safely delete a cluster when it is not in use. You are also charged for an HDInsight cluster, even when it is not in use. Since the charges for the cluster are many times more than the charges for storage, it makes economic sense to delete clusters when they are not in use.
-
-To delete a cluster, see [Delete an HDInsight cluster using your browser, PowerShell, or the Azure CLI](../hdinsight-delete-cluster.md).
-
-## Next steps
-
-In this quickstart, you used an example from the Apache [storm-starter](https://github.com/apache/storm/tree/v2.0.0/examples/storm-starter) project to create and monitor an Apache Storm topology to an existing Apache Storm cluster. Advance to the next article to learn the basics of managing and monitoring Apache Storm topologies.
-
-> [!div class="nextstepaction"]
->[Deploy and manage Apache Storm topologies on Azure HDInsight](./apache-storm-deploy-monitor-topology-linux.md)
hdinsight Apache Storm Scp Programming Guide https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-scp-programming-guide.md
- Title: SCP.NET programming guide for Storm in Azure HDInsight
-description: Learn how to use SCP.NET to create .NET-based Storm topologies for use with Storm running in Azure HDInsight.
--- Previously updated : 01/13/2020---
-# SCP programming guide for Apache Storm in Azure HDInsight
-
-SCP is a platform for building real time, reliable, consistent, and high-performance data processing applications. It's built on top of [Apache Storm](https://storm.incubator.apache.org/), which is a stream-processing system designed by open-source software communities. Nathan Marz created Storm. It was published as open source by Twitter. Storm uses [Apache ZooKeeper](https://zookeeper.apache.org/), which is another Apache project that enables highly reliable distributed coordination and state management.
-
-The SCP project has ported not only Storm on Windows but also project-added extensions and customization for the Windows environment. The extensions include the .NET developer experience and .NET libraries. The customization includes Windows-based deployment.
-
-With the extensions and customization, you don't need to fork the open-source software projects. You can use derived environments that are built on top of Storm.
-
-## Processing model
-
-The data in SCP is modeled as continuous streams of tuples. Typically, the tuples:
-
-1. Flow into a queue.
-1. Are picked up and transformed by business logic hosted inside a Storm topology.
-1. Either have their output piped as tuples to another SCP system or are committed to stores like distributed file systems and databases like SQL Server.
--
-In Storm, an application topology defines a computation graph. Each node in a topology contains processing logic. Links between nodes indicate data flow.
-
-Nodes that inject input data into the topology are called _spouts_. You can use them to sequence the data. The input data could come from a source like file logs, a transactional database, or a system performance counter.
-
-Nodes that have both input and output data flows are called _bolts_. They do the actual data filtering, selections, and aggregation.
-
-SCP supports best-efforts, at-least-once, and exactly once data processing.
-
-In a distributed stream-processing application, errors might happen during data processing. Such errors include a network outage, a machine failure, or an error in your code. At-least-once processing ensures all data is processed at least once by automatically replaying the same data when an error happens.
-
-At-least-once processing is simple and reliable, and it suits many applications. But when an application requires exact counting, at-least-once processing is insufficient because the same data could be replayed in the application topology. In that case, exactly once processing makes sure the result is correct even when the data is replayed and processed multiple times.
-
-SCP lets .NET developers create real-time data processing applications while using a Java Virtual Machine (JVM) with Storm. A JVM and .NET communicate via TCP local sockets. Each spout/bolt is a .NET/Java process pair, where the user logic runs in a .NET process as a plug-in.
-
-To build a data processing application on top of SCP, follow these steps:
-
-1. Design and implement spouts to pull in data from queues.
-1. Design and implement bolts that process the input data and save it to external stores like a database.
-1. Design the topology, then submit and run it.
-
-The topology defines vertexes and the data that flows between them. SCP takes a topology specification and deploys it on a Storm cluster, where each vertex runs on one logical node. The Storm task scheduler takes care of failover and scaling.
-
-This article uses some simple examples to walk through how to build data processing applications with SCP.
-
-## SCP plug-in interface
-
-SCP plug-ins are standalone applications. They can run inside Visual Studio during development and be plugged into the Storm pipeline after production deployment.
-
-Writing an SCP plug-in is the same as writing any other Windows console application. The SCP.NET platform declares some interfaces for spout/bolt. Your plug-in code implements these interfaces. The main purpose of this design is to let you focus on your business logic while letting the SCP.NET platform handle other things.
-
-Your plug-in code implements one of the following interfaces. Which interface depends on whether the topology is transactional or nontransactional and whether the component is a spout or a bolt.
-
-* **ISCPSpout**
-* **ISCPBolt**
-* **ISCPTxSpout**
-* **ISCPBatchBolt**
-
-### ISCPPlugin
-
-**ISCPPlugin** is the common interface for many plug-ins. Currently, it's a dummy interface.
-
-```csharp
-public interface ISCPPlugin
-{
-}
-```
-
-### ISCPSpout
-
-**ISCPSpout** is the interface for a nontransactional spout.
-
-```csharp
-public interface ISCPSpout : ISCPPlugin
-{
- void NextTuple(Dictionary<string, Object> parms);
- void Ack(long seqId, Dictionary<string, Object> parms);
- void Fail(long seqId, Dictionary<string, Object> parms);
-}
-```
-
-When **NextTuple** is called, your C# code may emit one or more tuples. If there's nothing to emit, this method should return without emitting anything.
-
-The **NextTuple**, **Ack**, and **Fail** methods are all called in a tight loop in a single thread of a C# process. When there are no tuples to emit, have **NextTuple** sleep for a short amount of time like 10 milliseconds. This sleep helps avoid wasting CPU availability.
-
-The **Ack** and **Fail** methods are called only when a specification file enables the acknowledgment mechanism. The *seqId* parameter identifies the tuple that is acknowledged or has failed. If acknowledgment is enabled in a nontransactional topology, the following **Emit** function should be used in a spout:
-
-```csharp
-public abstract void Emit(string streamId, List<object> values, long seqId);
-```
-
-If a nontransactional topology doesn't support acknowledgment, **Ack** and **Fail** can be left as empty functions.
-
-The *parms* input parameter in these functions specifies an empty dictionary and is reserved for future use.
-
-### ISCPBolt
-
-**ISCPBolt** is the interface for a nontransactional bolt.
-
-```csharp
-public interface ISCPBolt : ISCPPlugin
-{
-void Execute(SCPTuple tuple);
-}
-```
-
-When a new tuple is available, the **Execute** function is called to process it.
-
-### ISCPTxSpout
-
-**ISCPTxSpout** is the interface for a transactional spout.
-
-```csharp
-public interface ISCPTxSpout : ISCPPlugin
-{
- void NextTx(out long seqId, Dictionary<string, Object> parms);
- void Ack(long seqId, Dictionary<string, Object> parms);
- void Fail(long seqId, Dictionary<string, Object> parms);
-}
-```
-
-Just like their nontransactional counterparts, **NextTx**, **Ack**, and **Fail** are all called in a tight loop in a single thread of a C# process. When there are no tuples to emit, have **NextTx** sleep for a short amount of time like 10 milliseconds. This sleep helps avoid wasting CPU availability.
-
-When **NextTx** is called to start a new transaction, the *seqId* output parameter identifies the transaction. The transaction is also used in **Ack** and **Fail**. Your **NextTx** method can emit data to the Java side. The data is stored in ZooKeeper to support replay. Because ZooKeeper has limited capacity, your code should emit only metadata and not bulk data in a transactional spout.
-
-Because Storm automatically replays a failed transaction, **Fail** usually won't be called. But if SCP can check the metadata emitted by a transactional spout, it can call **Fail** when the metadata is invalid.
-
-The *parms* input parameter in these functions specifies an empty dictionary and is reserved for future use.
-
-### ISCPBatchBolt
-
-**ISCPBatchBolt** is the interface for a transactional bolt.
-
-```csharp
-public interface ISCPBatchBolt : ISCPPlugin
-{
- void Execute(SCPTuple tuple);
- void FinishBatch(Dictionary<string, Object> parms);
-}
-```
-
-The **Execute** method is called when a new tuple arrives at the bolt. The **FinishBatch** method is called when this transaction ends. The *parms* input parameter is reserved for future use.
-
-For a transactional topology, **StormTxAttempt** is an important class. It has two members: **TxId** and **AttemptId**. The **TxId** member identifies a specific transaction. A transaction might be attempted multiple times if it fails and is replayed.
-
-SCP.NET creates a new **ISCPBatchBolt** object to process each **StormTxAttempt** object, just like what Storm does in Java. This design's purpose is to support parallel transaction processing. After a transaction attempt is complete, the corresponding **ISCPBatchBolt** object is destroyed and garbage collected.
-
-## Object model
-
-SCP.NET also provides a simple set of key objects for developers to program with. The objects are **Context**, **StateStore**, and **SCPRuntime**. They're discussed in this section.
-
-### Context
-
-The **Context** object provides a running environment to an application. Each **ISCPPlugin** instance of **ISCPSpout**, **ISCPBolt**, **ISCPTxSpout**, or **ISCPBatchBolt** has a corresponding **Context** instance. The functionality provided by **Context** is divided into these two parts:
-
-* The static part, which is available in the whole C# process
-* The dynamic part, which is available only for the specific **Context** instance
-
-### Static part
-
-```csharp
-public static ILogger Logger = null;
-public static SCPPluginType pluginType;
-public static Config Config { get; set; }
-public static TopologyContext TopologyContext { get; set; }
-```
-
-The **Logger** object is provided for logging purposes.
-
-The **pluginType** object indicates the plug-in type of the C# process. If the process is run in local test mode without Java, the plug-in type is **SCP_NET_LOCAL**.
-
-```csharp
-public enum SCPPluginType
-{
- SCP_NET_LOCAL = 0,
- SCP_NET_SPOUT = 1,
- SCP_NET_BOLT = 2,
- SCP_NET_TX_SPOUT = 3,
- SCP_NET_BATCH_BOLT = 4
- }
-```
-
-The **Config** property gets configuration parameters from the Java side, which passes them when a C# plug-in is initialized. The **Config** parameters are divided into two parts: **stormConf** and **pluginConf**.
-
-```csharp
-public Dictionary<string, Object> stormConf { get; set; }
-public Dictionary<string, Object> pluginConf { get; set; }
-```
-
-The **stormConf** part is parameters defined by Storm, and the **pluginConf** part is parameters defined by SCP. Here's an example:
-
-```csharp
-public class Constants
-{
- … …
-
- // constant string for pluginConf
- public static readonly String NONTRANSACTIONAL_ENABLE_ACK = "nontransactional.ack.enabled";
-
- // constant string for stormConf
- public static readonly String STORM_ZOOKEEPER_SERVERS = "storm.zookeeper.servers";
- public static readonly String STORM_ZOOKEEPER_PORT = "storm.zookeeper.port";
-}
-```
-
-The **TopologyContext** type gets the topology context. It's most useful for multiple parallel components. Here's an example:
-
-```csharp
-//demo how to get TopologyContext info
-if (Context.pluginType != SCPPluginType.SCP_NET_LOCAL)
-{
- Context.Logger.Info("TopologyContext info:");
- TopologyContext topologyContext = Context.TopologyContext;
- Context.Logger.Info("taskId: {0}", topologyContext.GetThisTaskId());
- taskIndex = topologyContext.GetThisTaskIndex();
- Context.Logger.Info("taskIndex: {0}", taskIndex);
- string componentId = topologyContext.GetThisComponentId();
- Context.Logger.Info("componentId: {0}", componentId);
- List<int> componentTasks = topologyContext.GetComponentTasks(componentId);
- Context.Logger.Info("taskNum: {0}", componentTasks.Count);
-}
-```
-
-### Dynamic part
-
-The following interfaces are pertinent to a certain **Context** instance, which is created by the SCP.NET platform and passed to your code:
-
-```csharp
-// Declare the Output and Input Stream Schemas
-
-public void DeclareComponentSchema(ComponentStreamSchema schema);
-
-// Emit tuple to default stream.
-public abstract void Emit(List<object> values);
-
-// Emit tuple to the specific stream.
-public abstract void Emit(string streamId, List<object> values);
-```
-
-For a nontransactional spout that supports acknowledgment, the following method is provided:
-
-```csharp
-// for nontransactional spout that supports ack
-public abstract void Emit(string streamId, List<object> values, long seqId);
-```
-
-A nontransactional bolt that supports acknowledgment should explicitly call **Ack** or **Fail** with the tuple it received. When emitting a new tuple, the bolt must also specify the tuple's anchors. The following methods are provided:
-
-```csharp
-public abstract void Emit(string streamId, IEnumerable<SCPTuple> anchors, List<object> values);
-public abstract void Ack(SCPTuple tuple);
-public abstract void Fail(SCPTuple tuple);
-```
-
-### StateStore
-
-The **StateStore** object provides metadata services, monotonic sequence generation, and wait-free coordination. You can build higher-level distributed concurrency abstractions on **StateStore**. These abstractions include distributed locks, distributed queues, barriers, and transaction services.
-
-SCP applications can use the **State** object to serialize information in [Apache ZooKeeper](https://zookeeper.apache.org/). This ability is especially valuable for a transactional topology. If a transactional spout stops responding and restarts, **State** can retrieve the necessary information from ZooKeeper and restart the pipeline.
-
-The **StateStore** object has these principal methods:
-
-```csharp
-/// <summary>
-/// Static method to retrieve a state store of the given path and connStr
-/// </summary>
-/// <param name="storePath">StateStore path</param>
-/// <param name="connStr">StateStore address</param>
-/// <returns>Instance of StateStore</returns>
-public static StateStore Get(string storePath, string connStr);
-
-/// <summary>
-/// Create a new state object in this state store instance
-/// </summary>
-/// <returns>State from StateStore</returns>
-public State Create();
-
-/// <summary>
-/// Retrieve all states that were previously uncommitted, excluding all exited states
-/// </summary>
-/// <returns>Uncommitted states</returns>
-public IEnumerable<State> GetUnCommitted();
-
-/// <summary>
-/// Get all the states in the StateStore
-/// </summary>
-/// <returns>All the states</returns>
-public IEnumerable<State> States();
-
-/// <summary>
-/// Get state or registry object
-/// </summary>
-/// <param name="info">Registry name (registry only)</param>
-/// <typeparam name="T">Type, registry or state</typeparam>
-/// <returns>Return registry or state</returns>
-public T Get<T>(string info = null);
-
-/// <summary>
-/// List all the committed states
-/// </summary>
-/// <returns>Registries containing the committed state </returns>
-public IEnumerable<Registry> Committed();
-
-/// <summary>
-/// List all the exited states in the StateStore
-/// </summary>
-/// <returns>Registries containing the exited states</returns>
-public IEnumerable<Registry> Aborted();
-
-/// <summary>
-/// Retrieve an existing state object from this state store instance
-/// </summary>
-/// <returns>State from StateStore</returns>
-/// <typeparam name="T">stateId, id of the State</typeparam>
-public State GetState(long stateId)
-```
-
-The **State** object has these principal methods:
-
-```csharp
-/// <summary>
-/// Set the status of the state object to commit
-/// </summary>
-public void Commit(bool simpleMode = true);
-
-/// <summary>
-/// Set the status of the state object to exit
-/// </summary>
-public void Abort();
-
-/// <summary>
-/// Put an attribute value under the given key
-/// </summary>
-/// <param name="key">Key</param>
-/// <param name="attribute">State attribute</param>
- public void PutAttribute<T>(string key, T attribute);
-
-/// <summary>
-/// Get the attribute value associated with the given key
-/// </summary>
-/// <param name="key">Key</param>
-/// <returns>State attribute</returns>
- public T GetAttribute<T>(string key);
-```
-
-When **simpleMode** is set to **true**, the **Commit** method deletes the corresponding ZNode in ZooKeeper. Otherwise, the method deletes the current ZNode and adds a new node in the COMMITTED\_PATH.
-
-### SCPRuntime
-
-The **SCPRuntime** class provides the following two methods:
-
-```csharp
-public static void Initialize();
-
-public static void LaunchPlugin(newSCPPlugin createDelegate);
-```
-
-The **Initialize** method initializes the SCP runtime environment. In this method, the C# process connects to the Java side to get configuration parameters and topology context.
-
-The **LaunchPlugin** method starts the message-processing loop. In this loop, the C# plug-in receives messages from the Java side. These messages include tuples and control signals. The plug-in then processes the messages, perhaps by calling the interface method provided by your code.
-
-The input parameter for **LaunchPlugin** is a delegate. The method can return an object that implements the **ISCPSpout**, **ISCPBolt**, **ISCPTxSpout**, or **ISCPBatchBolt** interface.
-
-```csharp
-public delegate ISCPPlugin newSCPPlugin(Context ctx, Dictionary<string, Object> parms);
-```
-
-For **ISCPBatchBolt**, you can get a **StormTxAttempt** object from the *parms* parameter and use it to judge whether the attempt is a replayed attempt. The check for a replay attempt is often done at the commit bolt. The HelloWorldTx example later in this article demonstrates this check.
-
-SCP plug-ins can usually run in two modes: local test mode and regular mode.
-
-#### Local test mode
-
-In this mode, the SCP plug-ins in your C# code run inside Visual Studio during the development phase. You can use the **ILocalContext** interface in this mode. The interface provides methods to serialize the emitted tuples to local files and read them back into RAM.
-
-```csharp
-public interface ILocalContext
-{
- List<SCPTuple> RecvFromMsgQueue();
- void WriteMsgQueueToFile(string filepath, bool append = false);
- void ReadFromFileToMsgQueue(string filepath);
-}
-```
-
-#### Regular mode
-
-In this mode, the Storm Java process runs the SCP plug-ins. Here's an example:
-
-```csharp
-namespace Scp.App.HelloWorld
-{
-public class Generator : ISCPSpout
-{
- … …
- public static Generator Get(Context ctx, Dictionary<string, Object> parms)
- {
- return new Generator(ctx);
- }
-}
-
-class HelloWorld
-{
- static void Main(string[] args)
- {
- /* Setting the environment variable here can change the log file name */
- System.Environment.SetEnvironmentVariable("microsoft.scp.logPrefix", "HelloWorld");
-
- SCPRuntime.Initialize();
- SCPRuntime.LaunchPlugin(new newSCPPlugin(Generator.Get));
- }
-}
-}
-```
-
-## Topology Specification language
-
-SCP Topology Specification is a domain-specific language (DSL) for describing and configuring SCP topologies. It's based on [Storm's Clojure DSL](https://storm.incubator.apache.org/documentation/Clojure-DSL.html) and is extended by SCP.
-
-You can submit topology specifications directly to a Storm cluster for execution via the **runSpec** command.
-
-SCP.NET has added the following functions to define transactional topologies:
-
-| New function | Parameters | Description |
-| | | |
-| **tx-topolopy** |*topology-name*<br />*spout-map*<br />*bolt-map* |Defines a transactional topology with the topology name, spouts definition map, and bolts definition map. |
-| **scp-tx-spout** |*exec-name*<br />*args*<br />*fields* |Defines a transactional spout. The function runs the application that's specified by *exec-name* and uses *args*.<br /><br />The *fields* parameter specifies the output fields for the spout. |
-| **scp-tx-batch-bolt** |*exec-name*<br />*args*<br />*fields* |Defines a transactional batch bolt. The function runs the application that's specified by *exec-name* and uses *args.*<br /><br />The *fields* parameter specifies the output fields for the bolt. |
-| **scp-tx-commit-bolt** |*exec-name*<br />*args*<br />*fields* |Defines a transactional commit bolt. The function runs the application that's specified by *exec-name* and uses *args*.<br /><br />The *fields* parameter specifies the output fields for the bolt. |
-| **nontx-topology** |*topology-name*<br />*spout-map*<br />*bolt-map* |Defines a nontransactional topology with the topology name, spouts definition map, and bolts definition map. |
-| **scp-spout** |*exec-name*<br />*args*<br />*fields*<br />*parameters* |Defines a nontransactional spout. The function runs the application that's specified by *exec-name* and uses *args*.<br /><br />The *fields* parameter specifies the output fields for the spout.<br /><br />The *parameters* parameter is optional. Use it to specify parameters like "nontransactional.ack.enabled". |
-| **scp-bolt** |*exec-name*<br />*args*<br />*fields*<br />*parameters* |Defines a nontransactional bolt. The function runs the application that's specified by *exec-name* and uses *args*.<br /><br />The *fields* parameter specifies the output fields for the bolt<br /><br />The *parameters* parameter is optional. Use it to specify parameters like "nontransactional.ack.enabled". |
-
-SCP.NET defines the following keywords:
-
-| Keyword | Description |
-| | |
-| **:name** |The topology name |
-| **:topology** |The topology using the functions in the previous table and built-in functions |
-| **:p** |The parallelism hint for each spout or bolt |
-| **:config** |Whether to configure parameters or update the existing ones |
-| **:schema** |The schema of the stream |
-
-SCP.NET also defines these frequently used parameters:
-
-| Parameter | Description |
-| | |
-| "plugin.name" |The .exe file name of the C# plug-in |
-| "plugin.args" |The plug-in arguments |
-| "output.schema" |The output schema |
-| "nontransactional.ack.enabled" |Whether acknowledgment is enabled for a nontransactional topology |
-
-The **runSpec** command is deployed together with the bits. Here is the command usage:
-
-```csharp
-.\bin\runSpec.cmd
-usage: runSpec [spec-file target-dir [resource-dir] [-cp classpath]]
-ex: runSpec examples\HelloWorld\HelloWorld.spec specs examples\HelloWorld\Target
-```
-
-The *resource-dir* parameter is optional. Specify it when you want to plug in a C# application. The specified directory contains the application, dependencies, and configurations.
-
-The *classpath* parameter is also optional. It specifies the Java classpath if the specification file contains a Java spout or bolt.
-
-## Miscellaneous features
-
-### Input and output schema declarations
-
-Your C# processes can emit tuples. To do so, the platform serializes tuples into **byte[]** objects and transfers the objects to the Java side. Storm then transfers these tuples to the targets.
-
-In downstream components, C# processes receive tuples back from the Java side and convert them to the platform's original types. All these operations are hidden by the platform.
-
-To support serialization and deserialization, your code needs to declare the schema of the input and output. The schema is defined as a dictionary. The stream ID is the dictionary key. The key value is the types of the columns. A component can declare multiple streams.
-
-```csharp
-public class ComponentStreamSchema
-{
- public Dictionary<string, List<Type>> InputStreamSchema { get; set; }
- public Dictionary<string, List<Type>> OutputStreamSchema { get; set; }
- public ComponentStreamSchema(Dictionary<string, List<Type>> input, Dictionary<string, List<Type>> output)
- {
- InputStreamSchema = input;
- OutputStreamSchema = output;
- }
-}
-```
-
-The following function is added to a **Context** object:
-
-```csharp
-public void DeclareComponentSchema(ComponentStreamSchema schema)
-```
-
-Developers must ensure that the emitted tuples obey the schema defined for a stream. Otherwise, the system will throw a runtime exception.
-
-### Multistream support
-
-SCP lets your code emit to or receive from multiple distinct streams at the same time. The **Context** object reflects this support as the **Emit** method's optional stream ID parameter.
-
-Two methods in the SCP.NET **Context** object have been added. They emit one or more tuples to specific streams. The *streamId* parameter is a string. Its value must be the same in both C# code and the topology-definition specification.
-
-```csharp
-/* Emit tuple to the specific stream. */
-public abstract void Emit(string streamId, List<object> values);
-
-/* for nontransactional spout only */
-public abstract void Emit(string streamId, List<object> values, long seqId);
-```
-
-Emitting to a nonexistent stream causes runtime exceptions.
-
-### Fields grouping
-
-The built-in fields grouping in Storm doesn't work properly in SCP.NET. On the Java proxy side, the data type of all fields is actually **byte[]**. The fields grouping uses the **byte[]** object's hash code to do the grouping. The hash code is the address of this object in RAM. So the grouping will be wrong for multibyte objects that share the same content but not the same address.
-
-SCP.NET adds a customized grouping method, and it uses the content of the **byte[]** object to do the grouping. In a specification file, the syntax looks like this example:
-
-```csharp
-(bolt-spec
- {
- "spout_test" (scp-field-group :non-tx [0,1])
- }
- …
-)
-```
-
-In the preceding specification file:
-
-* `scp-field-group` specifies that the grouping is a customized field grouping implemented by SCP.
-* `:tx` or `:non-tx` specifies whether the topology is transactional. You need this information because the starting index is different between transactional and nontransactional topologies.
-* `[0,1]` specifies a hash set of field IDs that start with zero.
-
-### Hybrid topology
-
-Native Storm code is written in Java. SCP.NET has enhanced Storm to let you write C# code to handle your business logic. But SCP.NET also supports hybrid topologies, which contain not only C# spouts/bolts but also Java spouts/bolts.
-
-### Specify Java spout/bolt in a specification file
-
-You can use **scp-spout** and **scp-bolt** in a specification file to specify Java spouts and bolts. Here's an example:
-
-```csharp
-(spout-spec
- (microsoft.scp.example.HybridTopology.Generator.)
- :p 1)
-```
-
-Here `microsoft.scp.example.HybridTopology.Generator` is the name of the Java spout class.
-
-### Specify the Java classpath in a runSpec command
-
-If you want to submit topology that contains Java spouts or bolts, first compile them to produce JAR files. Then specify the Java classpath that contains the JAR files when you submit topology. Here's an example:
-
-```csharp
-bin\runSpec.cmd examples\HybridTopology\HybridTopology.spec specs examples\HybridTopology\net\Target -cp examples\HybridTopology\java\target\*
-```
-
-Here, `examples\HybridTopology\java\target\` is the folder containing the Java spout/bolt JAR file.
-
-### Serialization and deserialization between Java and C#
-
-An SCP component includes the Java side and the C# side. To interact with native Java spouts/bolts, serialization and deserialization must occur between the Java side and the C# side, as illustrated in the following graph:
--
-#### Serialization in the Java side and deserialization in the C# side
-
-First provide the default implementation for serialization in the Java side and deserialization in the C# side.
-
-Specify the Java side's serialization method in a specification file.
-
-```csharp
-(scp-bolt
- {
- "plugin.name" "HybridTopology.exe"
- "plugin.args" ["displayer"]
- "output.schema" {}
- "customized.java.serializer" ["microsoft.scp.storm.multilang.CustomizedInteropJSONSerializer"]
- })
-```
-
-Specify the C# side's deserialization method in your C# code.
-
-```csharp
-Dictionary<string, List<Type>> inputSchema = new Dictionary<string, List<Type>>();
-inputSchema.Add("default", new List<Type>() { typeof(Person) });
-this.ctx.DeclareComponentSchema(new ComponentStreamSchema(inputSchema, null));
-this.ctx.DeclareCustomizedDeserializer(new CustomizedInteropJSONDeserializer());
-```
-
-If the data type isn't too complex, this default implementation should handle most cases. Here are cases where you can plug in your own implementation:
-
-* Your data type is too complex for the default implementation.
-* The performance of your default implementation doesn't meet your requirements.
-
-The serialization interface in the Java side is defined as:
-
-```csharp
-public interface ICustomizedInteropJavaSerializer {
- public void prepare(String[] args);
- public List<ByteBuffer> serialize(List<Object> objectList);
-}
-```
-
-The deserialization interface in the C# side is defined as:
-
-```csharp
-public interface ICustomizedInteropCSharpDeserializer
-{
- List<Object> Deserialize(List<byte[]> dataList, List<Type> targetTypes);
-}
-```
-
-#### Serialization in the C# side and deserialization in the Java side
-
-Specify the C# side's serialization method in your C# code.
-
-```csharp
-this.ctx.DeclareCustomizedSerializer(new CustomizedInteropJSONSerializer());
-```
-
-Specify the Java side's deserialization method in a specification file.
-
-```csharp
-(scp-spout
- {
- "plugin.name" "HybridTopology.exe"
- "plugin.args" ["generator"]
- "output.schema" {"default" ["person"]}
- "customized.java.deserializer" ["microsoft.scp.storm.multilang.CustomizedInteropJSONDeserializer" "microsoft.scp.example.HybridTopology.Person"]
- }
-)
-```
-
-Here, `"microsoft.scp.storm.multilang.CustomizedInteropJSONDeserializer"` is the name of the deserializer, and `"microsoft.scp.example.HybridTopology.Person"` is the target class the data is deserialized to.
-
-You can also plug in your own implementation of a C# serializer and a Java deserializer.
-
-This code is the interface for the C# serializer:
-
-```csharp
-public interface ICustomizedInteropCSharpSerializer
-{
- List<byte[]> Serialize(List<object> dataList);
-}
-```
-
-This code is the interface for the Java deserializer:
-
-```csharp
-public interface ICustomizedInteropJavaDeserializer {
- public void prepare(String[] targetClassNames);
- public List<Object> Deserialize(List<ByteBuffer> dataList);
-}
-```
-
-## SCP host mode
-
-In this mode, you can compile your code as a DLL and use SCPHost.exe as provided by SCP to submit a topology. A specification file looks like this code:
-
-```csharp
-(scp-spout
- {
- "plugin.name" "SCPHost.exe"
- "plugin.args" ["HelloWorld.dll" "Scp.App.HelloWorld.Generator" "Get"]
- "output.schema" {"default" ["sentence"]}
- })
-```
-
-Here, `"plugin.name"` is specified as `"SCPHost.exe"`, which is provided by the SCP SDK. SCPHost.exe accepts three parameters in the following order:
-
-1. The DLL name, which is `"HelloWorld.dll"` in this example.
-1. The class name, which is `"Scp.App.HelloWorld.Generator"` in this example.
-1. The name of a public static method, which can be invoked to get an instance of **ISCPPlugin**.
-
-In host mode, compile your code as a DLL for invocation by the SCP platform. Because the platform can then get full control of the whole processing logic, we recommend you submit topology in SCP host mode. Doing so simplifies the development experience. It also brings you more flexibility and better backward compatibility for later releases.
-
-## SCP programming examples
-
-### HelloWorld
-
-The following simple HelloWorld example shows a taste of SCP.NET. It uses a nontransactional topology with a spout called **generator** and two bolts called **splitter** and **counter**. The **generator** spout randomly generates sentences and emit these sentences to **splitter**. The **splitter** bolt splits the sentences into words and emits these words to the **counter** bolt. The **counter** bolt uses a dictionary to record the occurrence of each word.
-
-This example has two specification files: HelloWorld.spec and HelloWorld\_EnableAck.spec. The C# code can find out whether acknowledgment is enabled by getting the `pluginConf` object from the Java side.
-
-```csharp
-/* demo how to get pluginConf info */
-if (Context.Config.pluginConf.ContainsKey(Constants.NONTRANSACTIONAL_ENABLE_ACK))
-{
- enableAck = (bool)(Context.Config.pluginConf[Constants.NONTRANSACTIONAL_ENABLE_ACK]);
-}
-Context.Logger.Info("enableAck: {0}", enableAck);
-```
-
-If acknowledgment is enabled in the spout, a dictionary caches the tuples that haven't been acknowledged. If `Fail` is called, the failed tuple is replayed.
-
-```csharp
-public void Fail(long seqId, Dictionary<string, Object> parms)
-{
- Context.Logger.Info("Fail, seqId: {0}", seqId);
- if (cachedTuples.ContainsKey(seqId))
- {
- /* get the cached tuple */
- string sentence = cachedTuples[seqId];
-
- /* replay the failed tuple */
- Context.Logger.Info("Re-Emit: {0}, seqId: {1}", sentence, seqId);
- this.ctx.Emit(Constants.DEFAULT_STREAM_ID, new Values(sentence), seqId);
- }
- else
- {
- Context.Logger.Warn("Fail(), can't find cached tuple for seqId {0}!", seqId);
- }
-}
-```
-
-### HelloWorldTx
-
-The following HelloWorldTx example demonstrates how to implement transactional topology. The example has one spout called **generator**, a batch bolt called **partial-count**, and a commit bolt called **count-sum**. The example also has three existing text files: DataSource0.txt, DataSource1.txt, and DataSource2.txt.
-
-In each transaction, the **generator** spout randomly selects two files from the existing three files and emits the two file names to the **partial-count** bolt. The **partial-count** bolt:
-
-1. Gets a file name from the received tuple.
-1. Opens the corresponding file.
-1. Counts the number of words in the file.
-1. Emits the word count to the **count-sum** bolt.
-
-The **count-sum** bolt summarizes the total count.
-
-To achieve exactly once semantics, the **count-sum** commit bolt needs to judge whether it's a replayed transaction. In this example, it has the following static member variable:
-
-```csharp
-public static long lastCommittedTxId = -1;
-```
-
-When an **ISCPBatchBolt** instance is created, it gets the value of the `txAttempt` object from input parameters.
-
-```csharp
-public static CountSum Get(Context ctx, Dictionary<string, Object> parms)
-{
- /* for transactional topology, we can get txAttempt from the input parms */
- if (parms.ContainsKey(Constants.STORM_TX_ATTEMPT))
- {
- StormTxAttempt txAttempt = (StormTxAttempt)parms[Constants.STORM_TX_ATTEMPT];
- return new CountSum(ctx, txAttempt);
- }
- else
- {
- throw new Exception("null txAttempt");
- }
-}
-```
-
-When `FinishBatch` is called, `lastCommittedTxId` is updated if it isn't a replayed transaction.
-
-```csharp
-public void FinishBatch(Dictionary<string, Object> parms)
-{
- /* judge whether it is a replayed transaction */
- bool replay = (this.txAttempt.TxId <= lastCommittedTxId);
-
- if (!replay)
- {
- /* If it is not replayed, update the totalCount and lastCommittedTxId value */
- totalCount = totalCount + this.count;
- lastCommittedTxId = this.txAttempt.TxId;
- }
- … …
-}
-```
-
-### HybridTopology
-
-This topology contains a Java spout and a C# bolt. It uses the default serialization and deserialization implementation provided by the SCP platform. See the file HybridTopology.spec in the examples\\HybridTopology folder for the specification file details. Also see SubmitTopology.bat for how to specify the Java classpath.
-
-### SCPHostDemo
-
-This example is in essence the same as HelloWorld. The only difference is that your code is compiled as a DLL and the topology is submitted by using SCPHost.exe. See the SCP host mode section for a more detailed explanation.
-
-## Next steps
-
-For examples of Apache Storm topologies created using SCP, see the following articles:
-
-* [Develop C# topologies for Apache Storm on HDInsight using Visual Studio](apache-storm-develop-csharp-visual-studio-topology.md)
-* [Process events from Azure Event Hubs with Apache Storm on HDInsight](apache-storm-develop-csharp-event-hub-topology.md)
-* [Process vehicle sensor data from Event Hubs using Apache Storm on HDInsight](https://github.com/hdinsight/hdinsight-storm-examples/tree/master/IotExample)
-* [Extract, transform, and load (ETL) from Azure Event Hubs to Apache HBase](https://github.com/hdinsight/hdinsight-storm-examples/blob/master/RealTimeETLExample)
hdinsight Apache Storm Write Data Lake Store https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-storm-write-data-lake-store.md
- Title: 'Tutorial: HDInsight Apache Storm to Storage - Azure/Data Lake'
-description: Tutorial - Learn how to use Apache Storm to write to the HDFS-compatible storage for Azure HDInsight.
--- Previously updated : 06/24/2019--
-# Tutorial: Write to Apache Hadoop HDFS from Apache Storm on Azure HDInsight
-
-This tutorial demonstrates how to use Apache Storm to write data to the HDFS-compatible storage used by Apache Storm on HDInsight. HDInsight can use both Azure Storage and Azure Data Lake Storage as HDFS-compatible storage. Storm provides an [HdfsBolt](https://storm.apache.org/releases/current/javadocs/org/apache/storm/hdfs/bolt/HdfsBolt.html) component that writes data to HDFS. This document provides information on writing to either type of storage from the HdfsBolt.
-
-The example topology used in this document relies on components that are included with Storm on HDInsight. It may require modification to work with Azure Data Lake Storage when used with other Apache Storm clusters.
-
-In this tutorial, you learn how to:
-
-> [!div class="checklist"]
-> * Configure the cluster with script action
-> * Build and package the topology
-> * Deploy and run the topology
-> * View output data
-> * Stop the topology
-
-## Prerequisites
-
-* [Java Developer Kit (JDK) version 8](/azure/developer/java/fundamentals/java-support-on-azure)
-
-* [Apache Maven](https://maven.apache.org/download.cgi) properly [installed](https://maven.apache.org/install.html) according to Apache. Maven is a project build system for Java projects.
-
-* An SSH client. For more information, see [Connect to HDInsight (Apache Hadoop) using SSH](../hdinsight-hadoop-linux-use-ssh-unix.md).
-
-* The [URI scheme](../hdinsight-hadoop-linux-information.md#URI-and-scheme) for your clusters primary storage. This would be `wasb://` for Azure Storage, `abfs://` for Azure Data Lake Storage Gen2 or `adl://` for Azure Data Lake Storage Gen1. If secure transfer is enabled for Azure Storage, the URI would be `wasbs://`. See also, [secure transfer](../../storage/common/storage-require-secure-transfer.md).
-
-### Example configuration
-
-The following YAML is an excerpt from the `resources/writetohdfs.yaml` file included in the example. This file defines the Storm topology using the [Flux](https://storm.apache.org/releases/current/flux.html) framework for Apache Storm.
-
-```yaml
-components:
- - id: "syncPolicy"
- className: "org.apache.storm.hdfs.bolt.sync.CountSyncPolicy"
- constructorArgs:
- - 1000
-
- # Rotate files when they hit 5 MB
- - id: "rotationPolicy"
- className: "org.apache.storm.hdfs.bolt.rotation.FileSizeRotationPolicy"
- constructorArgs:
- - 5
- - "MB"
-
- - id: "fileNameFormat"
- className: "org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat"
- configMethods:
- - name: "withPath"
- args: ["${hdfs.write.dir}"]
- - name: "withExtension"
- args: [".txt"]
-
- - id: "recordFormat"
- className: "org.apache.storm.hdfs.bolt.format.DelimitedRecordFormat"
- configMethods:
- - name: "withFieldDelimiter"
- args: ["|"]
-
-# spout definitions
-spouts:
- - id: "tick-spout"
- className: "com.microsoft.example.TickSpout"
- parallelism: 1
--
-# bolt definitions
-bolts:
- - id: "hdfs-bolt"
- className: "org.apache.storm.hdfs.bolt.HdfsBolt"
- configMethods:
- - name: "withConfigKey"
- args: ["hdfs.config"]
- - name: "withFsUrl"
- args: ["${hdfs.url}"]
- - name: "withFileNameFormat"
- args: [ref: "fileNameFormat"]
- - name: "withRecordFormat"
- args: [ref: "recordFormat"]
- - name: "withRotationPolicy"
- args: [ref: "rotationPolicy"]
- - name: "withSyncPolicy"
- args: [ref: "syncPolicy"]
-```
-
-This YAML defines the following items:
-
-* `syncPolicy`: Defines when files are synched/flushed to the file system. In this example, every 1000 tuples.
-* `fileNameFormat`: Defines the path and file name pattern to use when writing files. In this example, the path is provided at runtime using a filter, and the file extension is `.txt`.
-* `recordFormat`: Defines the internal format of the files written. In this example, fields are delimited by the `|` character.
-* `rotationPolicy`: Defines when to rotate files. In this example, no rotation is performed.
-* `hdfs-bolt`: Uses the previous components as configuration parameters for the `HdfsBolt` class.
-
-For more information on the Flux framework, see [https://storm.apache.org/releases/current/flux.html](https://storm.apache.org/releases/current/flux.html).
-
-## Configure the cluster
-
-By default, Storm on HDInsight does not include the components that `HdfsBolt` uses to communicate with Azure Storage or Data Lake Storage in Storm's classpath. Use the following script action to add these components to the `extlib` directory for Storm on your cluster:
-
-| Property | Value |
-|||
-|Script type |- Custom|
-|Bash script URI |`https://hdiconfigactions.blob.core.windows.net/linuxstormextlibv01/stormextlib.sh`|
-|Node type(s) |Nimbus, Supervisor|
-|Parameters |None|
-
-For information on using this script with your cluster, see the [Customize HDInsight clusters using script actions](./../hdinsight-hadoop-customize-cluster-linux.md) document.
-
-## Build and package the topology
-
-1. Download the example project from [https://github.com/Azure-Samples/hdinsight-storm-azure-data-lake-store](https://github.com/Azure-Samples/hdinsight-storm-azure-data-lake-store) to your development environment.
-
-2. From a command prompt, terminal, or shell session, change directories to the root of the downloaded project. To build and package the topology, use the following command:
-
- ```cmd
- mvn compile package
- ```
-
- Once the build and packaging completes, there is a new directory named `target`, that contains a file named `StormToHdfs-1.0-SNAPSHOT.jar`. This file contains the compiled topology.
-
-## Deploy and run the topology
-
-1. Use the following command to copy the topology to the HDInsight cluster. Replace `CLUSTERNAME` with the name of the cluster.
-
- ```cmd
- scp target\StormToHdfs-1.0-SNAPSHOT.jar sshuser@CLUSTERNAME-ssh.azurehdinsight.net:StormToHdfs-1.0-SNAPSHOT.jar
- ```
-
-1. Once the upload completes, use the following to connect to the HDInsight cluster using SSH. Replace `CLUSTERNAME` with the name of the cluster.
-
- ```cmd
- ssh sshuser@CLUSTERNAME-ssh.azurehdinsight.net
- ```
-
-1. Once connected, use the following command to create a file named `dev.properties`:
-
- ```bash
- nano dev.properties
- ```
-
-1. Use the following text as the contents of the `dev.properties` file. Revise as needed based on your [URI scheme](../hdinsight-hadoop-linux-information.md#URI-and-scheme).
-
- ```
- hdfs.write.dir: /stormdata/
- hdfs.url: wasbs:///
- ```
-
- To save the file, use __Ctrl + X__, then __Y__, and finally __Enter__. The values in this file set the storage URL and the directory name that data is written to.
-
-1. Use the following command to start the topology:
-
- ```bash
- storm jar StormToHdfs-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --remote -R /writetohdfs.yaml --filter dev.properties
- ```
-
- This command starts the topology using the Flux framework by submitting it to the Nimbus node of the cluster. The topology is defined by the `writetohdfs.yaml` file included in the jar. The `dev.properties` file is passed as a filter, and the values contained in the file are read by the topology.
-
-## View output data
-
-To view the data, use the following command:
-
- ```bash
- hdfs dfs -ls /stormdata/
- ```
-
-A list of the files created by this topology is displayed. The following list is an example of the data returned by the previous commands:
-
-```output
-Found 23 items
--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-0-1561407909895.txt--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-1-1561407915577.txt--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-10-1561407943327.txt--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-11-1561407946312.txt--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-12-1561407949320.txt--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-13-1561407952662.txt--rw-r--r-- 1 storm supergroup 5242880 2019-06-24 20:25 /stormdata/hdfs-bolt-3-14-1561407955502.txt
-```
-
-## Stop the topology
-
-Storm topologies run until stopped, or the cluster is deleted. To stop the topology, use the following command:
-
-```bash
-storm kill hdfswriter
-```
-
-## Clean up resources
-
-To clean up the resources created by this tutorial, you can delete the resource group. Deleting the resource group also deletes the associated HDInsight cluster, and any other resources associated with the resource group.
-
-To remove the resource group using the Azure portal:
-
-1. In the Azure portal, expand the menu on the left side to open the menu of services, and then choose __Resource Groups__ to display the list of your resource groups.
-2. Locate the resource group to delete, and then right-click the __More__ button (...) on the right side of the listing.
-3. Select __Delete resource group__, and then confirm.
-
-## Next steps
-
-In this tutorial, you learned how to use Apache Storm to write data to the HDFS-compatible storage used by Apache Storm on HDInsight.
-
-> [!div class="nextstepaction"]
-> Discover other [Apache Storm examples for HDInsight](apache-storm-example-topology.md)
hdinsight Apache Troubleshoot Storm https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/apache-troubleshoot-storm.md
- Title: Troubleshoot Storm by using Azure HDInsight
-description: Get answers to common questions about using Apache Storm with Azure HDInsight.
-keywords: Azure HDInsight, Storm, FAQ, troubleshooting guide, common problems
-- Previously updated : 05/13/2022---
-# Troubleshoot Apache Storm by using Azure HDInsight
-
-Learn about the top issues and their resolutions for working with [Apache Storm](https://storm.apache.org/) payloads in [Apache Ambari](https://ambari.apache.org/).
-
-## How do I access the Storm UI on a cluster?
-
-You have two options for accessing the Storm UI from a browser:
-
-### Apache Ambari UI
-
-1. Go to the Ambari dashboard.
-2. In the list of services, select **Storm**.
-3. In the **Quick Links** menu, select **Storm UI**.
-
-### Direct link
-
-You can access the Storm UI at the following URL:
-
-`https://<cluster DNS name>/stormui`
-
-Example: `https://stormcluster.azurehdinsight.net/stormui`
-
-## How do I transfer Storm event hub spout checkpoint information from one topology to another?
-
-When you develop topologies that read from Azure Event Hubs by using the HDInsight Storm event hub spout .jar file, you must deploy a topology that has the same name on a new cluster. However,
-you must retain the checkpoint data that was committed to [Apache ZooKeeper](https://zookeeper.apache.org/) on the old cluster.
-
-### Where checkpoint data is stored
-
-Checkpoint data for offsets is stored by the event hub spout in ZooKeeper in two root paths:
--- Nontransactional spout checkpoints are stored in `/eventhubspout`.--- Transactional spout checkpoint data is stored in `/transactional`.-
-### How to restore
-
-To get the scripts and libraries that you use to export data out of ZooKeeper and then import the data back to ZooKeeper with a new name, see [HDInsight Storm examples](https://github.com/hdinsight/hdinsight-storm-examples/tree/master/tools/zkdatatool-1.0).
-
-The lib folder has .jar files that contain the implementation for the export/import operation. The bash folder has an example script that demonstrates how to export data from the ZooKeeper server on the old cluster, and then import it back to the ZooKeeper server on the new cluster.
-
-Run the [stormmeta.sh](https://github.com/hdinsight/hdinsight-storm-examples/blob/master/tools/zkdatatool-1.0/bash/stormmeta.sh) script from the ZooKeeper nodes to export and then import data. Update the script to the correct Hortonworks Data Platform (HDP) version. (We are working on making these scripts generic in HDInsight. Generic scripts can run from any node on the cluster without modifications by the user.)
-
-The export command writes the metadata to an Apache Hadoop Distributed File System (HDFS) path (in Azure Blob Storage or Azure Data Lake Storage) at a location that you set.
-
-### Examples
-
-#### Export offset metadata
-
-1. Use SSH to go to the ZooKeeper cluster on the cluster from which the checkpoint offset needs to be exported.
-2. Run the following command (after you update the HDP version string) to export ZooKeeper offset data to the `/stormmetadta/zkdata` HDFS path:
-
- ```apache
- java -cp ./*:/etc/hadoop/conf/*:/usr/hdp/2.5.1.0-56/hadoop/*:/usr/hdp/2.5.1.0-56/hadoop/lib/*:/usr/hdp/2.5.1.0-56/hadoop-hdfs/*:/usr/hdp/2.5.1.0-56/hadoop-hdfs/lib/*:/etc/failover-controller/conf/*:/etc/hadoop/* com.microsoft.storm.zkdatatool.ZkdataImporter export /eventhubspout /stormmetadata/zkdata
- ```
-
-#### Import offset metadata
-
-1. Use SSH to go to the ZooKeeper cluster on the cluster from which the checkpoint offset needs to be imported.
-2. Run the following command (after you update the HDP version string) to import ZooKeeper offset data from the HDFS path `/stormmetadata/zkdata` to the ZooKeeper server on the target cluster:
-
- ```apache
- java -cp ./*:/etc/hadoop/conf/*:/usr/hdp/2.5.1.0-56/hadoop/*:/usr/hdp/2.5.1.0-56/hadoop/lib/*:/usr/hdp/2.5.1.0-56/hadoop-hdfs/*:/usr/hdp/2.5.1.0-56/hadoop-hdfs/lib/*:/etc/failover-controller/conf/*:/etc/hadoop/* com.microsoft.storm.zkdatatool.ZkdataImporter import /eventhubspout /home/sshadmin/zkdata
- ```
-
-#### Delete offset metadata so that topologies can start processing data from the beginning, or from a timestamp that the user chooses
-
-1. Use SSH to go to the ZooKeeper cluster on the cluster from which the checkpoint offset needs to be deleted.
-2. Run the following command (after you update the HDP version string) to delete all ZooKeeper offset data in the current cluster:
-
- ```apache
- java -cp ./*:/etc/hadoop/conf/*:/usr/hdp/2.5.1.0-56/hadoop/*:/usr/hdp/2.5.1.0-56/hadoop/lib/*:/usr/hdp/2.5.1.0-56/hadoop-hdfs/*:/usr/hdp/2.5.1.0-56/hadoop-hdfs/lib/*:/etc/failover-controller/conf/*:/etc/hadoop/* com.microsoft.storm.zkdatatool.ZkdataImporter delete /eventhubspout
- ```
-
-## How do I locate Storm binaries on a cluster?
-
-Storm binaries for the current HDP stack are in `/usr/hdp/current/storm-client`. The location is the same both for head nodes and for worker nodes.
-
-There might be multiple binaries for specific HDP versions in /usr/hdp (for example, `/usr/hdp/2.5.0.1233/storm`). The `/usr/hdp/current/storm-client` folder is symlinked to the latest version that is running on the cluster.
-
-For more information, see [Connect to an HDInsight cluster by using SSH](../hdinsight-hadoop-linux-use-ssh-unix.md) and [Apache Storm](https://storm.apache.org/).
-
-## How do I determine the deployment topology of a Storm cluster?
-
-First, identify all components that are installed with HDInsight Storm. A Storm cluster consists of four node categories:
-
-* Gateway nodes
-* Head nodes
-* ZooKeeper nodes
-* Worker nodes
-
-### Gateway nodes
-
-A gateway node is a gateway and reverse proxy service that enables public access to an active Ambari management service. It also handles Ambari leader election.
-
-### Head nodes
-
-Storm head nodes run the following
-* Nimbus
-* Ambari server
-* Ambari Metrics server
-* Ambari Metrics Collector
-
-### ZooKeeper nodes
-
-HDInsight comes with a three-node ZooKeeper quorum. The quorum size is fixed, and cannot be reconfigured.
-
-Storm services in the cluster are configured to automatically use the ZooKeeper quorum.
-
-### Worker nodes
-
-Storm worker nodes run the following
-* Supervisor
-* Worker Java virtual machines (JVMs), for running topologies
-* Ambari agent
-
-## How do I locate Storm event hub spout binaries for development?
-
-For more information about using Storm event hub spout .jar files with your topology, see the following resources.
-
-### Java-based topology
-
-[Process events from Azure Event Hubs with Apache Storm on HDInsight (Java)](https://github.com/Azure-Samples/hdinsight-java-storm-eventhub)
-
-### C#-based topology (Mono on HDInsight 3.4+ Linux Storm clusters)
-
-[Process events from Azure Event Hubs with Apache Storm on HDInsight (C#)](./apache-storm-develop-csharp-event-hub-topology.md)
-
-### Latest Apache Storm event hub spout binaries for HDInsight 3.5+ Linux Storm clusters
-
-To learn how to use the latest Storm event hub spout that works with HDInsight 3.5+ Linux Storm clusters, see the [mvn-repo readme file](https://github.com/hdinsight/mvn-repo/blob/master/README.md).
-
-### Source code examples
-
-See [examples](https://github.com/Azure-Samples/hdinsight-java-storm-eventhub) of how to read and write from Azure Event Hub using an Apache Storm topology (written in Java) on an Azure HDInsight cluster.
-
-## How do I locate Storm Log4J 2 configuration files on clusters?
-
-To identify [Apache Log4j 2](https://logging.apache.org/log4j/2.x/) configuration files for Storm services.
-
-### On head nodes
-
-The Nimbus Log4J configuration is read from `/usr/hdp/\<HDP version>/storm/log4j2/cluster.xml`.
-
-### On worker nodes
-
-The supervisor Log4J configuration is read from `/usr/hdp/\<HDP version>/storm/log4j2/cluster.xml`.
-
-The worker Log4J configuration file is read from `/usr/hdp/\<HDP version>/storm/log4j2/worker.xml`.
-
-Examples:
-`/usr/hdp/2.6.0.2-76/storm/log4j2/cluster.xml`
-`/usr/hdp/2.6.0.2-76/storm/log4j2/worker.xml`
---
-## Not a leader exception
-
-When submitting a topology, user may receive an error message similar to: `Topology submission exception, cause not a leader, the current leader is NimbusInfo`.
-
-To resolve, user may need to file a ticket to have the nodes restarted/rebooted. For more information, see [https://community.hortonworks.com/content/supportkb/150287/error-ignoring-exception-while-trying-to-get-leade.html](https://community.hortonworks.com/content/supportkb/150287/error-ignoring-exception-while-trying-to-get-leade.html).
---
-## Next steps
-
-If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support:
--- Get answers from Azure experts through [Azure Community Support](https://azure.microsoft.com/support/community/).--- Connect with [@AzureSupport](https://twitter.com/azuresupport) - the official Microsoft Azure account for improving customer experience. Connecting the Azure community to the right resources: answers, support, and experts.--- If you need more help, you can submit a support request from the [Azure portal](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade/). Select **Support** from the menu bar or open the **Help + support** hub. For more detailed information, review [How to create an Azure support request](../../azure-portal/supportability/how-to-create-azure-support-request.md). Access to Subscription Management and billing support is included with your Microsoft Azure subscription, and Technical Support is provided through one of the [Azure Support Plans](https://azure.microsoft.com/support/plans/).
hdinsight Migrate Storm To Spark https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/hdinsight/storm/migrate-storm-to-spark.md
- Title: Migrate Azure HDInsight 3.6 Apache Storm to HDInsight 4.0 Apache Spark
-description: The differences and migration flow for migrating Apache Storm workloads to Spark Streaming or Spark Structured Streaming.
-- Previously updated : 01/16/2019-
-# Migrate Azure HDInsight 3.6 Apache Storm to HDInsight 4.0 Apache Spark
-
-This document describes how to migrate Apache Storm workloads on HDInsight 3.6 to HDInsight 4.0. HDInsight 4.0 doesn't support the Apache Storm cluster type and you will need to migrate to another streaming data platform. Two suitable options are Apache Spark Streaming and Spark Structured Streaming. This document describes the differences between these platforms and also recommends a workflow for migrating Apache Storm workloads.
-
-## Storm migration paths in HDInsight
-
-If you want to migrate from Apache Storm on HDInsight 3.6 you have multiple options:
-
-* Spark Streaming on HDInsight 4.0
-* Spark Structured Streaming on HDInsight 4.0
-* Azure Stream Analytics
-
-This document provides a guide for migrating from Apache Storm to Spark Streaming and Spark Structured Streaming.
--
-## Comparison between Apache Storm and Spark Streaming, Spark Structured Streaming
-
-Apache Storm can provide different levels of guaranteed message processing. For example, a basic Storm application can guarantee at-least-once processing, and [Trident](https://storm.apache.org/releases/current/Trident-API-Overview.html) can guarantee exactly once processing. Spark Streaming and Spark Structured Streaming guarantee that any input event is processed exactly once, even if a node failure occurs. Storm has a model that processes each single event, and you can also use the Micro Batch model with Trident. Spark Streaming and Spark Structured Streaming provide Micro-Batch processing model.
-
-| |Storm |Spark streaming | Spark structured streaming|
-|||||
-|**Event processing guarantee**|At least once <br> Exactly Once (Trident) |[Exactly Once](https://spark.apache.org/docs/latest/streaming-programming-guide.html)|[Exactly Once](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html)|
-|**Processing Model**|Real-time <br> Micro Batch (Trident) |Micro Batch |Micro Batch |
-|**Event time support**|[Yes](https://storm.apache.org/releases/2.0.0/Windowing.html)|No|[Yes](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html)|
-|**Languages**|Java, etc.|Scala, Java, Python|Python, R, Scala, Java, SQL|
-
-### Spark streaming vs Spark structured streaming
-
-Spark Structured Streaming is replacing Spark Streaming (DStreams). Structured Streaming will continue to receive enhancements and maintenance, while DStreams will be in maintenance mode only. Structured Streaming does not have as many features as DStreams for the sources and sinks that it supports out of the box, so evaluate your requirements to choose the appropriate Spark stream processing option.
-
-## Streaming (Single event) processing vs Micro-Batch processing
-
-Storm provides a model that processes each single event. This means that all incoming records will be processed as soon as they arrive. Spark Streaming applications must wait a fraction of a second to collect each micro-batch of events before sending that batch on for processing. In contrast, an event-driven application processes each event immediately. Spark Streaming latency is typically under a few seconds. The benefits of the micro-batch approach are more efficient data processing and simpler aggregate calculations.
--
-## Storm architecture and components
-
-Storm topologies are composed of multiple components that are arranged in a directed acyclic graph (DAG). Data flows between the components in the graph. Each component consumes one or more data streams, and can optionally emit one or more streams.
-
-|Component |Description |
-|||
-|Spout|Brings data into a topology. They emit one or more streams into the topology.|
-|Bolt|Consumes streams emitted from spouts or other bolts. Bolts might optionally emit streams into the topology. Bolts are also responsible for writing data to external services or storage, such as HDFS, Kafka, or HBase.|
--
-Storm consists of the following three daemons, which keep the Storm cluster functioning.
-
-|Daemon |Description |
-|||
-|Nimbus|Similar to Hadoop JobTracker, it's responsible for distributing code around the cluster and assigning tasks to machines and monitoring for failures.|
-|Zookeeper|Used for cluster coordination.|
-|Supervisor|Listens for work assigned to its machine and starts and stops worker processes based on directives from Nimbus. Each worker process executes a subset of a topology. User's application logic (Spouts and Bolt) run here.|
--
-## Spark Streaming architecture and components
-
-The following steps summarize how components work together in Spark Streaming (DStreams) and Spark Structured Streaming:
-
-* When Spark Streaming is launched, the driver launches the task in the executor.
-* The executor receives a stream from a streaming data source.
-* When the executor receives data streams, it splits the stream into blocks and keeps them in memory.
-* Blocks of data are replicated to other executors.
-* The processed data is then stored in the target data store.
--
-## Spark Streaming (DStream) workflow
-
-As each batch interval elapses, a new RDD is produced that contains all the data from that interval. The continuous sets of RDDs are collected into a DStream. For example, if the batch interval is one second long, your DStream emits a batch every second containing one RDD that contains all the data ingested during that second. When processing the DStream, the temperature event appears in one of these batches. A Spark Streaming application processes the batches that contain the events and ultimately acts on the data stored in each RDD.
--
-For details on the different transformations available with Spark Streaming, see [Transformations on DStreams](https://spark.apache.org/docs/latest/streaming-programming-guide.html#transformations-on-dstreams).
-
-## Spark Structured Streaming
-
-Spark Structured Streaming represents a stream of data as a table that is unbounded in depth. The table continues to grow as new data arrives. This input table is continuously processed by a long-running query, and the results are sent to an output table.
-
-In Structured Streaming, data arrives at the system and is immediately ingested into an input table. You write queries (using the DataFrame and Dataset APIs) that perform operations against this input table.
-
-The query output yields a *results table*, which contains the results of your query. You can draw data from the results table for an external datastore, such a relational database.
-
-The timing of when data is processed from the input table is controlled by the trigger interval. By default, the trigger interval is zero, so Structured Streaming tries to process the data as soon as it arrives. In practice, this means that as soon as Structured Streaming is done processing the run of the previous query, it starts another processing run against any newly received data. You can configure the trigger to run at an interval, so that the streaming data is processed in time-based batches.
---
-## General migration flow
-
-The recommended migration flow from Storm to Spark assumes the following initial architecture:
-
-* Kafka is used as the streaming data source
-* Kafka and Storm are deployed on the same virtual network
-* The data processed by Storm is written to a data sink, such as Azure Storage or Azure Data Lake Storage Gen2.
-
- :::image type="content" source="./media/migrate-storm-to-spark/presumed-current-environment.png" alt-text="diagram of presumed current environment" border="false":::
-
-To migrate your application from Storm to one of the Spark streaming APIs, do the following:
-
-1. **Deploy a new cluster.** Deploy a new HDInsight 4.0 Spark cluster in the same virtual network and deploy your Spark Streaming or Spark Structured Streaming application on it and test it thoroughly.
-
- :::image type="content" source="./media/migrate-storm-to-spark/new-spark-deployment.png" alt-text="new spark deployment in HDInsight" border="false":::
-
-1. **Stop consuming on the old Storm cluster.** In the existing Storm, stop consuming data from the streaming data source and wait it for the data to finish writing to the target sink.
-
- :::image type="content" source="./media/migrate-storm-to-spark/stop-consuming-current-cluster.png" alt-text="stop consuming on current cluster" border="false":::
-
-1. **Start consuming on the new Spark cluster.** Start streaming data from a newly deployed HDInsight 4.0 Spark cluster. At this time, the process is taken over by consuming from the latest Kafka offset.
-
- :::image type="content" source="./media/migrate-storm-to-spark/start-consuming-new-cluster.png" alt-text="start consuming on new cluster" border="false":::
-
-1. **Remove the old cluster as needed.** Once the switch is complete and working properly, remove the old HDInsight 3.6 Storm cluster as needed.
-
- :::image type="content" source="./media/migrate-storm-to-spark/remove-old-clusters1.png" alt-text="remove old HDInsight clusters as needed" border="false":::
-
-## Next steps
-
-For more information about Storm, Spark Streaming, and Spark Structured Streaming, see the following documents:
-
-* [Overview of Apache Spark Streaming](../spark/apache-spark-streaming-overview.md)
-* [Overview of Apache Spark Structured Streaming](../spark/apache-spark-structured-streaming-overview.md)
healthcare-apis Get Started With Iot https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/healthcare-apis/iot/get-started-with-iot.md
Previously updated : 07/19/2022 Last updated : 08/02/2022 # Get started with the MedTech service in the Azure Health Data Services
-This article outlines the basic steps to get started with the Azure MedTech service in the [Azure Health Data Services](../healthcare-apis-overview.md). The MedTech service ingests health data from a medical device using the Azure Event Hubs service. It then persists the data to the Azure Fast Healthcare Interoperability Resources (FHIR&#174;) service as Observation resources. This data processing procedure makes it possible to link FHIR service Observations to patient and device resources.
+This article will show you how to get started with the Azure MedTech service in the [Azure Health Data Services](../healthcare-apis-overview.md). There are six steps you need to follow to be able to deploy and process MedTech service to ingest health data from a medical device using Azure Event Hubs service, persist the data to Azure Fast Healthcare Interoperability Resources (FHIR&#174;) service as Observation resources, and link FHIR service Observations to patient and device resources. This article provides an architecture overview to help you follow the six steps of the implementation process.
-The following diagram shows the four-step data flow that enables the MedTech service to receive data from a device and send it to the FHIR service.
+## Architecture overview of MedTech service
-- Step 1 introduces the subscription and permissions prerequisites needed.
+The following diagram outlines the basic architectural path that enables the MedTech service to receive data from a medical device and send it to the FHIR service. This diagram shows how the six-step implementation process is divided into three key development stages: deployment, post-deployment, and data processing.
-- Step 2 shows how Azure services are provisioned for the MedTech services.
+[![Diagram showing MedTech service architectural overview.](media/iot-get-started/get-started-with-iot.png)](media/iot-get-started/get-started-with-iot.png#lightbox)
-- Step 3 represents the flow of data sent from devices to the event hub and the MedTech service.
+### Deployment
-- Step 4 demonstrates the path needed to verify data sent to the FHIR service.
+- Step 1 introduces the subscription and permissions prerequisites required.
-[![MedTech service data flow diagram.](media/iot-get-started/get-started-with-iot.png)](media/iot-get-started/get-started-with-iot.png#lightbox)
+- Step 2 shows how Azure services are provisioned for MedTech services.
-Follow these four steps and you'll be able to deploy the MedTech service effectively:
+- Step 3 presents the configuration process.
-## Step 1: Prerequisites for using the Azure Health Data Services
+### Post-deployment
-Before you can begin sending data from a device, you need to determine if you have the appropriate Azure subscription and Azure RBAC (Role-Based Access Control) roles. If you already have the appropriate subscription and roles, you can skip this step.
+- Step 4 outlines how to connect to other services.
+
+### Data processing
+
+- Step 5 represents the data flow from a device to an event hub and is processed through the five parts of the MedTech service.
+
+- Step 6 demonstrates the path to verify processed data sent from MedTech service to the FHIR service.
+
+## Get started implementing MedTech service
+
+Follow these six steps to set up and start using MedTech service.
+
+## Step 1: Prerequisites for deployment
+
+In order to begin deployment, you need to determine if you have: an Azure subscription and correct Azure RBAC (Role-Based Access Control) role assignments. If you already have the appropriate subscription and roles, you can skip this step.
- If you don't have an Azure subscription, see [Subscription decision guide](/azure/cloud-adoption-framework/decision-guides/subscriptions/). - You must have the appropriate RBAC roles for the subscription resources you want to use. The roles required for a user to complete the provisioning would be Contributor AND User Access Administrator OR Owner. The Contributor role allows the user to provision resources, and the User Access Administrator role allows the user to grant access so resources can send data between them. The Owner role can perform both. For more information, see [Azure role-based access control](/azure/cloud-adoption-framework/ready/considerations/roles).
-## Step 2: Provision services and obtain permissions
+## Step 2: Provision services for deployment
-After obtaining the required prerequisites, you must create a workspace and provision instances of the Event Hubs service, FHIR service, and MedTech service. You must also give the Event Hubs permission to read data from your device and give the MedTech service permission to read and write to the FHIR service.
+After obtaining the required prerequisites, the next phase of deployment is to create a workspace and provision instances of the Event Hubs service, FHIR service, and MedTech service. You must also give the Event Hubs permission to read data from your device and give the MedTech service permission to read and write to the FHIR service. There are four parts of this provisioning process.
### Create a resource group and workspace
You must provision a [FHIR service](../fhir/fhir-portal-quickstart.md) instance
Once the FHIR service is provisioned, you must give the MedTech service permission to read and write to FHIR service. This permission enables the data to be persisted in the FHIR service store using system-assigned managed identity. See details on how to assign the **FHIR Data Writer** role to the MedTech service from the [FHIR service](deploy-iot-connector-in-azure.md#granting-access-to-the-fhir-service).
-By design, the MedTech service retrieves data from the specified event hub using the system-assigned managed identity. For more information on how to assign the role to the MedTech service from [Event Hubs](deploy-iot-connector-in-azure.md#granting-access-to-the-device-message-event-hub).
+By design, the MedTech service retrieves data from the specified event hub using the system-assigned managed identity. For more information on how to assign the role to the MedTech service from [Event Hubs](deploy-iot-connector-in-azure.md#granting-access-to-the-device-message-event-hub).
### Provision a MedTech service instance in the workspace
You must provision a MedTech service instance from the [Azure portal](deploy-iot
The MedTech service persists the data to the FHIR store using the system-managed identity. See details on how to assign the role to the MedTech service from the [FHIR service](deploy-iot-connector-in-azure.md#granting-access-to-the-fhir-service).
-## Step 3: Send the data
+## Step 3: Configure MedTech for deployment
+
+After you have fulfilled the prerequisites and provisioned your services, the next phase of deployment is to configure MedTech services to ingest data, set up device mappings, and set up destination mappings. These configuration settings will ensure that the data can be translated from your device to Observations in the FHIR service. There are four parts in this configuration process.
+
+### Configuring MedTech service to ingest data
+
+MedTech service must be configured to ingest data it will receive from an event hub. First you must begin the official deployment process at the Azure portal. For more information about configuring MedTech service using the Azure portal, see [Deployment using the Azure portal](deploy-iot-connector-in-azure.md#prerequisites).
+
+Once you have starting using the portal and added MedTech service to your workspace, you must then configure MedTech service to ingest data from an event hub. For more information about configuring MedTech service to ingest data, see [Configure the MedTech service to ingest data](deploy-iot-connector-in-azure.md#configure-the-medtech-service-to-ingest-data).
+
+### Configuring device mappings
+
+You must configure MedTech to map it to the device you want to receive data from. Each device has unique settings that MedTech service must use. For more information on how to use Device mappings, see [How to use Device mappings](./how-to-use-device-mappings.md).
+
+- Azure Health Data Services provides an open source tool you can use called [IoMT Connector Data Mapper](https://github.com/microsoft/iomt-fhir/tree/main/tools/data-mapper) that will help you map your device's data structure to a form that MedTech can use. For more information on device content mapping, see [Device Content Mapping](https://github.com/microsoft/iomt-fhir/blob/main/docs/Configuration.md#device-content-mapping).
+
+- When you are deploying MedTech service, you must set specific device mapping properties. For more information on device mapping properties, see [Configure the Device mapping properties](deploy-iot-connector-in-azure.md#configure-the-device-mapping-properties).
+
+### Configuring destination mappings
+
+Once your device's data is properly mapped to your device's data format, you must then map it to an Observation in the FHIR service. For an overview of FHIR destination mappings, see [How to use the FHIR destination mappings](how-to-use-fhir-mappings.md).
+
+For step-by-step destination property mapping, see [Configure destination properties](deploy-iot-connector-in-azure.md#configure-destination-properties
+).
+
+### Create and deploy the MedTech service
+
+If you have completed the prerequisites, provisioning, and configuration, you are now ready to deploy the MedTech service. Create and deploy your MedTech service by following deployment the procedure at [Create your MedTech service](deploy-iot-connector-in-azure.md#create-your-medtech-service).
+
+## Step 4: Connect to required services (post deployment)
+
+When you complete the final [deployment procedure](deploy-iot-connector-in-azure.md#create-your-medtech-service) and don't get any errors, you must link MedTech service to an Event Hubs and the FHIR service. This will enable a connection from MedTech service to an Event Hubs instance and the FHIR service, so that data can flow smoothly from device to FHIR Observation. In order to do this, the Event Hubs instance for device message flow must be granted access via role assignment, so MedTech service can receive Event Hubs data. You must also grant access to The FHIR service via role assignments in order for MedTech to receive the data. There are two parts of the process to connect to required services.
+
+For more information about granting access via role assignments, see [Granting the MedTech service access to the device message event hub and FHIR service](deploy-iot-connector-in-azure.md#granting-the-medtech-service-access-to-the-device-message-event-hub-and-fhir-service).
+
+### Granting access to the device message event hub
+
+The Event Hubs instance for device message event hub must be granted access using managed identity in order for the MedTech service to receive data sent to the event hub from a device. The step-by-step procedure for doing this is at [Granting access to the device message event hub](deploy-iot-connector-in-azure.md#granting-access-to-the-device-message-event-hub).
+
+For more information about authorizing access to Event Hubs resources, see [Authorize access with Azure Active Directory](../../event-hubs/authorize-access-azure-active-directory.md).
+
+For more information about application roles, see [Authentication and Authorization for Azure Health Data Services](../authentication-authorization.md).
+
+### Granting access to FHIR service
+
+You must also grant access via role assignments to the FHIR service. This will enable FHIR service to receive data from the MedTech service by granting access using managed identity. The step-by-step procedure for doing this is at [Granting access to the FHIR service](deploy-iot-connector-in-azure.md#granting-access-to-the-fhir-service).
+
+For more information about assigning roles to the FHIR services, see [Configure Azure RBAC role for Azure Health Data Services](../configure-azure-rbac.md).
+
+For more information about application roles, see [Authentication and Authorization for Azure Health Data Services](../authentication-authorization.md).
+
+## Step 5: Send the data for processing
+
+When MedTech service is deployed and connected to the Event Hubs and FHIR services, it is ready to process data from a device and translate it into a FHIR service Observation. There are three parts of the sending process.
+
+### Data sent from Device to Event Hubs
+
+The data is sent to an Event Hub instance so that it can wait until MedTech service is ready to receive it. The data transfer needs to be asynchronous because it is sent over the Internet and delivery times cannot be precisely measured. Normally the data won't sit on an event hub longer than 24 hours.
+
+For more information about Event Hubs, see [Event Hubs](../../event-hubs/event-hubs-about.md).
+
+For more information on Event Hubs data retention, see [Event Hubs quotas](../../event-hubs/event-hubs-quotas.md)
+
+### Data Sent from Event Hubs to MedTech
+
+MedTech requests the data from the Event Hubs instance and the data is sent from the event hub to MedTech. This procedure is called ingestion.
+
+### MedTech processes the data
+
+MedTech processes the data in five steps:
-When the relevant services are provisioned, you can send event data from the device to MedTech service using an event hub. The event data is routed in the following manner:
+- Ingest
+- Normalize
+- Group
+- Transform
+- Persist
-- Data is sent from your device to the event hub.
+If the processing was successful and you did not get any error messages, your device data is now a FHIR service [Observation](http://hl7.org/fhir/observation.html) resource.
-- After the data is received by the event hub, MedTech service reads it. Then it transforms the data into a FHIR service [Observation](http://hl7.org/fhir/observation.html) resource using the data mapping you supplied.
+For more details on the data flow through MedTech, see [MedTech service data flow](iot-data-flow.md).
-## Step 4: Verify the data
+## Step 6: Verify the processed data
-If the data isn't mapped or if the mapping isn't authored properly, the data is skipped. If there are no problems with the [device mapping](./how-to-use-device-mappings.md) or the [FHIR destination mapping](how-to-use-fhir-mappings.md), the data is persisted in the FHIR service.
+You can verify that the data was processed correctly by checking to see if there is now a new Observation resource in the FHIR service. If the data isn't mapped or if the mapping isn't authored properly, the data will be skipped. If there are any problems, check the [device mapping](how-to-use-device-mappings.md) or the [FHIR destination mapping](how-to-use-fhir-mappings.md).
### Metrics
healthcare-apis How To Use Custom Functions https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/healthcare-apis/iot/how-to-use-custom-functions.md
Title: Custom Functions in the MedTech service - Azure Health Data Services
-description: This article describes how to use Custom Functions with MedTech service Device mappings templates.
+ Title: How to use custom functions in the MedTech service - Azure Health Data Services
+description: This article describes how to use custom functions with MedTech service device mapping.
Previously updated : 02/16/2022 Last updated : 08/05/2022
-# How to use Custom Functions
+# How to use custom functions
-> [!TIP]
-> Check out the [IoMT Connector Data Mapper](https://github.com/microsoft/iomt-fhir/tree/master/tools/data-mapper) tool for editing, testing, and troubleshooting the MedTech service Device and FHIR destination mappings. Export mappings for uploading to the MedTech service in the Azure portal or use with the [open-source version](https://github.com/microsoft/iomt-fhir) of the MedTech service.
-
-This article describes how to use the MedTech service Customer Functions.
-
-Many functions are available when using **JmesPath** as the expression language. Besides the functions available as part of the JmesPath specification, many custom functions may also be used. This article describes MedTech service-specific custom functions for use with the Device mappings template during the normalization process.
+Many functions are available when using **JmesPath** as the expression language. Besides the functions available as part of the JmesPath specification, many more custom functions may also be used. This article describes MedTech service-specific custom functions for use with the MedTech service device mapping during the device message normalization process.
-> [!TIP]
+> [!NOTE]
+>
> For more information on JmesPath functions, see the JmesPath [specification](https://jmespath.org/specification.html#built-in-functions).
+>[!TIP]
+>
+> Check out the [IoMT Connector Data Mapper](https://github.com/microsoft/iomt-fhir/tree/master/tools/data-mapper) tool for editing, testing, and troubleshooting the MedTech service Device and FHIR destination mappings. Export mappings for uploading to the MedTech service in the Azure portal or use with the [open-source version](https://github.com/microsoft/iomt-fhir) of the MedTech service.
+ ## Function signature Each function has a signature that follows the JmesPath specification. This signature can be represented as:
Examples:
## Next steps
-In this article, you learned how to use the MedTech service Custom Functions. To learn how to use Custom Functions with Device mappings, see
+In this article, you learned how to use the MedTech service custom functions. To learn how to use custom functions with the MedTech service device mapping, see
>[!div class="nextstepaction"]
->[How to use Device mappings](how-to-use-device-mappings.md)
+>[How to use device mappings](how-to-use-device-mappings.md)
-(FHIR&#174;) is a registered trademark of [HL7](https://hl7.org/fhir/) and is used with the permission of HL7.
+FHIR&#174; is a registered trademark of Health Level Seven International, registered in the U.S. Trademark Office and is used with their permission.
iot-edge Production Checklist https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/iot-edge/production-checklist.md
Additionally, the **Container engine** makes calls to container registries over
This checklist is a starting point for firewall rules:
- | FQDN (\* = wildcard) | Outbound TCP Ports | Usage |
+ | FQDN (`*` = wildcard) | Outbound TCP Ports | Usage |
| -- | -- | -- | | `mcr.microsoft.com` | 443 | Microsoft Container Registry |
- | `\*.data.mcr.microsoft.com` | 443 | Data endpoint providing content delivery |
+ | `*.data.mcr.microsoft.com` | 443 | Data endpoint providing content delivery |
| `*.cdn.azcr.io` | 443 | Deploy modules from the Marketplace to devices | | `global.azure-devices-provisioning.net` | 443 | [Device Provisioning Service](../iot-dps/about-iot-dps.md) access (optional) |
- | `\*.azurecr.io` | 443 | Personal and third-party container registries |
- | `\*.blob.core.windows.net` | 443 | Download Azure Container Registry image deltas from blob storage |
- | `\*.azure-devices.net` | 5671, 8883, 443<sup>1</sup> | IoT Hub access |
- | `\*.docker.io` | 443 | Docker Hub access (optional) |
+ | `*.azurecr.io` | 443 | Personal and third-party container registries |
+ | `*.blob.core.windows.net` | 443 | Download Azure Container Registry image deltas from blob storage |
+ | `*.azure-devices.net` | 5671, 8883, 443<sup>1</sup> | IoT Hub access |
+ | `*.docker.io` | 443 | Docker Hub access (optional) |
<sup>1</sup>Open port 8883 for secure MQTT or port 5671 for secure AMQP. If you can only make connections via port 443 then either of these protocols can be run through a WebSocket tunnel.
iot-hub Iot Hub Csharp Csharp File Upload https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/iot-hub/iot-hub-csharp-csharp-file-upload.md
At the end of this article you run two .NET console apps:
* A registered device. Register one in the [Azure portal](iot-hub-create-through-portal.md#register-a-new-device-in-the-iot-hub).
-* The sample applications you run in this article are written using C#. For the Azure IoT C# samples, we recommend you have the .NET Core SDK 3.1 or greater on your development machine.
+* The sample applications you run in this article are written using C# with .NET Core.
You can download the .NET Core SDK for multiple platforms from [.NET](https://dotnet.microsoft.com/download).
You can use the portal to view the uploaded file in the storage container you co
1. Select the folder named after your device. 1. Select the blob that you uploaded your file to. In this article, it's the blob named **TestPayload.txt**.
- :::image type="content" source="./media/iot-hub-csharp-csharp-file-upload/view-uploaded-file.png" alt-text="Screenshot of selecting the uploaded file in the Azure portal.":::
+ :::image type="content" source="./media/iot-hub-csharp-csharp-file-upload/view-uploaded-file.png" alt-text="Screenshot of selecting the uploaded file in the Azure portal." lightbox="./media/iot-hub-csharp-csharp-file-upload/view-uploaded-file.png":::
1. View the blob properties on the page that opens. You can select **Download** to download the file and view its contents locally.
iot-hub Iot Hub Devguide Messages Read Builtin https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/iot-hub/iot-hub-devguide-messages-read-builtin.md
The product integrations you can use with the built-in Event Hub-compatible endp
* [Azure Functions](../azure-functions/index.yml). See [Azure IoT Hub bindings for Azure Functions](../azure-functions/functions-bindings-event-iot.md). * [Azure Stream Analytics](../stream-analytics/index.yml). See [Stream data as input into Stream Analytics](../stream-analytics/stream-analytics-define-inputs.md#stream-data-from-iot-hub). * [Time Series Insights](../time-series-insights/index.yml). See [Add an IoT hub event source to your Time Series Insights environment](../time-series-insights/how-to-ingest-data-iot-hub.md).
-* [Apache Storm spout](../hdinsight/storm/apache-storm-develop-csharp-event-hub-topology.md). You can view the [spout source](https://github.com/apache/storm/tree/master/external/storm-eventhubs) on GitHub.
* [Apache Spark integration](../hdinsight/spark/apache-spark-ipython-notebook-machine-learning.md). * [Apache Kafka](https://kafka.apache.org/). For more information, see the [Apache Kafka developer guide for Event Hubs](../event-hubs/apache-kafka-developer-guide.md). * [Azure Databricks](/azure/azure-databricks/).
You can use the Event Hubs SDKs to read from the built-in endpoint in environmen
For more detail, see the [Process IoT Hub device-to-cloud messages using routes](tutorial-routing.md) tutorial.
-* If you want to route your device-to-cloud messages to custom endpoints, see [Use message routes and custom endpoints for device-to-cloud messages](iot-hub-devguide-messages-read-custom.md).
+* If you want to route your device-to-cloud messages to custom endpoints, see [Use message routes and custom endpoints for device-to-cloud messages](iot-hub-devguide-messages-read-custom.md).
iot-hub Iot Hub Java Java File Upload https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/iot-hub/iot-hub-java-java-file-upload.md
mvn exec:java -Dexec.mainClass="com.mycompany.app.App"
You can use the portal to view the uploaded file in the storage container you configured:
-![Uploaded file](media/iot-hub-java-java-upload/uploaded-file.png)
## Receive a file upload notification
In this section, you create a Java console app that receives file upload notific
4. Save and close the `pom.xml` file. 5. Get the IoT Hub service connection string.
- [!INCLUDE [iot-hub-include-find-service-connection-string](../../includes/iot-hub-include-find-service-connection-string.md)]
+ [!INCLUDE [iot-hub-include-find-service-connection-string](../../includes/iot-hub-include-find-service-connection-string.md)]
6. Using a text editor, open the `my-app\src\main\java\com\mycompany\app\App.java` file and replace the code with the following.
iot-hub Iot Hub Node Node File Upload https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/iot-hub/iot-hub-node-node-file-upload.md
You can use the portal to view the uploaded file in the storage container you co
1. Select the folder named after your device. 1. Select the blob that you uploaded your file to. In this article, it's the blob with the same name as your file.
- :::image type="content" source="./media/iot-hub-node-node-file-upload/view-uploaded-file.png" alt-text="Screenshot of viewing the uploaded file in the Azure portal.":::
+ :::image type="content" source="./media/iot-hub-node-node-file-upload/view-uploaded-file.png" alt-text="Screenshot of viewing the uploaded file in the Azure portal." lightbox="./media/iot-hub-node-node-file-upload/view-uploaded-file.png":::
1. View the blob properties on the page that opens. You can select **Download** to download the file and view its contents locally.
iot-hub Iot Hub Python Python File Upload https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/iot-hub/iot-hub-python-python-file-upload.md
At the end of this article, you run the Python console app:
## Prerequisites
-* Make sure that port 8883 is open in your firewall. The device sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see [Connecting to IoT Hub (MQTT)](iot-hub-mqtt-support.md#connecting-to-iot-hub).
-
-## Create an IoT hub
+* An IoT Hub. Create one with the [CLI](iot-hub-create-using-cli.md) or the [Azure portal](iot-hub-create-through-portal.md).
+* A registered device. Register one in the [Azure portal](iot-hub-create-through-portal.md#register-a-new-device-in-the-iot-hub).
-## Register a new device in the IoT hub
-
+* Make sure that port 8883 is open in your firewall. The device sample in this article uses MQTT protocol, which communicates over port 8883. This port may be blocked in some corporate and educational network environments. For more information and ways to work around this issue, see [Connecting to IoT Hub (MQTT)](iot-hub-mqtt-support.md#connecting-to-iot-hub).
[!INCLUDE [iot-hub-associate-storage](../../includes/iot-hub-include-associate-storage.md)]
Now you're ready to run the application.
2. The following screenshot shows the output from the **FileUpload** app:
- ![Output from simulated-device app](./media/iot-hub-python-python-file-upload/run-device-app.png)
+ :::image type="content" source="./media/iot-hub-python-python-file-upload/run-device-app.png" alt-text="Screenshot showing output from running the FileUpload app." border="true" lightbox="./media/iot-hub-python-python-file-upload/run-device-app.png":::
3. You can use the portal to view the uploaded file in the storage container you configured:
- ![Uploaded file](./media/iot-hub-python-python-file-upload/view-blob.png)
+ :::image type="content" source="./media/iot-hub-python-python-file-upload/view-blob.png" alt-text="Screenshot of the container in the Azure portal that shows the uploaded file." border="true" lightbox="./media/iot-hub-python-python-file-upload/view-blob.png":::
## Next steps
key-vault How To Configure Key Rotation https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/key-vault/keys/how-to-configure-key-rotation.md
Our recommendation is to rotate encryption keys at least every two years to meet
For more information about objects in Key Vault are versioned, see [Key Vault objects, identifiers, and versioning](../general/about-keys-secrets-certificates.md#objects-identifiers-and-versioning).
-## Integarion with Azure services
+## Integration with Azure services
This feature enables end-to-end zero-touch rotation for encryption at rest for Azure services with customer-managed key (CMK) stored in Azure Key Vault. Please refer to specific Azure service documentation to see if the service covers end-to-end rotation. For more information about data encryption in Azure, see:
load-balancer Load Balancer Floating Ip https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-balancer/load-balancer-floating-ip.md
If you want to reuse the backend port across multiple rules, you must enable Flo
When Floating IP is enabled, Azure changes the IP address mapping to the Frontend IP address of the Load Balancer frontend instead of backend instance's IP. Without Floating IP, Azure exposes the VM instances' IP. Enabling Floating IP changes the IP address mapping to the Frontend IP of the load Balancer to allow for more flexibility. Learn more [here](load-balancer-multivip-overview.md).
+In the diagrams below, you see how IP address mapping works before and after enabling Floating IP:
++ Floating IP can be configured on a Load Balancer rule via the Azure portal, REST API, CLI, PowerShell, or other client. In addition to the rule configuration, you must also configure your virtual machine's Guest OS in order to use Floating IP. ## Floating IP Guest OS configuration
Floating IP can be configured on a Load Balancer rule via the Azure portal, REST
In order to function, the Guest OS for the virtual machine needs to be configured to receive all traffic bound for the frontend IP and port of the load balancer. To accomplish this requires: * a loopback network interface to be added * configuring the loopback with the frontend IP address of the load balancer
-* ensure the system can send/receive packets on interfaces that do not have the IP address assigned to that interface (on Windows, this requires setting interfaces to use the "weak host" model; on Linux this model is normally used by default)
+* ensure the system can send/receive packets on interfaces that don't have the IP address assigned to that interface (on Windows, this requires setting interfaces to use the "weak host" model; on Linux this model is normally used by default)
The host firewall also needs to be open to receiving traffic on the frontend IP port. > [!NOTE]
load-balancer Load Balancer Ha Ports Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-balancer/load-balancer-ha-ports-overview.md
For NVA HA scenarios, HA ports offer the following advantages:
The following diagram presents a hub-and-spoke virtual network deployment. The spokes force-tunnel their traffic to the hub virtual network and through the NVA, before leaving the trusted space. The NVAs are behind an internal Standard Load Balancer with an HA ports configuration. All traffic can be processed and forwarded accordingly. When configured as show in the following diagram, an HA Ports load-balancing rule additionally provides flow symmetry for ingress and egress traffic. >[!NOTE] > If you are using NVAs, confirm with their providers how to best use HA ports and to learn which scenarios are supported.
load-balancer Load Balancer Tcp Reset https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-balancer/load-balancer-tcp-reset.md
# Load Balancer TCP Reset and Idle Timeout
-You can use [Standard Load Balancer](./load-balancer-overview.md) to create a more predictable application behavior for your scenarios by enabling TCP Reset on Idle for a given rule. Load Balancer's default behavior is to silently drop flows when the idle timeout of a flow is reached. Enabling this feature will cause Load Balancer to send bidirectional TCP Resets (TCP RST packet) on idle timeout. This will inform your application endpoints that the connection has timed out and is no longer usable. Endpoints can immediately establish a new connection if needed.
+You can use [Standard Load Balancer](./load-balancer-overview.md) to create a more predictable application behavior for your scenarios by enabling TCP Reset on Idle for a given rule. Load Balancer's default behavior is to silently drop flows when the idle timeout of a flow is reached. Enabling TCP reset will cause Load Balancer to send bidirectional TCP Resets (TCP RST packet) on idle timeout. This will inform your application endpoints that the connection has timed out and is no longer usable. Endpoints can immediately establish a new connection if needed.
-![Load Balancer TCP reset](media/load-balancer-tcp-reset/load-balancer-tcp-reset.png)
## TCP reset
You change this default behavior and enable sending TCP Resets on idle timeout o
Endpoints receiving TCP RST packets close the corresponding socket immediately. This provides an immediate notification to the endpoints that the release of the connection has occurred and any future communication on the same TCP connection will fail. Applications can purge connections when the socket closes and reestablish connections as needed without waiting for the TCP connection to eventually time out.
-For many scenarios, this may reduce the need to send TCP (or application layer) keepalives to refresh the idle timeout of a flow.
+For many scenarios, TCP reset may reduce the need to send TCP (or application layer) keepalives to refresh the idle timeout of a flow.
-If your idle durations exceed those of allowed by the configuration or your application shows an undesirable behavior with TCP Resets enabled, you may still need to use TCP keepalives (or application layer keepalives) to monitor the liveness of the TCP connections. Further, keepalives can also remain useful for when the connection is proxied somewhere in the path, particularly application layer keepalives.
+If your idle durations exceed configuration limits or your application shows an undesirable behavior with TCP Resets enabled, you may still need to use TCP keepalives, or application layer keepalives, to monitor the liveness of the TCP connections. Further, keepalives can also remain useful for when the connection is proxied somewhere in the path, particularly application layer keepalives.
-Carefully examine the entire end to end scenario to decide whether you benefit from enabling TCP Resets, adjusting the idle timeout, and if additional steps may be required to ensure the desired application behavior.
+By carefully examining the entire end to end scenario, you can determine the benefits from enabling TCP Resets and adjusting the idle timeout. Then you decide if more steps may be required to ensure the desired application behavior.
## Configurable TCP idle timeout
Azure Load Balancer has the following idle timeout range:
- 4 minutes to 100 minutes for Outbound Rules - 4 minutes to 30 minutes for Load Balancer rules and Inbound NAT rules
-By default, it is set to 4 minutes. If a period of inactivity is longer than the timeout value, there's no guarantee that the TCP or HTTP session is maintained between the client and your cloud service.
+By default, it's set to 4 minutes. If a period of inactivity is longer than the timeout value, there's no guarantee that the TCP or HTTP session is maintained between the client and your cloud service.
When the connection is closed, your client application may receive the following error message: "The underlying connection was closed: A connection that was expected to be kept alive was closed by the server."
TCP keep-alive works for scenarios where battery life isn't a constraint. It isn
## Limitations - TCP reset only sent during TCP connection in ESTABLISHED state.-- TCP idle timeout does not affect load balancing rules on UDP protocol.-- TCP reset is not supported for ILB HA ports when a network virtual appliance is in the path. A workaround could be to use outbound rule with TCP reset from NVA.
+- TCP idle timeout doesn't affect load balancing rules on UDP protocol.
+- TCP reset isn't supported for ILB HA ports when a network virtual appliance is in the path. A workaround could be to use outbound rule with TCP reset from NVA.
## Next steps
load-balancer Outbound Rules https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-balancer/outbound-rules.md
With outbound rules, you have full declarative control over outbound internet co
Outbound rules will only be followed if the backend VM doesn't have an instance-level public IP address (ILPIP).
-![Load Balancer outbound rules](media/load-balancer-outbound-rules-overview/load-balancer-outbound-rules.png)
With outbound rules, you can explicitly define outbound **SNAT** behavior.
Outbound rules follow the same familiar syntax as load balancing and inbound NAT
An outbound rule configures outbound NAT for _all virtual machines identified by the backend pool_ to be translated to the _frontend_.
-The _parameters_ provide additional fine grained control over the outbound NAT algorithm.
+The _parameters_ provide fine grained control over the outbound NAT algorithm.
## <a name="scale"></a> Scale outbound NAT with multiple IP addresses
-Each additional IP address provided by a frontend provides additional 64,000 ephemeral ports for load balancer to use as SNAT ports.
+Each extra IP address provided by a frontend provides another 64,000 ephemeral ports for load balancer to use as SNAT ports.
Use multiple IP addresses to plan for large-scale scenarios. Use outbound rules to mitigate [SNAT exhaustion](troubleshoot-outbound-connection.md#configure-load-balancer-outbound-rules-to-maximize-snat-ports-per-vm).
A public IP prefix increases scaling of your deployment. The prefix can be added
The load balancer has control over the public IP prefix. The outbound rule will automatically use all public IP addresses contained within the public IP prefix for outbound connections.
-Each of the IP addresses within public IP prefix provides an additional 64,000 ephemeral ports per IP address for load balancer to use as SNAT ports.
+Each of the IP addresses within public IP prefix provides an extra 64,000 ephemeral ports per IP address for load balancer to use as SNAT ports.
## <a name="idletimeout"></a> Outbound flow idle timeout and TCP reset
Use this scenario to tailor outbound connections to originate from a set of publ
This public IP or prefix can be the same as used by a load-balancing rule.
-To use a different public IP or prefix than used by a load-balancing rule:
+To use a different public IP or prefix than what is used by a load-balancing rule:
1. Create public IP prefix or public IP address. 2. Create a public standard load balancer 3. Create a frontend referencing the public IP prefix or public IP address you wish to use. 4. Reuse a backend pool or create a backend pool and place the VMs into a backend pool of the public load balancer
-5. Configure an outbound rule on the public load balancer to enable outbound NAT for the VMs using the frontend. It is not recommended to use a load-balancing rule for outbound, disable outbound SNAT on the load-balancing rule.
+5. Configure an outbound rule on the public load balancer to enable outbound NAT for the VMs using the frontend. It isn't recommended to use a load-balancing rule for outbound, disable outbound SNAT on the load-balancing rule.
### <a name="scenario2out"></a>Scenario 2: Modify [SNAT](load-balancer-outbound-connections.md) port allocation
Each public IP address contributes up to 64,000 ephemeral ports. The number of V
You can use outbound rules to tune the SNAT ports given by default. You give more or less than the default [SNAT](load-balancer-outbound-connections.md) port allocation provides. Each public IP address from a frontend of an outbound rule contributes up to 64,000 ephemeral ports for use as [SNAT](load-balancer-outbound-connections.md) ports.
-Load balancer gives [SNAT](load-balancer-outbound-connections.md) ports in multiples of 8. If you provide a value not divisible by 8, the configuration operation is rejected. Each load balancing rule and inbound NAT rule will consume a range of 8 ports. If a load balancing or inbound NAT rule shares the same range of 8 as another, no additional ports will be consumed.
+Load balancer gives [SNAT](load-balancer-outbound-connections.md) ports in multiples of 8. If you provide a value not divisible by 8, the configuration operation is rejected. Each load balancing rule and inbound NAT rule will consume a range of eight ports. If a load balancing or inbound NAT rule shares the same range of 8 as another, no extra ports will be consumed.
-If you attempt to give more [SNAT](load-balancer-outbound-connections.md) ports than are available based on the number of public IP addresses, the configuration operation is rejected. For example, if you give 10,000 ports per VM and seven VMs in a backend pool share a single public IP, the configuration is rejected. Seven multiplied by 10,000 exceeds the 64,000 port limit. Add more public IP addresses to the frontend of the outbound rule to enable the scenario.
+If you attempt to give out more [SNAT](load-balancer-outbound-connections.md) ports than are available (based on the number of public IP addresses), the configuration operation is rejected. For example, if you give 10,000 ports per VM and seven VMs in a backend pool share a single public IP, the configuration is rejected. Seven multiplied by 10,000 exceeds the 64,000 port limit. Add more public IP addresses to the frontend of the outbound rule to enable the scenario.
Revert to the [default port allocation](load-balancer-outbound-connections.md#preallocatedports) by specifying 0 for the number of ports. For more information on default SNAT port allocation, see [SNAT ports allocation table](./load-balancer-outbound-connections.md#preallocatedports).
Revert to the [default port allocation](load-balancer-outbound-connections.md#pr
#### Details
-Use a public standard load balancer to provide outbound NAT for a group of VMs. In this scenario, use an outbound rule by itself, without any additional rules configured.
+Use a public standard load balancer to provide outbound NAT for a group of VMs. In this scenario, use an outbound rule by itself, without configuring extra rules.
> [!NOTE] > **Azure Virtual Network NAT** can provide outbound connectivity for virtual machines without the need for a load balancer. See [What is Azure Virtual Network NAT?](../virtual-network/nat-gateway/nat-overview.md) for more information.
For more information, see [Outbound-only load balancer configuration](./egress-o
#### Details
-When using a public standard load balancer, the automatic outbound NAT provided matches the transport protocol of the load-balancing rule.
+With a public standard load balancer, the automatic outbound NAT provided matches the transport protocol of the load-balancing rule.
1. Disable outbound [SNAT](load-balancer-outbound-connections.md) on the load-balancing rule.
load-testing Concept Azure Load Testing Vnet Injection https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-testing/concept-azure-load-testing-vnet-injection.md
+
+ Title: Scenarios for VNET deployment
+
+description: Learn about the scenarios for deploying Azure Load Testing in a virtual network (VNET). This deployment enables you to load test private application endpoints and hybrid deployments.
+++++ Last updated : 08/03/2022++
+# Scenarios for deploying Azure Load Testing in a virtual network
+
+In this article, you'll learn about the scenarios for deploying Azure Load Testing Preview in a virtual network (VNET). This deployment is sometimes called VNET injection.
+
+This functionality enables the following usage scenarios:
+
+- Generate load to an [endpoint hosted in an Azure virtual network](#scenario-load-test-an-azure-hosted-private-endpoint).
+- Generate load to a [public endpoint with access restrictions](#scenario-load-test-a-public-endpoint-with-access-restrictions), such as restricting client IP addresses.
+- Generate load to an [on-premises service, not publicly accessible, that is connected to Azure via ExpressRoute (hybrid application deployment)](#scenario-load-test-an-on-premises-hosted-service-connected-via-azure-expressroute).
+
+When you deploy Azure Load Testing in a virtual network, the load test engine virtual machines are attached to the virtual network in your subscription. The load test engines can then communicate with the other resources in the virtual network, such as the private application endpoint. You are not billed for the test engine compute resources.
+
+> [!IMPORTANT]
+> When you deploy Azure Load Testing in a virtual network, you'll incur additional charges. Azure Load Testing deploys an [Azure Load Balancer](https://azure.microsoft.com/pricing/details/load-balancer/) and a [Public IP address](https://azure.microsoft.com/pricing/details/ip-addresses/) in your subscription and there might be a cost for generated traffic. For more information, see the [Virtual Network pricing information](https://azure.microsoft.com/pricing/details/virtual-network).
+
+The following diagram provides a technical overview:
++
+> [!IMPORTANT]
+> Azure Load Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
+
+## Scenario: Load test an Azure-hosted private endpoint
+
+In this scenario, you've deployed an application endpoint in a virtual network on Azure, which isn't publicly accessible. For example, the endpoint could be behind an internal load balancer, or running on a VM with a private IP address.
++
+When you deploy Azure Load Testing in the virtual network, the load test engines can now communicate with the application endpoint. If you've used separate subnets for the application endpoint and Azure Load Testing, make sure that communication between the subsets isn't blocked, for example by a network security group (NSG). Learn how [network security groups filter network traffic](/azure/virtual-network/network-security-group-how-it-works).
+
+## Scenario: Load test a public endpoint with access restrictions
+
+In this scenario, you've deployed a publicly available web service in Azure, or any other location. Access to the endpoint is restricted to specific client IP addresses. For example, the service could be running behind an [Azure Application Gateway](/azure/application-gateway/overview), hosted on [Azure App Service with access restrictions](/azure/app-service/app-service-ip-restrictions), or deployed behind a web application firewall.
++
+To restrict access to the endpoint for the load test engines, you need a range of public IP addresses for the test engine virtual machines. You deploy a [NAT Gateway resource](/azure/virtual-network/nat-gateway/nat-gateway-resource) in the virtual network, and then create and run a load test in the virtual network. A NAT gateway is a fully managed Azure service that provides source network address translation (SNAT).
+
+Attach the NAT gateway to the subnet in which the load test engines are injected. You can configure the public IP addresses used by the NAT gateway. These load test engine VMs use these IP addresses for generating load. You can then allowlist these IP addresses for restricting access to your application endpoint.
+
+## Scenario: Load test an on-premises hosted service, connected via Azure ExpressRoute
+
+In this scenario, you have an on-premises application endpoint, which isn't publicly accessible. The on-premises environment is connected to Azure by using Azure ExpressRoute.
++
+ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection with the help of a connectivity provider. Deploy Azure Load Testing in an Azure virtual network and then [connect the network to your ExpressRoute circuit](/azure/expressroute/expressroute-howto-linkvnet-portal-resource-manager). After you've set up the connection, the load test engines can connect to the on-premises hosted application endpoint.
+
+## Next steps
+
+- Learn how to [load test a private application endpoint](./how-to-test-private-endpoint.md).
+- Start using Azure Load Testing with the [Tutorial: Use a load test to identify performance bottlenecks](./tutorial-identify-bottlenecks-azure-portal.md).
load-testing How To Test Private Endpoint https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-testing/how-to-test-private-endpoint.md
+
+ Title: Load test private endpoints
+
+description: Learn how to deploy Azure Load Testing in a virtual network (VNET injection) to test private application endpoints and hybrid deployments.
++++ Last updated : 08/05/2022++++
+# Test private endpoints by deploying Azure Load Testing in an Azure virtual network
+
+In this article, learn how to test private application endpoints with Azure Load Testing Preview. You'll create an Azure Load Testing resource and enable it to generate load from within your virtual network (VNET injection).
+
+This functionality enables the following usage scenarios:
+
+- Generate load to an endpoint that is deployed in an Azure virtual network.
+- Generate load to a public endpoint with access restrictions, such as restricting client IP addresses.
+- Generate load to an on-premises service, not publicly accessible, that is connected to Azure via ExpressRoute.
+
+Learn more about the scenarios for [deploying Azure Load Testing in your virtual network](./concept-azure-load-testing-vnet-injection.md).
+
+The following diagram provides a technical overview:
++
+When you start the load test, Azure Load Testing service injects the following Azure resources in the virtual network that contains the application endpoint:
+
+- The test engine virtual machines. These VMs will invoke your application endpoint during the load test.
+- A public IP address.
+- A network security group (NSG).
+- An Azure Load Balancer.
+
+These resources are ephemeral and exist only for the duration of the load test run. If you restrict access to your virtual network, you need to [configure your virtual network](#configure-your-virtual-network) to enable communication between these Azure Load Testing and the injected VMs.
+
+> [!NOTE]
+> Virtual network support for Azure Load Testing is available in the following Azure regions: Australia East, East US, East US 2, and North Europe.
+
+> [!IMPORTANT]
+> Azure Load Testing is currently in preview. For legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability, see the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
+
+## Prerequisites
+
+- An existing virtual network and a subnet to use with Azure Load Testing.
+- The virtual network must be in the same subscription as the Azure Load Testing resource.
+- The subnet you use for Azure Load Testing must have enough unassigned IP addresses to accommodate the number of load test engines for your test. Learn more about [configuring your test for high-scale load](./how-to-high-scale-load.md).
+- The subnet shouldn't be delegated to any other Azure service. For example, it shouldn't be delegated to Azure Container Instances (ACI). Learn more about [subnet delegation](/azure/virtual-network/subnet-delegation-overview).
+
+## Configure your virtual network
+
+To test private endpoints, you need an Azure virtual network and at least one subnet. In this section, you'll configure your virtual network and subnet.
+
+### Create a subnet
+
+When you deploy Azure Load Testing in your virtual network, it's recommended to use different subnets for Azure Load Testing and the application endpoint. This approach enables you to configure network traffic access specifically for each purpose. Learn more about how to [add a subnet to a virtual network](/azure/virtual-network/virtual-network-manage-subnet#add-a-subnet).
+
+### Configure traffic access
+
+Azure Load Testing requires both inbound and outbound access for the injected VMs in your virtual network. If you plan to restrict traffic access to your virtual network, or are already using a network security group, configure the network security group for the subnet in which you deploy the load test.
+
+1. If you don't have an NSG yet, create one in the same region as your virtual network and associate it with your subnet. Follow these steps to [create a network security group](/azure/virtual-network/manage-network-security-group#create-a-network-security-group).
+
+1. Go to the [Azure portal](https://portal.azure.com) to view your network security groups. Search for and select **Network security groups**.
+
+1. Select the name of your network security group.
+
+1. Select **Inbound security rules** in the left navigation.
+
+1. Select **+ Add**, to add a new inbound security rule. Enter the following information to create a new rule, and then select **Add**.
+
+ | Field | Value |
+ | -- | -- |
+ | **Source** | *Service Tag* |
+ | **Source service tag** | *BatchNodeManagement* |
+ | **Source port ranges** | *\** |
+ | **Destination** | *Any* |
+ | **Destination port ranges** | *29876-29877* |
+ | **Name** | *batch-node-management-inbound* |
+ | **Description**| *Create, update, and delete of Azure Load Testing compute instances.* |
+
+1. Add a second inbound security rule using the following information:
+
+ | Field | Value |
+ | -- | -- |
+ | **Source** | *Service Tag* |
+ | **Source service tag** | *AzureLoadTestingInstanceManagement* |
+ | **Source port ranges** | *\** |
+ | **Destination** | *Any* |
+ | **Destination port ranges** | *8080* |
+ | **Name** | *azure-load-testing-inbound* |
+ | **Description**| *Create, update, and delete of Azure Load Testing compute instances.* |
+
+1. Select **Outbound security rules** in the left navigation.
+
+1. Select **+ Add**, to add a new outbound security rule. Enter the following information to create a new rule, and then select **Add**.
+
+ | Field | Value |
+ | -- | -- |
+ | **Source** | *Any* |
+ | **Source port ranges** | *\** |
+ | **Destination** | *Any* |
+ | **Destination port ranges** | *\** |
+ | **Name** | *azure-load-testing-outbound* |
+ | **Description**| *Used for various operations involved in orchestrating a load tests.* |
+
+## Configure your load test script
+
+The test engine VMs, which run the JMeter script, are injected in the virtual network that contains the application endpoint. You can now refer directly to the endpoint in the JMX file by using the private IP address or use [name resolution in your network](/azure/virtual-network/virtual-networks-name-resolution-for-vms-and-role-instances).
+
+For example, for an endpoint with IP address 10.179.0.7, in a virtual network with subnet range 10.179.0.0/18, the JMX file could have this information:
+
+```xml
+<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="Internal service homepage" enabled="true">
+ <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="Service homepage" enabled="true">
+ <collectionProp name="Arguments.arguments"/>
+ </elementProp>
+ <stringProp name="HTTPSampler.domain">10.179.0.7</stringProp>
+ <stringProp name="HTTPSampler.port">8081</stringProp>
+ <stringProp name="HTTPSampler.protocol"></stringProp>
+ <stringProp name="HTTPSampler.contentEncoding"></stringProp>
+ <stringProp name="HTTPSampler.path"></stringProp>
+ <stringProp name="HTTPSampler.method">GET</stringProp>
+</HTTPSamplerProxy>
+```
+
+## Set up a load test
+
+To create a load test for testing your private endpoint, you have to specify the virtual network details in the test creation wizard.
+
+1. Sign in to the [Azure portal](https://portal.azure.com) by using the credentials for your Azure subscription.
+
+1. Go to your Azure Load Testing resource, select **Tests** from the left pane, and then select **+ Create new test**.
+
+ :::image type="content" source="media/how-to-test-private-endpoint/create-new-test.png" alt-text="Screenshot that shows the Azure Load Testing page and the button for creating a new test.":::
+
+1. On the **Basics** tab, enter the **Test name** and **Test description** information. Optionally, you can select the **Run test after creation** checkbox.
+
+ :::image type="content" source="media/how-to-test-private-endpoint/create-new-test-basics.png" alt-text="Screenshot that shows the 'Basics' tab for creating a test.":::
+
+1. On the **Test plan** tab, select your Apache JMeter script, and then select **Upload** to upload the file to Azure.
+
+ You can select and upload other Apache JMeter configuration files or other files that are referenced in the JMX file. For example, if your test script uses CSV data sets, you can upload the corresponding *.csv* file(s).
+
+1. On the **Load** tab, select **Private** traffic mode, and then select your virtual network and subnet.
+
+ :::image type="content" source="media/how-to-test-private-endpoint/create-new-test-load-vnet.png" alt-text="Screenshot that shows the 'Load' tab for creating a test.":::
+
+ > [!IMPORTANT]
+ > When you deploy Azure Load Testing in a virtual network, you'll incur additional charges. Azure Load Testing deploys an [Azure Load Balancer](https://azure.microsoft.com/pricing/details/load-balancer/) and a [Public IP address](https://azure.microsoft.com/pricing/details/ip-addresses/) in your subscription and there might be a cost for generated traffic. For more information, see the [Virtual Network pricing information](https://azure.microsoft.com/pricing/details/virtual-network).
+
+1. Select **Review + create**. Review all settings, and then select **Create** to create the load test.
+
+## Troubleshooting
+
+### Starting the load test fails with `Test cannot be started`
+
+To start a load test, you must have sufficient permissions to deploy Azure Load Testing to the virtual network. You require the [Network Contributor](/azure/role-based-access-control/built-in-roles#network-contributor) role, or a parent of this role, on the virtual network. See [Check access for a user to Azure resources](/azure/role-based-access-control/check-access) to verify your permissions.
+
+If you're using the [Azure Load Testing REST API](/rest/api/loadtesting/) to start a load test, check that you're using a valid subnet ID. The subnet must be in the same Azure region as your Azure Load Testing resource.
+
+### The load test is stuck in `Provisioning` state and then goes to `Failed`
+
+1. Verify that your subscription is registered with `Microsoft.Batch`.
+
+ Run the following Azure CLI command to verify the status. The result should be `Registered`.
+
+ ```azurecli
+ az provider show --namespace Microsoft.Batch --query registrationState
+ ```
+
+1. Verify that Microsoft Batch node management and the Azure Load Testing IPs can make inbound connections to the test engine VMs.
+
+ 1. Enable [Network Watcher](/azure/network-watcher/network-watcher-monitoring-overview) for the virtual network region.
+
+ ```azurecli
+ az network watcher configure \
+ --resource-group NetworkWatcherRG \
+ --locations eastus \
+ --enabled
+ ```
+
+ 1. Create a temporary VM with a Public IP in the subnet you're using for the Azure Load Testing service. You'll only use this VM to diagnose the network connectivity and delete it afterwards. The VM can be of any type.
+
+ ```azurecli
+ az vm create \
+ --resource-group myResourceGroup \
+ --name myVm \
+ --image UbuntuLTS \
+ --generate-ssh-keys \
+ --subnet mySubnet
+ ```
+
+ 1. Test the inbound connectivity to the temporary VM from the `BatchNodeManagement` service tag.
+
+ 1. In the [Azure portal](https://portal.azure.com), go to **Network Watcher**.
+ 1. On the left pane, select **NSG Diagnostic**.
+ 1. Enter the details of the VM you created in the previous step.
+ 1. Select **Service Tag** for the **Source type**, and then select **BatchNodeManagement** for the **Service tag**.
+ 1. The **Destination IP address** is the IP address of the VM you created in previous step.
+ 1. For **Destination port**, you have to validate two ports: *29876* and *29877*. Enter one value at a time and move to the next step.
+ 1. Press **Check** to verify that the network security group isn't blocking traffic.
+
+ :::image type="content" source="media/how-to-test-private-endpoint/test-network-security-group-connectivity.png" alt-text="Screenshot that shows the NSG Diagnostic page to test network connectivity.":::
+
+ If the traffic status is **Denied**, [configure your virtual network](#configure-your-virtual-network) to allow traffic for the **BatchNodeManagement** service tag.
+
+ 1. Test the inbound connectivity to the temporary VM from the `AzureLoadTestingInstanceManagement` service tag.
+
+ 1. In the [Azure portal](https://portal.azure.com), go to **Network Watcher**.
+ 1. On the left pane, select **NSG Diagnostic**.
+ 1. Enter the details of the VM you created in the previous step.
+ 1. Select **Service Tag** for the **Source type**, and then select **AzureLoadTestingInstanceManagement** for the **Service tag**.
+ 1. The **Destination IP address** is the IP address of the VM you created in previous step.
+ 1. For **Destination port**, enter *8080*.
+ 1. Press **Check** to verify that the network security group isn't blocking traffic.
+
+ If the traffic status is **Denied**, [configure your virtual network](#configure-your-virtual-network) to allow traffic for the **AzureLoadTestingInstanceManagement** service tag.
+
+ 1. Delete the temporary VM you created earlier.
+
+### The test executes and results in a 100% error rate
+
+Possible cause: there are connectivity issues between the subnet in which you deployed Azure Load Testing and the subnet in which the application endpoint is hosted.
+
+1. You might deploy a temporary VM in the subnet used by Azure Load Testing and then use the [curl](https://curl.se/) tool to test connectivity to the application endpoint. Verify that there are no firewall or NSG rules that are blocking traffic.
+
+1. Verify the [Azure Load Testing results file](./how-to-export-test-results.md) for error response messages:
+
+ |Response message | Action |
+ |||
+ | **Non http response code java.net.unknownhostexception** | Possible cause is a DNS resolution issue. If youΓÇÖre using Azure Private DNS, verify that the DNS is set up correctly for the subnet in which Azure Load Testing instances are injected, and for the application subnet. |
+ | **Non http response code SocketTimeout** | Possible cause is when thereΓÇÖs a firewall blocking connections from the subnet in which Azure Load Testing instances are injected to your application subnet. |
+
+## Next steps
+
+- Learn more about the [scenarios for deploying Azure Load Testing in a virtual network](./concept-azure-load-testing-vnet-injection.md).
+- Learn how to [Monitor server-side application metrics](./how-to-monitor-server-side-metrics.md).
load-testing Overview What Is Azure Load Testing https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/load-testing/overview-what-is-azure-load-testing.md
For Azure-based applications, Azure Load Testing collects detailed resource metr
You can [automate regression testing](#enable-automated-load-testing) by running load tests as part of your continuous integration and continuous deployment (CI/CD) workflow.
+Azure Load Testing enables you to test private application endpoints or applications that you host on-premises. For more information, see the [scenarios for deploying Azure Load Testing in a virtual network](./concept-azure-load-testing-vnet-injection.md).
+ For more advanced load testing scenarios, you can [create a load test by reusing an existing Apache JMeter test script](how-to-create-and-run-load-test-with-jmeter-script.md), a popular open-source load and performance tool. For example, your test plan might consist of multiple application requests, or uses input data and parameters to make the test more dynamic. Learn more about the [key concepts for Azure Load Testing](./concept-load-testing-concepts.md).
Azure Load Testing test engines abstract the required infrastructure for [runnin
Azure Load Testing uses Apache JMeter version 5.4.3 for running load tests. You can use Apache JMeter plugins that are available on https://jmeter-plugins.org in your test script.
-The application can be hosted anywhere: in Azure, on-premises, or in other clouds. During the load test, the service collects the following resource metrics and displays them in a dashboard:
+The application can be hosted anywhere: in Azure, on-premises, or in other clouds. To load test services that have no public endpoint, [deploy Azure Load Testing in a virtual network](./how-to-test-private-endpoint.md).
+
+During the load test, the service collects the following resource metrics and displays them in a dashboard:
- *Client-side metrics* give you details reported by the test engine. These details include the number of virtual users, the request response time, or the number of requests per second.
machine-learning Concept Mlflow Models https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/concept-mlflow-models.md
with open(filename, 'wb') as f:
### Models
-A model in MLflow is also an artifact, as it matches the definition we introduced above. However, we make stronger assumptions about this type of artifacts. Such assumptions allow us to create a clear contract between the saved artifacts and what they mean. When you log your models as artifacts (simple files), you need to know what the model builder meant for each of them in order to know how to load the model for inference. When you log your models as a Model entity, you should be able to tell what it is based on the contract mentioned.
+A model in MLflow is also an artifact. However, we make stronger assumptions about this type of artifacts. Such assumptions provide a clear contract between the saved files and what they mean. When you log your models as artifacts (simple files), you need to know what the model builder meant for each of them in order to know how to load the model for inference. On the contrary, MLflow models can be loaded using the contract specified in the [The MLModel format](concept-mlflow-models.md#the-mlmodel-format).
-Logging models has the following advantages:
+In Azure Machine Learning, logging models has the following advantages:
> [!div class="checklist"]
-> * You don't need to provide an scoring script nor an environment for deployment.
-> * Swagger is enabled in endpoints automatically and the __Test__ feature can be used in Azure ML studio.
+> * You can deploy them on real-time or batch endpoints without providing an scoring script nor an environment.
+> * When deployed, Model's deployments have a Swagger generated automatically and the __Test__ feature can be used in Azure ML studio.
> * Models can be used as pipelines inputs directly.
-> * You can use the Responsable AI dashbord.
+> * You can use the [Responsible AI dashbord (preview)](how-to-responsible-ai-dashboard.md).
Models can get logged by: # [Using MLflow SDK](#tab/mlflow) ```python
+import mlflow
mlflow..sklearn.log_model(sklearn_estimator, "classifier") ```
mlflow..sklearn.log_model(sklearn_estimator, "classifier")
[!INCLUDE [sdk v1](../../includes/machine-learning-sdk-v1.md)]
-Logging models using Azure ML SDK v1 is not possible. We recommend to use MLflow SDK.
+> [!IMPORTANT]
+> Azure ML SDK v1 doesn't have the *model* concept.
# [Using the outputs folder](#tab/outputs)
mlflow.sklearn.save_model(sklearn_estimator, "outputs/classifier")
-## The MLModel format
+## The MLmodel format
-MLflow adopts the MLModel format as a way to create a contract between the artifacts and what they represent. The MLModel format stores assets in a folder. Among them, there is a particular file named MLModel. This file is the single source of truth about how a model can be loaded and used.
+MLflow adopts the MLmodel format as a way to create a contract between the artifacts and what they represent. The MLmodel format stores assets in a folder. Among them, there is a particular file named MLmodel. This file is the single source of truth about how a model can be loaded and used.
+
+![a sample MLflow model in MLmodel format](media/concept-mlflow-models/mlflow-mlmodel.png)
The following example shows how the `MLmodel` file for a computer version model trained with `fastai` may look like:
Signatures are indicated when the model gets logged and persisted in the `MLmode
There are two types of signatures:
-* **Column-based signature** corresponding to signatures that operate to tabular data. Models with this signature can expect to receive `pandas.DataFrame` objects as inputs.
-* **Tensor-based signature:** corresponding to signatures that operate with n-dimensional arrays or tensors. Models with this signature can expect to receive a `numpy.ndarray` as inputs (or a dictionary of `numpy.ndarray` in the case of named-tensors).
+* **Column-based signature** corresponding to signatures that operate to tabular data. For models with this signature, MLflow supplies `pandas.DataFrame` objects as inputs.
+* **Tensor-based signature:** corresponding to signatures that operate with n-dimensional arrays or tensors. For models with this signature, MLflow supplies `numpy.ndarray` as inputs (or a dictionary of `numpy.ndarray` in the case of named-tensors).
The following example corresponds to a computer vision model trained with `fastai`. This model receives a batch of images represented as tensors of shape `(300, 300, 3)` with the RGB representation of them (unsigned integers). It outputs batches of predictions (probabilities) for two classes.
signature:
``` > [!TIP]
-> Azure Machine Learning generates Swagger endpoints for MLflow models with a signature available. This makes easier to test deployed endpoints using the Azure ML studio.
+> Azure Machine Learning generates Swagger for model's deployment in MLflow format with a signature available. This makes easier to test deployed endpoints using the Azure ML studio.
### Model's environment
Models created as MLflow models can be loaded back directly from the run where t
There are two workflows available for loading models:
-* **Loading back the same object and types that were logged:**: You can load models using MLflow SDK and obtain an instance of the model with types belonging to the training library. For instance, an ONNX model will return a `ModelProto` while a decision tree trained with Scikit-Learn model will return a `DecisionTreeClassifier` object. Use `mlflow.<flavor>.load_model()` to do so.
-* **Loading back a model for running inference:** You can load models using MLflow SDK and obtain a wrapper where MLflow warranties there will be a `predict` function. It doesn't matter which flavor you are using, every MLflow model needs to implement this contract. Furthermore, MLflow warranties that this function can be called using arguments of type `pandas.DataFrame`, `numpy.ndarray` or `dict[strin, numpyndarray]` (depending on the signature of the model). MLflow handles the type conversion to the input type the model actually expects. Use `mlflow.pyfunc.load_model()` to do so.
+* **Loading back the same object and types that were logged:** You can load models using MLflow SDK and obtain an instance of the model with types belonging to the training library. For instance, an ONNX model will return a `ModelProto` while a decision tree trained with Scikit-Learn model will return a `DecisionTreeClassifier` object. Use `mlflow.<flavor>.load_model()` to do so.
+* **Loading back a model for running inference:** You can load models using MLflow SDK and obtain a wrapper where MLflow warranties there will be a `predict` function. It doesn't matter which flavor you are using, every MLflow model needs to implement this contract. Furthermore, MLflow warranties that this function can be called using arguments of type `pandas.DataFrame`, `numpy.ndarray` or `dict[string, numpyndarray]` (depending on the signature of the model). MLflow handles the type conversion to the input type the model actually expects. Use `mlflow.pyfunc.load_model()` to do so.
## Start logging models
machine-learning Concept Mlflow https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/concept-mlflow.md
[MLflow](https://www.mlflow.org) is an open-source framework, designed to manage the complete machine learning lifecycle. Its ability to train and serve models on different platforms allows you to use a consistent set of tools regardless of where your experiments are running: locally on your computer, on a remote compute target, a virtual machine or an Azure Machine Learning compute instance. > [!TIP]
-> Azure Machine Learning workspaces are MLflow compatible, meaning that you can use Azure Machine Learning workspaces in the same way you use an MLflow Tracking Server. Such compatibility has the following advantages:
+> Azure Machine Learning workspaces are MLflow-compatible, meaning that you can use Azure Machine Learning workspaces in the same way you use an MLflow Tracking Server. Such compatibility has the following advantages:
> * You can use Azure Machine Learning workspaces as your tracking server for any experiment you are running with MLflow, regardless if they run on Azure Machine Learning or not. You only need to configure MLflow to point to the workspace where the tracking should happen. > * You can run any training routine that uses MLflow in Azure Machine Learning without changes. Model mangagement and model deployment capabilities are also supported.
MLflow can manage the complete machine learning lifecycle using four core capabi
## Tracking with MLflow
-Azure Machine Learning uses MLflow Tracking for metric logging and artifact storage for your experiments, whether you created the experiment via the Azure Machine Learning Python SDK, Azure Machine Learning CLI or the Azure Machine Learning studio. Learn more at [Log & view metrics and log files with MLflow](how-to-log-view-metrics.md).
+Azure Machine Learning uses MLflow Tracking for metric logging and artifact storage for your experiments, whether you created the experiment via the Azure Machine Learning Python SDK, Azure Machine Learning CLI or the Azure Machine Learning studio. We recommend using MLflow for tracking experiments. To get you started, see [Log & view metrics and log files with MLflow](how-to-log-view-metrics.md).
> [!NOTE] > Unlike the Azure Machine Learning SDK v1, there's no logging functionality in the SDK v2 (preview), and it is recommended to use MLflow for logging and tracking.
-With MLflow Tracking you can connect Azure Machine Learning as the backend of your MLflow experiments. By doing so, you can:
+With MLflow Tracking you can connect Azure Machine Learning as the backend of your MLflow experiments. The workspace provides a centralized, secure, and scalable location to store training metrics and models. This includes:
-+ Track and log experiment metrics and artifacts in your [Azure Machine Learning workspace](./concept-azure-machine-learning-v2.md#workspace).
- + If you're using Azure Machine Learning computes, they're already configured to work with MLflow for tracking. Just import `mlflow` in your training routine and start using it.
- + Azure Machine Learning also supports remote tracking of experiments by configuring MLflow to point to the Azure Machine Learning workspace. By doing so, you can leverage the capabilities of Azure Machine Learning while keeping your experiments where they are.
-+ Lift and shift existing MLflow experiments to Azure Machine Learning. The workspace provides a centralized, secure, and scalable location to store training metrics and models.
+* [Track ML experiments and models running locally or in the cloud](how-to-use-mlflow-cli-runs.md) with MLflow in Azure Machine Learning.
+* [Track Azure Databricks ML experiments](how-to-use-mlflow-azure-databricks.md) with MLflow in Azure Machine Learning.
+* [Track Azure Synapse Analytics ML experiments](how-to-use-mlflow-azure-databricks.md) with MLflow in Azure Machine Learning.
> [!IMPORTANT]
-> - MLflow in R support is limited to tracking experiment's metrics and parameters on Azure Machine Learning jobs. Artifacts and models can't be tracked using the MLflow R SDK. You can save them locally and then have Azure Machine Learning to capture for you as a workaround. RStudio or Jupyter Notebooks with R kernels are not supported. View the following [R example about using the MLflow tracking client with Azure Machine Learning](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/r).
-> - MLflow in Java support is limited to tracking experiment's metrics and parameters on Azure Machine Learning jobs. Artifacts and models can't be tracked using the MLflow Java SDK. You can save them locally and then have Azure Machine Learning to capture for you as a workaround. View the following [Java example about using the MLflow tracking client with the Azure Machine Learning](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/java/iris).
+> - MLflow in R support is limited to tracking experiment's metrics and parameters on Azure Machine Learning jobs. RStudio or Jupyter Notebooks with R kernels are not supported. Artifacts and models can't be tracked using the MLflow R SDK. As an alternative, you can save them locally using [`mlflow_save_model.crate`](https://mlflow.org/docs/latest/R-api.html#mlflow-save-model-crate) in the `outputs` folder. Then, use Azure ML CLI or Azure ML studio for model registration. View the following [R example about using the MLflow tracking client with Azure Machine Learning](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/r).
+> - MLflow in Java support is limited to tracking experiment's metrics and parameters on Azure Machine Learning jobs. Artifacts and models can't be tracked using the MLflow Java SDK. View the following [Java example about using the MLflow tracking client with the Azure Machine Learning](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/single-step/java/iris).
+
+To learn how to use MLflow to query experiments and runs in Azure Machine Learning, see [Manage experiments and runs with MLflow](how-to-track-experiments-mlflow.md)
## Model Registries with MLflow
machine-learning Concept Responsible Ml https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/concept-responsible-ml.md
[!INCLUDE [dev v2](../../includes/machine-learning-dev-v2.md)]
-The societal implications of AI and the responsibility of organizations to anticipate and mitigate unintended consequences of AI technology are significant. Organizations are finding the need to create internal policies, practices, and tools to guide their AI efforts, whether they're deploying third-party AI solutions or developing their own. At Microsoft, we've recognized six principles that we believe should guide AI development and use: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. For us, these principles are the cornerstone of a responsible and trustworthy approach to AI, especially as intelligent technology becomes more prevalent in the products and services we use every day. Azure Machine Learning currently supports tools for various these principles, making it seamless for ML developers and data scientists to implement Responsible AI in practice.
+The societal implications of AI and the responsibility of organizations to anticipate and mitigate unintended consequences of AI technology are significant. Organizations are finding the need to create internal policies, practices, and tools to guide their AI efforts, whether they're deploying third-party AI solutions or developing their own. At Microsoft, we've recognized six principles that we believe should guide AI development and use: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. For us, these principles are the cornerstone of a responsible and trustworthy approach to AI, especially as intelligent technology becomes more prevalent in the products and services we use every day. Azure Machine Learning currently supports various tools for these principles, making it seamless for ML developers and data scientists to implement Responsible AI in practice.
:::image type="content" source="./media/concept-responsible-ml/concept-responsible-ml.png" alt-text="Responsible A I principles - fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability.":::
machine-learning How To Log View Metrics https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/how-to-log-view-metrics.md
Title: Log & view parameters, metrics and files with MLflow
description: Enable logging on your ML training runs to monitor real-time run metrics with MLflow, and to help diagnose errors and warnings. --++
machine-learning How To Manage Models https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/how-to-manage-models.md
The code snippets in this section cover how to:
These snippets use `custom` and `mlflow`. -- `custom` is a type that refers to a model file.
+- `custom` is a type that refers to a model file or folder trained with a custom standard not currently supported by Azure ML.
- `mlflow` is a type that refers to a model trained with [mlflow](how-to-use-mlflow-cli-runs.md). MLflow trained models are in a folder that contains the *MLmodel* file, the *model* file, the *conda dependencies* file, and the *requirements.txt* file. ### Register your model as an asset in Machine Learning by using the CLI
machine-learning How To Monitor Tensorboard https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/how-to-monitor-tensorboard.md
description: Launch TensorBoard to visualize experiment job histories and identi
--++ Last updated 10/21/2021
machine-learning How To Setup Authentication https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/how-to-setup-authentication.md
Authenticating with a service principal uses the [Azure Identity package for Pyt
* `AZURE_CLIENT_SECRET` - The password/credential generated for the service principal. > [!TIP]
-> During development, consider using the [python-dotenv](https://pypi.org/project/python-dotenv/) package to set these environment variables. Python-dotenv loads environment variables from `.env` files. The standard `.gitignore` file for Python automatically excludes `.env` files, so they shouldn't be checked into any github repos during development.
+> During development, consider using the [python-dotenv](https://pypi.org/project/python-dotenv/) package to set these environment variables. Python-dotenv loads environment variables from `.env` files. The standard `.gitignore` file for Python automatically excludes `.env` files, so they shouldn't be checked into any GitHub repos during development.
The following example demonstrates using python-dotenv to load the environment variables from a `.env` file and then using `DefaultAzureCredential` to create the credential object:
machine-learning How To Track Monitor Analyze Runs https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/how-to-track-monitor-analyze-runs.md
description: Learn how to start, monitor, and track your machine learning experi
--++ Last updated 06/24/2022
machine-learning Reference Machine Learning Cloud Parity https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/reference-machine-learning-cloud-parity.md
In the list of global Azure regions, there are several regions that serve specif
* Azure Government regions **US-Arizona** and **US-Virginia**. * Azure China 21Vianet region **China-East-2**.
-Azure Machine Learning is still in devlopment in Airgap Regions.
+Azure Machine Learning is still in development in Airgap Regions.
The information in the rest of this document provides information on what features of Azure Machine Learning are available in these regions, along with region-specific information on using these features. ## Azure Government
The information in the rest of this document provides information on what featur
| Interpretability SDK | GA | YES | YES | | **Training** | | | | | [Experimentation log streaming](how-to-track-monitor-analyze-runs.md) | GA | YES | YES |
-| [Reinforcement Learning](how-to-use-reinforcement-learning.md) | Public Preview | NO | NO |
+| [Reinforcement Learning](./v1/how-to-use-reinforcement-learning.md) | Public Preview | NO | NO |
| [Experimentation UI](how-to-track-monitor-analyze-runs.md) | Public Preview | YES | YES | | [.NET integration ML.NET 1.0](/dotnet/machine-learning/tutorials/object-detection-model-builder) | GA | YES | YES | | **Inference** | | | |
machine-learning Reference Yaml Job Sweep https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/reference-yaml-job-sweep.md
---++ Last updated 03/31/2022
The source JSON schema can be found at https://azuremlschemas.azureedge.net/late
| | - | -- | - | | `max_total_trials` | integer | The maximum time in seconds the job is allowed to run. Once this limit is reached the system will cancel the job. | `1000` | | `max_concurrent_trials` | integer | | Defaults to `max_total_trials`. |
-| `timeout` | integer | The maximum time in seconds the entire sweep job is allowed to run. Once this limit is reached the system will cancel the sweep job, including all its trials. | `100800` |
+| `timeout` | integer | The maximum time in seconds the entire sweep job is allowed to run. Once this limit is reached the system will cancel the sweep job, including all its trials. | `604800` |
| `trial_timeout` | integer | The maximum time in seconds each trial job is allowed to run. Once this limit is reached the system will cancel the trial. | | ### Attributes of the `trial` key
machine-learning How To Use Reinforcement Learning https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/machine-learning/v1/how-to-use-reinforcement-learning.md
+
+ Title: Train and deploy a reinforcement learning model (preview)
+
+description: Learn how to use Azure Machine Learning reinforcement learning (preview) to train an RL agent to play Pong.
+++++ Last updated : 08/05/2022++++
+# Reinforcement learning (preview) with Azure Machine Learning
++
+> [!WARNING]
+> Azure Machine Learning reinforcement learning via the [`azureml.contrib.train.rl`](/python/api/azureml-contrib-reinforcementlearning/azureml.contrib.train.rl) package will no longer be supported after June 2022. We recommend customers use the [Ray on Azure Machine Learning library](https://github.com/microsoft/ray-on-aml) for reinforcement learning experiments with Azure Machine Learning. For an example, see the notebook [Reinforcement Learning in Azure Machine Learning - Pong problem](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.ipynb).
+
+In this article, you learn how to train a reinforcement learning (RL) agent to play the video game Pong. You use the open-source Python library [Ray RLlib](https://docs.ray.io/en/master/rllib/) with Azure Machine Learning to manage the complexity of distributed RL.
+
+In this article you learn how to:
+> [!div class="checklist"]
+> * Set up an experiment
+> * Define head and worker nodes
+> * Create an RL estimator
+> * Submit an experiment to start a job
+> * View results
+
+This article is based on the [RLlib Pong example](https://aka.ms/azureml-rl-pong) that can be found in the Azure Machine Learning notebook [GitHub repository](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/reinforcement-learning/README.md).
+
+## Prerequisites
+
+Run this code in either of these environments. We recommend you try Azure Machine Learning compute instance for the fastest start-up experience. You can quickly clone and run the reinforcement sample notebooks on an Azure Machine Learning compute instance.
+
+ - Azure Machine Learning compute instance
+
+ - Learn how to clone sample notebooks in [Tutorial: Train and deploy a model](../tutorial-train-deploy-notebook.md).
+ - Clone the **how-to-use-azureml** folder instead of **tutorials**
+ - Run the virtual network setup notebook located at `/how-to-use-azureml/reinforcement-learning/setup/devenv_setup.ipynb` to open network ports used for distributed reinforcement learning.
+ - Run the sample notebook `/how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.ipynb`
+
+ - Your own Jupyter Notebook server
+
+ - Install the [Azure Machine Learning SDK](/python/api/overview/azure/ml/install).
+ - Install the [Azure Machine Learning RL SDK](/python/api/azureml-contrib-reinforcementlearning/): `pip install --upgrade azureml-contrib-reinforcementlearning`
+ - Create a [workspace configuration file](../how-to-configure-environment.md#workspace).
+ - Run the virtual network to open network ports used for distributed reinforcement learning.
++
+## How to train a Pong-playing agent
+
+Reinforcement learning (RL) is an approach to machine learning that learns by doing. While other machine learning techniques learn by passively taking input data and finding patterns within it, RL uses **training agents** to actively make decisions and learn from their outcomes.
+
+Your training agents learn to play Pong in a **simulated environment**. Training agents make a decision every frame of the game to move the paddle up, down, or stay in place. It looks at the state of the game (an RGB image of the screen) to make a decision.
+
+RL uses **rewards** to tell the agent if its decisions are successful. In this example, the agent gets a positive reward when it scores a point and a negative reward when a point is scored against it. Over many iterations, the training agent learns to choose the action, based on its current state, that optimizes for the sum of expected future rewards. It's common to use **deep neural networks** (DNN) to perform this optimization in RL.
+
+Training ends when the agent reaches an average reward score of 18 in a training epoch. This means that the agent has beaten its opponent by an average of at least 18 points in matches up to 21.
+
+The process of iterating through simulation and retraining a DNN is computationally expensive, and requires a lot of data. One way to improve performance of RL jobs is by **parallelizing work** so that multiple training agents can act and learn simultaneously. However, managing a distributed RL environment can be a complex undertaking.
+
+Azure Machine Learning provides the framework to manage these complexities to scale out your RL workloads.
+
+## Set up the environment
+
+Set up the local RL environment by:
+1. Loading the required Python packages
+1. Initializing your workspace
+1. Creating an experiment
+1. Specifying a configured virtual network.
+
+### Import libraries
+
+Import the necessary Python packages to run the rest of this example.
+
+```python
+# Azure ML Core imports
+import azureml.core
+from azureml.core import Workspace
+from azureml.core import Experiment
+from azureml.core.compute import AmlCompute
+from azureml.core.compute import ComputeTarget
+from azureml.core.runconfig import EnvironmentDefinition
+from azureml.widgets import RunDetails
+from azureml.tensorboard import Tensorboard
+
+# Azure ML Reinforcement Learning imports
+from azureml.contrib.train.rl import ReinforcementLearningEstimator, Ray
+from azureml.contrib.train.rl import WorkerConfiguration
+```
+
+### Initialize a workspace
+
+Initialize a [workspace](../concept-workspace.md) object from the `config.json` file created in the [prerequisites section](#prerequisites). If you are executing this code in an Azure Machine Learning Compute Instance, the configuration file has already been created for you.
+
+```Python
+ws = Workspace.from_config()
+```
+
+### Create a reinforcement learning experiment
+
+Create an [experiment](/python/api/azureml-core/azureml.core.experiment.experiment) to track your reinforcement learning job. In Azure Machine Learning, experiments are logical collections of related trials to organize job logs, history, outputs, and more.
+
+```python
+experiment_name='rllib-pong-multi-node'
+
+exp = Experiment(workspace=ws, name=experiment_name)
+```
+
+### Specify a virtual network
+
+For RL jobs that use multiple compute targets, you must specify a virtual network with open ports that allow worker nodes and head nodes to communicate with each other.
+
+The virtual network can be in any resource group, but it should be in the same region as your workspace. For more information on setting up your virtual network, see the workspace setup notebook in the prerequisites section. Here, you specify the name of the virtual network in your resource group.
+
+```python
+vnet = 'your_vnet'
+```
+
+## Define head and worker compute targets
+
+This example uses separate compute targets for the Ray head and workers nodes. These settings let you scale your compute resources up and down depending on your workload. Set the number of nodes, and the size of each node, based on your needs.
+
+### Head computing target
+
+You can use a GPU-equipped head cluster to improve deep learning performance. The head node trains the neural network that the agent uses to make decisions. The head node also collects data points from the worker nodes to train the neural network.
+
+The head compute uses a single [`STANDARD_NC6` virtual machine](../../virtual-machines/nc-series.md) (VM). It has 6 virtual CPUs to distribute work across.
++
+```python
+from azureml.core.compute import AmlCompute, ComputeTarget
+
+# choose a name for the Ray head cluster
+head_compute_name = 'head-gpu'
+head_compute_min_nodes = 0
+head_compute_max_nodes = 2
+
+# This example uses GPU VM. For using CPU VM, set SKU to STANDARD_D2_V2
+head_vm_size = 'STANDARD_NC6'
+
+if head_compute_name in ws.compute_targets:
+ head_compute_target = ws.compute_targets[head_compute_name]
+ if head_compute_target and type(head_compute_target) is AmlCompute:
+ print(f'found head compute target. just use it {head_compute_name}')
+else:
+ print('creating a new head compute target...')
+ provisioning_config = AmlCompute.provisioning_configuration(vm_size = head_vm_size,
+ min_nodes = head_compute_min_nodes,
+ max_nodes = head_compute_max_nodes,
+ vnet_resourcegroup_name = ws.resource_group,
+ vnet_name = vnet_name,
+ subnet_name = 'default')
+
+ # create the cluster
+ head_compute_target = ComputeTarget.create(ws, head_compute_name, provisioning_config)
+
+ # can poll for a minimum number of nodes and for a specific timeout.
+ # if no min node count is provided it will use the scale settings for the cluster
+ head_compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)
+
+ # For a more detailed view of current AmlCompute status, use get_status()
+ print(head_compute_target.get_status().serialize())
+```
++
+### Worker computing cluster
+
+This example uses four [`STANDARD_D2_V2` VMs](../../virtual-machines/nc-series.md) for the worker compute target. Each worker node has 2 available CPUs for a total of 8 available CPUs.
+
+GPUs aren't necessary for the worker nodes since they aren't performing deep learning. The workers run the game simulations and collect data.
+
+```python
+# choose a name for your Ray worker cluster
+worker_compute_name = 'worker-cpu'
+worker_compute_min_nodes = 0
+worker_compute_max_nodes = 4
+
+# This example uses CPU VM. For using GPU VM, set SKU to STANDARD_NC6
+worker_vm_size = 'STANDARD_D2_V2'
+
+# Create the compute target if it hasn't been created already
+if worker_compute_name in ws.compute_targets:
+ worker_compute_target = ws.compute_targets[worker_compute_name]
+ if worker_compute_target and type(worker_compute_target) is AmlCompute:
+ print(f'found worker compute target. just use it {worker_compute_name}')
+else:
+ print('creating a new worker compute target...')
+ provisioning_config = AmlCompute.provisioning_configuration(vm_size = worker_vm_size,
+ min_nodes = worker_compute_min_nodes,
+ max_nodes = worker_compute_max_nodes,
+ vnet_resourcegroup_name = ws.resource_group,
+ vnet_name = vnet_name,
+ subnet_name = 'default')
+
+ # create the cluster
+ worker_compute_target = ComputeTarget.create(ws, worker_compute_name, provisioning_config)
+
+ # can poll for a minimum number of nodes and for a specific timeout.
+ # if no min node count is provided it will use the scale settings for the cluster
+ worker_compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)
+
+ # For a more detailed view of current AmlCompute status, use get_status()
+ print(worker_compute_target.get_status().serialize())
+```
+
+## Create a reinforcement learning estimator
+
+Use the [ReinforcementLearningEstimator](/python/api/azureml-contrib-reinforcementlearning/azureml.contrib.train.rl.reinforcementlearningestimator) to submit a training job to Azure Machine Learning.
+
+Azure Machine Learning uses estimator classes to encapsulate job configuration information. This lets you specify how to configure a script execution.
+
+### Define a worker configuration
+
+The WorkerConfiguration object tells Azure Machine Learning how to initialize the worker cluster that runs the entry script.
+
+```python
+# Pip packages we will use for both head and worker
+pip_packages=["ray[rllib]==0.8.3"] # Latest version of Ray has fixes for isses related to object transfers
+
+# Specify the Ray worker configuration
+worker_conf = WorkerConfiguration(
+
+ # Azure ML compute cluster to run Ray workers
+ compute_target=worker_compute_target,
+
+ # Number of worker nodes
+ node_count=4,
+
+ # GPU
+ use_gpu=False,
+
+ # PIP packages to use
+ pip_packages=pip_packages
+)
+```
+
+### Define script parameters
+
+The entry script `pong_rllib.py` accepts a list of parameters that defines how to execute the training job. Passing these parameters through the estimator as a layer of encapsulation makes it easy to change script parameters and run configurations independently of each other.
+
+Specifying the correct `num_workers` makes the most out of your parallelization efforts. Set the number of workers to the same as the number of available CPUs. For this example, you can use the following calculation:
+
+The head node is a [Standard_NC6](../../virtual-machines/nc-series.md) with 6 vCPUs. The worker cluster is 4 [Standard_D2_V2 VMs](../../cloud-services/cloud-services-sizes-specs.md#dv2-series) with 2 CPUs each, for a total of 8 CPUs. However, you must subtract 1 CPU from the worker count since 1 must be dedicated to the head node role.
+
+6 CPUs + 8 CPUs - 1 head CPU = 13 simultaneous workers. Azure Machine Learning uses head and worker clusters to distinguish compute resources. However, Ray does not distinguish between head and workers, and all CPUs are available as worker threads.
++
+```python
+training_algorithm = "IMPALA"
+rl_environment = "PongNoFrameskip-v4"
+
+# Training script parameters
+script_params = {
+
+ # Training algorithm, IMPALA in this case
+ "--run": training_algorithm,
+
+ # Environment, Pong in this case
+ "--env": rl_environment,
+
+ # Add additional single quotes at the both ends of string values as we have spaces in the
+ # string parameters, outermost quotes are not passed to scripts as they are not actually part of string
+ # Number of GPUs
+ # Number of ray workers
+ "--config": '\'{"num_gpus": 1, "num_workers": 13}\'',
+
+ # Target episode reward mean to stop the training
+ # Total training time in seconds
+ "--stop": '\'{"episode_reward_mean": 18, "time_total_s": 3600}\'',
+}
+```
+
+### Define the reinforcement learning estimator
+
+Use the parameter list and the worker configuration object to construct the estimator.
+
+```python
+# RL estimator
+rl_estimator = ReinforcementLearningEstimator(
+
+ # Location of source files
+ source_directory='files',
+
+ # Python script file
+ entry_script="pong_rllib.py",
+
+ # Parameters to pass to the script file
+ # Defined above.
+ script_params=script_params,
+
+ # The Azure ML compute target set up for Ray head nodes
+ compute_target=head_compute_target,
+
+ # Pip packages
+ pip_packages=pip_packages,
+
+ # GPU usage
+ use_gpu=True,
+
+ # RL framework. Currently must be Ray.
+ rl_framework=Ray(),
+
+ # Ray worker configuration defined above.
+ worker_configuration=worker_conf,
+
+ # How long to wait for whole cluster to start
+ cluster_coordination_timeout_seconds=3600,
+
+ # Maximum time for the whole Ray job to run
+ # This will cut off the job after an hour
+ max_run_duration_seconds=3600,
+
+ # Allow the docker container Ray runs in to make full use
+ # of the shared memory available from the host OS.
+ shm_size=24*1024*1024*1024
+)
+```
+
+### Entry script
+
+The [entry script](https://aka.ms/azure-rl-pong-script) `pong_rllib.py` trains a neural network using the [OpenAI Gym environment](https://github.com/openai/gym/) `PongNoFrameSkip-v4`. OpenAI Gyms are standardized interfaces to test reinforcement learning algorithms on classic Atari games.
+
+This example uses a training algorithm known as [IMPALA](https://arxiv.org/abs/1802.01561) (Importance Weighted Actor-Learner Architecture). IMPALA parallelizes each individual learning actor to scale across many compute nodes without sacrificing speed or stability.
+
+[Ray Tune](https://ray.readthedocs.io/en/latest/tune.html) orchestrates the IMPALA worker tasks.
+
+```python
+import ray
+import ray.tune as tune
+from ray.rllib import train
+
+import os
+import sys
+
+from azureml.core import Run
+from utils import callbacks
+
+DEFAULT_RAY_ADDRESS = 'localhost:6379'
+
+if __name__ == "__main__":
+
+ # Parse arguments
+ train_parser = train.create_parser()
+
+ args = train_parser.parse_args()
+ print("Algorithm config:", args.config)
+
+ if args.ray_address is None:
+ args.ray_address = DEFAULT_RAY_ADDRESS
+
+ ray.init(address=args.ray_address)
+
+ tune.run(run_or_experiment=args.run,
+ config={
+ "env": args.env,
+ "num_gpus": args.config["num_gpus"],
+ "num_workers": args.config["num_workers"],
+ "callbacks": {"on_train_result": callbacks.on_train_result},
+ "sample_batch_size": 50,
+ "train_batch_size": 1000,
+ "num_sgd_iter": 2,
+ "num_data_loader_buffers": 2,
+ "model": {
+ "dim": 42
+ },
+ },
+ stop=args.stop,
+ local_dir='./logs')
+```
+
+### Logging callback function
++
+The entry script uses a utility function to define a [custom RLlib callback function](https://docs.ray.io/en/latest/rllib-training.html#callbacks-and-custom-metrics) to log metrics to your Azure Machine Learning workspace. Learn how to view these metrics in the [Monitor and view results](#monitor-and-view-results) section.
+
+```python
+'''RLlib callbacks module:
+ Common callback methods to be passed to RLlib trainer.
+'''
+from azureml.core import Run
+
+def on_train_result(info):
+ '''Callback on train result to record metrics returned by trainer.
+ '''
+ run = Run.get_context()
+ run.log(
+ name='episode_reward_mean',
+ value=info["result"]["episode_reward_mean"])
+ run.log(
+ name='episodes_total',
+ value=info["result"]["episodes_total"])
+```
+
+## Submit a job
+
+[Run](/python/api/azureml-core/azureml.core.run%28class%29) handles the run history of in-progress or complete jobs.
+
+```python
+run = exp.submit(config=rl_estimator)
+```
+> [!NOTE]
+> The run may take up to 30 to 45 minutes to complete.
+
+## Monitor and view results
+
+Use the Azure Machine Learning Jupyter widget to see the status of your jobs in real time. The widget shows two child jobs: one for head and one for workers.
+
+```python
+from azureml.widgets import RunDetails
+
+RunDetails(run).show()
+run.wait_for_completion()
+```
+
+1. Wait for the widget to load.
+1. Select the head job in the list of jobs.
+
+Select **Click here to see the job in Azure Machine Learning studio** for additional job information in the studio. You can access this information while the job is in progress or after it completes.
+
+![Line graph showing how job details widget](./media/how-to-use-reinforcement-learning/pong-run-details-widget.png)
+
+The **episode_reward_mean** plot shows the mean number of points scored per training epoch. You can see that the training agent initially performed poorly, losing its matches without scoring a single point (shown by a reward_mean of -21). Within 100 iterations, the training agent learned to beat the computer opponent by an average of 18 points.
+
+If you browse logs of the child job, you can see the evaluation results recorded in driver_log.txt file. You may need to wait several minutes before these metrics become available on the Job page.
+
+In short work, you have learned to configure multiple compute resources to train a reinforcement learning agent to play Pong very well against a computer opponent.
+
+## Next steps
+
+In this article, you learned how to train a reinforcement learning agent using an IMPALA learning agent. To see additional examples, go to the [Azure Machine Learning Reinforcement Learning GitHub repository](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/reinforcement-learning/README.md).
marketplace Marketplace Commercial Transaction Capabilities And Considerations https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/marketplace/marketplace-commercial-transaction-capabilities-and-considerations.md
description: This article describes pricing, billing, invoicing, and payout cons
Previously updated : 08/02/2022 Last updated : 08/05/2022
The transact publishing option is currently supported for the following offer ty
| Offer type | Billing cadence | Metered billing | Pricing model | | | - | - | - |
-| Azure Application <br>(Managed application) | Monthly | Yes | Usage-based |
+| Azure Application <br>(Managed application) | Monthly | Yes | Usage-based and flat rate |
| Azure Virtual Machine | Monthly [1] | No | Usage-based, BYOL | | Software as a service (SaaS) | One-time upfront monthly, annual [2,3] | Yes | Flat rate, per user, usage-based. | | Dynamics 365 apps on Dataverse and Power Apps [4] | Monthly and annual | No | Per user |
Use this flowchart to determine the appropriate *transactable* offer type and pr
Generally, SaaS offers are a good fit if your customers just want to subscribe to your service and use it online. With SaaS, the publisher (you) host the software in your cloud infrastructure, managing technical infrastructure, app software, and data management responsibilities.
-Virtual Machine and Azure Application offers are a good fit if you want customers to deploy, manage, and run your packaged app or service (as a VM Image and/or other Azure services in the ARM template) in their own cloud infrastructure.
+Virtual Machine and Azure Application offers are a good fit if you want customers to deploy, manage, and run your packaged app or service (as a VM Image and/or other Azure services in the ARM template) in their own Azure cloud infrastructure.
[![Illustration showing a flowchart for determining offer type and pricing plan.](media/commercial-marketplace-plans/offer-type-and-pricing-plan-flowchart.png)](media/commercial-marketplace-plans/offer-type-and-pricing-plan-flowchart.png#lightbox)
-&nbsp;&nbsp;&nbsp;<sup>(1)</sup> Attend [Microsoft Office Hours](https://go.microsoft.com/fwlink/?linkid=2185526) or [support](./support.md).<br>
-&nbsp;&nbsp;&nbsp;<sup>(2)</sup> VM offer images can be included in the Azure App offer to increase pricing flexibility.<br>
-&nbsp;&nbsp;&nbsp;<sup>(3)</sup> Customer pays the infrastructure costs since Azure services are deployed on the customer tenant for VM and Azure App offers.
+
+[1] Attend [Microsoft Office Hours](https://go.microsoft.com/fwlink/?linkid=2185526) or [support](./support.md).
+
+[2] VM offer images can be included in the Azure App offer to increase pricing and deployment flexibility.
+
+[3] This can include your own VM offers.
+
+[4] Customer pays additional infrastructure costs since Azure services are deployed on the customer tenant for VM and Azure App offers.
### Usage-based and subscription pricing
marketplace Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/marketplace/overview.md
The commercial marketplace is available in more than 100 countries and regions,
The following video provides more information about transacting in the commercial marketplace.
-<br />
-<iframe src=https://docs.microsoft.com/_themes/docs.theme/master/en-us/_themes/global/video-embed.html?id=ae2b72e2-6591-407f-8740-50cc4860e8ee width="1080" height="529"></iframe>
+> [!VIDEO ae2b72e2-6591-407f-8740-50cc4860e8ee]
## Why sell with Microsoft?
mysql Concepts Monitoring https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/mysql/flexible-server/concepts-monitoring.md
Last updated 9/21/2020
-# Monitor Azure Database for MySQL Flexible Servers with built-in metrics
+# Monitor Azure Database for MySQL Flexible Server
[!INCLUDE[applies-to-mysql-flexible-server](../includes/applies-to-mysql-flexible-server.md)]
-Azure Database for MySQL Flexible Server provides monitoring of servers through Azure Monitor. Metrics are numerical values that describe some aspect of the resources of your server at a particular time. Monitoring your server's resources helps you troubleshoot and optimize your workload by allowing you to monitor what matters the most to you. Monitoring the right metrics helps you keep the performance, reliability, and availability of your server and applications.
+Azure Database for MySQL Flexible Server provides monitoring of servers through Azure Monitor. Monitoring data about your servers helps you troubleshoot and optimize for your workload.
-In this article, you will learn about the various metrics available for your flexible server that give insight into the behavior of your server.
+In this article, you'll learn about the various metrics available and Server logs for your flexible server that give insight into the behavior of your server.
> [!NOTE] > This article contains references to the term *slave*, a term that Microsoft no longer uses. When the term is removed from the software, we'll remove it from this article.
-## Available metrics
+## Metrics
-Azure Database for MySQL Flexible Server provides various metrics to understand how your workload is performing and based on this data, you can understand the impact on your server and application. For example, in flexible server, you can monitor **Host CPU percent**, **Active Connections**, **IO percent**, and **Host Memory Percent** to identify when there is a performance impact. From there, you may have to optimize your workload, scale vertically by changing compute tiers, or scaling horizontally by using read replica.
+Metrics are numerical values that describe some aspect of the resources of your server at a particular time. Monitoring your server's resources helps you troubleshoot and optimize your workload by allowing you to monitor what matters the most to you. Monitoring the right metrics helps you keep the performance, reliability, and availability of your server and applications.
+
+Azure Database for MySQL Flexible Server provides various metrics to understand how your workload is performing and based on this data, you can understand the impact on your server and application.
All Azure metrics have a one-minute frequency, and each metric provides 30 days of history. You can configure alerts on the metrics. For step-by-step guidance, see [How to set up alerts](./how-to-alert-on-metric.md). Other tasks include setting up automated actions, performing advanced analytics, and archiving history. For more information, see the [Azure Metrics Overview](../../azure-monitor/data-platform.md).
-### List of metrics
+
+## List of metrics
These metrics are available for Azure Database for MySQL: |Metric display name|Metric|Unit|Description|
These metrics are available for Azure Database for MySQL:
|Aborted Connections|aborted_connections|Count|Total number of failed attempts to connect to your MySQL server, for example, failed connection due to bad credentials. For more information on aborted connections, you can refer to this [documentation](https://dev.mysql.com/doc/refman/5.7/en/communication-errors.html).| |Queries|queries|Count|Total number of queries executed per minute on your server. Total count of queries per minute on your server from your database workload and Azure MySQL processes.| +
+## Server logs
+
+In Azure Database for MySQL Server ΓÇô Flexible Server, users can configure and download server logs to assist with troubleshooting efforts. With this feature enabled, a flexible server starts capturing events of the selected log type and writes them to a file. You can then use the Azure portal and Azure CLI to download the files to work with them.
+The server logs feature is disabled by default. For information about how to enable server logs, see [How to enable and download server logs for Azure Database for MySQL - Flexible Server](./how-to-server-logs-portal.md)
+
+To perform a historical analysis of your data, in the Azure portal, on the Diagnostics settings pane for your server, add a diagnostic setting to send the logs to Log Analytics workspace, Azure Storage, or event hubs. For more information, see [Set up diagnostics](./tutorial-query-performance-insights.md#set-up-diagnostics).
+
+**Server logs retention**
+
+When logging is enabled for an Azure Database for MySQL flexible server, logs are available up to seven days from their creation.
+If the total size of the available logs exceeds 7 GB, then the oldest files are deleted until space is available.
+The 7-GB storage limit for server logs is available free of cost and can't be extended.
+Logs are rotated every 24 hours or 7 GB, whichever comes first.
++ ## Next steps - See [How to set up alerts](./how-to-alert-on-metric.md) for guidance on creating an alert on a metric.-- Learn more about [scaling IOPS](./concepts/../concepts-compute-storage.md#iops) to improve performance.
+- How to enable and download server logs for Azure Database for MySQL - Flexible Server from [Azure portal](./how-to-server-logs-portal.md) or [Azure CLI](./how-to-server-logs-cli.md)
mysql How To Server Logs Cli https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/mysql/flexible-server/how-to-server-logs-cli.md
+
+ Title: 'Monitoring - List and Download Server logs using Azure CLI'
+description: This article describes how to download and list server logs using Azure CLI.
+++++ Last updated : 08/05/2022+
+# List and Download Server logs using Azure CLI
+
+[[!INCLUDE[applies-to-mysql-flexible-server](../includes/applies-to-mysql-flexible-server.md)]
+
+This article shows you how to list and download server flexible server using Azure CLI.
+
+## Prerequisites
+
+This article requires that you're running the Azure CLI version 2.39.0 or later locally. To see the version installed, run the `az --version` command. If you need to install or upgrade, see [Install Azure CLI](/cli/azure/install-azure-cli).
+
+You'll need to sign-in to your account using the [az login](/cli/azure/reference-index#az-login) command. Note the **id** property, which refers to **Subscription ID** for your Azure account.
+
+```azurecli-interactive
+az login
+```
+
+Select the specific subscription under your account using [az account set](/cli/azure/account) command. Make a note of the **id** value from the **az login** output to use as the value for **subscription** argument in the command. If you have multiple subscriptions, choose the appropriate subscription in which the resource should be billed. To get all your subscription, use [az account list](/cli/azure/account#az-account-list).
+
+```azurecli
+az account set --subscription <subscription id>
+```
+
+## List Server logs using Azure CLI
+
+Once you're configured the prerequisites and connected to your required subscription.
+You can list the server logs from your flexible server by below command.
++
+```azurecli
+az mysql flexible-server server-logs list --resource-group <myresourcegroup> --server-name <serverlogdemo> --out <table>
+```
+
+Here are the details for the above command
+
+LastModifiedTime | Name | ResourceGroup | SizeInKb | TypePropertiesType | Url
+||||||
+2022-08-01T11:09:48+00:00 | mysql-slow-serverlogdemo-2022073111.log | myresourcegroup | 10947 | slowlog | https://00000000000.file.core.windows.net/0000000serverlog/slowlogs/mysql-slow-serverlogdemo-2022073111.log?
+2022-08-02T11:10:00+00:00 | mysql-slow-serverlogdemo-2022080111.log | myresourcegroup | 10927 | slowlog | https://00000000000.file.core.windows.net/0000000serverlog/slowlogs/mysql-slow-serverlogdemo-2022080111.log?
+2022-08-03T11:10:12+00:00 | mysql-slow-serverlogdemo-2022080211.log | myresourcegroup | 10936 | slowlog | https://00000000000.file.core.windows.net/0000000serverlog/slowlogs/mysql-slow-serverlogdemo-2022080211.log?
+2022-08-03T11:12:00+00:00 | mysql-slow-serverlogdemo-2022080311.log | myresourcegroup | 8920 | slowlog | https://00000000000.file.core.windows.net/0000000serverlog/slowlogs/mysql-slow-serverlogdemo-2022080311.log?
++
+Above list shows LastModifiedTime, Name, ResourceGroup, SizeInKb and Download Url of the Server Logs available.
+Default LastModifiedTime is set to 72 hours, for listing files older than 72 hours, use flag `--file-last-written <Time:HH>`
+
+```azurecli
+az mysql flexible-server server-logs list --resource-group <myresourcegroup> --server-name <serverlogdemo> --out table --file-last-written <144>
+```
+
+## Downloading Server logs using Azure CLI
+
+Below command will download the mentioned server logs to your current directory.
+
+```azurecli
+az mysql flexible-server server-logs download --resource-group <myresourcegroup> --server-name <serverlogdemo> --name <mysql-slow-serverlogdemo-2022073111.log>
+```
+
+## Next Steps
+- To enable and disable Server logs from portal, you can refer to the [article].(./how-to-server-logs-portal.md)
+- Learn more about [Configure slow logs using Azure CLI](./tutorial-query-performance-insights.md#configure-slow-query-logs-by-using-the-azure-cli)
mysql How To Server Logs Portal https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/mysql/flexible-server/how-to-server-logs-portal.md
+
+ Title: 'How to enable and download server logs for Azure Database for MySQL - Flexible Server'
+description: This article describes how to download and list server logs using Azure portal.
+++++ Last updated : 08/05/2022+
+# Enable, list and download server logs for Azure Database for MySQL - Flexible Server
++
+You can use server logs to help monitor and troubleshoot an instance of Azure Database for MySQL - Flexible Server, and to gain detailed insights into the activities that have run on your servers.
+By default, the server logs feature in Azure Database for MySQL - Flexible Server is disabled. However, after you enable the feature, a flexible server starts capturing events of the selected log type and writes them to a file. You can then use the Azure portal or the Azure CLI to download the files to assist with your troubleshooting efforts.
+This article explains how to enable the server logs feature in Azure Database for MySQL - Flexible Server and download server log files. It also provides information about how to disable the feature.
+
+In this tutorial, youΓÇÖll learn how to:
+- Enable the server logs feature.
+- Disable the server logs feature.
+- Download server log files.
+
+## Prerequisites
+
+To complete this tutorial, you need an existing Azure Database for MySQL flexible server. If you need to create a new server, see [Create an Azure Database for MySQL flexible server](./quickstart-create-server-portal.md).
+
+## Enable Server logs
+
+To enable the server logs feature, perform the following steps.
+
+1. In the [Azure portal](https://portal.azure.com), select your MySQL flexible server.
+
+2. On the left pane, under **Monitoring**, select **Server logs**.
+
+ :::image type="content" source="./media/how-to-server-logs-portal/1-how-to-serverlog.png" alt-text="Screenshot showing Azure Database for My SQL - Server Logs.":::
+
+3. To enable server logs, under **Server logs**, select **Enable**.
+
+ :::image type="content" source="./media/how-to-server-logs-portal/2-how-to-serverlog.png" alt-text="Screenshot showing Enable Server Logs.":::
+
+>[!Note]
+> You can also enable server logs in the Azure portal, on the [Server parameters](./how-to-configure-server-parameters-portal.md) pane for your server, by setting the value of the log_output parameter to FILE.
+> For more information on the log_output parameter, in the MySQL documentation, see topic Server System Variables ([version 5.7](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_log_output) or [version 8.0](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_log_output)).
+
+4. To enable the slow_query_log log, under **Select logs to enable**, select **slow_query_log**.
+
+ :::image type="content" source="./media/how-to-server-logs-portal/3-how-to-serverlog.png" alt-text="Screenshot showing select slow log - Server Logs.":::
+
+To configure slow_logs on your Azure Database for MySQL flexible server, see [Query Performance Insight.](./tutorial-query-performance-insights.md)
++
+## Download Server logs
+
+To download server logs, perform the following steps.
+> [!Note]
+> After enabling logs, the log files will be available to download after few minutes.
+
+1. Under **Name**, select the log file you want to download, and then, under **Action**, select **Download**.
+
+ :::image type="content" source="./media/how-to-server-logs-portal/4-how-to-serverlog.png" alt-text="Screenshot showing Server Logs - Download.":::
+
+ For HA enabled Azure database for MySQL Flexible server, server logs for standby server can be identified by another four-letter identifier after the hostname of the server as shown below.
+
+ :::image type="content" source="./media/how-to-server-logs-portal/5-how-to-serverlog.png" alt-text="Screenshot showing server Logs - HA logs.":::
+
+2. To download multiple log files at one time, under **Name**, select the files you want to download, and then above **Name**, select **Download**.
+
+ :::image type="content" source="./media/how-to-server-logs-portal/6-how-to-serverlog.png" alt-text="Screenshot showing server Logs - Download all.":::
++
+## Disable Server Logs
+
+1. From your Azure portal, select Server logs from Monitoring server pane.
+
+2. For disabling Server logs to file, Uncheck Enable. (The setting will disable logging for all the log_types available)
+
+ :::image type="content" source="./media/how-to-server-logs-portal/7-how-to-serverlog.png" alt-text="Screenshot showing server Logs - Disable.":::
+
+3. Select Save
+
+ :::image type="content" source="./media/how-to-server-logs-portal/8-how-to-serverlog.png" alt-text="Screenshot showing server Logs - Save.":::
++
+## Next steps
+- Learn more about [How to enable slow query logs](./tutorial-query-performance-insights.md#configure-slow-query-logs-by-using-the-azure-portal)
+- List and download [Server logs using Azure CLI](./how-to-server-logs-cli.md)
mysql Concepts High Availability https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/mysql/single-server/concepts-high-availability.md
Here are some failure scenarios that require user action to recover:
| **Scenario** | **Recovery plan** | | - | - | | <b> Region failure | Failure of a region is a rare event. However, if you need protection from a region failure, you can configure one or more read replicas in other regions for disaster recovery (DR). (See [this article](how-to-read-replicas-portal.md) about creating and managing read replicas for details). In the event of a region-level failure, you can manually promote the read replica configured on the other region to be your production database server. |
-| <b> Logical/user errors | Recovery from user errors, such as accidentally dropped tables or incorrectly updated data, involves performing a [point-in-time recovery](concepts-backup.md) (PITR), by restoring and recovering the data until the time just before the error had occurred.<br> <br> If you want to restore only a subset of databases or specific tables rather than all databases in the database server, you can restore the database server in a new instance, export the table(s) via [mysqldump](concepts-migrate-dump-restore.md), and then use [restore](concepts-migrate-dump-restore.md#restore-your-mysql-database-using-command-line-or-mysql-workbench) to restore those tables into your database. |
+| <b> Logical/user errors | Recovery from user errors, such as accidentally dropped tables or incorrectly updated data, involves performing a [point-in-time recovery](concepts-backup.md) (PITR), by restoring and recovering the data until the time just before the error had occurred.<br> <br> If you want to restore only a subset of databases or specific tables rather than all databases in the database server, you can restore the database server in a new instance, export the table(s) via [mysqldump](concepts-migrate-dump-restore.md), and then use [restore](concepts-migrate-dump-restore.md#restore-your-mysql-database-using-command-line) to restore those tables into your database. |
mysql Concepts Migrate Dump Restore https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/mysql/single-server/concepts-migrate-dump-restore.md
To step through this how-to guide, you need to have:
Most common use-cases are: - **Moving from other managed service provider** - Most managed service provider may not provide access to the physical storage file for security reasons so logical backup and restore is the only option to migrate.-- **Migrating from on-premises environment or Virtual machine** - Azure Database for MySQL doesn't support restore of physical backups which makes logical backup and restore as the ONLY approach.-- **Moving your backup storage from locally redundant to geo-redundant storage** - Azure Database for MySQL allows configuring locally redundant or geo-redundant storage for backup is only allowed during server create. Once the server is provisioned, you cannot change the backup storage redundancy option. In order to move your backup storage from locally redundant storage to geo-redundant storage, dump and restore is the ONLY option. -- **Migrating from alternative storage engines to InnoDB** - Azure Database for MySQL supports only InnoDB Storage engine, and therefore does not support alternative storage engines. If your tables are configured with other storage engines, convert them into the InnoDB engine format before migration to Azure Database for MySQL.
+- **Migrating from on-premises environment or Virtual machine** - Azure Database for MySQL doesn't support restore of physical backups, which makes logical backup and restore as the ONLY approach.
+- **Moving your backup storage from locally redundant to geo-redundant storage** - Azure Database for MySQL allows configuring locally redundant or geo-redundant storage for backup is only allowed during server create. Once the server is provisioned, you can't change the backup storage redundancy option. In order to move your backup storage from locally redundant storage to geo-redundant storage, dump and restore is the ONLY option.
+- **Migrating from alternative storage engines to InnoDB** - Azure Database for MySQL supports only InnoDB Storage engine, and therefore doesn't support alternative storage engines. If your tables are configured with other storage engines, convert them into the InnoDB engine format before migration to Azure Database for MySQL.
For example, if you have a WordPress or WebApp using the MyISAM tables, first convert those tables by migrating into InnoDB format before restoring to Azure Database for MySQL. Use the clause `ENGINE=InnoDB` to set the engine used when creating a new table, then transfer the data into the compatible table before the restore.
Add the connection information into your MySQL Workbench.
## Preparing the target Azure Database for MySQL server for fast data loads To prepare the target Azure Database for MySQL server for faster data loads, the following server parameters and configuration needs to be changed.-- max_allowed_packet ΓÇô set to 1073741824 (i.e. 1GB) to prevent any overflow issue due to long rows.
+- max_allowed_packet ΓÇô set to 1073741824 (that is, 1 GB) to prevent any overflow issue due to long rows.
- slow_query_log ΓÇô set to OFF to turn off the slow query log. This will eliminate the overhead caused by slow query logging during data loads. - query_store_capture_mode ΓÇô set to NONE to turn off the Query Store. This will eliminate the overhead caused by sampling activities by Query Store. - innodb_buffer_pool_size ΓÇô Scale up the server to 32 vCore Memory Optimized SKU from the Pricing tier of the portal during migration to increase the innodb_buffer_pool_size. Innodb_buffer_pool_size can only be increased by scaling up compute for Azure Database for MySQL server. - innodb_io_capacity & innodb_io_capacity_max - Change to 9000 from the Server parameters in Azure portal to improve the IO utilization to optimize for migration speed. - innodb_write_io_threads & innodb_write_io_threads - Change to 4 from the Server parameters in Azure portal to improve the speed of migration.-- Scale up Storage tier ΓÇô The IOPs for Azure Database for MySQL server increases progressively with the increase in storage tier. For faster loads, you may want to increase the storage tier to increase the IOPs provisioned. Please do remember the storage can only be scaled up, not down.
+- Scale up Storage tier ΓÇô The IOPs for Azure Database for MySQL server increases progressively with the increase in storage tier. For faster loads, you may want to increase the storage tier to increase the IOPs provisioned. Do remember the storage can only be scaled up, not down.
Once the migration is completed, you can revert back the server parameters and compute tier configuration to its previous values.
$ mysqldump --opt -u [uname] -p[pass] [dbname] > [backupfile.sql]
The parameters to provide are: - [uname] Your database username-- [pass] The password for your database (note there is no space between -p and the password)
+- [pass] The password for your database (note there's no space between -p and the password)
- [dbname] The name of your database - [backupfile.sql] The filename for your database backup - [--opt] The mysqldump option
-For example, to back up a database named 'testdb' on your MySQL server with the username 'testuser' and with no password to a file testdb_backup.sql, use the following command. The command backs up the `testdb` database into a file called `testdb_backup.sql`, which contains all the SQL statements needed to re-create the database. Make sure that the username 'testuser' has at least the SELECT privilege for dumped tables, SHOW VIEW for dumped views, TRIGGER for dumped triggers, and LOCK TABLES if the --single-transaction option is not used.
+For example, to back up a database named 'testdb' on your MySQL server with the username 'testuser' and with no password to a file testdb_backup.sql, use the following command. The command backs up the `testdb` database into a file called `testdb_backup.sql`, which contains all the SQL statements needed to re-create the database. Make sure that the username 'testuser' has at least the SELECT privilege for dumped tables, SHOW VIEW for dumped views, TRIGGER for dumped triggers, and LOCK TABLES if the `--single-transaction` option isn't used.
```bash GRANT SELECT, LOCK TABLES, SHOW VIEW ON *.* TO 'testuser'@'hostname' IDENTIFIED BY 'password';
To select specific tables in your database to back up, list the table names sepa
```bash $ mysqldump -u root -p testdb table1 table2 > testdb_tables_backup.sql ```
-To back up more than one database at once, use the --database switch and list the database names separated by spaces.
+To back up more than one database at once, use the `--database` switch and list the database names separated by spaces.
```bash $ mysqldump -u root -p --databases testdb1 testdb3 testdb5 > testdb135_backup.sql ```
-### Restore your MySQL database using command-line or MySQL Workbench
-Once you have created the target database, you can use the mysql command or MySQL Workbench to restore the data into the specific newly created database from the dump file.
+### Restore your MySQL database using command-line
+Once you've created the target database, you can use the mysql command to restore the data into the specific newly created database from the dump file.
```bash mysql -h [hostname] -u [uname] -p[pass] [db_to_restore] < [backupfile.sql] ``` In this example, restore the data into the newly created database on the target Azure Database for MySQL server.
-Here is an example for how to use this **mysql** for **Single Server** :
+Here's an example for how to use this **mysql** for **Single Server** :
```bash $ mysql -h mydemoserver.mysql.database.azure.com -u myadmin@mydemoserver -p testdb < testdb_backup.sql ```
-Here is an example for how to use this **mysql** for **Flexible Server** :
+Here's an example for how to use this **mysql** for **Flexible Server** :
```bash $ mysql -h mydemoserver.mysql.database.azure.com -u myadmin -p testdb < testdb_backup.sql ```
+>[!Note]
+>You can also use [MySQL Workbench client utility to restore MySQL database](./concepts-migrate-import-export.md#import-and-export-data-by-using-mysql-workbench).
## Dump and restore using PHPMyAdmin Follow these steps to dump and restore a database using PHPMyadmin.
Follow these steps to dump and restore a database using PHPMyadmin.
### Export with PHPMyadmin To export, you can use the common tool phpMyAdmin, which you may already have installed locally in your environment. To export your MySQL database using PHPMyAdmin: 1. Open phpMyAdmin.
-2. Select your database. Click the database name in the list on the left.
-3. Click the **Export** link. A new page appears to view the dump of database.
-4. In the Export area, click the **Select All** link to choose the tables in your database.
-5. In the SQL options area, click the appropriate options.
-6. Click the **Save as file** option and the corresponding compression option and then click the **Go** button. A dialog box should appear prompting you to save the file locally.
+2. Select your database. Select the database name in the list on the left.
+3. Select the **Export** link. A new page appears to view the dump of database.
+4. In the Export area, select the **Select All** link to choose the tables in your database.
+5. In the SQL options area, select the appropriate options.
+6. Select the **Save as file** option and the corresponding compression option and then select the **Go** button. A dialog box should appear prompting you to save the file locally.
### Import using PHPMyAdmin Importing your database is similar to exporting. Do the following actions: 1. Open phpMyAdmin.
-2. In the phpMyAdmin setup page, click **Add** to add your Azure Database for MySQL server. Provide the connection details and login information.
-3. Create an appropriately named database and select it on the left of the screen. To rewrite the existing database, click the database name, select all the check boxes beside the table names, and select **Drop** to delete the existing tables.
-4. Click the **SQL** link to show the page where you can type in SQL commands, or upload your SQL file.
+2. In the phpMyAdmin setup page, select **Add** to add your Azure Database for MySQL server. Provide the connection details and log in information.
+3. Create an appropriately named database and select it on the left of the screen. To rewrite the existing database, select the database name, select all the check boxes beside the table names, and select **Drop** to delete the existing tables.
+4. Select the **SQL** link to show the page where you can type in SQL commands, or upload your SQL file.
5. Use the **browse** button to find the database file.
-6. Click the **Go** button to export the backup, execute the SQL commands, and re-create your database.
+6. Select the **Go** button to export the backup, execute the SQL commands, and re-create your database.
## Known Issues For known issues, tips and tricks, we recommend you to look at our [techcommunity blog](https://techcommunity.microsoft.com/t5/azure-database-for-mysql/tips-and-tricks-in-using-mysqldump-and-mysql-restore-to-azure/ba-p/916912).
For known issues, tips and tricks, we recommend you to look at our [techcommunit
## Next steps - [Connect applications to Azure Database for MySQL](./how-to-connection-string.md). - For more information about migrating databases to Azure Database for MySQL, see the [Database Migration Guide](https://github.com/Azure/azure-mysql/tree/master/MigrationGuide).-- If you are looking to migrate large databases with database sizes more than 1 TBs, you may want to consider using community tools like **mydumper/myloader** which supports parallel export and import. Learn [how to migrate large MySQL databases](https://techcommunity.microsoft.com/t5/azure-database-for-mysql/best-practices-for-migrating-large-databases-to-azure-database/ba-p/1362699).
+- If you're looking to migrate large databases with database sizes more than 1 TBs, you may want to consider using community tools like **mydumper/myloader** which supports parallel export and import. Learn [how to migrate large MySQL databases](https://techcommunity.microsoft.com/t5/azure-database-for-mysql/best-practices-for-migrating-large-databases-to-azure-database/ba-p/1362699).
network-watcher Connection Monitor Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/network-watcher/connection-monitor-overview.md
For networks whose sources are on-premises VMs, the following issues can be dete
* Source unable to connect to destination. Target not reachable through ICMP. * Certificate-related issues: * Client certificate required to authenticate agent.
- * Certificate relocation list isn't accessible.
+ * Certificate revocation list isn't accessible.
* Host name of the endpoint doesn't match the certificate's subject or subject alternate name. * Root certificate is missing in source's Local Computer Trusted Certification Authorities store. * SSL certificate is expired, invalid, revoked, or incompatible.
postgresql Howto Ingest Azure Data Factory https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/postgresql/hyperscale/howto-ingest-azure-data-factory.md
Title: Azure Data Factory description: Step-by-step guide for using Azure Data Factory for ingestion on Hyperscale Citus -+
private-5g-core Collect Required Information For A Site https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/private-5g-core/collect-required-information-for-a-site.md
Collect all the values in the following table to define the packet core instance
|The data subnet default gateway. You identified this in [Allocate subnets and IP addresses](complete-private-mobile-network-prerequisites.md#allocate-subnets-and-ip-addresses) and it must match the value you used when deploying the AKS-HCI cluster. | **N6 gateway** (for 5G) or **SGi gateway** (for 4G). | | The network address of the subnet from which dynamic IP addresses must be allocated to user equipment (UEs), given in CIDR notation. You won't need this address if you don't want to support dynamic IP address allocation for this site. You identified this in [Allocate user equipment (UE) IP address pools](complete-private-mobile-network-prerequisites.md#allocate-user-equipment-ue-ip-address-pools). The following example shows the network address format. </br></br>`198.51.100.0/24` </br></br>Note that the UE subnets aren't related to the access subnet. |**Dynamic UE IP pool prefixes**| | The network address of the subnet from which static IP addresses must be allocated to user equipment (UEs), given in CIDR notation. You won't need this address if you don't want to support static IP address allocation for this site. You identified this in [Allocate user equipment (UE) IP address pools](complete-private-mobile-network-prerequisites.md#allocate-user-equipment-ue-ip-address-pools). The following example shows the network address format. </br></br>`198.51.100.0/24` </br></br>Note that the UE subnets aren't related to the access subnet. |**Static UE IP pool prefixes**|
- |Whether Network Address and Port Translation (NAPT) should be enabled for this data network. NAPT allows you to translate a large pool of private IP addresses for UEs to a small number of public IP addresses. The translation is performed at the point where traffic enters the data network, maximizing the utility of a limited supply of public IP addresses. |**NAPT**|
+ |Whether Network Address and Port Translation (NAPT) should be enabled for this data network. NAPT allows you to translate a large pool of private IP addresses for UEs to a small number of public IP addresses. The translation is performed at the point where traffic enters the data network, maximizing the utility of a limited supply of public IP addresses.</br></br>If you want to use [UE-to-UE traffic](private-5g-core-overview.md#ue-to-ue-traffic) in this data network, keep NAPT disabled. |**NAPT**|
## Next steps
private-5g-core Packet Core Dashboards https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/private-5g-core/packet-core-dashboards.md
You can access the following packet core dashboards:
- The **HTTP stats dashboard** for each network function shows statistics for the HTTP requests and responses shown by that network function. You can use the **Serving Endpoint**, **Client Operation**, and **Server Operation** filters to control which operations are shown.
- :::image type="content" source="media/packet-core-dashboards/packet-core-http-stats-dashboard.png" alt-text="Screenshot of the H T T P stats dashboard. Panels related to H T T P statistics for the Session Management Function are shown." lightbox="media/packet-core-dashboards/packet-core-http-stats-dashboard.png":::
+ :::image type="content" source="media/packet-core-dashboards/packet-core-http-stats-dashboard.png" alt-text="Screenshot of the H T T P stats dashboard. Panels related to H T T P statistics for the Session Management Function are shown." lightbox="media/packet-core-dashboards/packet-core-http-stats-dashboard.png":::
+
+ - The **4G Interfaces dashboard** displays request and response statistics recorded by each of the packet core instance's 4G interfaces. Note that this dashboard is only available for packet core instances supporting 4G devices.
+
+ :::image type="content" source="media/packet-core-dashboards/packet-core-4g-interfaces-dashboard.png" alt-text="Screenshot of the 4G Interfaces dashboard. Panels related to activity on the packet core instance's 4G interfaces are shown." lightbox="media/packet-core-dashboards/packet-core-4g-interfaces-dashboard.png":::
## Panels and rows
private-5g-core Private 5G Core Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/private-5g-core/private-5g-core-overview.md
Azure Private 5G Core supports the following algorithms for ciphering and integr
- 128-bit Snow3G - 128-bit Advanced Encryption System (AES) encryption
+### UE-to-UE traffic
+
+Azure Private 5G Core supports traffic flow from UE to UE through the user plane, allowing machine-to-machine (M2M) communication between 5G devices for a range of applications including robot control.
+
+An external router is responsible for hairpinning traffic from UE to UE over the N6 interface. This means that traffic leaving the UPF destined to a UE IP address will be routed back to the UPFΓÇÖs N6 IP address.
+ ### Index to RAT/Frequency Selection Priority (RFSP) The packet core instance can provide a RAN with an RFSP Index. The RAN can match the RFSP Index to its local configuration to apply specific radio resource management (RRM) policies, such as cell reselection or frequency layer redirection.
+### Multi-Operator Core Network (MOCN)
+
+Multi-operator Core Network (MOCN) aims to maximize resource usage by sharing a RAN between multiple core networks. Azure Private 5G Core supports MOCN, allowing multiple public land mobile networks (PLMNs) to be shared by a gNodeB (for 5G deployments) or eNodeB (for 4G deployments).
+
+In the context of private mobile networks, a single RAN can connect to both a private and a standard macro network, with traffic automatically routed to the appropriate core network based on the PLMN ID.
+ ## Flexible integration with Azure private multi-access edge compute (MEC) partners Each packet core instance is standards-compliant and compatible with several radio access network (RAN) partners in the Azure private MEC ecosystem.
private-link Private Endpoint Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/private-link/private-endpoint-overview.md
A private-link resource is the destination target of a specified private endpoin
| Azure SQL Database | Microsoft.Sql/servers | SQL Server (sqlServer) | | Azure Storage | Microsoft.Storage/storageAccounts | Blob (blob, blob_secondary)<BR> Table (table, table_secondary)<BR> Queue (queue, queue_secondary)<BR> File (file, file_secondary)<BR> Web (web, web_secondary)<BR> Dfs (dfs, dfs_secondary) | | Azure File Sync | Microsoft.StorageSync/storageSyncServices | File Sync Service |
-| Azure Synapse | Microsoft.Synapse/privateLinkHubs | synapse |
+| Azure Synapse | Microsoft.Synapse/privateLinkHubs | web |
| Azure Synapse Analytics | Microsoft.Synapse/workspaces | SQL, SqlOnDemand, Dev | | Azure App Service | Microsoft.Web/hostingEnvironments | hosting environment | | Azure App Service | Microsoft.Web/sites | sites |
purview Available Metadata https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/purview/available-metadata.md
+
+ Title: Available metadata for Power BI in the Microsoft Purview governance portal
+description: This reference article provides a list of metadata that is available for a Power BI tenant in the Microsoft Purview governance portal.
+++++ Last updated : 08/02/2022++
+# Available metadata
+
+This article has a list of the metadata that is available for a Power BI tenant in the Microsoft Purview governance portal.
+
+## Power BI
+
+| Metadata | Population method | Source of truth | Asset type | Editable | Upstream metadata |
+|--|-|-|--|-||
+| Classification | Manual | Microsoft Purview | All types | Yes | N/A |
+| Sensitivity Labels | Automatic | Microsoft Purview | All types | No | |
+| Glossary terms | Manual | Microsoft Purview | All types | Yes | N/A |
+| Collection | Automatic | Microsoft Purview | All types | | N/A |
+| Hierarchy | Automatic | Microsoft Purview | All types | No | N/A |
+| qualifiedName | Automatic | Microsoft Purview | All types | No | |
+| Asset Description | Automatic/Manual* | Microsoft Purview | All types | Yes | N/A |
+| Contacts - Expert | Manual | Microsoft Purview | All types | Yes | N/A |
+| Contacts - Owner | Manual | Microsoft Purview | All types | Yes | N/A |
+| name | Automatic | Power BI | Power BI Dashboard | Yes | dashboard.DisplayName |
+| isReadOnly | Automatic | Power BI | Power BI Dashboard | No | dashboard.IsReadOnly |
+| EmbedUrl | Automatic | Power BI | Power BI Dashboard | No | dashboard.EmbedUrl |
+| tileNames | Automatic | Power BI | Power BI Dashboard | No | TileTitles |
+| Lineage | Automatic | Power BI | Power BI Dashboard | No | N/A |
+| name | Automatic | Power BI | Power BI dataflow | Yes | dataflow.Name |
+| configured by | Automatic | Power BI | Power BI dataflow | No | dataflow.ConfiguredBy |
+| description | Automatic | Power BI | Power BI dataflow | Yes | dataflow.Description |
+| ModelUrl | Automatic | Power BI | Power BI dataflow | No | dataflow.ModelUrl |
+| ModifiedBy | Automatic | Power BI | Power BI dataflow | No | dataflow.ModifiedBy |
+| ModifiedDateTime | Automatic | Power BI | Power BI dataflow | No | dataflow.ModifiedDateTime |
+| Endorsement | Automatic | Power BI | Power BI dataflow | No | dataflow.EndorsementDetails |
+| name | Automatic | Power BI | Power BI Dataset | Yes | dataset.Name |
+| IsRefreshable | Automatic | Power BI | Power BI Dataset | No | dataset.IsRefreshable |
+| configuredBy | Automatic | Power BI | Power BI Dataset | No | dataset.ConfiguredBy |
+| contentProviderType | Automatic | Power BI | Power BI Dataset | No | dataset.ContentProviderType |
+| createdDate | Automatic | Power BI | Power BI Dataset | No | dataset.CreatedDateTime |
+| targetStorageMode | Automatic | Power BI | Power BI Dataset | No | dataset.TargetStorageMode |
+| Schema | Automatic/Manual | Power BI | Power BI Dataset | | |
+| Lineage | Automatic | Microsoft Purview | Power BI Dataset | No | |
+| description | Automatic | Power BI | Power BI Dataset | Yes | dataset.Description |
+| Endorsement | Automatic | Power BI | Power BI Dataset | No | dataset.EndorsementDetails |
+| name | Automatic | Power BI | Power BI Report | Yes | report.Name |
+| description | Automatic | Power BI | Power BI Report | Yes | report.Description |
+| createdDateTime | Automatic | Power BI | Power BI Report | No | report.CreatedDateTime |
+| WebUrl | Automatic | Power BI | Power BI Report | No | report.WebUrl |
+| EmbedUrl | Automatic | Power BI | Power BI Report | No | report.EmbedUrl |
+| PBIDatasetId | Automatic | Power BI | Power BI Report | No | report.DatasetId; |
+| modifiedBy | Automatic | Power BI | Power BI Report | No | report.ModifiedBy |
+| modifiedDateTime | Automatic | Power BI | Power BI Report | No | report.ModifiedDateTime |
+| reportType | Automatic | Power BI | Power BI Report | No | report.ReportType |
+| Endorsement | Automatic | Power BI | Power BI Report | No | report.EndorsementDetails |
+| Lineage | Automatic | Microsoft Purview | Power BI Report | No | N/A |
+| name | Automatic | Power BI | Power BI Workspace | Yes | workspace.Name |
+| Description | Automatic | Power BI | Power BI Workspace | Yes | workspace.Description |
+| state | Automatic | Power BI | Power BI Workspace | No | workspace.State |
+| type | Automatic | Power BI | Power BI Workspace | No | ResourceType.Workspace |
+| IsReadOnly | Automatic | Power BI | Power BI Workspace | No | workspace.IsReadOnly |
+| IsOnDedicatedCapacity | Automatic | Power BI | Power BI Workspace | No | workspace.IsOnDedicatedCapacity |
++
+## Next steps
+
+- [Register and scan a Power BI tenant](register-scan-power-bi-tenant.md)
+- [Register and scan Power BI across tenants](register-scan-power-bi-tenant-cross-tenant.md)
+- [Register and scan Power BI troubleshooting](register-scan-power-bi-tenant-troubleshoot.md)
purview Create A Custom Classification And Classification Rule https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/purview/create-a-custom-classification-and-classification-rule.md
the following logic:
- All underscores (\_) are replaced with spaces.
-As an example, if you named your classification **CONTOSO.HR.EMPLOYEE\_ID**, the friendly name is stored
+As an example, if you named your classification **contoso.hr.employee\_ID**, the friendly name is stored
in the system as **Hr.Employee ID**. :::image type="content" source="media/create-a-custom-classification-and-classification-rule/contoso-hr-employee-id.png" alt-text="Contoso.hr.employee_id" border="true":::
purview Deployment Best Practices https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/purview/deployment-best-practices.md
Title: 'Deployment best practices for Microsoft Purview (formerly Azure Purview)'
-description: This article provides best practices for deploying Microsoft Purview (formerly Azure Purview). The Microsoft Purview Data Map and governance portal enable any user to register, discover, understand, and consume data sources.
+description: This article provides best practices for deploying Microsoft Purview (formerly Azure Purview) in your data estate. The Microsoft Purview Data Map and governance portal enable any user to register, discover, understand, and consume data sources.
Previously updated : 11/23/2020 Last updated : 08/04/2022 # Microsoft Purview (formerly Azure Purview) deployment best practices
-This article identifies common tasks that can help you deploy Microsoft Purview (formerly Azure Purview) into production. These tasks can be completed in phases, over the course of a month or more. Even organizations who have already deployed Microsoft Purview can use this guide to ensure they're getting the most out of their investment.
+This article is a guide to successfully deploying Microsoft Purview (formerly Azure Purview) into production in your data estate. It's intended to help you strategize and phase your deployment from research to hardening your production environment, and is best used in tandem with our [deployment checklist](tutorial-azure-purview-checklist.md).
-A well-planned deployment of a data governance platform (such as Microsoft Purview), can give the following benefits:
+If you're looking for a strictly technical deployment guide, use the [deployment checklist](tutorial-azure-purview-checklist.md).
+
+If you're creating a plan to deploy Microsoft Purview and want to consider best practices as you develop your deployment strategy, then follow the article below. This guide outlines tasks can be completed in phases over the course of a month or more to develop your deployment process for Microsoft Purview. Even organizations who have already deployed Microsoft Purview can use this guide to ensure they're getting the most out of their investment.
+
+A well-planned deployment of your data governance platform, can give the following benefits:
- Better data discovery - Improved analytic collaboration-- Maximized return on investment.
+- Maximized return on investment
+
+This guide provides insight on a full deployment lifecycle, from initial planning to a mature environment by following these stages:
-## Prerequisites
+| Stage | Description |
+|-|-|
+|[Identify objectives and goals](#identify-objectives-and-goals)|Consider what your entire organization wants and needs from data governance.|
+|[Gathering questions](#gathering-questions)|What questions might you and your team have as you get started, and where can you look to begin addressing them?|
+|[Deployment models](#deployment-models)|Customize your Microsoft Purview deployment to your data estate.|
+|[Create a process to move to production](#create-a-process-to-move-to-production)|Create a phased deployment strategy tailored to your organization.|
+|[Platform hardening](#platform-hardening)|Continue to grow your deployment to maturity.|
-- Access to Microsoft Azure with a development or production subscription-- Ability to create Azure resources including Microsoft Purview-- Access to data sources such as Azure Data Lake Storage or Azure SQL in test, development, or production environments
- - For Data Lake Storage, the required role to scan is Reader Role
- - For SQL, the identity must be able to query tables for sampling of classifications
-- Access to Microsoft 365 Defender for Cloud or ability to collaborate with Microsoft 365 Defender for Cloud Admin for data labeling
+Many of Microsoft Purview's applications and features have their own individual best practices pages as well. They're referenced often throughout this deployment guide, but you can find all of them in the table of contents under **Concepts** and then **Best practices and guidelines**.
## Identify objectives and goals Many organizations have started their data governance journey by developing individual solutions that cater to specific requirements of isolated groups and data domains across the organization. Although experiences may vary depending on the industry, product, and culture, most organizations find it difficult to maintain consistent controls and policies for these types of solutions.
-Some of the common data governance objectives that you might want to identify in the early phases, include:
+Some of the common data governance objectives that you might want to identify in the early phases to create a comprehensive data governance experience include:
* Maximizing the business value of your data * Enabling a data culture where data consumers can easily find, interpret, and trust data
The general approach is to break down those overarching objectives into various
|Workflow|The platform must have the ability to create and modify workflow so that it's easy to scale out and automate various tasks within the platform.| |Integration|Other third-party technologies such as ticketing or orchestration must be able to integrate into the platform via script or REST APIs.|
-## Top questions to ask
+### Identify key scenarios
+
+Microsoft Purview governance services can be used to centrally manage data governance across an organizationΓÇÖs data estate spanning cloud and on-premises environments. To have a successful implementation, you must identify key scenarios that are critical to the business. These scenarios can cross business unit boundaries or affect multiple user personas either upstream or downstream.
+
+These scenarios can be written up in various ways, but you should include at least these five dimensions:
+
+1. Persona ΓÇô Who are the users?
+2. Source system ΓÇô What are the data sources such as Azure Data Lake Storage Gen2 or Azure SQL Database?
+3. Impact Area ΓÇô What is the category of this scenario?
+4. Detail scenarios ΓÇô How the users use Microsoft Purview to solve problems?
+5. Expected outcome ΓÇô What is the success criteria?
+
+The scenarios must be specific, actionable, and executable with measurable results. Some example scenarios that you can use:
+
+|Scenario|Detail|Persona|
+||||
+|Catalog business-critical assets|I need to have information about each data sets to have a good understanding of what it is. This scenario includes both business and technical metadata data about the data set in the catalog. The data sources include Azure Data Lake Storage Gen2, Azure Synapse DW, and/or Power BI. This scenario also includes on-premises resources such as SQL Server.|Business Analyst, Data Scientist, Data Engineer|
+|Discover business-critical assets|I need to have a search engine that can search through all metadata in the catalog. I should be able to search using technical term, business term with either simple or complex search using wildcard.|Business Analyst, Data Scientist, Data Engineer, Data Admin|
+|Track data to understand its origin and troubleshoot data issues|I need to have data lineage to track data in reports, predictions, or models back to its original source. I also need to understand the changes made to the data, and where the data has resided throughout the data life cycle. This scenario needs to support prioritized data pipelines Azure Data Factory and Databricks.|Data Engineer, Data Scientist|
+|Enrich metadata on critical data assets|I need to enrich the data set in the catalog with technical metadata that is generated automatically. Classification and labeling are some examples.|Data Engineer, Domain/Business Owner|
+|Govern data assets with friendly user experience|I need to have a Business glossary for business-specific metadata. The business users can use Microsoft Purview for self-service scenarios to annotate their data and enable the data to be discovered easily via search.|Domain/Business Owner, Business Analyst, Data Scientist, Data Engineer|
+
+### Integration points with Microsoft Purview
-Once your organization agrees on the high-level objectives and goals, there will be many questions from multiple groups. ItΓÇÖs crucial to gather these questions in order to craft a plan to address all of the concerns. Some example questions that you may run into during the initial phase:
+ItΓÇÖs likely that a mature organization already has an existing data catalog. The key question is whether to continue to use the existing technology and sync with the Microsoft Purview Data Map and Data Catalog or not. To handle syncing with existing products in an organization, [Microsoft Purview provides Atlas REST APIs](tutorial-using-rest-apis.md). Atlas APIs provide a powerful and flexible mechanism handling both push and pull scenarios. Information can be published to Microsoft Purview using Atlas APIs for bootstrapping or to push latest updates from another system into Microsoft Purview. The information available in Microsoft Purview can also be read using Atlas APIs and then synced back to existing products.
-1. What are the main organization data sources and data systems?
-2. For data sources that aren't supported yet by Microsoft Purview, what are my options?
-3. How many Microsoft Purview instances do we need?
-4. Who are the users?
-5. Who can scan new data sources?
-6. Who can modify content inside of Microsoft Purview?
-7. What process can I use to improve the data quality in Microsoft Purview?
-8. How to bootstrap the platform with existing critical assets, glossary terms, and contacts?
-9. How to integrate with existing systems?
-10. How to gather feedback and build a sustainable process?
+For other integration scenarios such as ticketing, custom user interface, and orchestration you can use Atlas APIs and Kafka endpoints. In general, there are four integration points with Microsoft Purview:
+
+* **Data Asset** ΓÇô This enables Microsoft Purview to scan a storeΓÇÖs assets in order to enumerate what those assets are and collect any readily available metadata about them. So for SQL this could be a list of DBs, tables, stored procedures, views and config data about them kept in places like `sys.tables`. For something like Azure Data Factory (ADF) this could be enumerating all the pipelines and getting data on when they were created, last run, current state.
+* **Lineage** ΓÇô This enables Microsoft Purview to collect information from an analysis/data mutation system on how data is moving around. For something like Spark this could be gathering information from the execution of a notebook to see what data the notebook ingested, how it transformed it and where it outputted it. For something like SQL, it could be analyzing query logs to reverse engineer what mutation operations were executed and what they did. We support both push and pull based lineage depending on the needs.
+* **Classification** ΓÇô This enables Microsoft Purview to take physical samples from data sources and run them through our classification system. The classification system figures out the semantics of a piece of data. For example, we may know that a file is a Parquet file and has three columns and the third one is a string. But the classifiers we run on the samples will tell us that the string is a name, address, or phone number. Lighting up this integration point means that we've defined how Microsoft Purview can open up objects like notebooks, pipelines, parquet files, tables, and containers.
+* **Embedded Experience** ΓÇô Products that have a ΓÇ£studioΓÇ¥ like experience (such as ADF, Synapse, SQL Studio, PBI, and Dynamics) usually want to enable users to discover data they want to interact with and also find places to output data. Microsoft PurviewΓÇÖs catalog can help to accelerate these experiences by providing an embedding experience. This experience can occur at the API or the UX level at the partnerΓÇÖs option. By embedding a call to Microsoft Purview, the organization can take advantage of Microsoft PurviewΓÇÖs map of the data estate to find data assets, see lineage, check schemas, look at ratings, contacts etc.
-While you might not have the answer to most of these questions right away, it can help your organization to frame this project and ensure all ΓÇ£must-haveΓÇ¥ requirements can be met.
+## Gathering questions
-## Include the right stakeholders
+Once your organization agrees on the high-level objectives and goals, there will be many questions from multiple groups. ItΓÇÖs crucial to gather these questions in order to craft a plan to address all of the concerns. Make sure to [include relevant groups](#include-the-right-stakeholders) as you gather these questions. You can use our documentation to start answering them.
-To ensure the success of implementing Microsoft Purview for the entire enterprise, itΓÇÖs important to involve the right stakeholders. Only a few people are involved in the initial phase. However, as the scope expands, you'll require more personas to contribute to the project and provide feedback.
+Some example questions that you may run into during the initial phase:
+
+- What are the main data sources and data systems in our organization?
+- [What data sources are supported?](microsoft-purview-connector-overview.md)
+- For data sources that aren't supported yet by Microsoft Purview, what are my options?
+- [How should we budget for Microsoft Purview?](concept-guidelines-pricing.md)
+- [How many Microsoft Purview instances do we need?](concept-best-practices-accounts.md)
+- [Who will use Microsoft Purview, and what roles will they have?](catalog-permissions.md)
+- [Who can scan new data sources?](catalog-permissions.md)
+- [Who can modify content inside of Microsoft Purview?](catalog-permissions.md)
+- What processes can I use to improve the data quality in Microsoft Purview?
+- How to bootstrap the platform with existing critical assets, [glossary terms](concept-best-practices-glossary.md), and contacts?
+- How to integrate with existing systems?
+- [How can we secure Microsoft Purview?](concept-best-practices-security.md)
+- How can we gather feedback and build a sustainable process?
+- [What can we do in a disaster situation?](concept-best-practices-migration.md)
+- [We're already using Azure Data Catalog, can we migrate to Microsoft Purview?](../data-catalog/data-catalog-migration-to-azure-purview.md)
+
+Even if you might not have the answer to most of these questions right away, gathering questions can help your organization to frame this project and ensure all ΓÇ£must-haveΓÇ¥ requirements can be met.
+
+### Include the right stakeholders
+
+To ensure the success of implementing Microsoft Purview for your entire organization, itΓÇÖs important to involve the right stakeholders. Only a few people are involved in the initial phase. However, as the scope expands, you'll require more personas to contribute to the project and provide feedback.
Some key stakeholders that you may want to include:
Some key stakeholders that you may want to include:
|**DevOps**|Line-of-Business application development and implementation; may include writing scripts and orchestration capabilities| |**Data Security Specialist**|Assess overall network and data security, which involves data coming in and out of Microsoft Purview|
-## Identify key scenarios
-
-Microsoft Purview governance services can be used to centrally manage data governance across an organizationΓÇÖs data estate spanning cloud and on-premises environments. To have a successful implementation, you must identify key scenarios that are critical to the business. These scenarios can cross business unit boundaries or impact multiple user personas either upstream or downstream.
-
-These scenarios can be written up in various ways, but you should include at least these five dimensions:
-
-1. Persona ΓÇô Who are the users?
-2. Source system ΓÇô What are the data sources such as Azure Data Lake Storage Gen2 or Azure SQL Database?
-3. Impact Area ΓÇô What is the category of this scenario?
-4. Detail scenarios ΓÇô How the users use Microsoft Purview to solve problems?
-5. Expected outcome ΓÇô What is the success criteria?
-
-The scenarios must be specific, actionable, and executable with measurable results. Some example scenarios that you can use:
-
-|Scenario|Detail|Persona|
-||||
-|Catalog business-critical assets|I need to have information about each data sets to have a good understanding of what it is. This scenario includes both business and technical metadata data about the data set in the catalog. The data sources include Azure Data Lake Storage Gen2, Azure Synapse DW, and/or Power BI. This scenario also includes on-premises resources such as SQL Server.|Business Analyst, Data Scientist, Data Engineer|
-|Discover business-critical assets|I need to have a search engine that can search through all metadata in the catalog. I should be able to search using technical term, business term with either simple or complex search using wildcard.|Business Analyst, Data Scientist, Data Engineer, Data Admin|
-|Track data to understand its origin and troubleshoot data issues|I need to have data lineage to track data in reports, predictions, or models back to its original source and understand the changes and where the data has resided through the data life cycle. This scenario needs to support prioritized data pipelines Azure Data Factory and Databricks.|Data Engineer, Data Scientist|
-|Enrich metadata on critical data assets|I need to enrich the data set in the catalog with technical metadata that is generated automatically. Classification and labeling are some examples.|Data Engineer, Domain/Business Owner|
-|Govern data assets with friendly user experience|I need to have a Business glossary for business-specific metadata. The business users can use Microsoft Purview for self-service scenarios to annotate their data and enable the data to be discovered easily via search.|Domain/Business Owner, Business Analyst, Data Scientist, Data Engineer|
- ## Deployment models If you have only one small group using Microsoft Purview with basic consumption use cases, the approach could be as simple as having one Microsoft Purview instance to service the entire group. However, you may also wonder whether your organization needs more than one Microsoft Purview instance. And if using multiple Microsoft Purview instances, how can employees promote the assets from one stage to another.
However, there are exceptions to this pattern:
3. **Conglomerates and federated model** ΓÇô Conglomerates often have many business units (BUs) that operate separately, and, in some cases, they won't even share billing with each other. In those cases, the organization will end up creating a Microsoft Purview instance for each BU. This model isn't ideal, but may be necessary, especially because BUs are often not willing to share billing. 4. **Compliance** ΓÇô There are some strict compliance regimes, which treat even metadata as sensitive and require it to be in a specific geography. If a company has multiple geographies, the only solution is to have multiple Microsoft Purview instances, one for each geography.
-### Create a process to move to production
-
-Some organizations may decide to keep things simple by working with a single production version of Microsoft Purview. They probably donΓÇÖt need to go beyond discovery, search, and browse scenarios. If some assets have incorrect glossary terms, itΓÇÖs quite forgiving to let people self-correct. However, most organizations that want to deploy Microsoft Purview across various business units will want to have some form of process and control.
-
-Another important aspect to include in your production process is how classifications and labels can be migrated. Microsoft Purview has over 90 system classifiers. You can apply system or custom classifications on file, table, or column assets. Classifications are like subject tags and are used to mark and identify content of a specific type found within your data estate during scanning. Sensitivity labels are used to identify the categories of classification types within your organizational data, and then group the policies you wish to apply to each category. It makes use of the same sensitive information types as Microsoft 365, allowing you to stretch your existing security policies and protection across your entire content and data estate. It can scan and automatically classify documents. For example, if you have a file named multiple.docx and it has a National ID number in its content, Microsoft Purview will add classification such as EU National Identification Number in the Asset Detail page.
-
-In the Microsoft Purview Data Map, there are several areas where the Catalog Administrators need to ensure consistency and maintenance best practices over its life cycle:
+For more information, see our [accounts architecture best practices guide](concept-best-practices-accounts.md) and our [default account guide](concept-default-purview-account.md).
-* **Data assets** ΓÇô Data sources will need to be rescanned across environments. ItΓÇÖs not recommended to scan only in development and then regenerate them using APIs in Production. The main reason is that the Microsoft Purview scanners do a lot more ΓÇ£wiringΓÇ¥ behind the scenes on the data assets, which could be complex to move them to a different Microsoft Purview instance. ItΓÇÖs much easier to just add the same data source in production and scan the sources again. The general best practice is to have documentation of all scans, connections, and authentication mechanisms being used.
-* **Scan rule sets** ΓÇô This is your collection of rules assigned to specific scan such as file type and classifications to detect. If you donΓÇÖt have that many scan rule sets, itΓÇÖs possible to just re-create them manually again via Production. This will require an internal process and good documentation. However, if your rule sets change on a daily or weekly basis, this could be addressed by exploring the REST API route.
-* **Custom classifications** ΓÇô Your classifications may not also change regularly. During the initial phase of deployment, it may take some time to understand various requirements to come up with custom classifications. However, once settled, this will require little change. So the recommendation here's to manually migrate any custom classifications over or use the REST API.
-* **Glossary** ΓÇô ItΓÇÖs possible to export and import glossary terms via the UX. For automation scenarios, you can also use the REST API.
-* **Resource set pattern policies** ΓÇô This functionality is advanced for any typical organizations to apply. In some cases, your Azure Data Lake Storage has folder naming conventions and specific structure that may cause problems for Microsoft Purview to generate the resource set. Your business unit may also want to change the resource set construction with more customizations to fit the business needs. For this scenario, itΓÇÖs best to keep track of all changes via REST API, and document the changes through external versioning platform.
-* **Role assignment** ΓÇô This is where you control who has access to Microsoft Purview and which permissions they have. Microsoft Purview also has REST API to support export and import of users and roles but this isn't Atlas API-compatible. The recommendation is to assign an Azure Security Group and manage the group membership instead.
-
-### Plan and implement different integration points with Microsoft Purview
+## Create a process to move to production
-ItΓÇÖs likely that a mature organization already has an existing data catalog. The key question is whether to continue to use the existing technology and sync with the Microsoft Purview Data Map and Data Catalog or not. To handle syncing with existing products in an organization, Microsoft Purview provides Atlas REST APIs. Atlas APIs provide a powerful and flexible mechanism handling both push and pull scenarios. Information can be published to Microsoft Purview using Atlas APIs for bootstrapping or to push latest updates from another system into Microsoft Purview. The information available in Microsoft Purview can also be read using Atlas APIs and then synced back to existing products.
+Some organizations may decide to keep things simple by working with a single production version of Microsoft Purview. They probably donΓÇÖt need to go beyond discovery, search, and browse scenarios. However, most organizations that want to deploy Microsoft Purview across various business units will want to have some form of process and control.
-For other integration scenarios such as ticketing, custom user interface, and orchestration you can use Atlas APIs and Kafka endpoints. In general, there are four integration points with Microsoft Purview:
+Below we've provided a potential four phase deployment plan that includes tasks, helpful links, and acceptance criteria for each phase:
-* **Data Asset** ΓÇô This enables Microsoft Purview to scan a storeΓÇÖs assets in order to enumerate what those assets are and collect any readily available metadata about them. So for SQL this could be a list of DBs, tables, stored procedures, views and config data about them kept in places like `sys.tables`. For something like Azure Data Factory (ADF) this could be enumerating all the pipelines and getting data on when they were created, last run, current state.
-* **Lineage** ΓÇô This enables Microsoft Purview to collect information from an analysis/data mutation system on how data is moving around. For something like Spark this could be gathering information from the execution of a notebook to see what data the notebook ingested, how it transformed it and where it outputted it. For something like SQL, it could be analyzing query logs to reverse engineer what mutation operations were executed and what they did. We support both push and pull based lineage depending on the needs.
-* **Classification** ΓÇô This enables Microsoft Purview to take physical samples from data sources and run them through our classification system. The classification system figures out the semantics of a piece of data. For example, we may know that a file is a Parquet file and has three columns and the third one is a string. But the classifiers we run on the samples will tell us that the string is a name, address, or phone number. Lighting up this integration point means that we've defined how Microsoft Purview can open up objects like notebooks, pipelines, parquet files, tables, and containers.
-* **Embedded Experience** ΓÇô Products that have a ΓÇ£studioΓÇ¥ like experience (such as ADF, Synapse, SQL Studio, PBI, and Dynamics) usually want to enable users to discover data they want to interact with and also find places to output data. Microsoft PurviewΓÇÖs catalog can help to accelerate these experiences by providing an embedding experience. This experience can occur at the API or the UX level at the partnerΓÇÖs option. By embedding a call to Microsoft Purview, the organization can take advantage of Microsoft PurviewΓÇÖs map of the data estate to find data assets, see lineage, check schemas, look at ratings, contacts etc.
+1. [Phase 1: Pilot](#phase-1-pilot)
+1. [Phase 2: Minimum viable product](#phase-2-minimum-viable-product)
+1. [Phase 3: Pre-production](#phase-3-pre-production)
+1. [Phase 4: Production](#phase-4-production)
-## Phase 1: Pilot
+### Phase 1: Pilot
In this phase, Microsoft Purview must be created and configured for a small set of users. Usually, it's just a group of 2-3 people working together to run through end-to-end scenarios. They're considered the advocates of Microsoft Purview in their organization. The main goal of this phase is to ensure key functionalities can be met and the right stakeholders are aware of the project.
-### Tasks to complete
+#### Tasks to complete
|Task|Detail|Duration| |||| |Gather & agree on requirements|Discussion with all stakeholders to gather a full set of requirements. Different personas must participate to agree on a subset of requirements to complete for each phase of the project.|One Week|
-|Navigating Microsoft Purview|Understand how to use Microsoft Purview from the home page.|One Day|
-|Configure ADF for lineage|Identify key pipelines and data assets. Gather all information required to connect to an internal ADF account.|One Day|
-|Scan a data source such as Azure Data Lake Storage|Add the data source and set up a scan. Ensure the scan successfully detects all assets.|Two Day|
-|Search and browse|Allow end users to access Microsoft Purview and perform end-to-end search and browse scenarios.|One Day|
+|[Navigating the Microsoft Purview governance portal](use-azure-purview-studio.md)|Understand how to use Microsoft Purview from the home page.|One Day|
+|[Configure ADF for lineage](how-to-link-azure-data-factory.md)|Identify key pipelines and data assets. Gather all information required to connect to an internal ADF account.|One Day|
+|Scan a data source such as [Azure Data Lake Storage Gen2](register-scan-adls-gen2.md) or a [SQL server.](tutorial-register-scan-on-premises-sql-server.md)|Add the data source and set up a scan. Ensure the scan successfully detects all assets.|Two Day|
+|[Search](how-to-search-catalog.md) and [browse](how-to-browse-catalog.md)|Allow end users to access Microsoft Purview and perform end-to-end search and browse scenarios.|One Day|
+
+#### Other helpful links
-### Acceptance criteria
+- [Create a Microsoft Purview account](create-catalog-portal.md)
+- [Create a collection](quickstart-create-collection.md)
+- [Concept: Permissions and access](catalog-permissions.md)
+- [Microsoft Purview product glossary](reference-azure-purview-glossary.md)
+
+#### Acceptance criteria
* Microsoft Purview account is created successfully in organization subscription under the organization tenant. * A small group of users with multiple roles can access Microsoft Purview.
In this phase, Microsoft Purview must be created and configured for a small set
* Presentation and demo to raise awareness to key stakeholders. * Buy-in from management to approve more resources for MVP phase.
-## Phase 2: Minimum viable product
+### Phase 2: Minimum viable product
Once you have the agreed requirements and participated business units to onboard Microsoft Purview, the next step is to work on a Minimum Viable Product (MVP) release. In this phase, you'll expand the usage of Microsoft Purview to more users who will have more needs horizontally and vertically. There will be key scenarios that must be met horizontally for all users such as glossary terms, search, and browse. There will also be in-depth requirements vertically for each business unit or group to cover specific end-to-end scenarios such as lineage from Azure Data Lake Storage to Azure Synapse DW to Power BI.
-### Tasks to complete
+#### Tasks to complete
|Task|Detail|Duration| ||||
Once you have the agreed requirements and participated business units to onboard
|[Scan Power BI](register-scan-power-bi-tenant.md)|If your organization uses Power BI, you can scan Power BI in order to gather all data assets being used by Data Scientists or Data Analysts that have requirements to include lineage from the storage layer.|1-2 Weeks| |[Import glossary terms](how-to-create-import-export-glossary.md)|In most cases, your organization may already develop a collection of glossary terms and term assignment to assets. This will require an import process into Microsoft Purview via .csv file.|One Week| |Add contacts to assets|For top assets, you may want to establish a process to either allow other personas to assign contacts or import via REST APIs.|One Week|
-|Add sensitive labels and scan|This might be optional for some organizations, depending on the usage of Labeling from Microsoft 365.|1-2 Weeks|
-|Get classification and sensitive insights|For reporting and insight in Microsoft Purview, you can access this functionality to get various reports and provide presentation to management.|One Day|
+|[Add sensitive labels and scan](how-to-automatically-label-your-content.md)|This might be optional for some organizations, depending on the usage of Labeling from Microsoft 365.|1-2 Weeks|
+|[Get classification and sensitive insights](concept-insights.md)|For reporting and insight in Microsoft Purview, you can access this functionality to get various reports and provide presentation to management.|One Day|
|Onboard more users using Microsoft Purview managed users|This step will require the Microsoft Purview Admin to work with the Azure Active Directory Admin to establish new Security Groups to grant access to Microsoft Purview.|One Week|
-### Acceptance criteria
+#### Other helpful links
+
+- [Collections architecture best practices](concept-best-practices-collections.md)
+- [Classification best practices](concept-best-practices-classification.md)
+- [Glossary best practices](concept-best-practices-glossary.md)
+- [Labeling best practices](concept-best-practices-sensitivity-labels.md)
+- [Asset lifecycle best practices](concept-best-practices-asset-lifecycle.md)
+
+#### Acceptance criteria
* Successfully onboard a larger group of users to Microsoft Purview (50+) * Scan business critical data sources
Once you have the agreed requirements and participated business units to onboard
* Successfully test important labeling on key assets * Successfully met minimum scenarios for participated business unitsΓÇÖ users
-## Phase 3: Pre-production
+### Phase 3: Pre-production
Once the MVP phase has passed, itΓÇÖs time to plan for pre-production milestone. Your organization may decide to have a separate instance of Microsoft Purview for pre-production and production, or keep the same instance but restrict access. Also in this phase, you may want to include scanning on on-premises data sources such as SQL Server. If there's any gap in data sources not supported by Microsoft Purview, it's time to explore the Atlas API to understand other options.
-### Tasks to complete
+#### Tasks to complete
|Task|Detail|Duration| ||||
-|Refine your scan using scan rule set|Your organization will have many data sources for pre-production. ItΓÇÖs important to pre-define key criteria for scanning so that classifications and file extension can be applied consistently across the board.|1-2 Days|
-|Assess region availability for scan|Depending on the region of the data sources and organizational requirements on compliance and security, you may want to consider what regions must be available for scanning.|One Day|
-|Understand firewall concept when scanning|This step requires some exploration of how the organization configures its firewall and how Microsoft Purview can authenticate itself to access the data sources for scanning.|One Day|
-|Understand Private Link concept when scanning|If your organization uses Private Link, you must lay out the foundation of network security to include Private Link as a part of the requirements.|One Day|
+|[Refine your scan using scan rule set](create-a-scan-rule-set.md)|Your organization will have many data sources for pre-production. ItΓÇÖs important to pre-define key criteria for scanning so that classifications and file extension can be applied consistently across the board.|1-2 Days|
+|[Assess region availability for scan for each of your sources by checking source pages](microsoft-purview-connector-overview.md)|Depending on the region of the data sources and organizational requirements on compliance and security, you may want to consider what regions must be available for scanning.|One Day|
+|[Understand firewall concept when scanning](concept-best-practices-security.md#network-security)|This step requires some exploration of how the organization configures its firewall and how Microsoft Purview can authenticate itself to access the data sources for scanning.|One Day|
+|[Understand Private Link concept when scanning](catalog-private-link.md)|If your organization uses Private Link, you must lay out the foundation of network security to include Private Link as a part of the requirements.|One Day|
|[Scan on-premises SQL Server](register-scan-on-premises-sql-server.md)|This is optional if you have on-premises SQL Server. The scan will require setting up [Self-hosted Integration Runtime](manage-integration-runtimes.md) and adding SQL Server as a data source.|1-2 Weeks|
-|Use Microsoft Purview REST API for integration scenarios|If you have requirements to integrate Microsoft Purview with other third party technologies such as orchestration or ticketing system, you may want to explore REST API area.|1-4 Weeks|
-|Understand Microsoft Purview pricing|This step will provide the organization important financial information to make decision.|1-5 Days|
+|[Use Microsoft Purview REST API for integration scenarios](tutorial-using-rest-apis.md)|If you have requirements to integrate Microsoft Purview with other third party technologies such as orchestration or ticketing system, you may want to explore REST API area.|1-4 Weeks|
+|[Understand Microsoft Purview pricing](concept-guidelines-pricing.md)|This step will provide the organization important financial information to make decision.|1-5 Days|
-### Acceptance criteria
+#### Other helpful links
+
+- [Labeling best practices](concept-best-practices-sensitivity-labels.md)
+- [Network architecture best practices](concept-best-practices-network.md)
+
+#### Acceptance criteria
* Successfully onboard at least one business unit with all of users * Scan on-premises data source such as SQL Server * POC at least one integration scenario using REST API * Complete a plan to go to production, which should include key areas on infrastructure and security
-## Phase 4: Production
+### Phase 4: Production
The above phases should be followed to create an effective data lifecycle management, which is the foundation for better governance programs. Data governance will help your organization prepare for the growing trends such as AI, Hadoop, IoT, and blockchain. It's just the start for many things data and analytics, and there's plenty more that can be discussed. The outcome of this solution would deliver: * **Business Focused** - A solution that is aligned to business requirements and scenarios over technical requirements. * **Future Ready** - A solution will maximize default features of the platform and use standardized industry practices for configuration or scripting activities to support the advancements/evolution of the platform.
-### Tasks to complete
+#### Tasks to complete
|Task|Detail|Duration| |||| |Scan production data sources with Firewall enabled|If this is optional when firewall is in place but itΓÇÖs important to explore options to hardening your infrastructure.|1-5 Days|
-|Enable Private Link|If this is optional when Private Link is used. Otherwise, you can skip this as itΓÇÖs a must-have criterion when Private is enabled.|1-5 Days|
-|Create automated workflow|Workflow is important to automate process such as approval, escalation, review and issue management.|2-3 Weeks|
-|Operation documentation|Data governance isn't a one-time project. It's an ongoing program to fuel data-driven decision making and creating opportunities for business. It's critical to document key procedure and business standards.|One Week|
+|[Enable Private Link](catalog-private-link.md)|If this is optional when Private Link is used. Otherwise, you can skip this as itΓÇÖs a must-have criterion when Private is enabled.|1-5 Days|
+|[Create automated workflow](concept-workflow.md)|Workflow is important to automate process such as approval, escalation, review and issue management.|2-3 Weeks|
+|Create operation documentation|Data governance isn't a one-time project. It's an ongoing program to fuel data-driven decision making and creating opportunities for business. It's critical to document key procedure and business standards.|One Week|
+
+#### Other helpful links
-### Acceptance criteria
+- [Manage workflow runs](how-to-workflow-manage-runs.md)
+- [Workflow requests and approvals](how-to-workflow-manage-requests-approvals.md)
+- [Manage integration runtimes](manage-integration-runtimes.md)
+- [Request access to a data asset](how-to-request-access.md)
+
+#### Acceptance criteria
* Successfully onboard all business unit and their users * Successfully meet infrastructure and security requirements for production
More hardening steps can be taken:
* Increase security posture by enabling scan on firewall resources or use Private Link * Fine-tune scope scan to improve scan performance
-* Use REST APIs to export critical metadata and properties for backup and recovery
-* Use workflow to automate ticketing and eventing to avoid human errors
+* [Use REST APIs](tutorial-atlas-2-2-apis.md) to export critical metadata and properties for backup and recovery
+* [Use workflow](how-to-workflow-business-terms-approval.md) to automate ticketing and eventing to avoid human errors
+* [Use policies](concept-data-owner-policies.md) to manage access to data assets through the Microsoft Purview governance portal.
+
+## Lifecycle considerations
+
+Another important aspect to include in your production process is how classifications and labels can be migrated. Microsoft Purview has over 90 system classifiers. You can apply system or custom classifications on file, table, or column assets. Classifications are like subject tags and are used to mark and identify content of a specific type found within your data estate during scanning. Sensitivity labels are used to identify the categories of classification types within your organizational data, and then group the policies you wish to apply to each category. It makes use of the same sensitive information types as Microsoft 365, allowing you to stretch your existing security policies and protection across your entire content and data estate. It can scan and automatically classify documents. For example, if you have a file named multiple.docx and it has a National ID number in its content, Microsoft Purview will add classification such as EU National Identification Number in the Asset Detail page.
+
+In the Microsoft Purview Data Map, there are several areas where the Catalog Administrators need to ensure consistency and maintenance best practices over its life cycle:
+
+* **Data assets** ΓÇô Data sources will need to be rescanned across environments. ItΓÇÖs not recommended to scan only in development and then regenerate them using APIs in Production. The main reason is that the Microsoft Purview scanners do a lot more ΓÇ£wiringΓÇ¥ behind the scenes on the data assets, which could be complex to move them to a different Microsoft Purview instance. ItΓÇÖs much easier to just add the same data source in production and scan the sources again. The general best practice is to have documentation of all scans, connections, and authentication mechanisms being used.
+* **Scan rule sets** ΓÇô This is your collection of rules assigned to specific scan such as file type and classifications to detect. If you donΓÇÖt have that many scan rule sets, itΓÇÖs possible to just re-create them manually again via Production. This will require an internal process and good documentation. However, if your rule sets change on a daily or weekly basis, this could be addressed by exploring the REST API route.
+* **Custom classifications** ΓÇô Your classifications may not also change regularly. During the initial phase of deployment, it may take some time to understand various requirements to come up with custom classifications. However, once settled, this will require little change. So the recommendation here's to manually migrate any custom classifications over or use the REST API.
+* **Glossary** ΓÇô ItΓÇÖs possible to export and import glossary terms via the UX. For automation scenarios, you can also use the REST API.
+* **Resource set pattern policies** ΓÇô This functionality is advanced for any typical organizations to apply. In some cases, your Azure Data Lake Storage has folder naming conventions and specific structure that may cause problems for Microsoft Purview to generate the resource set. Your business unit may also want to change the resource set construction with more customizations to fit the business needs. For this scenario, itΓÇÖs best to keep track of all changes via REST API, and document the changes through external versioning platform.
+* **Role assignment** ΓÇô This is where you control who has access to Microsoft Purview and which permissions they have. Microsoft Purview also has REST API to support export and import of users and roles but this isn't Atlas API-compatible. The recommendation is to assign an Azure Security Group and manage the group membership instead.
## Moving tenants
purview Manage Data Sources https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/purview/manage-data-sources.md
Use the following steps to register a new source.
3. Select **Register**.
+>[!IMPORTANT]
+>Most data sources have additional information and prerequisites to register and scan them in Microsoft Purview. For a list of all available sources, and links to source-specific instructions for registeration and scanning, see our [supported sources article.](microsoft-purview-connector-overview.md#microsoft-purview-data-map-available-data-sources)
+ ## View sources
-You can view all registered sources on the **Data Map** tab of the Microsoft Purview governance portal. There are two view types: map view and list view.
+You can view all registered sources on the **Data Map** tab of the Microsoft Purview governance portal.
+There are two view types:
+
+- [The map view](#map-view)
+- [The list view](#table-view)
### Map view
To create a hierarchy of collections, assign higher-level collections as a paren
:::image type="content" source="media/manage-data-sources/collections.png" alt-text="A hierarchy of collections in the Microsoft Purview governance portal":::
-You can remove sources from a hierarchy by selecting *None* for the parent. Unparented sources are grouped in a dotted box in the map view with no arrows linking them to parents.
+>[!TIP]
+>You can remove sources from a hierarchy by selecting *None* for the parent. Unparented sources are grouped in a dotted box in the map view with no arrows linking them to parents.
## Next steps
purview Microsoft Purview Connector Overview https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/purview/microsoft-purview-connector-overview.md
Previously updated : 06/17/2022 Last updated : 08/03/2022
The table below shows the supported capabilities for each data source. Select th
|| SQL Server on Azure-Arc| No |No | No |[Yes (Preview)](how-to-data-owner-policies-arc-sql-server.md) | No | || [Teradata](register-scan-teradata-source.md)| [Yes](register-scan-teradata-source.md#register)| [Yes](register-scan-teradata-source.md#scan)| [Yes*](register-scan-teradata-source.md#lineage) | No| No | |File|[Amazon S3](register-scan-amazon-s3.md)|[Yes](register-scan-amazon-s3.md)| [Yes](register-scan-amazon-s3.md)| Limited* | No| No |
+||[HDFS](register-scan-hdfs.md)|[Yes](register-scan-hdfs.md)| [Yes](register-scan-hdfs.md)| No | No| No |
|Services and apps| [Erwin](register-scan-erwin-source.md)| [Yes](register-scan-erwin-source.md#register)| No | [Yes](register-scan-erwin-source.md#lineage)| No| No | || [Looker](register-scan-looker-source.md)| [Yes](register-scan-looker-source.md#register)| No | [Yes](register-scan-looker-source.md#lineage)| No| No | || [Power BI](register-scan-power-bi-tenant.md)| [Yes](register-scan-power-bi-tenant.md)| No | [Yes](how-to-lineage-powerbi.md)| No| No |
purview Register Scan Hdfs https://github.com/MicrosoftDocs/azure-docs/commits/main/articles/purview/register-scan-hdfs.md