+| 31019| ServiceBusHeartBeat| This event indicates that we successfully sent a request to your tenant's Service Bus instance.|

+* **Passwordless authentication** - Implement passwordless authentication using the Microsoft Authenticator app or FIDO2 Security keys:

+| [Migrate from ADAL to MSAL](recommendation-migrate-from-adal-to-msal.md) | Applications | All licenses | Generally available* |

+*The Migrate from ADAL to MSAL recommendation is generally available, but rolling out in phases. If you don't see this recommendation in your tenant, check back later.

+ Title: Azure Active Directory SSO integration with BlueConic

+description: Learn how to configure single sign-on between Azure Active Directory and BlueConic.

+# Azure Active Directory SSO integration with BlueConic

+In this article, you'll learn how to integrate BlueConic with Azure Active Directory (Azure AD). BlueConic is the customer data platform (CDP) that puts unified, privacy-compliant first-party data into the hands of business teams that want to transform customer relationships and unleash growth. When you integrate BlueConic with Azure AD, you can:

+* Control in Azure AD who has access to BlueConic.

+* Enable your users to be automatically signed-in to BlueConic with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+You'll configure and test Azure AD single sign-on for BlueConic in a test environment. BlueConic supports **IDP** initiated single sign-on.

+## Prerequisites

+To integrate Azure Active Directory with BlueConic, you need:

+* An Azure AD user account. If you don't already have one, you can [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* One of the following roles: Global Administrator, Cloud Application Administrator, Application Administrator, or owner of the service principal.

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* BlueConic single sign-on (SSO) enabled subscription.

+## Add application and assign a test user

+Before you begin the process of configuring single sign-on, you need to add the BlueConic application from the Azure AD gallery. You need a test user account to assign to the application and test the single sign-on configuration.

+### Add BlueConic from the Azure AD gallery

+Add BlueConic from the Azure AD application gallery to configure single sign-on with BlueConic. For more information on how to add application from the gallery, see the [Quickstart: Add application from the gallery](../manage-apps/add-application-portal.md).

+### Create and assign Azure AD test user

+Follow the guidelines in the [create and assign a user account](../manage-apps/add-application-portal-assign-users.md) article to create a test user account in the Azure portal called B.Simon.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, and assign roles. The wizard also provides a link to the single sign-on configuration pane in the Azure portal. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides).

+## Configure Azure AD SSO

+Complete the following steps to enable Azure AD single sign-on in the Azure portal.

+1. In the Azure portal, on the **BlueConic** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, select the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, perform the following steps:

+ a. In the **Identifier** textbox, type a URL using the following pattern:

+ `https://<SUBDOMAIN>.blueconic.net/saml/metadata`

+ b. In the **Reply URL** textbox, type a URL using the following pattern:

+ `https://<SUBDOMAIN>.blueconic.net/saml/acs`

+ > [!Note]

+ > These values are not real. Update these values with the actual Identifier and Reply URL. Contact [BlueConic support team](mailto:support@blueconic.com) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. On the **Set-up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Certificate (Base64)** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up BlueConic** section, copy the appropriate URL(s) based on your requirement.

+ 

+## Configure BlueConic SSO

+To configure single sign-on on **BlueConic** side, you need to send the downloaded **Certificate (Base64)** and appropriate copied URLs from Azure portal to [BlueConic support team](mailto:support@blueconic.com). They set this setting to have the SAML SSO connection set properly on both sides

+### Create BlueConic test user

+In this section, you create a user called Britta Simon in BlueConic. Work with [BlueConic support team](mailto:support@blueconic.com) to add the users in the BlueConic platform. Users must be created and activated before you use single sign-on.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+* Click on Test this application in Azure portal and you should be automatically signed in to the BlueConic for which you set up the SSO.

+* You can use Microsoft My Apps. When you click the BlueConic tile in the My Apps, you should be automatically signed in to the BlueConic for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Next steps

+Once you configure BlueConic you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: Azure Active Directory SSO integration with Getty Images

+description: Learn how to configure single sign-on between Azure Active Directory and Getty Images.

+# Azure Active Directory SSO integration with Getty Images

+In this article, you'll learn how to integrate Getty Images with Azure Active Directory (Azure AD). Getty Images finds the perfect image for your next project from the world's best photo library of creative stock photos, vector art illustrations and stock photography. When you integrate Getty Images with Azure AD, you can:

+* Control in Azure AD who has access to Getty Images.

+* Enable your users to be automatically signed-in to Getty Images with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+You'll configure and test Azure AD single sign-on for Getty Images in a test environment. Getty Images supports both **SP** and **IDP** initiated single sign-on and **Just In Time** user provisioning.

+> [!NOTE]

+> Identifier of this application is a fixed string value so only one instance can be configured in one tenant.

+## Prerequisites

+To integrate Azure Active Directory with Getty Images, you need:

+* An Azure AD user account. If you don't already have one, you can [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* One of the following roles: Global Administrator, Cloud Application Administrator, Application Administrator, or owner of the service principal.

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Getty Images single sign-on (SSO) enabled subscription.

+## Add application and assign a test user

+Before you begin the process of configuring single sign-on, you need to add the Getty Images application from the Azure AD gallery. You need a test user account to assign to the application and test the single sign-on configuration.

+### Add Getty Images from the Azure AD gallery

+Add Getty Images from the Azure AD application gallery to configure single sign-on with Getty Images. For more information on how to add application from the gallery, see the [Quickstart: Add application from the gallery](../manage-apps/add-application-portal.md).

+### Create and assign Azure AD test user

+Follow the guidelines in the [create and assign a user account](../manage-apps/add-application-portal-assign-users.md) article to create a test user account in the Azure portal called B.Simon.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, and assign roles. The wizard also provides a link to the single sign-on configuration pane in the Azure portal. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides).

+## Configure Azure AD SSO

+Complete the following steps to enable Azure AD single sign-on in the Azure portal.

+1. In the Azure portal, on the **Getty Images** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, select the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, perform the following steps:

+ a. In the **Identifier** textbox, type the URL:

+ `https://gettyimages.com/`

+ b. In the **Reply URL** textbox, type a URL using one of the following patterns:

+ | **Reply URL** |

+ |--|

+ | `https://www.gettyimages.com/sign-in/sso/acs` |

+ | `https://www.gettyimages.<Environment>/sign-in/sso/acs` |

+ | `https://www.gettyimages.com.<Environment>/sign-in/sso/acs` |

+1. If you wish to configure the application in **SP** initiated mode, then perform the following step:

+ In the **Sign on URL** textbox, type the URL:

+ `https://www.gettyimages.in/sign-in/sso`

+ > [!NOTE]

+ > The Reply URL value is not real. Update this value with the actual Reply URL. Contact [Getty Images support team](mailto:support@gettyimages.com) to get the value. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. Getty Images application expects the SAML assertions in a specific format, which requires you to add custom attribute mappings to your SAML token attributes configuration. The following screenshot shows the list of default attributes.

+ 

+1. In addition to above, Getty Images application expects few more attributes to be passed back in SAML response, which are shown below. These attributes are also pre populated but you can review them as per your requirements.

+ | Name | Source Attribute|

+ | | |

+ | groupid | user.groups |

+ > [!Note]

+ > The groupid claim is a constant value which will be provided to you by [Getty Images support team](mailto:support@gettyimages.com).

+1. On the **Set up single sign-on with SAML** page, in the **SAML Signing Certificate** section, select copy button to copy **App Federation Metadata Url** and save it on your computer.

+ 

+## Configure Getty Images SSO

+To configure single sign-on on **Getty Images** side, you need to send the **App Federation Metadata Url** to [Getty Images support team](mailto:support@gettyimages.com). They set this setting to have the SAML SSO connection set properly on both sides. For more information, please refer [this](https://developers.gettyimages.com/single-sign-on/) link.

+### Create Getty Images test user

+In this section, a user called B.Simon is created in Getty Images. Getty Images supports just-in-time user provisioning, which is enabled by default. There's no action item for you in this section. If a user doesn't already exist in Getty Images, a new one is commonly created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to Getty Images Sign-on URL where you can initiate the login flow.

+* Go to Getty Images Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the Getty Images for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the Getty Images tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the Getty Images for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Additional resources

+* [What is single sign-on with Azure Active Directory?](../manage-apps/what-is-single-sign-on.md)

+* [Plan a single sign-on deployment](../manage-apps/plan-sso-deployment.md).

+## Next steps

+Once you configure Getty Images you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: Azure Active Directory SSO integration with Imagen

+description: Learn how to configure single sign-on between Azure Active Directory and Imagen.

+# Azure Active Directory SSO integration with Imagen

+In this article, you'll learn how to integrate Imagen with Azure Active Directory (Azure AD). Imagen is the cloud-native media asset management platform built to manage, enrich and distribute your organization large-scale, mission-critical video files and drive value from your content. When you integrate Imagen with Azure AD, you can:

+* Control in Azure AD who has access to Imagen.

+* Enable your users to be automatically signed-in to Imagen with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+You'll configure and test Azure AD single sign-on for Imagen in a test environment. Imagen supports **SP** initiated single sign-on and **Just In Time** user provisioning.

+## Prerequisites

+To integrate Azure Active Directory with Imagen, you need:

+* An Azure AD user account. If you don't already have one, you can [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* One of the following roles: Global Administrator, Cloud Application Administrator, Application Administrator, or owner of the service principal.

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Imagen single sign-on (SSO) enabled subscription.

+## Add application and assign a test user

+Before you begin the process of configuring single sign-on, you need to add the Imagen application from the Azure AD gallery. You need a test user account to assign to the application and test the single sign-on configuration.

+### Add Imagen from the Azure AD gallery

+Add Imagen from the Azure AD application gallery to configure single sign-on with Imagen. For more information on how to add application from the gallery, see the [Quickstart: Add application from the gallery](../manage-apps/add-application-portal.md).

+### Create and assign Azure AD test user

+Follow the guidelines in the [create and assign a user account](../manage-apps/add-application-portal-assign-users.md) article to create a test user account in the Azure portal called B.Simon.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, and assign roles. The wizard also provides a link to the single sign-on configuration pane in the Azure portal. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides).

+## Configure Azure AD SSO

+Complete the following steps to enable Azure AD single sign-on in the Azure portal.

+1. In the Azure portal, on the **Imagen** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, select the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, perform the following steps:

+ a. In the **Identifier** textbox, type a URL using the following pattern:

+ `https://<SUBDOMAIN>.imagencloud.com/sp-entityid`

+ b. In the **Reply URL** textbox, type a URL using the following pattern:

+ `https://<SUBDOMAIN>.imagencloud.com/saml/module.php/saml/sp/saml2-acs.php/imagenweb`

+ c. In the **Sign on URL** textbox, type a URL using the following pattern:

+ `https://<SUBDOMAIN>.imagencloud.com/site/login`

+

+ > [!Note]

+ > These values are not real. Update these values with the actual Identifier, Reply URL and Sign on URL. Contact [Imagen support team](mailto:support@imagen.io) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. Imagen application expects the SAML assertions in a specific format, which requires you to add custom attribute mappings to your SAML token attributes configuration. The following screenshot shows the list of default attributes.

+ 

+1. In addition to above, Imagen application expects few more attributes to be passed back in SAML response, which are shown below. These attributes are also pre populated but you can review them as per your requirements.

+ | Name | Source Attribute|

+ | | |

+ | urn:oid:0.9.2342.19200300.100.1.3 | user.mail |

+ | urn:oid:1.3.6.1.4.1.5923.1.1.1.10 | user.userprincipalname |

+ | Groups | user.groups |

+1. On the **Set-up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Federation Metadata XML** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up Imagen** section, copy the appropriate URL(s) based on your requirement.

+ 

+## Configure Imagen SSO

+To configure single sign-on on **Imagen** side, you need to send the downloaded **Federation Metadata XML** and appropriate copied URLs from Azure portal to [Imagen support team](mailto:support@imagen.io). They set this setting to have the SAML SSO connection set properly on both sides.

+### Create Imagen test user

+In this section, a user called B.Simon is created in Imagen. Imagen supports just-in-time user provisioning, which is enabled by default. There's no action item for you in this section. If a user doesn't already exist in Imagen, a new one is commonly created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+* Click on **Test this application** in Azure portal. This will redirect to Imagen Sign-on URL where you can initiate the login flow.

+* Go to Imagen Sign-on URL directly and initiate the login flow from there.

+* You can use Microsoft My Apps. When you click the Imagen tile in the My Apps, this will redirect to Imagen Sign-on URL. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Additional resources

+* [What is single sign-on with Azure Active Directory?](../manage-apps/what-is-single-sign-on.md)

+* [Plan a single sign-on deployment](../manage-apps/plan-sso-deployment.md).

+## Next steps

+Once you configure Imagen you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: Azure Active Directory SSO integration with Informatica Intelligent Data Management Cloud

+description: Learn how to configure single sign-on between Azure Active Directory and Informatica Intelligent Data Management Cloud.

+# Azure Active Directory SSO integration with Informatica Intelligent Data Management Cloud

+In this article, you'll learn how to integrate Informatica Intelligent Data Management Cloud with Azure Active Directory (Azure AD). It is a SAML SSO Auth application to enable Informatica Intelligent Data Management Cloud on Azure Native Services. When you integrate Informatica Intelligent Data Management Cloud with Azure AD, you can:

+* Control in Azure AD who has access to Informatica Intelligent Data Management Cloud.

+* Enable your users to be automatically signed-in to Informatica Intelligent Data Management Cloud with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+You'll configure and test Azure AD single sign-on for Informatica Intelligent Data Management Cloud in a test environment. Informatica Intelligent Data Management Cloud supports both **SP** and **IDP** initiated single sign-on and **Just In Time** user provisioning.

+## Prerequisites

+To integrate Azure Active Directory with Informatica Intelligent Data Management Cloud, you need:

+* An Azure AD user account. If you don't already have one, you can [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* One of the following roles: Global Administrator, Cloud Application Administrator, Application Administrator, or owner of the service principal.

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* Informatica Intelligent Data Management Cloud single sign-on (SSO) enabled subscription.

+## Add application and assign a test user

+Before you begin the process of configuring single sign-on, you need to add the Informatica Intelligent Data Management Cloud application from the Azure AD gallery. You need a test user account to assign to the application and test the single sign-on configuration.

+### Add Informatica Intelligent Data Management Cloud from the Azure AD gallery

+Add Informatica Intelligent Data Management Cloud from the Azure AD application gallery to configure single sign-on with Informatica Intelligent Data Management Cloud. For more information on how to add application from the gallery, see the [Quickstart: Add application from the gallery](../manage-apps/add-application-portal.md).

+### Create and assign Azure AD test user

+Follow the guidelines in the [create and assign a user account](../manage-apps/add-application-portal-assign-users.md) article to create a test user account in the Azure portal called B.Simon.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, and assign roles. The wizard also provides a link to the single sign-on configuration pane in the Azure portal. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides).

+## Configure Azure AD SSO

+Complete the following steps to enable Azure AD single sign-on in the Azure portal.

+1. In the Azure portal, on the **Informatica Intelligent Data Management Cloud** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, select the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, if you have **Service Provider metadata file**, perform the following steps:

+ a. Click **Upload metadata file**.

+ 

+ b. Click on **folder logo** to select the metadata file and click **Upload**.

+ 

+ c. After the metadata file is successfully uploaded, the **Identifier** and **Reply URL** values get auto populated in **Basic SAML Configuration** section.

+ > [!NOTE]

+ > You will get the **Service Provider metadata file** from the **Configure Informatica Intelligent Data Management Cloud SSO** section, which is explained later in the tutorial. If the **Identifier** and **Reply URL** values do not get auto populated, then fill in the values manually according to your requirement.

+1. If you wish to configure the application in **SP** initiated mode, then perform the following step:

+ In the **Sign on URL** textbox, type a URL using the following pattern:

+ `https://<INFORMATICA_CLOUD_REGION>.informaticacloud.com/ma/sso/<INFORMATICA_CLOUD_ORG_ID>`

+ > [!NOTE]

+ > This value is not real. Update this value with the actual Sign on URL. Contact [Informatica Intelligent Data Management Cloud support team](mailto:support@informatica.com) to get the value. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. On the **Set-up single sign-on with SAML** page, in the **SAML Signing Certificate** section, find **Certificate (Base64)** and select **Download** to download the certificate and save it on your computer.

+ 

+1. On the **Set up Informatica Intelligent Data Management Cloud** section, copy the appropriate URL(s) based on your requirement.

+ 

+## Configure Informatica Intelligent Data Management Cloud SSO

+1. Log in to your Informatica Intelligent Data Management Cloud company site as an administrator.

+1. Go to **Administrator** > **SAML Setup** and perform the following steps:

+ 

+ 1. In the **Issuer** textbox, paste the **Azure AD Identifier** value, which you have copied from the Azure portal.

+ 1. In the **Single Sign-On Service URL** textbox, paste the **Login URL**, which you have copied from the Azure portal.

+ 1. In the **Single Logout Service URL** textbox, paste the **Logout URL**, which you have copied from the Azure portal.

+ 1. Open the downloaded **Certificate (Base64)** from the Azure portal into Notepad and paste the content into the **Signing Certificate** textbox.

+ 1. Click **Save** to save the details.

+ 1. **Download Service Provider Metadata** xml file and upload in the **Basic SAML Configuration** section in the Azure portal.

+### Create Informatica Intelligent Data Management Cloud test user

+In this section, a user called B.Simon is created in Informatica Intelligent Data Management Cloud. Informatica Intelligent Data Management Cloud supports just-in-time user provisioning, which is enabled by default. There's no action item for you in this section. If a user doesn't already exist in Informatica Intelligent Data Management Cloud, a new one is commonly created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to Informatica Intelligent Data Management Cloud Sign-on URL where you can initiate the login flow.

+* Go to Informatica Intelligent Data Management Cloud Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the Informatica Intelligent Data Management Cloud for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the Informatica Intelligent Data Management Cloud tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the Informatica Intelligent Data Management Cloud for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Additional resources

+* [What is single sign-on with Azure Active Directory?](../manage-apps/what-is-single-sign-on.md)

+* [Plan a single sign-on deployment](../manage-apps/plan-sso-deployment.md).

+## Next steps

+Once you configure Informatica Intelligent Data Management Cloud you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+ Title: Azure Active Directory SSO integration with TASC (beta)

+description: Learn how to configure single sign-on between Azure Active Directory and TASC (beta).

+# Azure Active Directory SSO integration with TASC (beta)

+In this article, you'll learn how to integrate TASC (beta) with Azure Active Directory (Azure AD). TASC (beta) is a psychological testing platform for selection and career development, enabling HR professionals to make better talent decisions in beta environment. When you integrate TASC (beta) with Azure AD, you can:

+* Control in Azure AD who has access to TASC (beta).

+* Enable your users to be automatically signed-in to TASC (beta) with their Azure AD accounts.

+* Manage your accounts in one central location - the Azure portal.

+You'll configure and test Azure AD single sign-on for TASC (beta) in a test environment. TASC (beta) supports both **SP** and **IDP** initiated single sign-on and **Just In Time** user provisioning.

+## Prerequisites

+To integrate Azure Active Directory with TASC (beta), you need:

+* An Azure AD user account. If you don't already have one, you can [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).

+* One of the following roles: Global Administrator, Cloud Application Administrator, Application Administrator, or owner of the service principal.

+* An Azure AD subscription. If you don't have a subscription, you can get a [free account](https://azure.microsoft.com/free/).

+* TASC (beta) single sign-on (SSO) enabled subscription.

+## Add application and assign a test user

+Before you begin the process of configuring single sign-on, you need to add the TASC (beta) application from the Azure AD gallery. You need a test user account to assign to the application and test the single sign-on configuration.

+### Add TASC (beta) from the Azure AD gallery

+Add TASC (beta) from the Azure AD application gallery to configure single sign-on with TASC (beta). For more information on how to add application from the gallery, see the [Quickstart: Add application from the gallery](../manage-apps/add-application-portal.md).

+### Create and assign Azure AD test user

+Follow the guidelines in the [create and assign a user account](../manage-apps/add-application-portal-assign-users.md) article to create a test user account in the Azure portal called B.Simon.

+Alternatively, you can also use the [Enterprise App Configuration Wizard](https://portal.office.com/AdminPortal/home?Q=Docs#/azureadappintegration). In this wizard, you can add an application to your tenant, add users/groups to the app, and assign roles. The wizard also provides a link to the single sign-on configuration pane in the Azure portal. [Learn more about Microsoft 365 wizards.](/microsoft-365/admin/misc/azure-ad-setup-guides).

+## Configure Azure AD SSO

+Complete the following steps to enable Azure AD single sign-on in the Azure portal.

+1. In the Azure portal, on the **TASC (beta)** application integration page, find the **Manage** section and select **single sign-on**.

+1. On the **Select a single sign-on method** page, select **SAML**.

+1. On the **Set up single sign-on with SAML** page, select the pencil icon for **Basic SAML Configuration** to edit the settings.

+ 

+1. On the **Basic SAML Configuration** section, perform the following steps:

+ a. In the **Identifier** textbox, type a URL using the following pattern:

+ `https://login.beta.tascnet.be/saml/<CustomerName>`

+ b. In the **Reply URL** textbox, type a URL using the following pattern:

+ `https://login.beta.tascnet.be/saml/<CustomerName>/acs`

+1. If you wish to configure the application in **SP** initiated mode, then perform the following step:

+ In the **Sign on URL** textbox, type a URL using the following pattern:

+ `https://login.beta.tascnet.be/saml/<CustomerName>`

+ > [!NOTE]

+ > These values are not real. Update these values with the actual Identifier, Reply URL and Sign on URL. Contact [TASC (beta) support team](mailto:support@cebir.be) to get these values. You can also refer to the patterns shown in the **Basic SAML Configuration** section in the Azure portal.

+1. TASC (beta) application expects the SAML assertions in a specific format, which requires you to add custom attribute mappings to your SAML token attributes configuration. The following screenshot shows the list of default attributes.

+ 

+1. In addition to above, TASC (beta) application expects few more attributes to be passed back in SAML response, which are shown below. These attributes are also pre populated but you can review them as per your requirements.

+ | Name | Source Attribute|

+ | | |

+ | groups | user.groups |

+1. On the **Set up single sign-on with SAML** page, in the **SAML Signing Certificate** section, select copy button to copy **App Federation Metadata Url** and save it on your computer.

+ 

+## Configure TASC (beta) SSO

+To configure single sign-on on **TASC (beta)** side, you need to send the **App Federation Metadata Url** to [TASC (beta) support team](mailto:support@cebir.be). They set this setting to have the SAML SSO connection set properly on both sides.

+### Create TASC (beta) test user

+In this section, a user called B.Simon is created in TASC (beta). TASC (beta) supports just-in-time user provisioning, which is enabled by default. There's no action item for you in this section. If a user doesn't already exist in TASC (beta), a new one is commonly created after authentication.

+## Test SSO

+In this section, you test your Azure AD single sign-on configuration with following options.

+#### SP initiated:

+* Click on **Test this application** in Azure portal. This will redirect to TASC (beta) Sign-on URL where you can initiate the login flow.

+* Go to TASC (beta) Sign-on URL directly and initiate the login flow from there.

+#### IDP initiated:

+* Click on **Test this application** in Azure portal and you should be automatically signed in to the TASC (beta) for which you set up the SSO.

+You can also use Microsoft My Apps to test the application in any mode. When you click the TASC (beta) tile in the My Apps, if configured in SP mode you would be redirected to the application sign-on page for initiating the login flow and if configured in IDP mode, you should be automatically signed in to the TASC (beta) for which you set up the SSO. For more information about the My Apps, see [Introduction to the My Apps](../user-help/my-apps-portal-end-user-access.md).

+## Additional resources

+* [What is single sign-on with Azure Active Directory?](../manage-apps/what-is-single-sign-on.md)

+* [Plan a single sign-on deployment](../manage-apps/plan-sso-deployment.md).

+## Next steps

+Once you configure TASC (beta) you can enforce session control, which protects exfiltration and infiltration of your organizationΓÇÖs sensitive data in real time. Session control extends from Conditional Access. [Learn how to enforce session control with Microsoft Cloud App Security](/cloud-app-security/proxy-deployment-aad).

+|  | [Secure Personally Identifiable Information | AffinitiQuest](https://affinitiquest.io/) |

+|  | [CTC's SELMID offering](https://ctc-insight.com/selmid) |

+ --set defaultBackend.image.digest=""

+For background, see [Versions](api-management-versions.md) & [Revisions](api-management-revisions.md).

+There are certain cases where responses being returned contain some portion of data that is expensive to determine and yet remains fresh for a reasonable amount of time. As an example, consider a service built by an airline that provides information relating to flight reservations, flight status, and so on. If the user is a member of the airlines points program, they would also have information relating to their current status and accumulated mileage. This user-related information might be stored in a different system, but it may be desirable to include it in responses returned about flight status and reservations. This can be done using a process called fragment caching. The primary representation can be returned from the origin server using some kind of token to indicate where the user-related information is to be inserted.

+Application Gateway v1 is built on some legacy components and customers have deployed the gateways in many different ways which makes the one click seamless migration a near impossible goal to achieve. Hence customer involvement is required for migration.

+4. Re-enable your features.

+> [!NOTE]

+> The movement of System assigned managed identity, and User-assigned managed identity takes place automatically with the Automation account.

+You must re-enable the features that you removed before the move:

+3. Machines that are enabled with your features are visible when you've connected the existing Log Analytics workspace.

+1. Use [this script](https://github.com/azureautomation/runbooks/blob/master/Utility/AzRunAs/IdentifyRunAsRunbooks.ps1) to find out if all runbooks in your Automation account are using the Run As account.

+NAME READY UP-TO-DATE AVAILABLE AGE

+deployment.apps/cluster-metadata-operator 1/1 1 1 3d19h

+deployment.apps/clusterconnect-agent 1/1 1 1 3d19h

+deployment.apps/clusteridentityoperator 1/1 1 1 3d19h

+deployment.apps/config-agent 1/1 1 1 3d19h

+deployment.apps/controller-manager 1/1 1 1 3d19h

+deployment.apps/extension-events-collector 1/1 1 1 3d19h

+deployment.apps/extension-manager 1/1 1 1 3d19h

+deployment.apps/flux-logs-agent 1/1 1 1 3d19h

+deployment.apps/kube-aad-proxy 1/1 1 1 3d19h

+deployment.apps/metrics-agent 1/1 1 1 3d19h

+deployment.apps/resource-sync-agent 1/1 1 1 3d19h

+

+NAME READY STATUS RESTARTS AGE

+pod/cluster-metadata-operator-74747b975-9phtz 2/2 Running 0 3d19h

+pod/clusterconnect-agent-cf4c7849c-88fmf 3/3 Running 0 3d19h

+pod/clusteridentityoperator-79bdfd945f-pt2rv 2/2 Running 0 3d19h

+pod/config-agent-67bcb94b7c-d67t8 1/2 Running 0 3d19h

+pod/controller-manager-559dd48b64-v6rmk 2/2 Running 0 3d19h

+pod/extension-events-collector-85f4fbff69-55zmt 2/2 Running 0 3d19h

+pod/extension-manager-7c7668446b-69gps 3/3 Running 0 3d19h

+pod/flux-logs-agent-fc7c6c959-vgqvm 1/1 Running 0 3d19h

+pod/kube-aad-proxy-84d668c44b-j457m 2/2 Running 0 3d19h

+pod/metrics-agent-58fb8554df-5ll67 2/2 Running 0 3d19h

+pod/resource-sync-agent-dbf5db848-c9lg8 2/2 Running 0 3d19h

+### Resource Provisioning Failed

+If you receive this error, it indicates that there was an error due to which the resource could not be provisioned successfully. Please check the status of the Azure Arc enabled Kubernetes service at the following dashboard: [Azure status](https://azure.status.microsoft/en-us/status). If the status is healthy and you continue to face issues while onboarding, please raise a support ticket. If the status is unhealthy, please wait until the status becomes healthy and try onboarding again after deleting the existing connected cluster Azure resource.

+### Service Timeout

+If you receive this error, it indicates that the service timed out while provisioning the certificates. Please check the status of the Azure Arc enabled Kubernetes service at the following dashboard: [Azure status](https://azure.status.microsoft/en-us/status). If the status is healthy and you continue to face issues while onboarding, please raise a support ticket. If the status is unhealthy, please wait until the status becomes healthy and try onboarding again after deleting the existing connected cluster Azure resource.

+1. On the **Select a method** page, under the **Add a single server** tile, and then select **Generate script**.

+1. On the **Prerequisites** page, review the information and then select **Next** to Resource details page.

+ 1. In the **Connectivity method** section, If the machine is communicating through a proxy server to connect to the internet, select **Proxy server** option and specify the proxy server IP address or the name and port number that the machine will use to communicate with the proxy server. Enter the value in the format `http://<proxyURL>:<proxyport>`. Else if the machine is communicating through a private endpoint then select **Private endpoint** option and appropriate private link scope in the drop-down list. Else if the machine is communicating through a public endpoint then select **Public endpoint** option.

+ 1. In the **Automanage machine best practices** section, you may enable automanage if you want to onboard and configure best practice services like Machine configuration and Insights, based on your server needs.

+ 1. Select **Next** to Tags page.

+1. Select **Next** to Download and run script page.

+1. (Optional) If you've configured action groups for this alert rule, you can add custom properties in key:value pairs to add more information to the alert notification payload in the <a name="custom-props">**Custom properties**</a> section. Add the property **Name** and **Value** for the custom property you want included in the payload.

+ The format for extracting values from the [common schema](alerts-common-schema.md), use a "$", and then the path of the common schema field inside curly brackets. For example: `${data.essentials.monitorCondition}`.

+ This example creates an "Additional Details" tag with data about the "evaluation window start time" and "evaluation window end time".

+ This example adds data about the reason the alert was fired or resolved.

+--data-urlencode 'client_secret=<your apps client secret>' \

+[Query Prometheus metrics using Azure workbooks](./prometheus-workbooks.md)

+> [!IMPORTANT]

+> Starting September 18, 2023, the Log Analytics Daily Cap will no longer exclude the below set of data types from the daily cap, and all billable data types will

+> be capped if the daily cap is met. This change improves your ability to fully contain costs from higher-than-expected data ingestion.

+> If you have a Daily Cap set on your workspace which has [Microsoft Defender for Servers](../../defender-for-cloud/plan-defender-for-servers-select-plan.md),

+> be sure that the cap is high enough to accomodate this change. Also, be sure to set an alert (see below) so that you are notified as soon as your Daily Cap is met.

+

+Until September 18, 2023, some data security-related data types collected [Microsoft Defender for Cloud](../../security-center/index.yml) or Microsoft Sentinel are collected despite any daily cap, when the [Microsoft Defender for Servers](../../defender-for-cloud/plan-defender-for-servers-select-plan.md) solution was enabled on a workspace after June 19, 2017. The following data types will be subject to this special exception from the daily cap:

+A [data collection endpoint](../essentials/data-collection-endpoint-overview.md) is required to accept the data from the script. After you configure the DCE and link it to a DCR, you can send data over HTTP from your application. The DCE needs to be in the same region as the Log Analytics workspace where the data will be sent or the data collection rule being used.

+> [Log Analytics script samples](../logs/queries.md)

+To share access to a dashboard, you must first publish it. When you do so, other users in your organization will be able to access and modify the dashboard based on their Azure RBAC roles.

+* [Create a dashboard](azure-portal-dashboards.md) in the Azure portal.

+> Runbooks must exist in the same resource group as the Automation Account.

+> The movement of System assigned managed identity, and User-assigned managed identity takes place automatically with the Automation account. For information, see [Move your Azure Automation account to another subscription](../../automation/how-to/move-account.md?toc=/azure/azure-resource-manager/toc.json).

+ Title: Delete items by partition key value using the Azure Cosmos DB SDK (preview)

+Because the number of documents to be deleted may be large, the operation runs in the background. Though the physical deletion operation runs in the background, the effects are available immediately, as the documents to be deleted won't appear in the results of queries or read operations.

+The delete by partition key operation is constrained to consume at most 10% of the total available RU/s on the container each second. This helps in limiting the resources used by this background task.

+#### [Python](#tab/python-example)

+Use [beta-version ( >= 4.4.0b1)](https://pypi.org/project/azure-cosmos/4.4.0b1/) of the Azure Cosmos DB Python SDK to delete items by partition key. The delete by partition key API will be marked as beta.

+```python

+# Suppose our container is partitioned by tenantId, and we want to delete all the data for a particular tenant Contoso

+# Delete by logical partition key

+container.delete_all_items_by_partition_key("Contoso")

+```

+Yes, once the delete by partition key operation starts, the documents to be deleted won't appear in the results of queries or read operations. This also means that you can write new a new document with the same ID and partition key as a document to be deleted without resulting in a conflict.

+When the delete by partition key operation is issued, only the documents that exist in the container at that point in time with the partition key value will be deleted. Any new documents that come in won't be in scope for the deletion.

+- [Continuous backup (point in time restore)](../continuous-backup-restore-introduction.md) - a restore that is triggered during an ongoing delete by partition key operation may contain the results of the documents to be deleted in the restored collection. It isn't recommended to use this preview feature if you have a scenario that requires continuous backup.

+| **Python SDK v4** | *>= 4..4.0b1 (must be beta version)* | <https://pypi.org/project/azure-cosmos/4.4.0b1/> |

+- [Query an Azure Cosmos DB container](how-to-query-container.md)

+- [Transactional batch operations in Azure Cosmos DB using the .NET SDK](transactional-batch.md)

+## [Python (Preview)](#tab/python)

+Support for Partial Document Update (Patch API) in the [Azure Cosmos DB Python SDK](nosql/sdk-python.md) is available in Preview starting with version *4.4.0b2*. You can download it from the [pip Registry](https://pypi.org/project/azure-cosmos/4.4.0b2/).

+> [!NOTE]

+> Find a complete Partial Document Update sample in the [python samples repository](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cosmos/azure-cosmos/samples/document_management.py#L105C8-L122) on GitHub.

+- Run a single patch operation:

+ ```python

+ operations =

+ [

+ { op: 'replace', path: '/price', value: 355.45 }

+ ]

+

+ response = container.patch_item(item='e379aea5-63f5-4623-9a9b-4cd9b33b91d5', partition_key='road-bikes', patch_operations=operations)

+

+ ```

+- Combine multiple patch operations:

+ ```python

+ operations =

+ [

+ { op: 'add', path: '/color', value: 'silver' },

+ { op: 'remove', path: '/used' }

+ ]

+

+ response = container.patch_item(item='e379aea5-63f5-4623-9a9b-4cd9b33b91d5', partition_key='road-bikes', patch_operations=operations)

+ ```

+- Use conditional patch syntax based on filter predicate:

+ ```python

+ filter = "from products p WHERE p.used = false"

+ operations =

+ [

+ { op: 'replace', path: '/price', value: 100.00 }

+ ]

+ try:

+ container.patch_item(item='e379aea5-63f5-4623-9a9b-4cd9b33b91d5', partition_key='road-bikes', patch_operations=operations, filter_predicate=filter)

+ except exceptions.CosmosHttpResponseError as e:

+ print('\nError occured. {0}'.format(e.message))

+

+ ```

+ "uri": "@{concat('https://synapse.azure.com/monitoring/pipelineruns/',pipeline().parameters.runId,'?factory=/subscriptions/',pipeline().parameters.subscription,'/resourceGroups/',pipeline().parameters.resourceGroup,'/providers/Microsoft.DataFactory/factories/',pipeline().DataFactory)}"

+ "uri": "@{concat('https://synapse.azure.com/monitoring/pipelineruns/',pipeline().parameters.runId,'?factory=/subscriptions/',pipeline().parameters.subscription,'/resourceGroups/',pipeline().parameters.resourceGroup,'/providers/Microsoft.DataFactory/factories/',pipeline().DataFactory)}"

+If you're using the new preview alerts experience as described in [Manage and respond to security alerts in Microsoft Defender for Cloud](managing-and-responding-alerts.md), you can create sample alerts from the security alerts page in the Azure portal.

+1. Select the relevant Microsoft Defender plan/s for which you want to see alerts.

+ After a few minutes, the alerts appear in the security alerts page. They also appear anywhere else that you've configured to receive your Microsoft Defender for Cloud security alerts (connected SIEMs, email notifications, and so on).

+**To simulate a Kubernetes control plane security alert**:

+1. Run the following command from the cluster:

+1. On the relevant Kubernetes cluster, locate the following alert `Microsoft Defender for Cloud test alert for K8S (not a threat)`

+- Ensure the Defender profile\extension is installed.

+## Simulate alerts for App Service

+1. Create a new website and wait 24 hours for it to be registered with Defender for Cloud, or use an existing web site.

+ 1. Open the app service resource pane and copy the domain for the URL from the default domain field.

+1. An alert is generated within about 1-2 hours.

+### Hash reputation analysis (available in all plans)

+The intended audience of this guide is cloud solution and infrastructure architects, security architects and analysts, and anyone who's involved in protecting cloud and hybrid servers and workloads.

+- Where is my data stored and what Log Analytics workspaces do I need?

+The following table shows an overview of the Defender for Servers deployment process:

+| Stage | Details |

+| | |

+| Start protecting resources | ΓÇó When you open Defender for Cloud in the portal, it starts protecting resources with free foundational CSPM assessments and recommendations.<br /><br />ΓÇó Defender for Cloud creates a default Log Analytics workspace with the *SecurityCenterFree* solution enabled.<br /><br />ΓÇó Recommendations start appearing in the portal. |

+| Enable Defender for Servers | ΓÇó When you enable a paid plan, Defender for Cloud enables the *Security* solution on its default workspace.<br /><br />ΓÇó Enable Defender for Servers Plan 1 (subscription only) or Plan 2 (subscription and workspace).<br /><br />ΓÇó After enabling a plan, decide how you want to install agents and extensions on Azure VMs in the subscription or workgroup.<br /><br />ΓÇóBy default, auto-provisioning is enabled for some extensions. |

+| Protect AWS/GCP machines | ΓÇó For a Defender for Servers deployment, you set up a connector, turn off plans you don't need, configure auto-provisioning settings, authenticate to AWS/GCP, and deploy the settings.<br /><br />ΓÇó Auto-provisioning includes the agents used by Defender for Cloud and the Azure Connected Machine agent for onboarding to Azure with Azure Arc.<br /><br />ΓÇó AWS uses a CloudFormation template.<br /><br />ΓÇó GCP uses a Cloud Shell template.<br /><br />ΓÇó Recommendations start appearing in the portal. |

+| Protect on-premises servers | ΓÇó Onboard them as Azure Arc machines and deploy agents with automation provisioning. |

+| Foundational CSPM | ΓÇó There are no charges when you use foundational CSPM with no plans enabled.<br /><br />ΓÇó AWS/GCP machines don't need to be set up with Azure Arc for foundational CSPM. On-premises machines do.<br /><br />ΓÇó Some foundational recommendations rely only agents: Antimalware / endpoint protection (Log Analytics agent or Azure Monitor agent) \| OS baselines recommendations (Log Analytics agent or Azure Monitor agent and Guest Configuration extension) \| System updates recommendation (Log Analytics agent) |

+When you enable [Microsoft Defender for Servers](defender-for-servers-introduction.md) on an Azure subscription or a connected AWS account, all of the connected machines are protected by Defender for Servers. You can enable Microsoft Defender for Servers at the Log Analytics workspace level, but only servers reporting to that workspace will be protected and billed and those servers won't receive some benefits, such as Microsoft Defender for Endpoint, vulnerability assessment, and just-in-time VM access.

+> [!NOTE]

+> There is no need to pre-install an operating system on the VM, the sensor installation includes the operating system image.

+> [!NOTE]

+> There is no need to pre-install an operating system on the VM, the sensor installation includes the operating system image.

+> [!NOTE]

+> There is no need to pre-install an operating system on the VM, the sensor installation includes the operating system image.

+> [!NOTE]

+> There is no need to pre-install an operating system on the VM, the sensor installation includes the operating system image.

+> [!NOTE]

+> There is no need to pre-install an operating system on the VM, the sensor installation includes the operating system image.

+> [!NOTE]

+> There is no need to pre-install an operating system on the VM, the sensor installation includes the operating system image.

+| 22.3.9 | 05/2023 | Patch | 04/2024 |

+### 22.3.9

+**Release date**: 05/2023

+**Supported until**: 04/2024

+This version includes bug fixes for stability improvements.

+description: Learn how to set up a connection between HDInsight Spark cluster and Azure SQL Database. To read data, write data, and stream data into an SQL database

+1. Use the following snippet to build a JDBC URL that you can pass to the Spark dataframe APIs. The code creates a `Properties` object to hold the parameters. Paste the snippet in a code cell and press **SHIFT + ENTER** to run.

+1. Use the following snippet to create a dataframe with the data from a table in your database. In this snippet, we use a `SalesLT.Address` table that is available as part of the **AdventureWorksLT** database. Paste the snippet in a code cell and press **SHIFT + ENTER** to run.

+ a. Start SSMS and connect to the Azure SQL Database by providing connection details as shown in the following screenshot.

+Use this component to run Python code. For more information about the architecture and design principles of Python, see [how run Python code in Azure Machine Learning designer](../v1/how-to-designer-python.md).

+ Title: What is the Azure Machine Learning designer(v2)?

+description: Learn what the Azure Machine Learning designer is and what tasks you can use it for. The drag-and-drop UI enables customer to build machine learning pipeline.

+# What is Azure Machine Learning designer(v2)?

+Azure Machine Learning designer is a drag-and-drop UI interface to build pipeline in Azure Machine Learning.

+As shown in below GIF, you can build a pipeline visually by drag and drop your building blocks and connect them.

+>[!Note]

+> Designer supports two types of components, classic prebuilt components（v1） and custom components(v2). These two types of components are NOT compatible.

+>

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>

+>This article applies to custom components.

+## Assets

+The building blocks of pipeline are called assets in Azure Machine Learning, which includes:

+ - [Data](./concept-data.md)

+ - Model

+ - [Component](./concept-component.md)

+Designer has an asset library on the left side, where you can access all the assets you need to create your pipeline. It shows both the assets you created in your workspace, and the assets shared in [registry](./how-to-share-models-pipelines-across-workspaces-with-registries.md) that you have permission to access.

+To see assets from a specific registry, select the Registry name filter above the asset library. The assets you created in your current workspace are in the registry = workspace. The assets provided by Azure Machine Learning are in the registry = azureml.

+Designer only shows the assets that you created and named in your workspace. You won't see any unnamed assets in the asset library. To learn how to create data and component assets, read these articles:

+- [How to create data asset](./how-to-create-data-assets.md)

+- [How to create component](./how-to-create-component-pipelines-ui.md#register-component-in-your-workspace)

+## Pipeline

+Designer is a tool that lets you create pipelines with your assets in a visual way. When you use designer, you'll encounter two concepts related to pipelines: pipeline draft and pipeline jobs.

+

+### Pipeline draft

+As you edit a pipeline in the designer, your progress is saved as a **pipeline draft**. You can edit a pipeline draft at any point by adding or removing components, configuring compute targets, creating parameters, and so on.

+A valid pipeline draft has these characteristics:

+- Data assets can only connect to components.

+- Components can only connect to either data assets or other components.

+- All required input ports for components must have some connection to the data flow.

+- All required parameters for each component must be set.

+When you're ready to run your pipeline draft, you submit a pipeline job.

+### Pipeline job

+Each time you run a pipeline, the configuration of the pipeline and its results are stored in your workspace as a **pipeline job**. You can go back to any pipeline job to inspect it for troubleshooting or auditing. **Clone** a pipeline job creates a new pipeline draft for you to continue editing.

+## Approaches to build pipeline in designer

+### Create new pipeline from scratch

+You can create a new pipeline and build from scratch. Remember to select the **Custom component** option when you create the pipeline in designer.

+### Clone an existing pipeline job

+If you would like to work based on an existing pipeline job in the workspace, you can easily clone it into a new pipeline draft to continue editing.

+After cloning, you can also know which pipeline job it's cloned from by selecting **Show lineage**.

+You can edit your pipeline and then submit again. After submitting, you can see the lineage between the job you submit and the original job by selecting **Show lineage** in the job detail page.

+## Next step

+- [Create pipeline with components (UI)](./how-to-create-component-pipelines-ui.md)

+ * Designer jobs are compartmentalized into [Components](./v1/concept-designer.md#component). Each component has its own environment that layers on top of the Azure Machine Learning base docker images. For more information on components, see the [Component reference](./component-reference/component-reference.md).

+You can check the logs and outputs of each component by right clicking the component, or select the component to open its detail pane. To learn more about how to debug your pipeline in UI, see [How to use debug pipeline failure](how-to-debug-pipeline-failure.md).

+> Designer supports two types of components, classic prebuilt components（v1） and custom components(v2). These two types of components are NOT compatible.

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>This article applies to custom components.

+ Select the **Gear icon**  at the top right of the canvas to open the **Settings** pane. Select the default compute target for your pipeline.

+1. In asset library, you can see **Data**, **Model** and **Components** tabs. Switch to **Components** tab, you can see the components registered from previous section.

+> Custom components and the designer classic prebuilt components cannot be used together.

+#### Try the optimized submission wizard(preview)

+There is an optimized pipeline submission wizard to streamline your pipeline submission experience. It provides spacious space to edit pipeline inputs and outputs, and the ability to modify runtime settings, such as the default compute, at the time of submission.

+To enable this feature:

+1. Navigate to Azure Machine Learning studio UI.

+2. Select Manage preview features (megaphone icon) among the icons on the top right side of the screen.

+3. In Managed preview feature panel, toggle on **Submit pipeline jobs using the optimized submission wizard**

+After enabling the pipeline submission wizard, select **Configure & Submit** button on the top right to submit the pipeline job.

+Then you will see a step-by-step wizard, which allows you to edit pipeline inputs & outputs, and change runtime settings before submitting the pipeline job. Follow the wizard to submit the pipeline job.

+ Title: 'How to use studio UI to debug pipeline failure'

+description: Learn how to debug compare pipeline failure with pipeline UI in studio.

+# How to use pipeline UI to debug Azure Machine Learning pipeline failures

+After submitting a pipeline, you'll see a link to the pipeline job in your Azure Machine Learning workspace. The link lands you in the pipeline job page in Azure Machine Learning studio, in which you can check result and debug your pipeline job.

+This article introduces how to use the pipeline job page to debug machine learning pipeline failures.

+> [!IMPORTANT]

+> Items marked (preview) in this article are currently in public preview.

+> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

+> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

+## Using outline to quickly find a node

+In pipeline job detail page, there's an outline left to the canvas, which shows the overall structure of your pipeline job. Hovering on any row, you can select the "Locate" button to locate that node in the canvas.

+You can filter failed or completed nodes, and filter by only components or dataset for further search. The left pane shows the matched nodes with more information including status, duration, and created time.

+You can also sort the filtered nodes.

+## Check logs and outputs of component

+If your pipeline fails or gets stuck on a node, first view the logs.

+1. You can select the specific node and open the right pane.

+1. Select **Outputs+logs** tab and you can explore all the outputs and logs of this node.

+ The **user_logs folder** contains information about user code generated logs. This folder is open by default, and the **std_log.txt** log is selected. The **std_log.txt** is where your code's logs (for example, print statements) show up.

+ The **system_logs folder** contains logs generated by Azure Machine Learning. Learn more about [View and download diagnostic logs](how-to-log-view-metrics.md#view-and-download-diagnostic-logs).

+ 

+ If you don't see those folders, this is due to the compute run time update isn't released to the compute cluster yet, and you can look at **70_driver_log.txt** under **azureml-logs** folder first.

+## Compare different pipelines to debug failure or other unexpected issues (preview)

+Pipeline comparison identifies the differences (including topology, component properties, and job properties) between multiple jobs. For example you can compare a successful pipeline and a failed pipeline, which helps you find what modifications make your pipeline fail.

+Two major scenarios where you can use pipeline comparison to help with debugging:

+- Debug your failed pipeline job by comparing it to a completed one.

+- Debug your failed node in a pipeline by comparing it to a similar completed one.

+To enable this feature:

+1. Navigate to Azure Machine Learning studio UI.

+2. Select **Manage preview features** (megaphone icon) among the icons on the top right side of the screen.

+3. In **Managed preview feature** panel, toggle on **Compare pipeline jobs to debug failures or unexpected issues** feature.

+### How to debug your failed pipeline job by comparing it to a completed one

+During iterative model development, you may have a baseline pipeline, and then do some modifications such as changing a parameter, dataset or compute resource, etc. If your new pipeline failed, you can use pipeline comparison to identify what has changed by comparing it to the baseline pipeline, which could help with figuring out why it failed.

+#### Compare a pipeline with its parent

+The first thing you should check when debugging is to locate the failed node and check the logs.

+For example, you may get an error message showing that your pipeline failed due to out-of-memory. If your pipeline is cloned from a completed parent pipeline, you can use pipeline comparison to see what has changed.

+1. Select **Show lineage**.

+1. Select the link under "Cloned From". This will open a new browser tab with the parent pipeline.

+ :::image type="content" source="./media/how-to-debug-pipeline-failure/cloned-from.png" alt-text="Screenshot showing the cloned from link, with the previous step, the lineage button highlighted." lightbox= "./media/how-to-debug-pipeline-failure/cloned-from.png":::

+1. Select **Add to compare** on the failed pipeline and the parent pipeline. This adds them in the comparison candidate list.

+ :::image type="content" source="./media/how-to-debug-pipeline-failure/comparison-list.png" alt-text="Screenshot showing the comparison list with a parent and child pipeline added." lightbox= "./media/how-to-debug-pipeline-failure/comparison-list.png":::

+### Compare topology

+Once the two pipelines are added to the comparison list, you have two options: **Compare detail** and **Compare graph**. **Compare graph** allows you to compare pipeline topology.

+**Compare graph** shows you the graph topology changes between pipeline A and B. The special nodes in pipeline A are highlighted in red and marked with "A only". The special nodes in pipeline B are in green and marked with "B only". The shared nodes are in gray. If there are differences on the shared nodes, what has been changed is shown on the top of node.

+There are three categories of changes with summaries viewable in the detail page, parameter change, input source, pipeline component. When the pipeline component is changed this means that there's a topology change inside or an inner node parameter change, you can select the folder icon on the pipeline component node to dig down into the details. Other changes can be detected by viewing the colored nodes in the compare graph.

+ :::image type="content" source="./media/how-to-debug-pipeline-failure/parameter-changed.png" alt-text="Screenshot showing the parameter changed and the component information tab." lightbox= "./media/how-to-debug-pipeline-failure/parameter-changed.png":::

+### Compare pipeline meta info and properties

+If you investigate the dataset difference and find that data or topology doesn't seem to be the root cause of failure, you can also check the pipeline details like pipeline parameter, output or run settings.

+**Compare graph** is used to compare pipeline topology, **Compare detail** is used to compare pipeline properties link meta info or settings.

+To access the detail comparison, go to the comparison list, select **Compare details** or select **Show compare details** on the pipeline comparison page.

+You'll see *Pipeline properties* and *Run properties*.

+- Pipeline properties include pipeline parameters, run and output setting, etc.

+- Run properties include job status, submit time and duration, etc.

+The following screenshot shows an example of using the detail comparison, where the default compute setting might have been the reason for failure.

+To quickly check the topology comparison, select the pipeline name and select **Compare graph**.

+### How to debug your failed node in a pipeline by comparing to similar completed node

+If you only updated node properties and changed nothing in the pipeline, then you can debug the node by comparing it with the jobs that are submitted from the same component.

+#### Find the job to compare with

+1. Find a successful job to compare with by viewing all runs submitted from the same component.

+ 1. Right select the failed node and select *View Jobs*. This gives you a list of all the jobs.

+

+ :::image type="content" source="./media/how-to-debug-pipeline-failure/view-jobs.png" alt-text="Screenshot that shows a failed node with view jobs highlighted." lightbox= "./media/how-to-debug-pipeline-failure/view-jobs.png":::

+ 1. Choose a completed job as a comparison target.

+1. After you found a failed and completed job to compare with, add the two jobs to the comparison candidate list.

+ 1. For the failed node, right select and select *Add to compare*.

+ 1. For the completed job, go to its parent pipeline and located the completed job. Then select *Add to compare*.

+1. Once the two jobs are in the comparison list, select **Compare detail** to show the differences.

+### Share the comparison results

+To share your comparison results select **Share** and copying the link. For example, you might find out that the dataset difference might of lead to the failure but you aren't a dataset specialist, you can share the comparison result with a data engineer on your team.

+## Next steps

+In this article, you learned how to debug pipeline failures. To learn more about how you can use the pipeline, see the following articles:

+- [How to build pipeline using python sdk v2](./how-to-create-component-pipeline-python.md)

+- [How to build pipeline using python CLI v2](./how-to-create-component-pipelines-cli.md)

+- [What is machine learning component](./concept-component.md)

+ Title: 'How to debug pipeline performance issues'

+description: How to debug pipeline performance issues by using profile feature

+# View profiling to debug pipeline performance issues (preview)

+Profiling (preview) feature can help you debug pipeline performance issues such as hang, long pole etc. Profiling will list the duration information of each step in a pipeline and provide a Gantt chart for visualization.

+Profiling enables you to:

+- Quickly find which node takes longer time than expected.

+- Identify the time spent of job on each status

+To enable this feature:

+1. Navigate to Azure Machine Learning studio UI.

+2. Select **Manage preview features** (megaphone icon) among the icons on the top right side of the screen.

+3. In **Managed preview feature** panel, toggle on **View profiling to debug pipeline performance issues** feature.

+## How to find the node that runs totally the longest

+1. On the Jobs page, select the job name and enter the job detail page.

+1. In the action bar, select **View profiling**. Profiling only works for root level pipeline. It will take a few minutes to load the next page.

+ :::image type="content" source="./media/how-to-debug-pipeline-performance/view-profiling.png" alt-text="Screenshot showing the pipeline at root level with the view profiling button highlighted." lightbox= "./media/how-to-debug-pipeline-performance/view-profiling.png":::

+1. After the profiler loads, you'll see a Gantt chart. By Default the critical path of a pipeline is shown. A critical path is a subsequence of steps that determine a pipeline job's total duration.

+ :::image type="content" source="./media/how-to-debug-pipeline-performance/critical-path.png" alt-text="Screenshot showing the Gantt chart and the critical path." lightbox= "./media/how-to-debug-pipeline-performance/critical-path.png":::

+1. To find the step that takes the longest, you can either view the Gantt chart or the table below it.

+ In the Gantt chart, the length of each bar shows how long the step takes, steps with a longer bar length take more time. You can also filter the table below by "total duration". When you select a row in the table, it shows you the node in the Gantt chart too. When you select a bar on the Gantt chart it will also highlight it in the table.

+ In the table, reuse is denoted with the recycling icon.

+ If you select the log icon next the node name it opens the detail page, which shows parameter, code, outputs, logs etc.

+ :::image type="content" source="./media/how-to-debug-pipeline-performance/detail-page-from-log-icon.png" alt-text="Screenshot highlighting the log icon and showing the detail page." lightbox= "./media/how-to-debug-pipeline-performance/detail-page-from-log-icon.png":::

+ If you're trying to make the queue time shorter for a node, you can change the compute node number and modify job priority to get more compute resources on this one.

+## How to find the node that runs the longest in each status

+Besides the total duration, you can also sort by durations for each status. For example, you can sort by *Preparing* duration to see which step spends the most time on image building. Then you can open the detail page to find that image building fails because of timeout issue.

+### What do I do if a duration issue identified

+Status and definitions:

+| Status | What does it mean? | Time estimation | Next step |

+||--|-|-|

+| Not started | Job is submitted from client side and accepted in Azure Machine Learning services. Time spent in this stage is mainly in Azure Machine Learning service scheduling and preprocessing. | If there's no backend service issue, this time should be short.| Open support case via Azure portal. |

+|Preparing | In this status, job is pending for some preparation on job dependencies, for example, environment image building.| If you're using curated or registered custom environment, this time should be short. | Check image building log. |

+|Inqueue | Job is pending for compute resource allocation. Time spent in this stage is mainly depending on the status of your compute cluster.| If you're using a cluster with enough compute resource, this time should be short. | Check with workspace admin whether to increase the max nodes of the target compute or change the job to another less busy compute. |

+|Running | Job is executing on remote compute. Time spent in this stage is mainly in two parts: <br> Runtime preparation: image pulling, docker starting and data preparation (mount or download). <br> User script execution. | This status is expected to be most time consuming one. | 1. Go to the source code check if there's any user error. <br> 2. View the monitoring tab of compute metrics (CPU, memory, networking etc.) to identify the bottleneck. <br> 3. Try online debug with [interactive endpoints](how-to-interactive-jobs.md) if the job is running or locally debug of your code. |

+| Finalizing | Job is in post processing after execution complete. Time spent in this stage is mainly for some post processes like: output uploading, metric/logs uploading and resources clean up.| It will be short for command job. However, might be very long for PRS/MPI job because for a distributed job, the finalizing status is from the first node starting finalizing to the last node done finalizing. | Change your step job output mode from upload to mount if you find unexpected long finalizing time, or open support case via Azure portal. |

+## Different view of Gantt chart

+- Critical path

+ - You'll see only the step jobs in the pipeline's critical path (jobs that have a dependency).

+ - By default the critical path of the pipeline job is shown.

+- Flatten view

+ - You'll see all step jobs.

+ - In this view, you'll see more nodes than in critical path.

+- Compact view

+ - You'll only see step jobs that are longer than 30 seconds.

+- Hierarchical view.

+ - You'll see all jobs including pipeline component jobs and step jobs.

+## Download the duration table

+To export the table, select **Export CSV**.

+## Next steps

+In this article, you learned how to debug pipeline failures. To learn more about how you can use the pipeline, see the following articles:

+- [How to build pipeline using python sdk v2](./how-to-create-component-pipeline-python.md)

+- [How to build pipeline using python CLI v2](./how-to-create-component-pipelines-cli.md)

+- [What is machine learning component](./concept-component.md)

+ Title: 'How to deploy pipeline as batch endpoint(preview)'

+description: Learn how to deploy pipeline component as batch endpoint to trigger the pipeline using REST endpoint

+# Deploy your pipeline as batch endpoint

+After building your machine learning pipeline, you can deploy your pipeline as a [batch endpoint(preview)](./concept-endpoints-batch.md#pipeline-component-deployment-preview) for following scenarios:

+- You want to run your machine learning pipeline from other platforms out of Azure Machine Learning (for example: custom Java code, Azure DevOps, GitHub Actions, Azure Data Factory). Batch endpoint lets you do this easily because it's a REST endpoint and doesn't depend on the language/platform.

+- You want to change the logic of your machine learning pipeline without affecting the downstream consumers who use a fixed URI interface.

+## Pipeline component deployment as batch endpoint

+Pipeline component deployment as batch endpoint is the feature allows you to achieve above goals. This is the equivalent feature with published pipeline/pipeline endpoint in SDK v1.

+To deploy your pipeline as batch endpoint, we recommend first convert your pipeline into a [pipeline component](./how-to-use-pipeline-component.md), and then deploy the pipeline component as a batch endpoint. Check below article to learn more.

+- [How to deploy pipeline component as batch endpoint](how-to-use-batch-pipeline-deployments.md)

+It's also possible to deploy your pipeline job as batch endpoint. In this case, Azure Machine Learning can accept that job as input to your batch endpoint and create the pipeline component automatically for you. Check below article to learn more.

+- [Deploy existing pipeline jobs to batch endpoints (preview)](how-to-use-batch-pipeline-from-job.md)

+## Next steps

+- [How to deploy a training pipeline with batch endpoints (preview)](how-to-use-batch-training-pipeline.md)

+- [How to deploy a pipeline to perform batch scoring with preprocessing (preview)](how-to-use-batch-scoring-pipeline.md)

+- [Access data from batch endpoints jobs](how-to-access-data-batch-endpoints-jobs.md)

+- [Troubleshooting batch endpoints](how-to-troubleshoot-batch-endpoints.md)

+ Title: Machine Learning Algorithm Cheat Sheet - designer

+description: A printable Machine Learning Algorithm Cheat Sheet helps you choose the right algorithm for your predictive model in Azure Machine Learning designer.

+adobe-target: true

+# Machine Learning Algorithm Cheat Sheet for Azure Machine Learning designer

+The **Azure Machine Learning Algorithm Cheat Sheet** helps you choose the right algorithm from the designer for a predictive analytics model.

+>[!Note]

+> Designer supports two types of components, classic prebuilt components (v1) and custom components (v2). These two types of components are NOT compatible.

+>

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>

+>This article applies to classic prebuilt components and not compatible with CLI v2 and SDK v2.

+Azure Machine Learning has a large library of algorithms from the ***classification***, ***recommender systems***, ***clustering***, ***anomaly detection***, ***regression***, and ***text analytics*** families. Each is designed to address a different type of machine learning problem.

+For more information, see [How to select algorithms](how-to-select-algorithms.md).

+## Download: Machine Learning Algorithm Cheat Sheet

+**Download the cheat sheet here: [Machine Learning Algorithm Cheat Sheet (11x17 in.)](https://download.microsoft.com/download/3/5/b/35bb997f-a8c7-485d-8c56-19444dafd757/azure-machine-learning-algorithm-cheat-sheet-july-2021.pdf)**

+Download and print the Machine Learning Algorithm Cheat Sheet in tabloid size to keep it handy and get help choosing an algorithm.

+## How to use the Machine Learning Algorithm Cheat Sheet

+The suggestions offered in this algorithm cheat sheet are approximate rules-of-thumb. Some can be bent, and some can be flagrantly violated. This cheat sheet is intended to suggest a starting point. DonΓÇÖt be afraid to run a head-to-head competition between several algorithms on your data. There is simply no substitute for understanding the principles of each algorithm and the system that generated your data.

+Every machine learning algorithm has its own style or inductive bias. For a specific problem, several algorithms may be appropriate, and one algorithm may be a better fit than others. But it's not always possible to know beforehand, which is the best fit. In cases like these, several algorithms are listed together in the cheat sheet. An appropriate strategy would be to try one algorithm, and if the results are not yet satisfactory, try the others.

+To learn more about the algorithms in Azure Machine Learning designer, go to the [Algorithm and component reference](../component-reference/component-reference.md).

+## Kinds of machine learning

+There are three main categories of machine learning: *supervised learning*, *unsupervised learning*, and *reinforcement learning*.

+### Supervised learning

+In supervised learning, each data point is labeled or associated with a category or value of interest. An example of a categorical label is assigning an image as either a ΓÇÿcatΓÇÖ or a ΓÇÿdogΓÇÖ. An example of a value label is the sale price associated with a used car. The goal of supervised learning is to study many labeled examples like these, and then to be able to make predictions about future data points. For example, identifying new photos with the correct animal or assigning accurate sale prices to other used cars. This is a popular and useful type of machine learning.

+### Unsupervised learning

+In unsupervised learning, data points have no labels associated with them. Instead, the goal of an unsupervised learning algorithm is to organize the data in some way or to describe its structure. Unsupervised learning groups data into clusters, as K-means does, or finds different ways of looking at complex data so that it appears simpler.

+### Reinforcement learning

+In reinforcement learning, the algorithm gets to choose an action in response to each data point. It is a common approach in robotics, where the set of sensor readings at one point in time is a data point, and the algorithm must choose the robotΓÇÖs next action. It's also a natural fit for Internet of Things applications. The learning algorithm also receives a reward signal a short time later, indicating how good the decision was. Based on this signal, the algorithm modifies its strategy in order to achieve the highest reward.

+## Next steps

+* See more information on [How to select algorithms](how-to-select-algorithms.md)

+* [Learn about studio in Azure Machine Learning and the Azure portal](../overview-what-is-azure-machine-learning.md).

+* [Tutorial: Build a prediction model in Azure Machine Learning designer](tutorial-designer-automobile-price-train-score.md).

+* [Learn about deep learning vs. machine learning](../concept-deep-learning-vs-machine-learning.md).

+| [Designer](concept-designer.md) | Drag-and-drop machine learning modeling tool formerly known as the visual interface |

+ + Added deprecation message for explain_model() and retrieve_model_explanations().

+ * Consume it directly in Azure Machine Learning solutions like, automated machine learning (automated ML) experiment runs, machine learning pipelines, or the [Azure Machine Learning designer](concept-designer.md).

+ Title: What is the Azure Machine Learning designer(v1)?

+description: Learn what the Azure Machine Learning designer is and what tasks you can use it for. The drag-and-drop UI enables model training and deployment.

+# What is Azure Machine Learning designer (v1)?

+Azure Machine Learning designer is a drag-and-drop interface used to train and deploy models in Azure Machine Learning. This article describes the tasks you can do in the designer.

+>[!Note]

+> Designer supports two types of components, classic prebuilt components（v1） and custom components(v2). These two types of components are NOT compatible.

+>

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>

+>This article applies to classic prebuilt components and not compatible with CLI v2 and SDK v2.

+ - To get started with the designer, see [Tutorial: Train a no-code regression model](tutorial-designer-automobile-price-train-score.md).

+ - To learn about the components available in the designer, see the [Algorithm and component reference](../algorithm-module-reference/module-reference.md).

+The designer uses your Azure Machine Learning [workspace](../concept-workspace.md) to organize shared resources such as:

+## Model training and deployment

+Use a visual canvas to build an end-to-end machine learning workflow. Train, test, and deploy models all in the designer:

+ + Publish a **training pipeline** to reuse a single pipeline to train multiple models while changing parameters and data assets.

+ + Publish a **batch inference pipeline** to make predictions on new data by using a previously trained model.

+## Pipeline

+A [pipeline](../concept-ml-pipelines.md) consists of data assets and analytical components, which you connect. Pipelines have many uses: you can make a pipeline that trains a single model, or one that trains multiple models. You can create a pipeline that makes predictions in real time or in batch, or make a pipeline that only cleans data. Pipelines let you reuse your work and organize your projects.

+### Pipeline draft

+As you edit a pipeline in the designer, your progress is saved as a **pipeline draft**. You can edit a pipeline draft at any point by adding or removing components, configuring compute targets, creating parameters, and so on.

+A valid pipeline has these characteristics:

+* Data assets can only connect to components.

+* components can only connect to either data assets or other components.

+* All input ports for components must have some connection to the data flow.

+* All required parameters for each component must be set.

+When you're ready to run your pipeline draft, you submit a pipeline job.

+### Pipeline job

+Each time you run a pipeline, the configuration of the pipeline and its results are stored in your workspace as a **pipeline job**. You can go back to any pipeline job to inspect it for troubleshooting or auditing. **Clone** a pipeline job to create a new pipeline draft for you to edit.

+Pipeline jobs are grouped into experiments to organize job history. You can set the experiment for every pipeline job.

+## Data

+A machine learning data asset makes it easy to access and work with your data. Several [sample data assets](samples-designer.md#datasets) are included in the designer for you to experiment with. You can [register](how-to-create-register-datasets.md) more data assets as you need them.

+## Component

+A component is an algorithm that you can perform on your data. The designer has several components ranging from data ingress functions to training, scoring, and validation processes.

+A component may have a set of parameters that you can use to configure the component's internal algorithms. When you select a component on the canvas, the component's parameters are displayed in the Properties pane to the right of the canvas. You can modify the parameters in that pane to tune your model. You can set the compute resources for individual components in the designer.

+For some help navigating through the library of machine learning algorithms available, see [Algorithm & component reference overview](../component-reference/component-reference.md). For help with choosing an algorithm, see the [Azure Machine Learning Algorithm Cheat Sheet](algorithm-cheat-sheet.md).

+## <a name="compute"></a> Compute resources

+Use compute resources from your workspace to run your pipeline and host your deployed models as online endpoints or pipeline endpoints (for batch inference). The supported compute targets are:

+| Compute target | Training | Deployment |

+| - |:-:|:-:|

+| Azure Machine Learning compute | Γ£ô | |

+| Azure Kubernetes Service | | Γ£ô |

+Compute targets are attached to your [Azure Machine Learning workspace](../concept-workspace.md). You manage your compute targets in your workspace in the [Azure Machine Learning studio](https://ml.azure.com).

+## Deploy

+To perform real-time inferencing, you must deploy a pipeline as an [online endpoint](../concept-endpoints-online.md). The online endpoint creates an interface between an external application and your scoring model. A call to an online endpoint returns prediction results to the application in real time. To make a call to an online endpoint, you pass the API key that was created when you deployed the endpoint. The endpoint is based on REST, a popular architecture choice for web programming projects.

+Online endpoints must be deployed to an Azure Kubernetes Service cluster.

+To learn how to deploy your model, see [Tutorial: Deploy a machine learning model with the designer](tutorial-designer-automobile-price-deploy.md).

+## Publish

+You can also publish a pipeline to a **pipeline endpoint**. Similar to an online endpoint, a pipeline endpoint lets you submit new pipeline jobs from external applications using REST calls. However, you cannot send or receive data in real time using a pipeline endpoint.

+Published pipelines are flexible, they can be used to train or retrain models, [perform batch inferencing](how-to-run-batch-predictions-designer.md), process new data, and much more. You can publish multiple pipelines to a single pipeline endpoint and specify which pipeline version to run.

+A published pipeline runs on the compute resources you define in the pipeline draft for each component.

+The designer creates the same [PublishedPipeline](/python/api/azureml-pipeline-core/azureml.pipeline.core.graph.publishedpipeline) object as the SDK.

+## Next steps

+* Learn the fundamentals of predictive analytics and machine learning with [Tutorial: Predict automobile price with the designer](tutorial-designer-automobile-price-train-score.md)

+* Learn how to modify existing [designer samples](samples-designer.md) to adapt them to your needs.

+If you use the [Designer](concept-designer.md) to create your ML pipelines, you may at any time click the **"..."** at the top-right of the Designer page and then select **Clone**. Cloning your pipeline allows you to iterate your pipeline design without losing your old versions.

+In this article, you learn how to troubleshoot when you get errors running a [machine learning pipeline](../concept-ml-pipelines.md) in the [Azure Machine Learning SDK](/python/api/overview/azure/ml/intro) and [Azure Machine Learning designer](concept-designer.md).

+ Title: Execute Python Script in the designer

+description: Learn how to use the Execute Python Script model in Azure Machine Learning designer to run custom operations written in Python.

+# Run Python code in Azure Machine Learning designer

+In this article, you'll learn how to use the [Execute Python Script](../algorithm-module-reference/execute-python-script.md) component to add custom logic to the Azure Machine Learning designer. In this how-to, you use the Pandas library to do simple feature engineering.

+You can use the in-built code editor to quickly add simple Python logic. You should use the zip file method to add more complex code, or to upload additional Python libraries.

+The default execution environment uses the Anacondas distribution of Python. See the [Execute Python Script component reference](../algorithm-module-reference/execute-python-script.md) page for a complete list of pre-installed packages.

+

+## Execute Python written in the designer

+### Add the Execute Python Script component

+1. Find the **Execute Python Script** component in the designer palette. It can be found in the **Python Language** section.

+1. Drag and drop the component onto the pipeline canvas.

+### Connect input datasets

+This article uses the **Automobile price data (Raw)** sample dataset.

+1. Drag and drop your dataset to the pipeline canvas.

+1. Connect the output port of the dataset to the top-left input port of the **Execute Python Script** component. The designer exposes the input as a parameter to the entry point script.

+ The right input port is reserved for zipped Python libraries.

+ 

+1. Carefully note the specific input port you use. The designer assigns the left input port to the variable `dataset1`, and the middle input port to `dataset2`.

+Input components are optional, since you can generate or import data directly in the **Execute Python Script** component.

+### Write your Python code

+The designer provides an initial entry point script for you to edit and enter your own Python code.

+In this example, you use Pandas to combine two of the automobile dataset columns - **Price** and **Horsepower** - to create a new column, **Dollars per horsepower**. This column represents how much you pay for each horsepower unit, which could become a useful information point to decide if a specific car is a good deal for its price.

+1. Select the **Execute Python Script** component.

+1. In the pane that appears to the right of the canvas, select the **Python script** text box.

+1. Copy and paste the following code into the text box:

+ ```python

+ import pandas as pd

+

+ def azureml_main(dataframe1 = None, dataframe2 = None):

+ dataframe1['Dollar/HP'] = dataframe1.price / dataframe1.horsepower

+ return dataframe1

+ ```

+ Your pipeline should look like this image:

+

+ 

+ The entry point script must contain the function `azureml_main`. The function has two function parameters that map to the two input ports for the **Execute Python Script** component.

+ The return value must be a Pandas Dataframe. You can return at most two dataframes as component outputs.

+

+1. Submit the pipeline.

+Now you have a dataset, which has a new **Dollars/HP** feature. This new feature could help to train a car recommender. This example shows feature extraction and dimensionality reduction.

+## Next steps

+Learn how to [import your own data](how-to-designer-import-data.md) in Azure Machine Learning designer.

+ Title: Transform data in the designer

+description: Learn how to import and transform data in Azure Machine Learning designer to create your own datasets.

+# Transform data in Azure Machine Learning designer

+In this article, you'll learn how to transform and save datasets in the Azure Machine Learning designer, to prepare your own data for machine learning.

+You'll use the sample [Adult Census Income Binary Classification](samples-designer.md) dataset to prepare two datasets: one dataset that includes adult census information from only the United States, and another dataset that includes census information from non-US adults.

+In this article, you'll learn how to:

+1. Transform a dataset to prepare it for training.

+1. Export the resulting datasets to a datastore.

+1. View the results.

+This how-to is a prerequisite for the [how to retrain designer models](how-to-retrain-designer.md) article. In that article, you'll learn how to use the transformed datasets to train multiple models, with pipeline parameters.

+## Transform a dataset

+In this section, you'll learn how to import the sample dataset, and split the data into US and non-US datasets. See [how to import data](how-to-designer-import-data.md) for more information about how to import your own data into the designer.

+### Import data

+Use these steps to import the sample dataset:

+1. Sign in to <a href="https://ml.azure.com?tabs=jre" target="_blank">ml.azure.com</a>, and select the workspace you want to use.

+1. Go to the designer. Select **Easy-to-use-prebuild components** to create a new pipeline.

+1. Select a default compute target to run the pipeline.

+1. To the left of the pipeline canvas, you'll see a palette of datasets and components. Select **Datasets**. Then view the **Samples** section.

+1. Drag and drop the **Adult Census Income Binary classification** dataset onto the canvas.

+1. Right-click the **Adult Census Income** dataset component, and select **Visualize** > **Dataset output**

+1. Use the data preview window to explore the dataset. Take special note of the "native-country" column values.

+### Split the data

+In this section, you'll use the [Split Data component](../algorithm-module-reference/split-data.md) to identify and split rows that contain "United-States" in the "native-country" column.

+1. To the left of the canvas, in the component palette, expand the **Data Transformation** section, and find the **Split Data** component.

+1. Drag the **Split Data** component onto the canvas, and drop that component below the dataset component.

+1. Connect the dataset component to the **Split Data** component.

+1. Select the **Split Data** component.

+1. To the right of the canvas in the component details pane, set **Splitting mode** to **Regular Expression**.

+1. Enter the **Regular Expression**: `\"native-country" United-States`.

+ The **Regular expression** mode tests a single column for a value. See the related [algorithm component reference page](../algorithm-module-reference/split-data.md) for more information on the Split Data component.

+Your pipeline should look like this:

+## Save the datasets

+Now that you set up your pipeline to split the data, you must specify where to persist the datasets. For this example, use the **Export Data** component to save your dataset to a datastore. See [Connect to Azure storage services](how-to-access-data.md) for more information about datastores.

+1. To the left of the canvas in the component palette, expand the **Data Input and Output** section, and find the **Export Data** component.

+1. Drag and drop two **Export Data** components below the **Split Data** component.

+1. Connect each output port of the **Split Data** component to a different **Export Data** component.

+ Your pipeline should look something like this:

+ .

+1. Select the **Export Data** component connected to the *left*-most port of the **Split Data** component.

+ For the **Split Data** component, the output port order matters. The first output port contains the rows where the regular expression is true. In this case, the first port contains rows for US-based income, and the second port contains rows for non-US based income.

+1. In the component details pane to the right of the canvas, set the following options:

+

+ **Datastore type**: Azure Blob Storage

+ **Datastore**: Select an existing datastore, or select "New datastore" to create one now.

+ **Path**: `/data/us-income`

+ **File format**: csv

+ > [!NOTE]

+ > This article assumes that you have access to a datastore registered to the current Azure Machine Learning workspace. See [Connect to Azure storage services](how-to-connect-data-ui.md#create-datastores) for datastore setup instructions.

+ You can create a datastore if you don't have one now. For example purposes, this article will save the datasets to the default blob storage account associated with the workspace. It will save the datasets into the `azureml` container, in a new folder named `data`.

+1. Select the **Export Data** component connected to the *right*-most port of the **Split Data** component.

+1. To the right of the canvas in the component details pane, set the following options:

+

+ **Datastore type**: Azure Blob Storage

+ **Datastore**: Select the same datastore as above

+ **Path**: `/data/non-us-income`

+ **File format**: csv

+1. Verify that the **Export Data** component connected to the left port of the **Split Data** has the **Path** `/data/us-income`.

+1. Verify that the **Export Data** component connected to the right port has the **Path** `/data/non-us-income`.

+ Your pipeline and settings should look like this:

+

+ .

+### Submit the job

+Now that you set up your pipeline to split and export the data, submit a pipeline job.

+1. Select **Submit** at the top of the canvas.

+1. Select **Create new** in the **Set up pipeline job**, to create an experiment.

+ Experiments logically group related pipeline jobs together. If you run this pipeline in the future, you should use the same experiment for logging and tracking purposes.

+1. Provide a descriptive experiment name - for example "split-census-data".

+1. Select **Submit**.

+## View results

+After the pipeline finishes running, you can navigate to your Azure portal blob storage to view your results. You can also view the intermediary results of the **Split Data** component to confirm that your data has been split correctly.

+1. Select the **Split Data** component.

+1. In the component details pane to the right of the canvas, select **Outputs + logs**.

+1. Select the visualize icon  next to **Results dataset1**.

+1. Verify that the "native-country" column contains only the value "United-States".

+1. Select the visualize icon  next to **Results dataset2**.

+1. Verify that the "native-country" column does not contain the value "United-States".

+## Clean up resources

+To continue with part two of this [Retrain models with Azure Machine Learning designer](how-to-retrain-designer.md) how-to, skip this section.

+## Next steps

+In this article, you learned how to transform a dataset, and save it to a registered datastore.

+Continue to the next part of this how-to series with [Retrain models with Azure Machine Learning designer](how-to-retrain-designer.md), to use your transformed datasets and pipeline parameters to train machine learning models.

+ Title: Use pipeline parameters to retrain models in the designer

+description: Retrain models with published pipelines and pipeline parameters in Azure Machine Learning designer.

+# Use pipeline parameters to retrain models in the designer

+In this how-to article, you learn how to use Azure Machine Learning designer to retrain a machine learning model using pipeline parameters. You will use published pipelines to automate your workflow and set parameters to train your model on new data. Pipeline parameters let you re-use existing pipelines for different jobs.

+In this article, you learn how to:

+> [!div class="checklist"]

+> * Train a machine learning model.

+> * Create a pipeline parameter.

+> * Publish your training pipeline.

+> * Retrain your model with new parameters.

+## Prerequisites

+* An Azure Machine Learning workspace

+* Complete part 1 of this how-to series, [Transform data in the designer](how-to-designer-transform-data.md)

+This article also assumes that you have some knowledge of building pipelines in the designer. For a guided introduction, complete the [tutorial](tutorial-designer-automobile-price-train-score.md).

+### Sample pipeline

+The pipeline used in this article is an altered version of a sample pipeline [Income prediction](samples-designer.md#classification) in the designer homepage. The pipeline uses the [Import Data](../algorithm-module-reference/import-data.md) component instead of the sample dataset to show you how to train models using your own data.

+

+## Create a pipeline parameter

+Pipeline parameters are used to build versatile pipelines which can be resubmitted later with varying parameter values. Some common scenarios are updating datasets or some hyper-parameters for retraining. Create pipeline parameters to dynamically set variables at runtime.

+Pipeline parameters can be added to data source or component parameters in a pipeline. When the pipeline is resubmitted, the values of these parameters can be specified.

+For this example, you will change the training data path from a fixed value to a parameter, so that you can retrain your model on different data. You can also add other component parameters as pipeline parameters according to your use case.

+1. Select the **Import Data** component.

+ > [!NOTE]

+ > This example uses the Import Data component to access data in a registered datastore. However, you can follow similar steps if you use alternative data access patterns.

+1. In the component detail pane, to the right of the canvas, select your data source.

+1. Enter the path to your data. You can also select **Browse path** to browse your file tree.

+1. Mouseover the **Path** field, and select the ellipses above the **Path** field that appear.

+1. Select **Add to pipeline parameter**.

+1. Provide a parameter name and a default value.

+ 

+1. Select **Save**.

+ > [!NOTE]

+ > You can also detach a component parameter from pipeline parameter in the component detail pane, similar to adding pipeline parameters.

+ >

+ > You can inspect and edit your pipeline parameters by selecting the **Settings** gear icon next to the title of your pipeline draft.

+ > - After detaching, you can delete the pipeline parameter in the **Setings** pane.

+ > - You can also add a pipeline parameter in the **Settings** pane, and then apply it on some component parameter.

+1. Submit the pipeline job.

+## Publish a training pipeline

+Publish a pipeline to a pipeline endpoint to easily reuse your pipelines in the future. A pipeline endpoint creates a REST endpoint to invoke pipeline in the future. In this example, your pipeline endpoint lets you reuse your pipeline to retrain a model on different data.

+1. Select **Publish** above the designer canvas.

+1. Select or create a pipeline endpoint.

+ > [!NOTE]

+ > You can publish multiple pipelines to a single endpoint. Each pipeline in a given endpoint is given a version number, which you can specify when you call the pipeline endpoint.

+1. Select **Publish**.

+## Retrain your model

+Now that you have a published training pipeline, you can use it to retrain your model on new data. You can submit jobs from a pipeline endpoint from the studio workspace or programmatically.

+### Submit jobs by using the studio portal

+Use the following steps to submit a parameterized pipeline endpoint job from the studio portal:

+1. Go to the **Endpoints** page in your studio workspace.

+1. Select the **Pipeline endpoints** tab. Then, select your pipeline endpoint.

+1. Select the **Published pipelines** tab. Then, select the pipeline version that you want to run.

+1. Select **Submit**.

+1. In the setup dialog box, you can specify the parameters values for the job. For this example, update the data path to train your model using a non-US dataset.

+

+### Submit jobs by using code

+You can find the REST endpoint of a published pipeline in the overview panel. By calling the endpoint, you can retrain the published pipeline.

+To make a REST call, you need an OAuth 2.0 bearer-type authentication header. For information about setting up authentication to your workspace and making a parameterized REST call, see [Build an Azure Machine Learning pipeline for batch scoring](../tutorial-pipeline-batch-scoring-classification.md#publish-and-run-from-a-rest-endpoint).

+## Next steps

+In this article, you learned how to create a parameterized training pipeline endpoint using the designer.

+For a complete walkthrough of how you can deploy a model to make predictions, see the [designer tutorial](tutorial-designer-automobile-price-train-score.md) to train and deploy a regression model.

+For how to publish and submit a job to pipeline endpoint using the SDK v1, see [this article](how-to-deploy-pipelines.md).

+ Title: Run batch predictions using Azure Machine Learning designer

+description: Learn how to create a batch prediction pipeline. Deploy the pipeline as a parameterized web service, and trigger it from any HTTP library.

+# Run batch predictions using Azure Machine Learning designer

+In this article, you learn how to use the designer to create a batch prediction pipeline. Batch prediction lets you continuously score large datasets on-demand using a web service that can be triggered from any HTTP library.

+In this how-to, you learn to do the following tasks:

+> [!div class="checklist"]

+> * Create and publish a batch inference pipeline

+> * Consume a pipeline endpoint

+> * Manage endpoint versions

+To learn how to set up batch scoring services using the SDK, see the accompanying [tutorial on pipeline batch scoring](../tutorial-pipeline-batch-scoring-classification.md).

+## Prerequisites

+This how-to assumes you already have a training pipeline. For a guided introduction to the designer, complete [part one of the designer tutorial](tutorial-designer-automobile-price-train-score.md).

+## Create a batch inference pipeline

+Your training pipeline must be run at least once to be able to create an inferencing pipeline.

+1. Go to the **Designer** tab in your workspace.

+1. Select the training pipeline that trains the model you want to use to make prediction.

+1. **Submit** the pipeline.

+

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/run-training-pipeline.png" alt-text="Screenshot showing the set up pipeline job with the experiment drop-down and submit button highlighted." lightbox= "./media/how-to-run-batch-predictions-designer/run-training-pipeline.png":::

+You'll see a submission list on the left of canvas. You can select the job detail link to go to the job detail page, and after the training pipeline job completes, you can create a batch inference pipeline.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/submission-list.png" alt-text="Screenshot showing the submitted job list." lightbox= "./media/how-to-run-batch-predictions-designer/submission-list.png":::

+1. In job detail page, above the canvas, select the dropdown **Create inference pipeline**. Select **Batch inference pipeline**.

+ > [!NOTE]

+ > Currently auto-generating inference pipeline only works for training pipeline built purely by the designer built-in components.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/create-batch-inference.png" alt-text="Screenshot of the create inference pipeline drop-down with batch inference pipeline highlighted." lightbox= "./media/how-to-run-batch-predictions-designer/create-batch-inference.png":::

+

+ It will create a batch inference pipeline draft for you. The batch inference pipeline draft uses the trained model as **MD-** node and transformation as **TD-** node from the training pipeline job.

+ You can also modify this inference pipeline draft to better handle your input data for batch inference.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/batch-inference-draft.png" alt-text="Screenshot showing a batch inference pipeline draft." lightbox= "./media/how-to-run-batch-predictions-designer/batch-inference-draft.png":::

+### Add a pipeline parameter

+To create predictions on new data, you can either manually connect a different dataset in this pipeline draft view or create a parameter for your dataset. Parameters let you change the behavior of the batch inferencing process at runtime.

+In this section, you create a dataset parameter to specify a different dataset to make predictions on.

+1. Select the dataset component.

+1. A pane will appear to the right of the canvas. At the bottom of the pane, select **Set as pipeline parameter**.

+

+ Enter a name for the parameter, or accept the default value.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/create-pipeline-parameter.png" alt-text="Screenshot of cleaned dataset tab with set as pipeline parameter checked." lightbox= "./media/how-to-run-batch-predictions-designer/create-pipeline-parameter.png":::

+1. Submit the batch inference pipeline and go to job detail page by selecting the job link in the left pane.

+## Publish your batch inference pipeline

+Now you're ready to deploy the inference pipeline. This will deploy the pipeline and make it available for others to use.

+1. Select the **Publish** button.

+1. In the dialog that appears, expand the drop-down for **PipelineEndpoint**, and select **New PipelineEndpoint**.

+1. Provide an endpoint name and optional description.

+ Near the bottom of the dialog, you can see the parameter you configured with a default value of the dataset ID used during training.

+1. Select **Publish**.

+## Consume an endpoint

+Now, you have a published pipeline with a dataset parameter. The pipeline will use the trained model created in the training pipeline to score the dataset you provide as a parameter.

+### Submit a pipeline job

+In this section, you'll set up a manual pipeline job and alter the pipeline parameter to score new data.

+1. After the deployment is complete, go to the **Endpoints** section.

+1. Select **Pipeline endpoints**.

+1. Select the name of the endpoint you created.

+1. Select **Published pipelines**.

+ This screen shows all published pipelines published under this endpoint.

+1. Select the pipeline you published.

+ The pipeline details page shows you a detailed job history and connection string information for your pipeline.

+

+1. Select **Submit** to create a manual run of the pipeline.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/submit-manual-run.png" alt-text="Screenshot of set up pipeline job with parameters highlighted." lightbox= "./media/how-to-run-batch-predictions-designer/submit-manual-run.png" :::

+

+1. Change the parameter to use a different dataset.

+

+1. Select **Submit** to run the pipeline.

+### Use the REST endpoint

+You can find information on how to consume pipeline endpoints and published pipeline in the **Endpoints** section.

+You can find the REST endpoint of a pipeline endpoint in the job overview panel. By calling the endpoint, you're consuming its default published pipeline.

+You can also consume a published pipeline in the **Published pipelines** page. Select a published pipeline and you can find the REST endpoint of it in the **Published pipeline overview** panel to the right of the graph.

+To make a REST call, you'll need an OAuth 2.0 bearer-type authentication header. See the following [tutorial section](../tutorial-pipeline-batch-scoring-classification.md#publish-and-run-from-a-rest-endpoint) for more detail on setting up authentication to your workspace and making a parameterized REST call.

+## Versioning endpoints

+The designer assigns a version to each subsequent pipeline that you publish to an endpoint. You can specify the pipeline version that you want to execute as a parameter in your REST call. If you don't specify a version number, the designer will use the default pipeline.

+When you publish a pipeline, you can choose to make it the new default pipeline for that endpoint.

+You can also set a new default pipeline in the **Published pipelines** tab of your endpoint.

+## Update pipeline endpoint

+If you make some modifications in your training pipeline, you may want to update the newly trained model to the pipeline endpoint.

+1. After your modified training pipeline completes successfully, go to the job detail page.

+1. Right click **Train Model** component and select **Register data**

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/register-train-model-as-dataset.png" alt-text="Screenshot of the train model component options with register data highlighted." lightbox= "./media/how-to-run-batch-predictions-designer/register-train-model-as-dataset.png" :::

+ Input name and select **File** type.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/register-train-model-as-dataset-2.png" alt-text="Screenshot of register as data asset with new data asset selected." lightbox= "./media/how-to-run-batch-predictions-designer/register-train-model-as-dataset-2.png" :::

+1. Find the previous batch inference pipeline draft, or you can just **Clone** the published pipeline into a new draft.

+1. Replace the **MD-** node in the inference pipeline draft with the registered data in the step above.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/update-inference-pipeline-draft.png" alt-text="Screenshot of updating the inference pipeline draft with the registered data in the step above." :::

+1. Updating data transformation node **TD-** is the same as the trained model.

+1. Then you can submit the inference pipeline with the updated model and transformation, and publish again.

+## Next steps

+* Follow the [designer tutorial to train and deploy a regression model](tutorial-designer-automobile-price-train-score.md).

+* For how to publish and run a published pipeline using the SDK v1, see the [How to deploy pipelines](how-to-deploy-pipelines.md?view=azureml-api-1&preserve-view=true) article.

+ Title: How to select a machine learning algorithm

+description: How to select Azure Machine Learning algorithms for supervised and unsupervised learning in clustering, classification, or regression experiments.

+# How to select algorithms for Azure Machine Learning

+A common question is ΓÇ£Which machine learning algorithm should I use?ΓÇ¥ The algorithm you select depends primarily on two different aspects of your data science scenario:

+ - **What you want to do with your data?** Specifically, what is the business question you want to answer by learning from your past data?

+ - **What are the requirements of your data science scenario?** Specifically, what is the accuracy, training time, linearity, number of parameters, and number of features your solution supports?

+ 

+>[!Note]

+> Designer supports two types of components, classic prebuilt components (v1) and custom components (v2). These two types of components are NOT compatible.

+>

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>

+>This article applies to classic prebuilt components and not compatible with CLI v2 and SDK v2.

+## Business scenarios and the Machine Learning Algorithm Cheat Sheet

+The [Azure Machine Learning Algorithm Cheat Sheet](./algorithm-cheat-sheet.md?WT.mc_id=docs-article-lazzeri) helps you with the first consideration: **What you want to do with your data**? On the Machine Learning Algorithm Cheat Sheet, look for task you want to do, and then find a [Azure Machine Learning designer](./concept-designer.md?WT.mc_id=docs-article-lazzeri) algorithm for the predictive analytics solution.

+Machine Learning designer provides a comprehensive portfolio of algorithms, such as [Multiclass Decision Forest](../algorithm-module-reference/multiclass-decision-forest.md?WT.mc_id=docs-article-lazzeri), [Recommendation systems](../algorithm-module-reference/evaluate-recommender.md?WT.mc_id=docs-article-lazzeri), [Neural Network Regression](../algorithm-module-reference/neural-network-regression.md?WT.mc_id=docs-article-lazzeri), [Multiclass Neural Network](../algorithm-module-reference/multiclass-neural-network.md?WT.mc_id=docs-article-lazzeri), and [K-Means Clustering](../algorithm-module-reference/k-means-clustering.md?WT.mc_id=docs-article-lazzeri). Each algorithm is designed to address a different type of machine learning problem. See the [Machine Learning designer algorithm and component reference](../component-reference/component-reference.md?WT.mc_id=docs-article-lazzeri) for a complete list along with documentation about how each algorithm works and how to tune parameters to optimize the algorithm.

+> [!NOTE]

+> Download the cheat sheet here: [Machine Learning Algorithm Cheat Sheet (11x17 in.)](https://download.microsoft.com/download/3/5/b/35bb997f-a8c7-485d-8c56-19444dafd757/azure-machine-learning-algorithm-cheat-sheet-july-2021.pdf)

+>

+>

+Along with guidance in the Azure Machine Learning Algorithm Cheat Sheet, keep in mind other requirements when choosing a machine learning algorithm for your solution. Following are additional factors to consider, such as the accuracy, training time, linearity, number of parameters and number of features.

+## Comparison of machine learning algorithms

+Some learning algorithms make particular assumptions about the structure of the data or the desired results. If you can find one that fits your needs, it can give you more useful results, more accurate predictions, or faster training times.

+The following table summarizes some of the most important characteristics of algorithms from the classification, regression, and clustering families:

+| **Algorithm** | **Accuracy** | **Training time** | **Linearity** | **Parameters** | **Notes** |

+| |::|::|::|::| |

+| **Classification family** | | | | | |

+| [Two-Class logistic regression](../algorithm-module-reference/two-class-logistic-regression.md?WT.mc_id=docs-article-lazzeri) |Good |Fast |Yes |4 | |

+| [Two-class decision forest](../algorithm-module-reference/two-class-decision-forest.md?WT.mc_id=docs-article-lazzeri) |Excellent |Moderate |No |5 |Shows slower scoring times. Suggest not working with One-vs-All Multiclass, because of slower scoring times caused by tread locking in accumulating tree predictions |

+| [Two-class boosted decision tree](../algorithm-module-reference/two-class-boosted-decision-tree.md?WT.mc_id=docs-article-lazzeri) |Excellent |Moderate |No |6 |Large memory footprint |

+| [Two-class neural network](../algorithm-module-reference/two-class-neural-network.md?WT.mc_id=docs-article-lazzeri) |Good |Moderate |No |8 | |

+| [Two-class averaged perceptron](../algorithm-module-reference/two-class-averaged-perceptron.md?WT.mc_id=docs-article-lazzeri) |Good |Moderate |Yes |4 | |

+| [Two-class support vector machine](../algorithm-module-reference/two-class-support-vector-machine.md?WT.mc_id=docs-article-lazzeri) |Good |Fast |Yes |5 |Good for large feature sets |

+| [Multiclass logistic regression](../algorithm-module-reference/multiclass-logistic-regression.md?WT.mc_id=docs-article-lazzeri) |Good |Fast |Yes |4 | |

+| [Multiclass decision forest](../algorithm-module-reference/multiclass-decision-forest.md?WT.mc_id=docs-article-lazzeri) |Excellent |Moderate |No |5 |Shows slower scoring times |

+| [Multiclass boosted decision tree](../algorithm-module-reference/multiclass-boosted-decision-tree.md?WT.mc_id=docs-article-lazzeri) |Excellent |Moderate |No |6 | Tends to improve accuracy with some small risk of less coverage |

+| [Multiclass neural network](../algorithm-module-reference/multiclass-neural-network.md?WT.mc_id=docs-article-lazzeri) |Good |Moderate |No |8 | |

+| [One-vs-all multiclass](../algorithm-module-reference/one-vs-all-multiclass.md?WT.mc_id=docs-article-lazzeri) | - | - | - | - |See properties of the two-class method selected |

+| **Regression family** | | | | | |

+| [Linear regression](../algorithm-module-reference/linear-regression.md?WT.mc_id=docs-article-lazzeri) |Good |Fast |Yes |4 | |

+| [Decision forest regression](../algorithm-module-reference/decision-forest-regression.md?WT.mc_id=docs-article-lazzeri)|Excellent |Moderate |No |5 | |

+| [Boosted decision tree regression](../algorithm-module-reference/boosted-decision-tree-regression.md?WT.mc_id=docs-article-lazzeri) |Excellent |Moderate |No |6 |Large memory footprint |

+| [Neural network regression](../algorithm-module-reference/neural-network-regression.md?WT.mc_id=docs-article-lazzeri) |Good |Moderate |No |8 | |

+| **Clustering family** | | | | | |

+| [K-means clustering](../algorithm-module-reference/k-means-clustering.md?WT.mc_id=docs-article-lazzeri) |Excellent |Moderate |Yes |8 |A clustering algorithm |

+## Requirements for a data science scenario

+Once you know what you want to do with your data, you need to determine additional requirements for your solution.

+Make choices and possibly trade-offs for the following requirements:

+- Accuracy

+- Training time

+- Linearity

+- Number of parameters

+- Number of features

+## Accuracy

+Accuracy in machine learning measures the effectiveness of a model as the proportion of true results to total cases. In Machine Learning designer, the [Evaluate Model component](../algorithm-module-reference/evaluate-model.md?WT.mc_id=docs-article-lazzeri) computes a set of industry-standard evaluation metrics. You can use this component to measure the accuracy of a trained model.

+Getting the most accurate answer possible isnΓÇÖt always necessary. Sometimes an approximation is adequate, depending on what you want to use it for. If that is the case, you may be able to cut your processing time dramatically by sticking with more approximate methods. Approximate methods also naturally tend to avoid overfitting.

+There are three ways to use the Evaluate Model component:

+- Generate scores over your training data in order to evaluate the model

+- Generate scores on the model, but compare those scores to scores on a reserved testing set

+- Compare scores for two different but related models, using the same set of data

+For a complete list of metrics and approaches you can use to evaluate the accuracy of machine learning models, see [Evaluate Model component](../algorithm-module-reference/evaluate-model.md?WT.mc_id=docs-article-lazzeri).

+## Training time

+In supervised learning, training means using historical data to build a machine learning model that minimizes errors. The number of minutes or hours necessary to train a model varies a great deal between algorithms. Training time is often closely tied to accuracy; one typically accompanies the other.

+In addition, some algorithms are more sensitive to the number of data points than others. You might choose a specific algorithm because you have a time limitation, especially when the data set is large.

+In Machine Learning designer, creating and using a machine learning model is typically a three-step process:

+1. Configure a model, by choosing a particular type of algorithm, and then defining its parameters or hyperparameters.

+2. Provide a dataset that is labeled and has data compatible with the algorithm. Connect both the data and the model to [Train Model component](../algorithm-module-reference/train-model.md?WT.mc_id=docs-article-lazzeri).

+3. After training is completed, use the trained model with one of the [scoring components](../algorithm-module-reference/score-model.md?WT.mc_id=docs-article-lazzeri) to make predictions on new data.

+## Linearity

+Linearity in statistics and machine learning means that there is a linear relationship between a variable and a constant in your dataset. For example, linear classification algorithms assume that classes can be separated by a straight line (or its higher-dimensional analog).

+Lots of machine learning algorithms make use of linearity. In Azure Machine Learning designer, they include:

+- [Multiclass logistic regression](../algorithm-module-reference/multiclass-logistic-regression.md?WT.mc_id=docs-article-lazzeri)

+- [Two-class logistic regression](../algorithm-module-reference/two-class-logistic-regression.md?WT.mc_id=docs-article-lazzeri)

+- [Support vector machines](../algorithm-module-reference/two-class-support-vector-machine.md?WT.mc_id=docs-article-lazzeri)

+Linear regression algorithms assume that data trends follow a straight line. This assumption isn't bad for some problems, but for others it reduces accuracy. Despite their drawbacks, linear algorithms are popular as a first strategy. They tend to be algorithmically simple and fast to train.

+

+***Nonlinear class boundary***: *Relying on a linear classification

+algorithm would result in low accuracy.*

+

+***Data with a nonlinear trend***: *Using a linear regression method would

+generate much larger errors than necessary.*

+## Number of parameters

+Parameters are the knobs a data scientist gets to turn when setting up an algorithm. They are numbers that affect the algorithmΓÇÖs behavior, such as error tolerance or number of iterations, or options between variants of how the algorithm behaves. The training time and accuracy of the algorithm can sometimes be sensitive to getting just the right settings. Typically, algorithms with large numbers of parameters require the most trial and error to find a good combination.

+Alternatively, there is the [Tune Model Hyperparameters component](../algorithm-module-reference/tune-model-hyperparameters.md?WT.mc_id=docs-article-lazzeri) in Machine Learning designer: The goal of this component is to determine the optimum hyperparameters for a machine learning model. The component builds and tests multiple models by using different combinations of settings. It compares metrics over all models to get the combinations of settings.

+While this is a great way to make sure youΓÇÖve spanned the parameter space, the time required to train a model increases exponentially with the number of parameters. The upside is that having many parameters typically indicates that an algorithm has greater flexibility. It can often achieve very good accuracy, provided you can find the right combination of parameter settings.

+## Number of features

+In machine learning, a feature is a quantifiable variable of the phenomenon you are trying to analyze. For certain types of data, the number of features can be very large compared to the number of data points. This is often the case with genetics or textual data.

+A large number of features can bog down some learning algorithms, making training time unfeasibly long. [Support vector machines](../algorithm-module-reference/two-class-support-vector-machine.md?WT.mc_id=docs-article-lazzeri) are particularly well suited to scenarios with a high number of features. For this reason, they have been used in many applications from information retrieval to text and image classification. Support vector machines can be used for both classification and regression tasks.

+Feature selection refers to the process of applying statistical tests to inputs, given a specified output. The goal is to determine which columns are more predictive of the output. The [Filter Based Feature Selection component](../algorithm-module-reference/filter-based-feature-selection.md?WT.mc_id=docs-article-lazzeri) in Machine Learning designer provides multiple feature selection algorithms to choose from. The component includes correlation methods such as Pearson correlation and chi-squared values.

+You can also use the [Permutation Feature Importance component](../algorithm-module-reference/permutation-feature-importance.md?WT.mc_id=docs-article-lazzeri) to compute a set of feature importance scores for your dataset. You can then leverage these scores to help you determine the best features to use in a model.

+## Next steps

+ - [Learn more about Azure Machine Learning designer](./concept-designer.md?WT.mc_id=docs-article-lazzeri)

+ - For descriptions of all the machine learning algorithms available in Azure Machine Learning designer, see [Machine Learning designer algorithm and component reference](../component-reference/component-reference.md?WT.mc_id=docs-article-lazzeri)

+ - To explore the relationship between deep learning, machine learning, and AI, see [Deep Learning vs. Machine Learning](../concept-deep-learning-vs-machine-learning.md?WT.mc_id=docs-article-lazzeri)

+ Title: Use pipeline parameters to build versatile pipelines

+description: How to use pipeline parameters in Azure Machine Learning designer.

+# Use pipeline parameters in the designer to build versatile pipelines

+Use pipeline parameters to build flexible pipelines in the designer. Pipeline parameters let you dynamically set values at runtime to encapsulate pipeline logic and reuse assets.

+Pipeline parameters are especially useful when resubmitting a pipeline job, [retraining models](how-to-retrain-designer.md), or [performing batch predictions](how-to-run-batch-predictions-designer.md).

+In this article, you learn how to do the following:

+> [!div class="checklist"]

+> * Create pipeline parameters

+> * Delete and manage pipeline parameters

+> * Trigger pipeline jobs while adjusting pipeline parameters

+## Prerequisites

+* An Azure Machine Learning workspace. See [Create workspace resources](../quickstart-create-resources.md).

+* For a guided introduction to the designer, complete the [designer tutorial](tutorial-designer-automobile-price-train-score.md).

+## Create pipeline parameter

+There are three ways to create a pipeline parameter in the designer:

+- Create a pipeline parameter in the settings panel, and bind it to a component.

+- Promote a component parameter to a pipeline parameter.

+- Promote a dataset to a pipeline parameter

+> [!NOTE]

+> Pipeline parameters only support basic data types like `int`, `float`, and `string`.

+### Option 1: Create a pipeline parameter in the settings panel

+In this section, you create a pipeline parameter in the settings panel.

+In this example, you create a pipeline parameter that defines how a pipeline fills in missing data using the **Clean missing data** component.

+1. Next to the name of your pipeline draft, select the **gear icon** to open the **Settings** panel.

+1. In the **Pipeline parameters** section, select the **+** icon.

+1. Enter a name for the parameter and a default value.

+ For example, enter `replace-missing-value` as parameter name and `0` as default value.

+ 

+After you create a pipeline parameter, you must [attach it to the component parameter](#attach-component-parameter-to-pipeline-parameter) that you want to dynamically set.

+### Option 2: Promote a component parameter

+The simplest way to create a pipeline parameter for a component value is to promote a component parameter. Use the following steps to promote a component parameter to a pipeline parameter:

+1. Select the component you want to attach a pipeline parameter to.

+1. In the component detail pane, mouseover the parameter you want to specify.

+1. Select the ellipses (**...**) that appear.

+1. Select **Add to pipeline parameter**.

+ 

+1. Enter a parameter name and default value.

+1. Select **Save**

+You can now specify new values for this parameter anytime you submit this pipeline.

+### Option 3: Promote a dataset to a pipeline parameter

+If you want to submit your pipeline with variable datasets, you must promote your dataset to a pipeline parameter:

+1. Select the dataset you want to turn into a pipeline parameter.

+1. In the detail panel of dataset, check **Set as pipeline parameter**.

+ 

+You can now specify a different dataset by using the pipeline parameter the next time you run the pipeline.

+## Attach and detach component parameter to pipeline parameter

+In this section, you will learn how to attach and detach component parameter to pipeline parameter.

+### Attach component parameter to pipeline parameter

+You can attach the same component parameters of duplicated components to the same pipeline parameter if you want to alter the value at one time when triggering the pipeline job.

+The following example has duplicated **Clean Missing Data** component. For each **Clean Missing Data** component, attach **Replacement value** to pipeline parameter **replace-missing-value**:

+1. Select the **Clean Missing Data** component.

+1. In the component detail pane, to the right of the canvas, set the **Cleaning mode** to "Custom substitution value".

+1. Mouseover the **Replacement value** field.

+1. Select the ellipses (**...**) that appear.

+1. Select the pipeline parameter `replace-missing-value`.

+ 

+You have successfully attached the **Replacement value** field to your pipeline parameter.

+### Detach component parameter to pipeline parameter

+After you attach **Replacement value** to pipeline parameter, it is non-actionable.

+You can detach component parameter to pipeline parameter by clicking the ellipses (**...**) next to the component parameter, and select **Detach from pipeline parameter**.

+ 

+## Update and delete pipeline parameters

+In this section, you learn how to update and delete pipeline parameters.

+### Update pipeline parameters

+Use the following steps to update a component pipeline parameter:

+1. At the top of the canvas, select the gear icon.

+1. In the **Pipeline parameters** section, you can view and update the name and default value for all of your pipeline parameter.

+### Delete a dataset pipeline parameter

+Use the following steps to delete a dataset pipeline parameter:

+1. Select the dataset component.

+1. Uncheck the option **Set as pipeline parameter**.

+### Delete component pipeline parameters

+Use the following steps to delete a component pipeline parameter:

+1. At the top of the canvas, select the gear icon.

+1. Select the ellipses (**...**) next to the pipeline parameter.

+ This view shows you which components the pipeline parameter is attached to.

+ 

+1. Select **Delete parameter** to delete the pipeline parameter.

+ > [!NOTE]

+ > Deleting a pipeline parameter will cause all attached component parameters to be detached and the value of detached component parameters will keep current pipeline parameter value.

+## Trigger a pipeline job with pipeline parameters

+In this section, you learn how to submit a pipeline job while setting pipeline parameters.

+### Resubmit a pipeline job

+After submitting a pipeline with pipeline parameters, you can resubmit a pipeline job with different parameters:

+1. Go to pipeline detail page. In the **Pipeline job overview** window, you can check current pipeline parameters and values.

+1. Select **Resubmit**.

+1. In the **Setup pipeline job**, specify your new pipeline parameters.

+

+### Use published pipelines

+You can also publish a pipeline to use its pipeline parameters. A **published pipeline** is a pipeline that has been deployed to a compute resource, which client applications can invoke via a REST endpoint.

+Published endpoints are especially useful for retraining and batch prediction scenarios. For more information, see [How to retrain models in the designer](how-to-retrain-designer.md) or [Run batch predictions in the designer](how-to-run-batch-predictions-designer.md).

+## Next steps

+In this article, you learned how to create pipeline parameters in the designer. Next, see how you can use pipeline parameters to [retrain models](how-to-retrain-designer.md) or perform [batch predictions](how-to-run-batch-predictions-designer.md).

+You can also learn how to [use pipelines programmatically with the SDK v1](how-to-deploy-pipelines.md?view=azureml-api-1&preserve-view=true).

+ 

+1. Select the **Gear icon**  next to the pipeline name.

+ Title: Example pipelines & datasets for the designer

+description: Learn how to use samples in Azure Machine Learning designer to jumps-start your machine learning pipelines.

+# Example pipelines & datasets for Azure Machine Learning designer

+Use the built-in examples in Azure Machine Learning designer to quickly get started building your own machine learning pipelines. The Azure Machine Learning designer [GitHub repository](https://github.com/Azure/MachineLearningDesigner) contains detailed documentation to help you understand some common machine learning scenarios.

+## Prerequisites

+* An Azure subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/)

+* An Azure Machine Learning workspace

+## Use sample pipelines

+The designer saves a copy of the sample pipelines to your studio workspace. You can edit the pipeline to adapt it to your needs and save it as your own. Use them as a starting point to jumpstart your projects.

+Here's how to use a designer sample:

+1. Sign in to <a href="https://ml.azure.com?tabs=jre" target="_blank">ml.azure.com</a>, and select the workspace you want to work with.

+1. Select **Designer**.

+1. Select a sample pipeline under the **New pipeline** section.

+ Select **Show more samples** for a complete list of samples.

+1. To run a pipeline, you first have to set default compute target to run the pipeline on.

+ 1. In the **Settings** pane to the right of the canvas, select **Select compute target**.

+ 1. In the dialog that appears, select an existing compute target or create a new one. Select **Save**.

+ 1. Select **Submit** at the top of the canvas to submit a pipeline job.

+ Depending on the sample pipeline and compute settings, jobs may take some time to complete. The default compute settings have a minimum node size of 0, which means that the designer must allocate resources after being idle. Repeated pipeline jobs will take less time since the compute resources are already allocated. Additionally, the designer uses cached results for each component to further improve efficiency.

+1. After the pipeline finishes running, you can review the pipeline and view the output for each component to learn more. Use the following steps to view component outputs:

+ 1. Right-click the component in the canvas whose output you'd like to see.

+ 1. Select **Visualize**.

+ Use the samples as starting points for some of the most common machine learning scenarios.

+## Regression

+Explore these built-in regression samples.

+| Sample title | Description |

+| | |

+| [Regression - Automobile Price Prediction (Basic)](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/regression-automobile-price-prediction-basic.md) | Predict car prices using linear regression. |

+| [Regression - Automobile Price Prediction (Advanced)](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/regression-automobile-price-prediction-compare-algorithms.md) | Predict car prices using decision forest and boosted decision tree regressors. Compare models to find the best algorithm.

+## Classification

+Explore these built-in classification samples. You can learn more about the samples by opening the samples and viewing the component comments in the designer.

+| Sample title | Description |

+| | |

+| [Binary Classification with Feature Selection - Income Prediction](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/binary-classification-feature-selection-income-prediction.md) | Predict income as high or low, using a two-class boosted decision tree. Use Pearson correlation to select features.

+| [Binary Classification with custom Python script - Credit Risk Prediction](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/binary-classification-python-credit-prediction.md) | Classify credit applications as high or low risk. Use the Execute Python Script component to weight your data.

+| [Binary Classification - Customer Relationship Prediction](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/binary-classification-customer-relationship-prediction.md) | Predict customer churn using two-class boosted decision trees. Use SMOTE to sample biased data.

+| [Text Classification - Wikipedia SP 500 Dataset](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/text-classification-wiki.md) | Classify company types from Wikipedia articles with multiclass logistic regression. |

+| Multiclass Classification - Letter Recognition | Create an ensemble of binary classifiers to classify written letters. |

+## Computer vision

+Explore these built-in computer vision samples. You can learn more about the samples by opening the samples and viewing the component comments in the designer.

+| Sample title | Description |

+| | |

+| Image Classification using DenseNet | Use computer vision components to build image classification model based on PyTorch DenseNet.|

+## Recommender

+Explore these built-in recommender samples. You can learn more about the samples by opening the samples and viewing the component comments in the designer.

+| Sample title | Description |

+| | |

+| Wide & Deep based Recommendation - Restaurant Rating Prediction | Build a restaurant recommender engine from restaurant/user features and ratings.|

+| Recommendation - Movie Rating Tweets | Build a movie recommender engine from movie/user features and ratings.|

+## Utility

+Learn more about the samples that demonstrate machine learning utilities and features. You can learn more about the samples by opening the samples and viewing the component comments in the designer.

+| Sample title | Description |

+| | |

+| Binary Classification using Vowpal Wabbit Model - Adult Income Prediction | Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning. This sample shows how to use Vowpal Wabbit model to build binary classification model.

+| [Use custom R script - Flight Delay Prediction](https://github.com/Azure/MachineLearningDesigner/blob/master/articles/samples/r-script-flight-delay-prediction.md) | Use customized R script to predict if a scheduled passenger flight will be delayed by more than 15 minutes.

+| Cross Validation for Binary Classification - Adult Income Prediction | Use cross validation to build a binary classifier for adult income.

+| Permutation Feature Importance | Use permutation feature importance to compute importance scores for the test dataset.

+| Tune Parameters for Binary Classification - Adult Income Prediction | Use Tune Model Hyperparameters to find optimal hyperparameters to build a binary classifier. |

+## Datasets

+When you create a new pipeline in Azure Machine Learning designer, a number of sample datasets are included by default. These sample datasets are used by the sample pipelines in the designer homepage.

+The sample datasets are available under **Datasets**-**Samples** category. You can find this in the component palette to the left of the canvas in the designer. You can use any of these datasets in your own pipeline by dragging it to the canvas.

+| Dataset&nbsp;name&nbsp;&nbsp;&nbsp;&nbsp;| Dataset description |

+|-|:--|

+| Adult Census Income Binary Classification dataset | A subset of the 1994 Census database, using working adults over the age of 16 with an adjusted income index of > 100.<br/>**Usage**: Classify people using demographics to predict whether a person earns over 50K a year.<br/> **Related Research**: Kohavi, R., Becker, B., (1996). [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science|

+|Automobile price data (Raw)|Information about automobiles by make and model, including the price, features such as the number of cylinders and MPG, as well as an insurance risk score.<br/> The risk score is initially associated with auto price. It is then adjusted for actual risk in a process known to actuaries as symboling. A value of +3 indicates that the auto is risky, and a value of -3 that it is probably safe.<br/>**Usage**: Predict the risk score by features, using regression or multivariate classification.<br/>**Related Research**: Schlimmer, J.C. (1987). [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science. |

+| CRM Appetency Labels Shared |Labels from the KDD Cup 2009 customer relationship prediction challenge ([orange_small_train_appetency.labels](https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train_appetency.labels)).|

+|CRM Churn Labels Shared|Labels from the KDD Cup 2009 customer relationship prediction challenge ([orange_small_train_churn.labels](https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Datas)).|

+|CRM Dataset Shared | This data comes from the KDD Cup 2009 customer relationship prediction challenge ([orange_small_train.data.zip](https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train.data.zip)). <br/>The dataset contains 50K customers from the French Telecom company Orange. Each customer has 230 anonymized features, 190 of which are numeric and 40 are categorical. The features are very sparse. |

+|CRM Upselling Labels Shared|Labels from the KDD Cup 2009 customer relationship prediction challenge ([orange_large_train_upselling.labels](https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train_upselling.labels)|

+|Flight Delays Data|Passenger flight on-time performance data taken from the TranStats data collection of the U.S. Department of Transportation ([On-Time](https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time)).<br/>The dataset covers the time period April-October 2013. Before uploading to the designer, the dataset was processed as follows: <br/>- The dataset was filtered to cover only the 70 busiest airports in the continental US <br/>- Canceled flights were labeled as delayed by more than 15 minutes <br/>- Diverted flights were filtered out <br/>- The following columns were selected: Year, Month, DayofMonth, DayOfWeek, Carrier, OriginAirportID, DestAirportID, CRSDepTime, DepDelay, DepDel15, CRSArrTime, ArrDelay, ArrDel15, Canceled|

+|German Credit Card UCI dataset|The UCI Statlog (German Credit Card) dataset ([Statlog+German+Credit+Data](https://archive.ics.uci.edu/ml/datasets/Statlog+(German+Credit+Data))), using the german.data file.<br/>The dataset classifies people, described by a set of attributes, as low or high credit risks. Each example represents a person. There are 20 features, both numerical and categorical, and a binary label (the credit risk value). High credit risk entries have label = 2, low credit risk entries have label = 1. The cost of misclassifying a low risk example as high is 1, whereas the cost of misclassifying a high risk example as low is 5.|

+|IMDB Movie Titles|The dataset contains information about movies that were rated in Twitter tweets: IMDB movie ID, movie name, genre, and production year. There are 17K movies in the dataset. The dataset was introduced in the paper "S. Dooms, T. De Pessemier and L. Martens. MovieTweetings: a Movie Rating Dataset Collected From Twitter. Workshop on Crowdsourcing and Human Computation for Recommender Systems, CrowdRec at RecSys 2013."|

+|Movie Ratings|The dataset is an extended version of the Movie Tweetings dataset. The dataset has 170K ratings for movies, extracted from well-structured tweets on Twitter. Each instance represents a tweet and is a tuple: user ID, IMDB movie ID, rating, timestamp, number of favorites for this tweet, and number of retweets of this tweet. The dataset was made available by A. Said, S. Dooms, B. Loni and D. Tikk for Recommender Systems Challenge 2014.|

+|Weather Dataset|Hourly land-based weather observations from NOAA ([merged data from 201304 to 201310](https://az754797.vo.msecnd.net/data/WeatherDataset.csv)).<br/>The weather data covers observations made from airport weather stations, covering the time period April-October 2013. Before uploading to the designer, the dataset was processed as follows: <br/> - Weather station IDs were mapped to corresponding airport IDs <br/> - Weather stations not associated with the 70 busiest airports were filtered out <br/> - The Date column was split into separate Year, Month, and Day columns <br/> - The following columns were selected: AirportID, Year, Month, Day, Time, TimeZone, SkyCondition, Visibility, WeatherType, DryBulbFarenheit, DryBulbCelsius, WetBulbFarenheit, WetBulbCelsius, DewPointFarenheit, DewPointCelsius, RelativeHumidity, WindSpeed, WindDirection, ValueForWindCharacter, StationPressure, PressureTendency, PressureChange, SeaLevelPressure, RecordType, HourlyPrecip, Altimeter|

+|Wikipedia SP 500 Dataset|Data is derived from Wikipedia (https://www.wikipedia.org/) based on articles of each S&P 500 company, stored as XML data. <br/>Before uploading to the designer, the dataset was processed as follows: <br/> - Extract text content for each specific company <br/> - Remove wiki formatting <br/> - Remove non-alphanumeric characters <br/> - Convert all text to lowercase <br/> - Known company categories were added <br/>Note that for some companies an article could not be found, so the number of records is less than 500.|

+|Restaurant Feature Data| A set of metadata about restaurants and their features, such as food type, dining style, and location. <br/>**Usage**: Use this dataset, in combination with the other two restaurant datasets, to train and test a recommender system.<br/> **Related Research**: Bache, K. and Lichman, M. (2013). [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science.|

+|Restaurant Ratings| Contains ratings given by users to restaurants on a scale from 0 to 2.<br/>**Usage**: Use this dataset, in combination with the other two restaurant datasets, to train and test a recommender system. <br/>**Related Research**: Bache, K. and Lichman, M. (2013). [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science.|

+|Restaurant Customer Data| A set of metadata about customers, including demographics and preferences. <br/>**Usage**: Use this dataset, in combination with the other two restaurant datasets, to train and test a recommender system. <br/> **Related Research**: Bache, K. and Lichman, M. (2013). [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml) Irvine, CA: University of California, School of Information and Computer Science.|

+## Clean up resources

+## Next steps

+Learn the fundamentals of predictive analytics and machine learning with [Tutorial: Predict automobile price with the designer](tutorial-designer-automobile-price-train-score.md)

+ Title: 'Tutorial: Designer - deploy no-code models'

+description: Deploy a machine learning model to predict car prices with the Azure Machine Learning designer.

+# Tutorial: Designer - deploy a machine learning model

+Use the designer to deploy a machine learning model to predict the price of cars. This tutorial is part two of a two-part series.

+>[!Note]

+> Designer supports two types of components, classic prebuilt components (v1) and custom components (v2). These two types of components are NOT compatible.

+>

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>

+>This article applies to classic prebuilt components and not compatible with CLI v2 and SDK v2.

+In [part one of the tutorial](tutorial-designer-automobile-price-train-score.md) you trained a linear regression model on car prices. In part two, you deploy the model to give others a chance to use it. In this tutorial, you:

+> [!div class="checklist"]

+> * Create a real-time inference pipeline.

+> * Create an inferencing cluster.

+> * Deploy the real-time endpoint.

+> * Test the real-time endpoint.

+## Prerequisites

+Complete [part one of the tutorial](tutorial-designer-automobile-price-train-score.md) to learn how to train and score a machine learning model in the designer.

+## Create a real-time inference pipeline

+To deploy your pipeline, you must first convert the training pipeline into a real-time inference pipeline. This process removes training components and adds web service inputs and outputs to handle requests.

+> [!NOTE]

+> Create inference pipeline only supports training pipelines which contain only the designer built-in components and must have a component like **Train Model** which outputs the trained model.

+### Create a real-time inference pipeline

+1. On pipeline job detail page, above the pipeline canvas, select **Create inference pipeline** > **Real-time inference pipeline**.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/create-real-time-inference.png" alt-text="Screenshot of create inference pipeline in pipeline job detail page.":::

+ Your new pipeline will now look like this:

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/real-time-inference-pipeline.png" alt-text="Screenshot showing the expected configuration of the pipeline after preparing it for deployment.":::

+ When you select **Create inference pipeline**, several things happen:

+

+ * The trained model is stored as a **Dataset** component in the component palette. You can find it under **My Datasets**.

+ * Training components like **Train Model** and **Split Data** are removed.

+ * The saved trained model is added back into the pipeline.

+ * **Web Service Input** and **Web Service Output** components are added. These components show where user data enters the pipeline and where data is returned.

+ > [!NOTE]

+ > By default, the **Web Service Input** will expect the same data schema as the component output data which connects to the same downstream port as it. In this sample, **Web Service Input** and **Automobile price data (Raw)** connect to the same downstream component, hence **Web Service Input** expect the same data schema as **Automobile price data (Raw)** and target variable column `price` is included in the schema.

+ > However, usually When you score the data, you won't know the target variable values. For such case, you can remove the target variable column in the inference pipeline using **Select Columns in Dataset** component. Make sure that the output of **Select Columns in Dataset** removing target variable column is connected to the same port as the output of the **Web Service Input** component.

+1. Select **Submit**, and use the same compute target and experiment that you used in part one.

+ If this is the first job, it may take up to 20 minutes for your pipeline to finish running. The default compute settings have a minimum node size of 0, which means that the designer must allocate resources after being idle. Repeated pipeline jobs will take less time since the compute resources are already allocated. Additionally, the designer uses cached results for each component to further improve efficiency.

+1. Go to the real-time inference pipeline job detail by selecting **Job detail** link in the left pane.

+1. Select **Deploy** in the job detail page.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/deploy-in-job-detail-page.png" alt-text="Screenshot showing deploying in job detail page.":::

+## Create an inferencing cluster

+In the dialog box that appears, you can select from any existing Azure Kubernetes Service (AKS) clusters to deploy your model to. If you don't have an AKS cluster, use the following steps to create one.

+1. Select **Compute** in the dialog box that appears to go to the **Compute** page.

+1. On the navigation ribbon, select **Inference Clusters** > **+ New**.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/new-inference-cluster.png" alt-text="Screenshot showing how to get to the new inference cluster pane.":::

+

+1. In the inference cluster pane, configure a new Kubernetes Service.

+1. Enter *aks-compute* for the **Compute name**.

+

+1. Select a nearby region that's available for the **Region**.

+1. Select **Create**.

+ > [!NOTE]

+ > It takes approximately 15 minutes to create a new AKS service. You can check the provisioning state on the **Inference Clusters** page.

+## Deploy the real-time endpoint

+After your AKS service has finished provisioning, return to the real-time inferencing pipeline to complete deployment.

+1. Select **Deploy** above the canvas.

+1. Select **Deploy new real-time endpoint**.

+1. Select the AKS cluster you created.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/setup-endpoint.png" alt-text="Screenshot showing how to set up a new real-time endpoint.":::

+ You can also change **Advanced** setting for your real-time endpoint.

+ |Advanced setting|Description|

+ |||

+ |Enable Application Insights diagnostics and data collection| Whether to enable Azure Application Insights to collect data from the deployed endpoints. </br> By default: false. |

+ |Scoring timeout| A timeout in milliseconds to enforce for scoring calls to the web service.</br>By default: 60000.|

+ |Auto scale enabled| Whether to enable autoscaling for the web service.</br>By default: true.|

+ |Min replicas| The minimum number of containers to use when autoscaling this web service.</br>By default: 1.|

+ |Max replicas| The maximum number of containers to use when autoscaling this web service.</br> By default: 10.|

+ |Target utilization|The target utilization (in percent out of 100) that the autoscaler should attempt to maintain for this web service.</br> By default: 70.|

+ |Refresh period|How often (in seconds) the autoscaler attempts to scale this web service.</br> By default: 1.|

+ |CPU reserve capacity|The number of CPU cores to allocate for this web service.</br> By default: 0.1.|

+ |Memory reserve capacity|The amount of memory (in GB) to allocate for this web service.</br> By default: 0.5.|

+1. Select **Deploy**.

+ A success notification from the notification center appears after deployment finishes. It might take a few minutes.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/deploy-notification.png" alt-text="Screenshot showing deployment notification.":::

+> [!TIP]

+> You can also deploy to **Azure Container Instance** (ACI) if you select **Azure Container Instance** for **Compute type** in the real-time endpoint setting box.

+> Azure Container Instance is used for testing or development. Use ACI for low-scale CPU-based workloads that require less than 48 GB of RAM.

+## Test the real-time endpoint

+After deployment finishes, you can view your real-time endpoint by going to the **Endpoints** page.

+1. On the **Endpoints** page, select the endpoint you deployed.

+ In the **Details** tab, you can see more information such as the REST URI, Swagger definition, status, and tags.

+ In the **Consume** tab, you can find sample consumption code, security keys, and set authentication methods.

+ In the **Deployment logs** tab, you can find the detailed deployment logs of your real-time endpoint.

+1. To test your endpoint, go to the **Test** tab. From here, you can enter test data and select **Test** verify the output of your endpoint.

+## Update the real-time endpoint

+You can update the online endpoint with new model trained in the designer. On the online endpoint detail page, find your previous training pipeline job and inference pipeline job.

+1. You can directly find and modify your training pipeline draft in the designer homepage.

+

+ Or you can open the training pipeline job link and then clone it into a new pipeline draft to continue editing.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/endpoint-train-job-link.png" alt-text="Screenshot showing training job link in endpoint detail page.":::

+1. After you submit the modified training pipeline, go to the job detail page.

+1. When the job completes, right click **Train Model** and select **Register data**.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/register-train-model-as-dataset.png" alt-text="Screenshot showing register trained model as dataset.":::

+ Input name and select **File** type.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/register-train-model-as-dataset-2.png" alt-text="Screenshot of register as a data asset with new data asset selected.":::

+1. After the dataset registers successfully, open your inference pipeline draft, or clone the previous inference pipeline job into a new draft. In the inference pipeline draft, replace the previous trained model shown as **MD-XXXX** node connected to the **Score Model** component with the newly registered dataset.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/modify-inference-pipeline.png" alt-text="Screenshot showing how to modify inference pipeline.":::

+1. If you need to update the data preprocessing part in your training pipeline, and would like to update that into the inference pipeline, the processing is similar as steps above.

+ You just need to register the transformation output of the transformation component as dataset.

+ Then manually replace the **TD-** component in inference pipeline with the registered dataset.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/replace-td-module.png" alt-text="Screenshot showing how to replace transformation component.":::

+1. After modifying your inference pipeline with the newly trained model or transformation, submit it. When the job is completed, deploy it to the existing online endpoint deployed previously.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/deploy-to-existing-endpoint.png" alt-text="Screenshot showing how to replace existing real-time endpoint.":::

+## Limitations

+* Due to datastore access limitation, if your inference pipeline contains **Import Data** or **Export Data** component, they'll be auto-removed when deploy to real-time endpoint.

+* If you have datasets in the real-time inference pipeline and want to deploy it to real-time endpoint, currently this flow only supports datasets registered from **Blob** datastore. If you want to use datasets from other type datastores, you can use Select Column to connect with your initial dataset with settings of selecting all columns, register the outputs of Select Column as File dataset and then replace the initial dataset in the real-time inference pipeline with this newly registered dataset.

+* If your inference graph contains "Enter Data Manually" component which is not connected to the same port as ΓÇ£Web service InputΓÇ¥ component, the "Enter Data Manually" component will not be executed during HTTP call processing. A workaround is to register the outputs of that "Enter Data Manually" component as dataset, then in the inference pipeline draft, replace the "Enter Data Manually" component with the registered dataset.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-deploy/real-time-inferencepipeline-limitation.png" alt-text="Screenshot showing how to modify inference pipeline containing enter data manually component.":::

+## Clean up resources

+## Next steps

+In this tutorial, you learned the key steps in how to create, deploy, and consume a machine learning model in the designer. To learn more about how you can use the designer see the following links:

+ Title: 'Tutorial: Designer - train a no-code regression model'

+description: Train a regression model that predicts car prices using the Azure Machine Learning designer.

+# Tutorial: Designer - train a no-code regression model

+Train a linear regression model that predicts car prices using the Azure Machine Learning designer. This tutorial is part one of a two-part series.

+This tutorial uses the Azure Machine Learning designer, for more information, see [What is Azure Machine Learning designer?](concept-designer.md)

+>[!Note]

+> Designer supports two types of components, classic prebuilt components (v1) and custom components (v2). These two types of components are NOT compatible.

+>

+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.

+>

+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.

+>

+>This article applies to classic prebuilt components and not compatible with CLI v2 and SDK v2.

+In part one of the tutorial, you learn how to:

+> [!div class="checklist"]

+> * Create a new pipeline.

+> * Import data.

+> * Prepare data.

+> * Train a machine learning model.

+> * Evaluate a machine learning model.

+In [part two](tutorial-designer-automobile-price-deploy.md) of the tutorial, you deploy your model as a real-time inferencing endpoint to predict the price of any car based on technical specifications you send it.

+> [!NOTE]

+>A completed version of this tutorial is available as a sample pipeline.

+>

+>To find it, go to the designer in your workspace. In the **New pipeline** section, select **Sample 1 - Regression: Automobile Price Prediction(Basic)**.

+## Create a new pipeline

+Azure Machine Learning pipelines organize multiple machine learning and data processing steps into a single resource. Pipelines let you organize, manage, and reuse complex machine learning workflows across projects and users.

+To create an Azure Machine Learning pipeline, you need an Azure Machine Learning workspace. In this section, you learn how to create both these resources.

+### Create a new workspace

+You need an Azure Machine Learning workspace to use the designer. The workspace is the top-level resource for Azure Machine Learning, it provides a centralized place to work with all the artifacts you create in Azure Machine Learning. For instruction on creating a workspace, see [Create workspace resources](../quickstart-create-resources.md).

+> [!NOTE]

+> If your workspace uses a Virtual network, there are additional configuration steps you must use to use the designer. For more information, see [Use Azure Machine Learning studio in an Azure virtual network](../how-to-enable-studio-virtual-network.md)

+### Create the pipeline

+>[!Note]

+> Designer supports two type of components, classic prebuilt components and custom components. These two types of components are not compatible.

+>

+>Classic prebuilt components provides prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.

+>

+>

+>Custom components allow you to provide your own code as a component. It supports sharing across workspaces and seamless authoring across Studio, CLI, and SDK interfaces.

+>

+>This article applies to classic prebuilt components.

+1. Sign in to <a href="https://ml.azure.com?tabs=jre" target="_blank">ml.azure.com</a>, and select the workspace you want to work with.

+1. Select **Designer** -> **Classic prebuilt**

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/launch-designer.png" alt-text="Screenshot of the visual workspace showing how to access the designer.":::

+1. Select **Create a new pipeline using classic prebuilt components**.

+1. Click the pencil icon beside the automatically generated pipeline draft name, rename it to *Automobile price prediction*. The name doesn't need to be unique.

+

+## Set the default compute target

+A pipeline jobs on a compute target, which is a compute resource that's attached to your workspace. After you create a compute target, you can reuse it for future jobs.

+> [!Important]

+> Attached compute is not supported, use [compute instances or clusters](../concept-compute-target.md#azure-machine-learning-compute-managed) instead.

+You can set a **Default compute target** for the entire pipeline, which will tell every component to use the same compute target by default. However, you can specify compute targets on a per-module basis.

+1. Select **Settings** to the right of the canvas to open the **Settings** pane.

+

+1. Select **Create Azure Machine Learning compute instance**.

+ * If you already have an available compute target, you can select it from the **Select Azure Machine Learning compute instance** drop-down to run this pipeline.

+ * Or, select "Serverless" to use [serverless compute (preview)](../how-to-use-serverless-compute.md).

+1. Enter a name for the compute resource.

+1. Select **Create**.

+ > [!NOTE]

+ > It takes approximately five minutes to create a compute resource. After the resource is created, you can reuse it and skip this wait time for future jobs.

+ >

+ > The compute resource autoscales to zero nodes when it's idle to save cost. When you use it again after a delay, you might experience approximately five minutes of wait time while it scales back up.

+## Import data

+There are several sample datasets included in the designer for you to experiment with. For this tutorial, use **Automobile price data (Raw)**.

+1. To the left of the pipeline canvas is a palette of datasets and components. Select **Component** -> **Sample data**.

+1. Select the dataset **Automobile price data (Raw)**, and drag it onto the canvas.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/automobile-data.png" alt-text="Gif of dragging the Automobile price data to the canvas.":::

+### Visualize the data

+You can visualize the data to understand the dataset that you'll use.

+1. Right-click the **Automobile price data (Raw)** and select **Preview Data**.

+1. Select the different columns in the data window to view information about each one.

+ Each row represents an automobile, and the variables associated with each automobile appear as columns. There are 205 rows and 26 columns in this dataset.

+## Prepare data

+Datasets typically require some preprocessing before analysis. You might have noticed some missing values when you inspected the dataset. These missing values must be cleaned so that the model can analyze the data correctly.

+### Remove a column

+When you train a model, you have to do something about the data that's missing. In this dataset, the **normalized-losses** column is missing many values, so you'll exclude that column from the model altogether.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Select Columns in Dataset** component.

+1. Drag the **Select Columns in Dataset** component onto the canvas. Drop the component below the dataset component.

+1. Connect the **Automobile price data (Raw)** dataset to the **Select Columns in Dataset** component. Drag from the dataset's output port, which is the small circle at the bottom of the dataset on the canvas, to the input port of **Select Columns in Dataset**, which is the small circle at the top of the component.

+ > [!TIP]

+ > You create a flow of data through your pipeline when you connect the output port of one component to an input port of another.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/connect-modules.gif" alt-text="Screenshot of connecting Automobile price data component to select columns in dataset component.":::

+1. Select the **Select Columns in Dataset** component.

+1. Click on the arrow icon under Settings to the right of the canvas to open the component details pane. Alternatively, you can double-click the **Select Columns in Dataset** component to open the details pane.

+1. Select **Edit column** to the right of the pane.

+1. Expand the **Column names** drop down next to **Include**, and select **All columns**.

+1. Select the **+** to add a new rule.

+1. From the drop-down menus, select **Exclude** and **Column names**.

+1. Enter *normalized-losses* in the text box.

+1. In the lower right, select **Save** to close the column selector.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/exclude-column.png" alt-text="Screenshot of select columns with exclude highlighted.":::

+1. In the **Select Columns in Dataset** component details pane, expand **Node info**.

+1. Select the **Comment** text box and enter *Exclude normalized losses*.

+ Comments will appear on the graph to help you organize your pipeline.

+### Clean missing data

+Your dataset still has missing values after you remove the **normalized-losses** column. You can remove the remaining missing data by using the **Clean Missing Data** component.

+> [!TIP]

+> Cleaning the missing values from input data is a prerequisite for using most of the components in the designer.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Clean Missing Data** component.

+1. Drag the **Clean Missing Data** component to the pipeline canvas. Connect it to the **Select Columns in Dataset** component.

+1. Select the **Clean Missing Data** component.

+1. Click on the arrow icon under Settings to the right of the canvas to open the component details pane. Alternatively, you can double-click the **Clean Missing Data** component to open the details pane.

+1. Select **Edit column** to the right of the pane.

+1. In the **Columns to be cleaned** window that appears, expand the drop-down menu next to **Include**. Select, **All columns**

+1. Select **Save**

+1. In the **Clean Missing Data** component details pane, under **Cleaning mode**, select **Remove entire row**.

+1. In the **Clean Missing Data** component details pane, expand **Node info**.

+1. Select the **Comment** text box and enter *Remove missing value rows*.

+ Your pipeline should now look something like this:

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/pipeline-clean.png" alt-text="Screenshot of automobile price data connected to select columns in dataset component, which is connected to clean missing data.":::

+## Train a machine learning model

+Now that you have the components in place to process the data, you can set up the training components.

+Because you want to predict price, which is a number, you can use a regression algorithm. For this example, you use a linear regression model.

+### Split the data

+Splitting data is a common task in machine learning. You'll split your data into two separate datasets. One dataset will train the model and the other will test how well the model performed.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Split Data** component.

+1. Drag the **Split Data** component to the pipeline canvas.

+1. Connect the left port of the **Clean Missing Data** component to the **Split Data** component.

+ > [!IMPORTANT]

+ > Make sure that the left output port of **Clean Missing Data** connects to **Split Data**. The left port contains the cleaned data. The right port contains the discarded data.

+1. Select the **Split Data** component.

+1. Click on the arrow icon under Settings to the right of the canvas to open the component details pane. Alternatively, you can double-click the **Split Data** component to open the details pane.

+1. In the **Split Data** details pane, set the **Fraction of rows in the first output dataset** to 0.7.

+ This option splits 70 percent of the data to train the model and 30 percent for testing it. The 70 percent dataset will be accessible through the left output port. The remaining data will be available through the right output port.

+1. In the **Split Data** details pane, expand **Node info**.

+1. Select the **Comment** text box and enter *Split the dataset into training set (0.7) and test set (0.3)*.

+### Train the model

+Train the model by giving it a dataset that includes the price. The algorithm constructs a model that explains the relationship between the features and the price as presented by the training data.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Linear Regression** component.

+1. Drag the **Linear Regression** component to the pipeline canvas.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Train Model** component.

+1. Drag the **Train Model** component to the pipeline canvas.

+1. Connect the output of the **Linear Regression** component to the left input of the **Train Model** component.

+1. Connect the training data output (left port) of the **Split Data** component to the right input of the **Train Model** component.

+ > [!IMPORTANT]

+ > Make sure that the left output port of **Split Data** connects to **Train Model**. The left port contains the training set. The right port contains the test set.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/pipeline-train-model.png" alt-text="Screenshot showing the Linear Regression connects to left port of Train Model and the Split Data connects to right port of Train Model.":::

+1. Select the **Train Model** component.

+1. Click on the arrow icon under Settings to the right of the canvas to open the component details pane. Alternatively, you can double-click the **Train Model** component to open the details pane.

+1. Select **Edit column** to the right of the pane.

+1. In the **Label column** window that appears, expand the drop-down menu and select **Column names**.

+1. In the text box, enter *price* to specify the value that your model is going to predict.

+ >[!IMPORTANT]

+ > Make sure you enter the column name exactly. Do not capitalize **price**.

+ Your pipeline should look like this:

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/pipeline-train-graph.png" alt-text="Screenshot showing the correct configuration of the pipeline after adding the Train Model component.":::

+### Add the Score Model component

+After you train your model by using 70 percent of the data, you can use it to score the other 30 percent to see how well your model functions.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Score Model** component.

+1. Drag the **Score Model** component to the pipeline canvas.

+1. Connect the output of the **Train Model** component to the left input port of **Score Model**. Connect the test data output (right port) of the **Split Data** component to the right input port of **Score Model**.

+### Add the Evaluate Model component

+Use the **Evaluate Model** component to evaluate how well your model scored the test dataset.

+1. In the datasets and component palette to the left of the canvas, click **Component** and search for the **Evaluate Model** component.

+1. Drag the **Evaluate Model** component to the pipeline canvas.

+1. Connect the output of the **Score Model** component to the left input of **Evaluate Model**.

+ The final pipeline should look something like this:

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/pipeline-final-graph.png" alt-text="Screenshot showing the correct configuration of the pipeline.":::

+## Submit the pipeline

+Now that your pipeline is all setup, you can submit a pipeline job to train your machine learning model. You can submit a valid pipeline job at any point, which can be used to review changes to your pipeline during development.

+1. At the top of the canvas, select **Submit**.

+1. In the **Set up pipeline job** dialog box, select **Create new**.

+ > [!NOTE]

+ > Experiments group similar pipeline jobs together. If you run a pipeline multiple times, you can select the same experiment for successive jobs.

+ 1. For **New experiment Name**, enter **Tutorial-CarPrices**.

+ 1. Select **Submit**.

+ 1. You'll see a submission list in the left pane of the canvas, and a notification will pop up at the top right corner of the page. You can select the **Job detail** link to go to job detail page for debugging.

+ :::image type="content" source="./media/how-to-run-batch-predictions-designer/submission-list.png" alt-text="Screenshot of the submitted jobs list with a success notification.":::

+ If this is the first job, it may take up to 20 minutes for your pipeline to finish running. The default compute settings have a minimum node size of 0, which means that the designer must allocate resources after being idle. Repeated pipeline jobs will take less time since the compute resources are already allocated. Additionally, the designer uses cached results for each component to further improve efficiency.

+### View scored labels

+In the job detail page, you can check the pipeline job status, results and logs.

+After the job completes, you can view the results of the pipeline job. First, look at the predictions generated by the regression model.

+1. Right-click the **Score Model** component, and select **Preview data** > **Scored dataset** to view its output.

+ Here you can see the predicted prices and the actual prices from the testing data.

+ :::image type="content" source="./media/tutorial-designer-automobile-price-train-score/score-result.png" alt-text="Screenshot of the output visualization highlighting the Scored Label column.":::

+### Evaluate models

+Use the **Evaluate Model** to see how well the trained model performed on the test dataset.

+1. Right-click the **Evaluate Model** component and select **Preview data** > **Evaluation results** to view its output.

+The following statistics are shown for your model:

+* **Mean Absolute Error (MAE)**: The average of absolute errors. An error is the difference between the predicted value and the actual value.

+* **Root Mean Squared Error (RMSE)**: The square root of the average of squared errors of predictions made on the test dataset.

+* **Relative Absolute Error**: The average of absolute errors relative to the absolute difference between actual values and the average of all actual values.

+* **Relative Squared Error**: The average of squared errors relative to the squared difference between the actual values and the average of all actual values.

+* **Coefficient of Determination**: Also known as the R squared value, this statistical metric indicates how well a model fits the data.

+For each of the error statistics, smaller is better. A smaller value indicates that the predictions are closer to the actual values. For the coefficient of determination, the closer its value is to one (1.0), the better the predictions.

+## Clean up resources

+Skip this section if you want to continue on with part 2 of the tutorial, [deploying models](tutorial-designer-automobile-price-deploy.md).

+## Next steps

+In part two, you'll learn how to deploy your model as a real-time endpoint.

+> [!div class="nextstepaction"]

+> [Continue to deploying models](tutorial-designer-automobile-price-deploy.md)

+| Version | What's New | Azure support start date | Azure support end date | Community Retirement date

+| | | | | |

+| [MySQL 5.7](https://dev.mysql.com/doc/relnotes/mysql/5.7/en/) | [Features](https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-31.html) | March 20, 2018 |September 2025 |October 2023|

+| [MySQL 8](https://mysqlserverteam.com/whats-new-in-mysql-8-0-generally-available/) | [Features](https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-21.html)) | December 11, 2019 | NA |April 2026|

+## What will happen to Azure Database for MySQL service after MySQL community version is retired in October 2023?

+In line with Oracle's announcement regarding the end-of-life (EOL) of [MySQL Community Version v5.7 in __October 2023__](https://www.oracle.com/us/support/library/lsp-tech-chart-069290.pdf) (Page 23), we at Azure are actively preparing for this important transition. This development specifically impacts customers who are currently utilizing Version 5.7 of Azure Database for MySQL - Single Server and Flexible Server.

+In response to the customer's requests, Microsoft has decided to prolong the support for Azure Database for MySQL beyond __October 2023__. During the extended support period, which will last until __September 2025__, Microsoft prioritizes the availability, reliability, and security of the service. While there are no specific guarantees regarding minor version upgrades, we implement essential modifications to ensure that the service remains accessible, dependable, and protected. Our plan includes:

+- Extended support for v5.7 on Azure Database for MySQL- Flexible Servers until __September 2025__, offering ample time for customers to plan and execute their upgrades to MySQL v8.0.

+- Extended support for v5.7 on Azure Database for MySQL- Single Servers until they're retired on __September 2024__. This extended support provides Azure Database for MySQL -Single Server customers ample time to migrate to Azure Database for MySQL - Flexible Server version 5.7 and then later upgrade to 8.0.

+Before we end our support of Azure Database for MySQL 5.7, there are several important timelines that you should pay attention.

+__Azure MySQL 5.7 Deprecation Timelines__

+|Timelines| Azure MySQL 5.7 Flexible end at |Azure MySQL 5.7 Single end at|

+||||

+|Creation of new servers using the Azure portal.| Jan 2024| Already ended as part of [Single Server deprecation](single-server/whats-happening-to-mysql-single-server.md)|

+|Creation of new servers using the Command Line Interface (CLI). | April 2024| September 2024|

+|Creation of replica servers for existing servers. | September 2025| September 2024|

+|Creation of servers using restore workflow for the existing servers| September 2025|September 2024|

+|Creation of new servers for migrating from Azure Database for MySQL - Single Server to Azure Database for MySQL - Flexible Server.| NA| September 2024|

+|Creation of new servers for migrating from Azure Database for MariaDB to Azure Database for MySQL - Flexible Server.| September 2025| NA|

+|Extended support for Azure Database for MySQL v5.7| September 2025| September 2024|

+To summarize, creating Azure Database for MySQL v5.7 - Flexible Server will conclude in __April 2024__. However, it's important to note that certain scenarios such as replica creation, point in time recovery, and migration from Azure Database for MySQL - Single Server or Azure Database for MariaDB to Azure Database for MySQL - Flexible Server, will allow you to create MySQL version 5.7 until the end of the extended support period.

+### FAQs

+__Q: What is the process for upgrading Azure database for MySQL - Flexible server from version v5.7 to v8.0?__

+A: Starting May 2023, Azure Database for MySQL - Flexible Server enables you to carry out an in-place upgrade from MySQL v5.7 to v8.0 utilizing the Major Version Upgrade (MVU) feature. For more detailed information, please consult the [Major version upgrade](flexible-server/how-to-upgrade.md) document.

+__Q: I'm currently using Azure database for MySQL - Single Sever version 5.7, how should I plan my upgrade?__

+A: Azure Database for MySQL - Single Server does not offer built-in support for major version upgrade from v5.7 to v8.0. As Azure Database for MySQL - Single Server is on deprecation path, there are no investments planned to support major version upgrade from v5.7 to v8.0. The recommended path to upgrade from v5.7 of Azure Database for MySQL - Single Server to v8.0 is to first [migrate your v5.7 Azure Database for MySQL - Single Server to v5.7 of Azure Database for MySQL - Flexible Server](single-server/whats-happening-to-mysql-single-server.md#migrate-from-single-server-to-flexible-server). Once the migration is completed and server is stabilized on Flexible Server, you can proceed with performing a [major version upgrade](flexible-server/how-to-upgrade.md) on the migrated Azure Database for MySQL - Flexible Server from v5.7 to v8.0. The extended support for v5.7 on Flexible Server will allow you to run on v5.7 longer and plan your upgrade to v8.0 on Flexible Server at a later point in time after migration from Single Server.

+__Q: Are there any expected downtime or performance impacts during the upgrade process?__

+A: Yes, it's expected that there will be some downtime during the upgrade process. The specific duration varies depending on factors such as the size and complexity of the database. We advise conducting a test upgrade on a nonproduction environment to assess the expected downtime and evaluate the potential performance impact. If you wish to minimize downtime for your applications during the upgrade, you can explore the option of [perform minimal downtime major version upgrade from MySQL 5.7 to MySQL 8.0 using read replica](flexible-server/how-to-upgrade.md#perform-minimal-downtime-major-version-upgrade-from-mysql-57-to-mysql-80-using-read-replicas).

+__Q: Can I roll back to MySQL v5.7 after upgrading to v8.0?__

+A: While it's generally not recommended to downgrade from MySQL v8.0 to v5.7, as the latter is nearing its End of Life status, we acknowledge that there may be specific scenarios where this flexibility becomes necessary. To ensure a smooth upgrade process and alleviate any potential concerns, it's strongly advised adhering to best practices by performing a comprehensive [on-demand backup](flexible-server/how-to-trigger-on-demand-backup.md) before proceeding with the upgrade to MySQL v8.0. This backup serves as a precautionary measure, allowing you to [restore your database](flexible-server/how-to-restore-server-portal.md) to its previous version on to another new Azure Database for MySQL -Flexible server in the event of any unexpected issues or complications with MySQL v8.0.

+__Q: What are the main advantages of upgrading to MySQL v8.0?__

+A: MySQL v8.0 comes with a host of improvements, including more efficient data dictionary, enhanced security, and other features like common table expressions and window functions. Details please refer to [MySQL 8.0 release notes](https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-32.html)

+__Q: Are there any compatibility issues to be aware of when upgrading to MySQL v8.0?__

+A: Some compatibility issues may arise due to changes in MySQL v8.0. It's important to test your applications with MySQL v8.0 before upgrading the production database. Check [MySQL's official documentation](https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html) for a detailed list of compatibility issues.

+__Q: What support is available if I encounter issues during the upgrade process?__

+A: If you have questions, get answers from community experts in [Microsoft Q&A](https://aka.ms/microsoft-azure-mysql-qa). If you have a support plan and you need technical help, create a [support request](https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest). You can also reach out to the Azure Database for MySQL product team at mailto:AskAzureDBforMySQL@service.microsoft.com.

+__Q: What will happen to my data during the upgrade?__

+A: While your data will remain unaffected during the upgrade process, it's highly advisable to create a backup of your data before proceeding with the upgrade. This precautionary measure will help mitigate the risk of potential data loss in the event of unforeseen complications.

+__Q: What will happen to the server 5.7 post Sep 2025__

+A: You refer to our [retired MySQL version support policy](concepts-version-policy.md#retired-mysql-engine-versions-not-supported-in-azure-database-for-mysql) to learn what will happen after Azure Database for MySQL 5.7 end of support

+__Q: I have a Azure Database for MariaDB or Azure database for MySQL -Single server, how can I create the server in 5.7 post April 2024 for migrating to Azure database for MySQL - flexible server?__

+A: If there's MariaDB\Single server in your subscription, this subscription is still permitted to create Azure Database for MySQL ΓÇô Flexible Server v5.7 for the purpose of migration to Azure Database for MySQL ΓÇô Flexible Server

+After you create a server, the compute tier, compute size, and storage size can be changed. Compute scaling requires a restart and takes between 60-120 seconds, while storage scaling doesn't require restart. You also can independently adjust the backup retention period up or down. For more information, see the [Scale resources](#scale-resources) section.

+| Compute size | vCores | Memory Size (GiB) | Max Supported IOPS | Max Connections | Temp Storage (SSD) GiB |

+|-|--|-| || |

+|Standard_B1s | 1 | 1 | 320 | 171 | 4 |

+|Standard_B1ms | 1 | 2 | 640 | 341 | 4 |

+|Standard_B2s | 2 | 4 | 1280 | 683 | 4 |

+|Standard_B2ms | 2 | 8 | 1700 | 1365 | 16 |

+|Standard_B4ms | 4 | 16 | 2400 | 2731 | 32 |

+|Standard_B8ms | 8 | 32 | 3100 | 5461 | 64 |

+|Standard_B12ms | 12 | 48 | 3800 | 8193 | 96 |

+|Standard_B16ms | 16 | 64 | 4300 | 10923 | 128 |

+|Standard_B20ms | 20 | 80 | 5000 | 13653 | 160 |

+|Standard_D2ads_v5 |2 |8 |3200 |1365 | 75 |

+|Standard_D2ds_v4 |2 |8 |3200 |1365 | 75 |

+|Standard_D4ads_v5 |4 |16 |6400 |2731 | 150 |

+|Standard_D4ds_v4 |4 |16 |6400 |2731 | 150 |

+|Standard_D8ads_v5 |8 |32 |12800 |5461 | 300 |

+|Standard_D8ds_v4 |8 |32 |12800 |5461 | 300 |

+|Standard_D16ads_v5 |16 |64 |20000 |10923 | 600 |

+|Standard_D16ds_v4 |16 |64 |20000 |10923 | 600 |

+|Standard_D32ads_v5 |32 |128 |20000 |21845 | 1200 |

+|Standard_D32ds_v4 |32 |128 |20000 |21845 | 1200 |

+|Standard_D48ads_v5 |48 |192 |20000 |32768 | 1800 |

+|Standard_D48ds_v4 |48 |192 |20000 |32768 | 1800 |

+|Standard_D64ads_v5 |64 |256 |20000 |43691 | 2400 |

+|Standard_D64ds_v4 |64 |256 |20000 |43691 | 2400 |

+|Standard_E2ds_v4 | 2 | 16 | 5000 | 2731 | 75 |

+|Standard_E2ads_v5 | 2 | 16 | 5000 | 2731 | 75 |

+|Standard_E4ds_v4 | 4 | 32 | 10000 | 5461 | 150 |

+|Standard_E4ads_v5 | 4 | 32 | 10000 | 5461 | 150 |

+|Standard_E8ds_v4 | 8 | 64 | 18000 | 10923 | 300 |

+|Standard_E8ads_v5 | 8 | 64 | 18000 | 10923 | 300 |

+|Standard_E16ds_v4 | 16 | 128 | 28000 | 21845 | 600 |

+|Standard_E16ads_v5 | 16 | 128 | 28000 | 21845 | 600 |

+|Standard_E32ds_v4 | 32 | 256 | 38000 | 43691 | 1200 |

+|Standard_E32ads_v5 | 32 | 256 | 38000 | 43691 | 1200 |

+|Standard_E48ds_v4 | 48 | 384 | 48000 | 65536 | 1800 |

+|Standard_E48ads_v5 | 48 | 384 | 48000 | 65536 | 1800 |

+|Standard_E64ds_v4 | 64 | 504 | 48000 | 86016 | 2400 |

+|Standard_E64ads_v5 | 64 | 504 | 48000 | 86016 | 2400 |

+|Standard_E80ids_v4 | 80 | 504 | 48000 | 86016 | 2400 |

+|Standard_E2ds_v5 | 2 | 16 | 5000 | 2731 | 75 |

+|Standard_E4ds_v5 | 4 | 32 | 10000 | 5461 | 150 |

+|Standard_E8ds_v5 | 8 | 64 | 18000 | 10923 | 300 |

+|Standard_E16ds_v5 | 16 | 128 | 28000 | 21845 | 600 |

+|Standard_E32ds_v5 | 32 | 256 | 38000 | 43691 | 1200 |

+|Standard_E48ds_v5 | 48 | 384 | 48000 | 65536 | 1800 |

+|Standard_E64ds_v5 | 64 | 512 | 48000 | 87383 | 2400 |

+|Standard_E96ds_v5 | 96 | 672 | 48000 | 100000 | 3600 |

+The minimum IOPS are 360 across all compute sizes and the maximum IOPS is determined by the selected compute size. To learn more about the maximum IOPS per compute size refer to the [table](#service-tiers-size-and-server-types).

+The maximum IOPS are dependent on the maximum available IOPS per compute size. Refer to the column *Max uncached disk throughput: IOPS/MBps* in the [B-series](../../virtual-machines/sizes-b-series-burstable.md), [Ddsv4-series](../../virtual-machines/ddv4-ddsv4-series.md), and [Edsv4-series](../../virtual-machines/edv4-edsv4-series.md)/ [Edsv5-series](../../virtual-machines/edv5-edsv5-series.md)] documentation.

+> **Minimum IOPS are 360 across all compute sizes<br>

+> **Maximum IOPS are determined by the selected compute size.

+The cornerstone of the Azure Database for MySQL - Flexible Server is its ability to achieve the best performance for tier 1 workloads, which can be improved by enabling server automatically scale performance (IO) of its database servers seamlessly depending on the workload needs. This is an opt-in feature that enables users to scale IOPS on demand without having to pre-provision a certain amount of IO per second. With the Autoscale IOPS featured enable, you can now enjoy worry free IO management in Azure Database for MySQL - Flexible Server because the server scales IOPs up or down automatically depending on workload needs.ΓÇ»

+Scaling storage and changing the backup retention period are online operations and don't require a server restart.

+- Reduce the backup retention period if a longer retention of backup isn't required.

+The following table shows the **entity types** currently available for mapping in Microsoft Sentinel, and the **attributes** available as **identifiers** for each entity type. These attributes appear in the **Identifiers** drop-down list in the [entity mapping](map-data-fields-to-entities.md) section of the [analytics rule wizard](detect-threats-custom.md).

+Each one of the identifiers in the **required identifiers** column is necessary to identify its entity. However, a required identifier might not, by itself, be sufficient to provide *unique* identification. The more identifiers used, the greater the likelihood of unique identification. You can use up to three identifiers for a single entity mapping.

+For best results&mdash;for guaranteed unique identification&mdash;you should use identifiers from the **strongest identifiers** column whenever possible. The use of multiple strong identifiers enables correlation between strong identifiers from varying data sources and schemas. This correlation in turn allows Microsoft Sentinel to provide more comprehensive insights for a given entity.

+The following section contains a more in-depth look at the full schemas of each entity type. You'll notice that many of these schemas include links to other entity types&mdash;for example, the User account schema includes a link to the Host entity type, since one attribute of a user account is the host it's defined on. These externally linked entities can't be used as identifiers for the purpose of entity mapping, but they are very useful in giving a complete picture of entities on entity pages and the investigation graph.

+> [!NOTE]

+> If the **Account** entity is defined using the **Name** identifier, and the Name value of a particular entity is one of the following generic, commonly built-in account names, then that entity will be dropped from its alert.

+> - ADMIN

+> - ADMINISTRATOR

+> - SYSTEM

+> - ROOT

+> - ANONYMOUS

+> - AUTHENTICATED USER

+> - NETWORK

+> - NULL

+> - LOCAL SYSTEM

+> - LOCALSYSTEM

+> - NETWORK SERVICE

+| Files | List\<Entity> | List of linked file entities on which the malware was found. Can contain the File entities inline or as reference.<br>See the [File](#file) entity for more details on structure. |

+| Processes | List\<Entity> | List of linked process entities on which the malware was found. This would often be used when the alert triggered on fileless activity.<br>See the [Process](#process) entity for more details on structure. |

+| ImageFile | Entity (File) | Can contain the File entity inline or as reference.<br>See the [File](#file) entity for more details on structure. |

+| Account | Entity | The account running the processes.<br>Can contain the Account entity inline or as reference.<br>See the [Account](#user-account) entity for more details on structure. |

+ > Reports in the Apache Parquet format present dates in the following format: `timestamp_millis [number of milliseconds since 1970-01-01 00:00:00 UTC`].

+ Title: Deploy Azure Stream Analytics jobs using Bicep files

+description: This article shows you how to generate and deploy Azure Stream Analytics jobs using Bicep files.

+# Deploy Stream Analytics jobs with Bicep files

+The Azure Stream Analytics (ASA) CI/CD npm package allows you to build, test, and deploy your Stream Analytics projects in your CI/CD pipeline using Bicep files. This article walks you through how to generate bicep files and deploy your ASA jobs.

+Follow this [guide](quick-create-visual-studio-code.md) to create a new Stream Analytics project using Visual Studio Code or export an existing one from the Azure portal.

+## Installation

+```powershell

+npm install -g azure-streamanalytics-cicd

+```

+## Build project with Bicep

+```powershell

+# Generate Bicep template

+azure-streamanalytics-cicd build --v2 --project <projectFullPath> [--outputPath <outputPath>] --type Bicep

+# Example:

+azure-streamanalytics-cicd build --v2 --project ./asaproj.json --outputPath ./Deploy --type Bicep

+```

+The **build** command does a keyword syntax check and generates Bicep files. If the project is built successfully, you see two files created under the output folder:

+* **Bicep template file**: [ProjectName].JobTemplate.bicep

+* **Bicep parameter file**: [ProjectName].JobTemplate.bicep.parameters.json

+The default values for **parameters.json** file come from your project settings. If you want to deploy to another environment, replace the values accordingly.

+The default values for all credentials are null. You're required to set the values before you deploy to Azure.

+## Deploy to Azure

+```powershell

+az login

+az account set --subscription "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

+az deployment group create --name demodeployment --resource-group {resource-group-name} --template-file .\DeployV2\{project-name}.JobTemplate.bicep --parameters .\DeployV2\{project-name}.JobTemplate.bicep.parameters.json

+```

+## Next steps

+* [Azure Stream Analytics CI/CD tool](cicd-overview.md)

+* [Automate builds, tests, and deployments of an Azure Stream Analytics job](cicd-tools.md)

+* [Set up CI/CD pipeline for Stream Analytics job using Azure Pipelines](set-up-cicd-pipeline.md)

+ Title: Continuous integration and continuous deployment of Azure Stream Analytics jobs

+description: This article gives an overview of setting up a continuous integration and deployment (CI/CD) pipeline for Azure Stream Analytics jobs.

+# Continuous integration and continuous deployment (CI/CD) of Stream Analytics jobs

+You can build, test and deploy your Azure Stream Analytics (ASA) job using a source control integration. Source control integration creates a workflow in which updating code would trigger a resource deployment to Azure. This article outlines the basic steps for creating a continuous integration and continuous delivery (CI/CD) pipeline.

+1. Create a Stream Analytics project using VS Code. You can either create a new project or export an existing job to your local machine using the ASA Tools extension for Visual Studio Code.

+ * [Quickstart: Create a Stream Analytics job using VS Code](./quick-create-visual-studio-code.md)

+ * [Export an existing job](visual-studio-code-explore-jobs.md)

+ Title: Export and manage Azure Stream Analytics jobs in Visual Studio Code

+description: This article shows you how to export and manage an Azure Stream Analytics job with the Azure Stream Analytics extension for Visual Studio Code.

+# Export and manage Stream Analytics jobs in Visual Studio Code

+The Azure Stream Analytics (ASA) extension for Visual Studio Code gives you a seamless experience for managing Stream Analytics jobs in the cloud. It can be used across Windows, Mac and Linux platforms. With the ASA extension, it allows you to:

+- Create, start, and stop a cloud job in Azure

+- Export existing jobs to a local machine

+- View and debug job using **Job Diagram** and **Monitor**

+## Export job to local machine

+1. Open your VS Code and select **Azure** icon on the activity bar. If you haven't installed the ASA extension, follow [this guide](./quick-create-visual-studio-code.md) to install.

+2. Select **STREAM ANALYTICS** in the explorer to locate the job you want to export.

+

+1. Select the **Azure** icon on Visual Studio Code Activity Bar and then expand **Stream Analytics node**. Your jobs should appear under your subscriptions.

+ 

+3. Right-click your job node, and choose the **Open Job View in Portal** node to open the job view in the Azure portal.

+ 

+ 

+2. You can view your job diagram and select **Job Summary** to view properties and information of your job.

+ 

+3. You can select **Test Connection** button to test connection to your input and output.

+ 

+4. You can also select **Locate Script** button to view your query.

+ 

+1. Select the arrow button, you can open the Metrics panel.

+ 

+2. You can interact with it and analyze your job with key metrics showing in chart. You can choose to view job-level metrics or nodes level metrics. And you can also decide which metrics you want them to show in the chart.

+ 

+ 

+ 

+* [Create an Azure Stream Analytics cloud job using Visual Studio Code extension](quick-create-visual-studio-code.md)

+ Title: Testing Azure Stream Analytics query locally against live stream input using Visual Studio Code extension.

+description: This article describes how to test query locally against live stream input using the Azure Stream Analytics Tools extension for Visual Studio Code.

+# Testing Stream Analytics query locally against live stream input with Visual Studio Code

+Testing your Stream Analytics query locally against live stream input is a crucial step in ensuring the accuracy and efficiency of your query logic. With the Azure Stream Analytics (ASA) extension for Visual Studio Code (VS Code), you have the necessary tools to perform this testing seamlessly. This article guides you through the steps of setting up your environment, configuring the live stream input, and executing your query locally for thorough testing.

+* Follow [this quickstart](quick-create-visual-studio-code.md) to install and create a Stream Analytics job using Visual Studio Code.

+1. Right-click the **Inputs** folder in your Stream Analytics project and select **ASA: Add Input**.

+ 

+ OR press **Ctrl+Shift+P** to open the command palette and enter **ASA: Add Input**. Then choose the query script for processing the input stream.

+ 

+2. Choose a live input source type from the drop-down list, **Select from your Azure Subscriptions** and enter a name for the input source.

+ 

+3. You see a new JSON file created under the **Input** folder. Open the file and **Select from your subscriptions** to help you configure your Event Hubs input.

+ 

+4. Select **Preview data** to validate if the live input is successfully configured.

+ 

+## Run query locally

+Open your query file, select **Run locally** or the **Play** button on the menu bar to open a **Local run option** window for configuring local run mode.

+

+Choose **Use Live Input and Local Output** to run job locally against live input stream and produce output results to a local folder. Then select **Run** to start running job locally.

+

+You can see a **Local test** window is opened and refreshed every 3 seconds. The output results are stored as JSON format in **LocalRunOutputs** folder.

+

+## Specify input partitions for local testing

+To avoid ingesting all data from Event Hubs, you can specify specific partitions for your Event Hubs during local testing.

+1. Select **Run locally** to open the local run option window.

+2. Under **Select the input partitions** section and choose **Custom**.

+3. Enter the specific partition numbers that you want to ingest data from for your local testing.

+

+By specifying the partitions, you can focus on testing your query logic without worrying about a large volume of data. This feature allows for more efficient and targeted testing of your Stream Analytics job.

+## Submit job to Azure

+Once you have completed testing your query locally, you can submit your Stream Analytics project to Azure and start running the job in the cloud.

+1. Select the **Submit to Azure** option or select the **Upload** button on the query file.

+2. In the submit options window, you can **update an existing job in Azure** or **create a new job** for the upload.

+3. Provide the necessary details, including your subscription, resource group, and region for the deployment.

+

+Once the submission process is complete, you can view your cloud job in a new window. From there, you can monitor its performance and manage its configurations as needed. You can also open the job in your browser by clicking on the link provided in the OUTPUT panel.

+

+* [Export and manage Stream Analytics jobs using Visual Studio Code extension](visual-studio-code-explore-jobs.md)

+* [Overview of running Stream Analytics jobs locally](visual-studio-code-local-run-all.md)

+* [Test Stream Analytics queries locally with sample data](visual-studio-code-local-run.md)

+* [Set up CI/CD pipelines using the ASA npm package](./cicd-overview.md)

+* Requires [manual installation](../../automation/extension-based-hybrid-runbook-worker-install.md).