-Azure Active Directory B2C (Azure AD B2C) allows you to enable CAPTCHA prevent to automated attacks on your consumer-facing applications. Azure AD B2CΓÇÖs CAPTCHA supports both audio and visual CAPTCHA challenges. You can enable this security feature in both sign-up and sign-in flows for your local accounts. CAPTCHA isn't applicable for social identity providers' sign-in.

-> [!NOTE]

-> - You can't add CAPTCHA to an MFA step in a sign-up only user flow.

-> - In an MFA flow, CAPTCHA is applicable where the MFA method you select is SMS or phone call, SMS only or Phone call only.

-1. In the `ContosoCustomPolicy.XML` file, locate the `AAD-UserUpdate` technical profile and then add a new technical profile by using the following code:

- <!--<Item Key="Operation">Write</Item>-->

-#Customer intent: As a developer building a single-page application (SPA) with a JavaScript framework, I want to implement OAuth 2.0 implicit flow for sign-in using Azure Active Directory B2C, so that I can securely authenticate users without server-to-server exchange and handle user flows like sign-up and profile management.

- - Use API connectors to [integrate with an anti-bot solution like reCAPTCHA](https://github.com/Azure-Samples/active-directory-b2c-node-sign-up-user-flow-captcha) (applies to sign-up flows).

-The Immersive Reader SDK [launchAsync](./reference.md#launchasync) `options` parameter contains the `-onPreferencesChanged` callback. This function is called anytime the user changes their preferences. The `value` parameter contains a string, which represents the user's current preferences. This string is then stored, for that user, by the host application.

-description: The Immersive Reader SDK contains a JavaScript library that allows you to integrate the Immersive Reader into your application.

-# Immersive Reader JavaScript SDK Reference (v1.4)

-You may use `npm`, `yarn`, or an `HTML` `<script>` element to include the library of the latest stable build in your web application:

-The SDK exposes the functions:

-<br>

-## launchAsync

-Launches the Immersive Reader within an `HTML` `iframe` element in your web application. The size of your content is limited to a maximum of 50 MB.

-#### launchAsync Parameters

-| Name | Type | Description |

-| `token` | string | The Microsoft Entra authentication token. For more information, see [How-To Create an Immersive Reader Resource](./how-to-create-immersive-reader.md). |

-| `subdomain` | string | The custom subdomain of your Immersive Reader resource in Azure. For more information, see [How-To Create an Immersive Reader Resource](./how-to-create-immersive-reader.md). |

-| `content` | [Content](#content) | An object containing the content to be shown in the Immersive Reader. |

-| `options` | [Options](#options) | Options for configuring certain behaviors of the Immersive Reader. Optional. |

-Returns a `Promise<LaunchResponse>`, which resolves when the Immersive Reader is loaded. The `Promise` resolves to a [`LaunchResponse`](#launchresponse) object.

-The returned `Promise` will be rejected with an [`Error`](#error) object if the Immersive Reader fails to load. For more information, see the [error codes](#error-codes).

-<br>

-## close

-Closes the Immersive Reader.

-An example use case for this function is if the exit button is hidden by setting ```hideExitButton: true``` in [options](#options). Then, a different button (for example a mobile header's back arrow) can call this ```close``` function when it's clicked.

-<br>

-## Immersive Reader Launch Button

-The SDK provides default styling for the button for launching the Immersive Reader. Use the `immersive-reader-button` class attribute to enable this styling. For more information, see [How-To Customize the Immersive Reader button](./how-to-customize-launch-button.md).

-```html

-<div class='immersive-reader-button'></div>

-```

-#### Optional attributes

-Use the following attributes to configure the look and feel of the button.

-| Attribute | Description |

-| | -- |

-| `data-button-style` | Sets the style of the button. Can be `icon`, `text`, or `iconAndText`. Defaults to `icon`. |

-| `data-locale` | Sets the locale. For example, `en-US` or `fr-FR`. Defaults to English `en`. |

-| `data-icon-px-size` | Sets the size of the icon in pixels. Defaults to 20px. |

-<br>

-## renderButtons

-The ```renderButtons``` function isn't necessary if you're using the [How-To Customize the Immersive Reader button](./how-to-customize-launch-button.md) guidance.

-This function styles and updates the document's Immersive Reader button elements. If ```options.elements``` is provided, then the buttons will be rendered within each element provided in ```options.elements```. Using the ```options.elements``` parameter is useful when you have multiple sections in your document on which to launch the Immersive Reader, and want a simplified way to render multiple buttons with the same styling, or want to render the buttons with a simple and consistent design pattern. To use this function with the [renderButtons options](#renderbuttons-options) parameter, call ```ImmersiveReader.renderButtons(options: RenderButtonsOptions);``` on page load as demonstrated in the below code snippet. Otherwise, the buttons will be rendered within the document's elements that have the class ```immersive-reader-button``` as shown in [How-To Customize the Immersive Reader button](./how-to-customize-launch-button.md).

-See the above [Optional Attributes](#optional-attributes) for more rendering options. To use these options, add any of the option attributes to each ```HTMLDivElement``` in your page HTML.

-#### renderButtons Parameters

-| Name | Type | Description |

-| `options` | [renderButtons options](#renderbuttons-options) | Options for configuring certain behaviors of the renderButtons function. Optional. |

-### renderButtons Options

-#### renderButtons Options Parameters

-| Setting | Type | Description |

-##### `elements`

-<br>

-Contains the response from the call to `ImmersiveReader.launchAsync`. A reference to the `HTML` `iframe` element that contains the Immersive Reader can be accessed via `container.firstChild`.

-#### LaunchResponse Parameters

-| Setting | Type | Description |

-

-#### Error Parameters

-| Setting | Type | Description |

-| code | String | One of a set of error codes. For more information, see [Error codes](#error-codes). |

-#### Error codes

-| Code | Description |

-| BadArgument | Supplied argument is invalid, see `message` parameter of the [Error](#error). |

-<br>

-#### Content Parameters

-| Name | Type | Description |

-<br>

-A single chunk of data, which will be passed into the Content of the Immersive Reader.

-#### Chunk Parameters

-| Name | Type | Description |

-| lang | String | Language of the text, the value is in IETF BCP 47-language tag format, for example, en, es-ES. Language will be detected automatically if not specified. For more information, see [Supported Languages](#supported-languages). |

-| MIME Type | Description |

-| text/html | HTML content. [Learn more](#html-support)|

-| application/mathml+xml | Mathematical Markup Language (MathML). [Learn more](./how-to/display-math.md).

-| application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Word .docx format document.

-<br>

-#### Options Parameters

-| Name | Type | Description |

-| uiLang | String | Language of the UI, the value is in IETF BCP 47-language tag format, for example, en, es-ES. Defaults to browser language if not specified. |

-| timeout | Number | Duration (in milliseconds) before [launchAsync](#launchasync) fails with a timeout error (default is 15,000 ms). This timeout only applies to the initial launch of the Reader page, when the Reader page opens successfully and the spinner starts. Adjustment of the timeout should'nt be necessary. |

-| uiZIndex | Number | Z-index of the `HTML` `iframe` element that will be created (default is 1000). |

-| useWebview | Boolean| Use a webview tag instead of an `HTML` `iframe` element, for compatibility with Chrome Apps (default is false). |

-| parent | Node | Node in which the `HTML` `iframe` element or `Webview` container is placed. If the element doesn't exist, iframe is placed in `body`. |

-| hideExitButton | Boolean | Hides the Immersive Reader's exit button arrow (default is false). This value should only be true if there's an alternative mechanism provided to exit the Immersive Reader (e.g a mobile toolbar's back arrow). |

-| cookiePolicy | [CookiePolicy](#cookiepolicy-options) | Setting for the Immersive Reader's cookie usage (default is *CookiePolicy.Disable*). It's the responsibility of the host application to obtain any necessary user consent following EU Cookie Compliance Policy. For more information, see [Cookie Policy Options](#cookiepolicy-options). |

-| preferences | String | String returned from onPreferencesChanged representing the user's preferences in the Immersive Reader. For more information, see [Settings Parameters](#settings-parameters) and [How-To Store User Preferences](./how-to-store-user-preferences.md). |

-| onPreferencesChanged | Function | Executes when the user's preferences have changed. For more information, see [How-To Store User Preferences](./how-to-store-user-preferences.md). |

-| disableGrammar | Boolean | Disable the Grammar experience. This option will also disable Syllables, Parts of Speech and Picture Dictionary, which depends on Parts of Speech. |

-| disableLanguageDetection | Boolean | Disable Language Detection to ensure the Immersive Reader only uses the language that is explicitly specified on the [Content](#content)/[Chunk[]](#chunk). This option should be used sparingly, primarily in situations where language detection isn't working, for instance, this issue is more likely to happen with short passages of fewer than 100 characters. You should be certain about the language you're sending, as text-to-speech won't have the correct voice. Syllables, Parts of Speech and Picture Dictionary won't work correctly if the language isn't correct. |

-> [!CAUTION]

-> **IMPORTANT** Do not attempt to programmatically change the values of the `-preferences` string sent to and from the Immersive Reader application as this may cause unexpected behavior resulting in a degraded user experience for your customers. Host applications should never assign a custom value to or manipulate the `-preferences` string. When using the `-preferences` string option, use only the exact value that was returned from the `-onPreferencesChanged` callback option.

-<br>

-#### ReadAloudOptions Parameters

-| Name | Type | Description |

-| voice | String | Voice, either "Female" or "Male". Not all languages support both genders. |

-| speed | Number | Playback speed, must be between 0.5 and 2.5, inclusive. |

-> [!NOTE]

-> Due to browser limitations, autoplay is not supported in Safari.

-<br>

-#### TranslationOptions Parameters

-| Name | Type | Description |

-| language | String | Sets the translation language, the value is in IETF BCP 47-language tag format, for example, fr-FR, es-MX, zh-Hans-CN. Required to automatically enable word or document translation. |

-Values available: For more information, see the Supported Languages section

-<br>

-#### DisplayOptions Parameters

-| Name | Type | Description |

-| fontFamily | String | Sets the chosen font ("Calibri", "ComicSans", or "Sitka"). |

-| themeOption | ThemeOption | Sets the chosen Theme of the reader ("Light", "Dark"). |

-<br>

-## CookiePolicy Options

-**The settings listed below are for informational purposes only**. The Immersive Reader stores its settings, or user preferences, in cookies. This *cookiePolicy* option **disables** the use of cookies by default to follow EU Cookie Compliance laws. If you want to re-enable cookies and restore the default functionality for Immersive Reader user preferences, your website or application will need proper consent from the user to enable cookies. Then, to re-enable cookies in the Immersive Reader, you must explicitly set the *cookiePolicy* option to *CookiePolicy.Enable* when launching the Immersive Reader. The table below describes what settings the Immersive Reader stores in its cookie when the *cookiePolicy* option is enabled.

-#### Settings Parameters

-| fontFamily | String | Sets the chosen font ("Calibri", "ComicSans", or "Sitka"). |

-| theme | String | Sets the chosen theme (e.g "Light", "Dark"...). |

-<br>

-## Supported Languages

-The translation feature of Immersive Reader supports many languages. For more information, see [Language Support](./language-support.md).

-<br>

-When formatting is enabled, the following content will be rendered as HTML in the Immersive Reader.

-| HTML | Supported Content |

-| Font Styles | Bold, Italic, Underline, Code, Strikethrough, Superscript, Subscript |

-| Unordered Lists | Disc, Circle, Square |

-| Ordered Lists | Decimal, Upper-Alpha, Lower-Alpha, Upper-Roman, Lower-Roman |

-Unsupported tags will be rendered comparably. Images and tables are currently not supported.

-<br>

-* Internet Explorer 11

-<br>

-## Next steps

-* Explore the [Immersive Reader SDK on GitHub](https://github.com/microsoft/immersive-reader-sdk)

-* [Quickstart: Create a web app that launches the Immersive Reader (C#)](./quickstarts/client-libraries.md?pivots=programming-language-csharp)

-description: This article will show you how to update the role assignment on existing Immersive Reader resources due to a security bug discovered in November 2021

-# Security Advisory: Update Role Assignment for Microsoft Entra authentication permissions

-A security bug has been discovered with Immersive Reader Microsoft Entra authentication configuration. We are advising that you change the permissions on your Immersive Reader resources as described below.

-A security bug was discovered that relates to Microsoft Entra authentication for Immersive Reader. When initially creating your Immersive Reader resources and configuring them for Microsoft Entra authentication, it is necessary to grant permissions for the Microsoft Entra application identity to access your Immersive Reader resource. This is known as a Role Assignment. The Azure role that was previously used for permissions was the [Cognitive Services User](../../role-based-access-control/built-in-roles.md#cognitive-services-user) role.

-During a security audit, it was discovered that this Cognitive Services User role has permissions to [List Keys](/rest/api/cognitiveservices/accountmanagement/accounts/list-keys). This is slightly concerning because Immersive Reader integrations involve the use of this Microsoft Entra access token in client web apps and browsers, and if the access token were to be stolen by a bad actor or attacker, there is a concern that this access token could be used to `list keys` of your Immersive Reader resource. If an attacker could `list keys` for your resource, then they would obtain the `Subscription Key` for your resource. The `Subscription Key` for your resource is used as an authentication mechanism and is considered a secret. If an attacker had the resource's `Subscription Key`, it would allow them to make valid and authenticated API calls to your Immersive Reader resource endpoint, which could lead to Denial of Service due to the increased usage and throttling on your endpoint. It would also allow unauthorized use of your Immersive Reader resource, which would lead to increased charges on your bill.

-In practice however, this attack or exploit is not likely to occur or may not even be possible. For Immersive Reader scenarios, customers obtain Microsoft Entra access tokens with an audience of `https://cognitiveservices.azure.com`. In order to successfully `list keys` for your resource, the Microsoft Entra access token would need to have an audience of `https://management.azure.com`. Generally speaking, this is not too much of a concern, since the access tokens used for Immersive Reader scenarios would not work to `list keys`, as they do not have the required audience. In order to change the audience on the access token, an attacker would have to hijack the token acquisition code and change the audience before the call is made to Microsoft Entra ID to acquire the token. Again, this is not likely to be exploited because, as an Immersive Reader authentication best practice, we advise that customers create Microsoft Entra access tokens on the web application backend, not in the client or browser. In those cases, since the token acquisition happens on the backend service, it's not as likely or perhaps even possible that attacker could compromise that process and change the audience.

-The real concern comes when or if any customer were to acquire tokens from Microsoft Entra ID directly in client code. We strongly advise against this, but since customers are free to implement as they see fit, it is possible that some customers are doing this.

-To mitigate the concerns about any possibility of using the Microsoft Entra access token to `list keys`, we have created a new built-in Azure role called `Cognitive Services Immersive Reader User` that does not have the permissions to `list keys`. This new role is not a shared role for the Azure AI services platform like `Cognitive Services User` role is. This new role is specific to Immersive Reader and will only allow calls to Immersive Reader APIs.

-We are advising that ALL customers migrate to using the new `Cognitive Services Immersive Reader User` role instead of the original `Cognitive Services User` role. We have provided a script below that you can run on each of your resources to switch over the role assignment permissions.

-If you do NOT do this, nothing will break. The old role will continue to function. The security impact for most customers is minimal. However, it is advised that you migrate to the new role to mitigate the security concerns discussed above. Applying this update is a security advisory recommendation; it is not a mandate.

-Any new Immersive Reader resources you create with our script at [How to: Create an Immersive Reader resource](./how-to-create-immersive-reader.md) will automatically use the new role.

-## Call to action

-If you created and configured an Immersive Reader resource using the instructions at [How to: Create an Immersive Reader resource](./how-to-create-immersive-reader.md) prior to February 2022, it is advised that you perform the operation below to update the role assignment permissions on ALL of your Immersive Reader resources. The operation involves running a script to update the role assignment on a single resource. If you have multiple resources, run this script multiple times, once for each resource.

-After you have updated the role using the script below, it is also advised that you rotate the subscription keys on your resource. This is in case your keys have been compromised by the exploit above, and somebody is actually using your resource with subscription key authentication without your consent. Rotating the keys will render the previous keys invalid and deny any further access. For customers using Microsoft Entra authentication, which should be everyone per current Immersive Reader SDK implementation, rotating the keys will have no impact on the Immersive Reader service, since Microsoft Entra access tokens are used for authentication, not the subscription key. Rotating the subscription keys is just another precaution.

-You can rotate the subscription keys on the [Azure portal](https://portal.azure.com). Navigate to your resource and then to the `Keys and Endpoint` blade. At the top, there are buttons to `Regenerate Key1` and `Regenerate Key2`.

-### Use Azure PowerShell environment to update your Immersive Reader resource Role assignment

- # Get the Azure AD application service principal

- throw "Error: Failed to find Azure AD application service principal"

-1. Run the function `Update-ImmersiveReaderRoleAssignment`, supplying the '<PARAMETER_VALUES>' placeholders below with your own values as appropriate.

- ```azurepowershell-interactive

- Update-ImmersiveReaderRoleAssignment -SubscriptionName '<SUBSCRIPTION_NAME>' -ResourceGroupName '<RESOURCE_GROUP_NAME>' -ResourceName '<RESOURCE_NAME>' -AADAppIdentifierUri '<AAD_APP_IDENTIFIER_URI>'

- The full command will look something like the following. Here we have put each parameter on its own line for clarity, so you can see the whole command. Do not copy or use this command as-is. Copy and use the command above with your own values. This example has dummy values for the '<PARAMETER_VALUES>' above. Yours will be different, as you will come up with your own names for these values.

- ```Update-ImmersiveReaderRoleAssignment```<br>

- ``` -SubscriptionName 'MyOrganizationSubscriptionName'```<br>

- ``` -ResourceGroupName 'MyResourceGroupName'```<br>

- ``` -ResourceName 'MyOrganizationImmersiveReader'```<br>

- ``` -AADAppIdentifierUri 'https://MyOrganizationImmersiveReaderAADApp'```<br>

- | ResourceGroupName |The name of the Resource Group that contains your Immersive Reader resource. |

- | AADAppIdentifierUri |The URI for your Azure AD app. |

-## Next steps

-* View the [Node.js quickstart](./quickstarts/client-libraries.md?pivots=programming-language-nodejs) to see what else you can do with the Immersive Reader SDK using Node.js

-* View the [Android tutorial](./how-to-launch-immersive-reader.md) to see what else you can do with the Immersive Reader SDK using Java or Kotlin for Android

-* View the [iOS tutorial](./how-to-launch-immersive-reader.md) to see what else you can do with the Immersive Reader SDK using Swift for iOS

-* View the [Python tutorial](./how-to-launch-immersive-reader.md) to see what else you can do with the Immersive Reader SDK using Python

-* Explore the [Immersive Reader SDK](https://github.com/microsoft/immersive-reader-sdk) and the [Immersive Reader SDK Reference](./reference.md)

-description: In this tutorial, you will build an iOS app from scratch and add the Picture to Immersive Reader functionality.

-In this tutorial, you will build an iOS app from scratch and integrate the Read API, and the Immersive Reader by using the Immersive Reader SDK. A full working sample of this tutorial is available [here](https://github.com/microsoft/immersive-reader-sdk/tree/master/js/samples/ios).

-If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/cognitive-services/) before you begin.

-* [Xcode](https://apps.apple.com/us/app/xcode/id497799835?mt=12)

-* An Immersive Reader resource configured for Microsoft Entra authentication. Follow [these instructions](./how-to-create-immersive-reader.md) to get set up. You will need some of the values created here when configuring the sample project properties. Save the output of your session into a text file for future reference.

-* Usage of this sample requires an Azure subscription to the Azure AI Vision service. [Create an Azure AI Vision resource in the Azure portal](https://portal.azure.com/#create/Microsoft.CognitiveServicesComputerVision).

-

-

-1. [Install CocoaPods](http://guides.cocoapods.org/using/getting-started.html) - Follow the getting started guide to install Cocoapods.

-You need some values from the Microsoft Entra authentication configuration prerequisite step above for this part. Refer back to the text file you saved of that session.

-ClientId => Azure AD ApplicationId

-ClientSecret => Azure AD Application Service Principal password

-In the main project folder, which contains the ViewController.swift file, create a Swift class file called Constants.swift. Replace the class with the following code, adding in your values where applicable. Keep this file as a local file that only exists on your machine and be sure not to commit this file into source control, as it contains secrets that should not be made public. It is recommended that you do not keep secrets in your app. Instead, we recommend using a backend service to obtain the token, where the secrets can be kept outside of the app and off of the device. The backend API endpoint should be secured behind some form of authentication (for example, [OAuth](https://oauth.net/2/)) to prevent unauthorized users from obtaining tokens to use against your Immersive Reader service and billing; that work is beyond the scope of this tutorial.

-Open AppDelegate.swift and replace the file with the following code.

-Rename ViewController.swift to PictureLaunchViewController.swift and replace the file with the following code.

- /// Retrieves the token for the Immersive Reader using Azure Active Directory authentication

- /// -onSuccess: A closure that gets called when the token is successfully recieved using Azure Active Directory authentication.

- /// -theToken: The token for the Immersive Reader recieved using Azure Active Directory authentication.

- /// -onFailure: A closure that gets called when the token fails to be obtained from the Azure Active Directory Authentication.

- /// -theError: The error that occurred when the token fails to be obtained from the Azure Active Directory Authentication.

-<br/>

-

-In Xcode, press Ctrl + R or select the play button to run the project and the app should launch on the specified simulator or device.

-

-Inside the app, take or upload a photo of text by pressing the 'Take Photo' button or 'Choose Photo from Library' button and the Immersive Reader will then launch displaying the text from the photo.

-

-## Next steps

-* Explore the [Immersive Reader SDK Reference](./reference.md)

-## Conversation summarization types

-Both of these capabilities are able to summarize around specific items of interest when specified.

-Custom Summarization enables users to build custom AI models to summarize unstructured text, such as contracts or novels. By creating a Custom Summarization project, developers can iteratively label data, train, evaluate, and improve model performance before making it available for consumption. The quality of the labeled data greatly impacts model performance. To simplify building and customizing your model, the service offers a custom web portal that can be accessed through the [Language studio](https://aka.ms/languageStudio). You can easily get started with the service by following the steps in this [quickstart](custom/quickstart.md).

-Document summarization uses natural language processing techniques to generate a summary for documents. There are two supported API approaches to automatic summarization: extractive and abstractive.

-There are two types of document summarization this API provides:

-* **Extractive summarization**: Produces a summary by extracting salient sentences within the document.

-* **Abstractive summarization**: Generates a summary that doesn't use the same words as in the document, but captures the main idea.

-* **Issue/resolution summarization**: A call center specific feature that gives a summary of issues and resolutions in conversations between customer-service agents and your customers.

-* **Chapter title summarization**: Segments a conversation into chapters based on the topics discussed in the conversation, and gives suggested chapter titles of the input conversation.

-* **Recap**: Summarizes a conversation into a brief paragraph.

-* **Narrative summarization**: Generates detail call notes, meeting notes or chat summaries of the input conversation.

-* **Follow-up tasks**: Gives a list of follow-up tasks discussed in the input conversation.

-The table above shows the total number of tokens available for each model type. It also determines the maximum number of tokens that can be used for the [system message](#system-message) and the model response. Additionally, the following also consume tokens:

-* The meta prompt (MP): if you limit responses from the model to the grounding data content (`inScope=True` in the API), the maximum number of tokens is 4,036 tokens. Otherwise (for example if `inScope=False`) the maximum is 3,444 tokens. This number is variable depending on the token length of the user question and conversation history. This estimate includes the base prompt and the query rewriting prompts for retrieval.

- "file_ids": ["file_123abc456"]

- "file_ids": ["file_123abc456"]

- "file_id": "file-1YGVTvNzc2JXajI5JU9F0HMD"

-image_data = client.files.content("file-abc123")

-In the following sections, you'll use the Azure CLI to assign roles, and obtain a bearer token to call the OpenAI resource. If you get stuck, links are provided in each section with all available options for each command in Azure Cloud Shell/Azure CLI.

-To sign-in to the Azure CLI, run the following command and complete the sign-in. You may need to do it again if your session has been idle for too long.

-## Assign yourself to the Cognitive Services User role

-Assigning yourself to the "Cognitive Services User" role will allow you to use your account for access to the specific Azure AI services resource.

-1. Get your user information

- ```azurecli

- export user=$(az account show --query "user.name" -o tsv)

- ```

-2. Assign yourself to ΓÇ£Cognitive Services UserΓÇ¥ role.

- ```azurecli

- export resourceId=$(az group show -g $RG --query "id" -o tsv)

- az role assignment create --role "Cognitive Services User" --assignee $user --scope $resourceId

- ```

- > [!NOTE]

- > Role assignment change will take ~5 mins to become effective.

-3. Acquire a Microsoft Entra access token. Access tokens expire in one hour. you'll then need to acquire another one.

- ```azurecli

- export accessToken=$(az account get-access-token --resource https://cognitiveservices.azure.com --query "accessToken" -o tsv)

- ```

-4. Make an API call

-Use the access token to authorize your API call by setting the `Authorization` header value.

-```bash

-curl ${endpoint%/}/openai/deployments/YOUR_DEPLOYMENT_NAME/completions?api-version=2023-05-15 \

-> We are retiring the standard non-neural training tier of custom voice from March 1, 2021 through February 29, 2024. If you used a non-neural custom voice with your Speech resource prior to March 1, 2021 then you can continue to do so until February 29, 2024. All other Speech resources can only use custom neural voice. After February 29, 2024, the non-neural custom voices won't be supported with any Speech resource.

->

-> The pricing for custom voice is different from custom neural voice. Go to the [pricing page](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) and check the pricing details in the collapsable "Deprecated" section. Custom voice (non-neural training) is referred as **Custom**.

-> The Speech to text REST API v2.0 is deprecated and will be retired by February 29, 2024. Please migrate your applications to the Speech to text REST API v3.2. Complete the steps in this article and then see the Speech to text REST API [v3.0 to v3.1](migrate-v3-0-to-v3-1.md) and [v3.1 to v3.2](migrate-v3-1-to-v3-2.md) migration guides for additional requirements.

-> We are retiring the standard non-neural training tier of custom voice from March 1, 2021 through February 29, 2024. If you used a non-neural custom voice with your Speech resource prior to March 1, 2021 then you can continue to do so until February 29, 2024. All other Speech resources can only use custom neural voice. After February 29, 2024, the non-neural custom voices won't be supported with any Speech resource.

->

-> The pricing for custom voice is different from custom neural voice. Go to the [pricing page](https://azure.microsoft.com/pricing/details/cognitive-services/speech-services/) and check the pricing details in the collapsable "Deprecated" section. Custom voice (non-neural training) is referred as **Custom**.

- 'to': ['fr', 'zu']

-Note that disabling SSH prevents SSH access from the public internet. But when a private virtual network is used, users can still SSH from within the virtual network.

-# Istio-based service mesh add-on for Azure Kubernetes Service (preview)

-* Managed lifecycle of mesh on how Istio versions are installed and later made available for upgrades.

-description: Deploy Istio-based service mesh add-on for Azure Kubernetes Service (preview)

-# Deploy Istio-based service mesh add-on for Azure Kubernetes Service (preview)

-### Verify Azure CLI and aks-preview extension versions

-The add-on requires:

-* Azure CLI version 2.49.0 or later installed. To install or upgrade, see [Install Azure CLI][azure-cli-install].

-* `aks-preview` Azure CLI extension of version 0.5.163 or later installed

-You can run `az --version` to verify above versions.

-To install the aks-preview extension, run the following command:

-```azurecli-interactive

-az extension add --name aks-preview

-```

-Run the following command to update to the latest version of the extension released:

-```azurecli-interactive

-az extension update --name aks-preview

-```

-istiod-asm-1-17-74f7f7c46c-xfdtl 1/1 Running 0 2m

-kubectl label namespace default istio.io/rev=asm-1-17

-> The default `istio-injection=enabled` labeling doesn't work. Explicit versioning (`istio.io/rev=asm-1-17`) is required.

-kubectl apply -f <(istioctl kube-inject -f sample.yaml -i aks-istio-system -r asm-1-17) -n foo

-kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.17/samples/bookinfo/platform/kube/bookinfo.yaml

-kubectl delete -f https://raw.githubusercontent.com/istio/istio/release-1.17/samples/bookinfo/platform/kube/bookinfo.yaml

-[istio-deploy-ingress]: istio-deploy-ingress.md

-description: Deploy external or internal ingresses for Istio service mesh add-on for Azure Kubernetes Service (preview)

-# Azure Kubernetes Service (AKS) external or internal ingresses for Istio service mesh add-on deployment (preview)

-description: Configure Istio-based service mesh add-on for Azure Kubernetes Service (preview)

-# Configure Istio-based service mesh add-on for Azure Kubernetes Service (preview)

-description: Plug in CA certificates for Istio-based service mesh add-on on Azure Kubernetes Service (preview)

-# Plug in CA certificates for Istio-based service mesh add-on on Azure Kubernetes Service (preview)

-In the Istio-based service mesh addon for Azure Kubernetes Service (preview), by default the Istio certificate authority (CA) generates a self-signed root certificate and key and uses them to sign the workload certificates. To protect the root CA key, you should use a root CA, which runs on a secure machine offline. You can use the root CA to issue intermediate certificates to the Istio CAs that run in each cluster. An Istio CA can sign workload certificates using the administrator-specified certificate and key, and distribute an administrator-specified root certificate to the workloads as the root of trust. This article addresses how to bring your own certificates and keys for Istio CA in the Istio-based service mesh add-on for Azure Kubernetes Service.

-### Verify Azure CLI and aks-preview extension versions

-The add-on requires:

-* Azure CLI version 2.49.0 or later installed. To install or upgrade, see [Install Azure CLI][install-azure-cli].

-* `aks-preview` Azure CLI extension of version 0.5.163 or later installed

-You can run `az --version` to verify above versions.

-To install the aks-preview extension, run the following command:

-```azurecli-interactive

-az extension add --name aks-preview

-```

-Run the following command to update to the latest version of the extension released:

-```azurecli-interactive

-az extension update --name aks-preview

-```

-description: Upgrade Istio-based service mesh add-on for Azure Kubernetes Service (preview).

-# Upgrade Istio-based service mesh add-on for Azure Kubernetes Service (preview)

-2. Hit enter to open in a new tab the auth portal.

-3. Enter in your Microsoft Credentials in the new page.

-4. Confirm that it's you trying to connect to Azure CLI. If you encounter any issues, skip to the Troubleshooting section.

-5. Verify the message "Device code authentication completed. Logged in to Azure." appears in your original terminal.

-### Troubleshooting: Can't Connect to Localhost

-Certain Azure security policies cause conflicts when trying to sign in. As a workaround, you can perform a curl request to the localhost url you were redirected to after you logged in.

-The workaround requires the Azure CLI for authentication. If you don't have it or aren't using GitHub Codespaces, install the [Azure CLI][install-azure-cli].

-1. Inside a terminal, run `az login --scope https://graph.microsoft.com/.default`

-2. Copy the "localhost" URL from the failed redirect

-3. In a new terminal window, type `curl` and paste your url

-4. If it works, code for a webpage saying "You have logged into Microsoft Azure!" appears

-5. Close the terminal and go back to the old terminal

-6. Copy and note down which subscription_id you want to use

-7. Paste in the subscription_ID to the command `az account set -n {sub}`

-2. Select which Azure subscription and region for your AKS Cluster.

-3. Wait as azd automatically runs the commands for pre-provision and post-provision steps.

-4. At the end, your output shows the newly created deployments and services.

-<!-- Image of Storefront Checkout -->

-1. Leave all settings on the other tabs set to their defaults.

-In previous tutorials, you packaged an application into a container image, uploaded the image to Azure Container Registry, and created a Kubernetes cluster. To complete this tutorial, you need the pre-created `aks-store-quickstart.yaml` Kubernetes manifest file. This file download was included with the application source code in a previous tutorial. Make sure you cloned the repo and changed directories into the cloned repo. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

-This tutorial requires Azure CLI version 2.34.1 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

-This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

-2. Make sure you're in the cloned *aks-store-demo* directory, and then open the manifest file with a text editor, such as `vi`:

-2. Make sure you're in the cloned *aks-store-demo* directory, and then open the manifest file with a text editor, such as `vi`:

-## Deploy the application

-* Deploy the application using the [`kubectl apply`][kubectl-apply] command, which parses the manifest file and creates the defined Kubernetes objects.

- Initially, the `EXTERNAL-IP` for the *store-front* service shows as *pending*.

->

-description: In this Azure Kubernetes Service (AKS) tutorial, you create an AKS cluster and use kubectl to connect to the Kubernetes main node.

-# Tutorial - Deploy an Azure Kubernetes Service (AKS) cluster

-In previous tutorials, you created a container image and uploaded it to an ACR instance. If you haven't completed these steps and want to follow along, start with [Tutorial 1 - Prepare application for AKS][aks-tutorial-prepare-app].

-* If you're using Azure CLI, this tutorial requires that you're running the Azure CLI version 2.0.53 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

-* If you're using Azure PowerShell, this tutorial requires that you're running Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

-This tutorial requires Azure CLI version 2.0.53 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].

-This tutorial requires Azure PowerShell version 5.9.0 or later. Run `Get-InstalledModule -Name Az` to find the version. If you need to install or upgrade, see [Install Azure PowerShell][azure-powershell-install].

-## Create an AKS cluster

-AKS clusters can use [Kubernetes role-based access control (Kubernetes RBAC)][k8s-rbac], which allows you to define access to resources based on roles assigned to users. Permissions are combined when users are assigned multiple roles. Permissions can be scoped to either a single namespace or across the whole cluster. For more information, see [Control access to cluster resources using Kubernetes RBAC and Azure Active Directory identities in AKS][aks-k8s-rbac].

-For information about AKS resource limits and region availability, see [Quotas, virtual machine size restrictions, and region availability in AKS][quotas-skus-regions].

-> [!NOTE]

-> To ensure your cluster operates reliably, you should run at least two nodes.

-### [Azure CLI](#tab/azure-cli)

-To allow an AKS cluster to interact with other Azure resources, the Azure platform automatically creates a cluster identity. In this example, the cluster identity is [granted the right to pull images][container-registry-integration] from the ACR instance you created in the previous tutorial. To execute the command successfully, you need to have an **Owner** or **Azure account administrator** role in your Azure subscription.

-* Create an AKS cluster using the [`az aks create`][az aks create] command. The following example creates a cluster named *myAKSCluster* in the resource group named *myResourceGroup*. This resource group was created in the [previous tutorial][aks-tutorial-prepare-acr] in the *eastus* region.

- ```azurecli-interactive

- az aks create \

- --resource-group myResourceGroup \

- --name myAKSCluster \

- --node-count 2 \

- --generate-ssh-keys \

- --attach-acr <acrName>

- ```

- > [!NOTE]

- > If you already generated SSH keys, you may encounter an error similar to `linuxProfile.ssh.publicKeys.keyData is invalid`. To proceed, retry the command without the `--generate-ssh-keys` parameter.

-### [Azure PowerShell](#tab/azure-powershell)

-To allow an AKS cluster to interact with other Azure resources, the Azure platform automatically creates a cluster identity. In this example, the cluster identity is [granted the right to pull images][container-registry-integration] from the ACR instance you created in the previous tutorial. To execute the command successfully, you need to have an **Owner** or **Azure account administrator** role in your Azure subscription.

-* Create an AKS cluster using the [`New-AzAksCluster`][new-azakscluster] cmdlet. The following example creates a cluster named *myAKSCluster* in the resource group named *myResourceGroup*. This resource group was created in the [previous tutorial][aks-tutorial-prepare-acr] in the *eastus* region.

- ```azurepowershell-interactive

- New-AzAksCluster -ResourceGroupName myResourceGroup -Name myAKSCluster -NodeCount 2 -GenerateSshKey -AcrNameToAttach <acrName>

- ```

- > [!NOTE]

- > If you already generated SSH keys, you may encounter an error similar to `linuxProfile.ssh.publicKeys.keyData is invalid`. To proceed, retry the command without the `-GenerateSshKey` parameter.

-To avoid needing an **Owner** or **Azure account administrator** role, you can also manually configure a service principal to pull images from ACR. For more information, see [ACR authentication with service principals](../container-registry/container-registry-auth-service-principal.md) or [Authenticate from Kubernetes with a pull secret](../container-registry/container-registry-auth-kubernetes.md). Alternatively, you can use a [managed identity](use-managed-identity.md) instead of a service principal for easier management.

-After a few minutes, the deployment completes and returns JSON-formatted information about the AKS deployment.

- The following example output shows a list of the cluster nodes:

- The following example output shows a list of the cluster nodes:

-The sample application you create in this tutorial uses the [*docker-compose-quickstart* YAML file](https://github.com/Azure-Samples/aks-store-demo/blob/main/docker-compose-quickstart.yml) in the [repository](https://github.com/Azure-Samples/aks-store-demo/tree/main) you cloned in the previous step.

-## Create container images and run application

-1. Create the container image, download the Redis image, and start the application using the `docker compose` command.

-[aks-tutorial-prepare-acr]: ./tutorial-kubernetes-prepare-acr.md

-* Public IP addresses are used for internal communication on port `3443` - for managing configuration (for example, through Azure Resource Manager). In the external VNet configuration, they are also used for runtime API traffic.

-* (Developer and Premium tiers) API Management service is moved to a different subnet, or [migrated](migrate-stv1-to-stv2.md) from the `stv1` to the `stv2` compute platform..

-1. In the [Azure portal], search for and select **App Services**, and then select your app.

-1. In the app's left menu, select **Configuration**.

-1. For **Basic Auth Publishing Credentials**, select **Off**, then select **Save**.

-When you disable basic authentication, deployment methods that depend on basic authentication stop working. The following table shows how various deployment methods behave when basic authentication is disabled, and if there's any fallback mechanism. For more information, see [Authentication types by deployment methods in Azure App Service](deploy-authentication-types.md).

-> App Service Build Service requires [basic authentication to be enabled](configure-basic-auth-disable.md) for the webhook to work. For more information, see [Deployment without basic authentication](configure-basic-auth-disable.md#deployment-without-basic-authentication).

-> When [basic authentication is disabled](configure-basic-auth-disable.md), FTP/S deployment doesn't work, and you can't view or configure FTP credentials in the app's Deployment Center.

-> When [basic authentication is disabled](configure-basic-auth-disable.md), Local Git deployment doesn't work, and you can't configure Local Git deployment in the app's Deployment Center.

-> The migration feature described in this article is used for in-place (same subnet) automated migration of App Service Environment v1 and v2 to App Service Environment v3. If you're looking for information on the side by side migration feature, see [Migrate to App Service Environment v3 by using the side by side migration feature](side-by-side-migrate.md). If you're looking for information on manual migration options, see [Manual migration options](migration-alternatives.md). For help deciding which migration option is right for you, see [Migration path decision tree](upgrade-to-asev3.md#migration-path-decision-tree). For more information on App Service Environment v3, see [App Service Environment v3 overview](overview.md).

-description: Learn how to migrate your App Service Environment v2 to App Service Environment v3 by using the side by side migration feature.

-# Use the side by side migration feature to migrate App Service Environment v2 to App Service Environment v3 (Preview)

-> The migration feature described in this article is used for side by side (different subnet) automated migration of App Service Environment v2 to App Service Environment v3 and is currently **in preview**.

-You can automatically migrate App Service Environment v2 to [App Service Environment v3](overview.md) by using the side by side migration feature. To learn more about the migration process and to see if your App Service Environment supports migration at this time, see the [overview of the side by side migration feature](side-by-side-migrate.md).

-The following command checks whether your App Service Environment is supported for migration. If you receive an error or if your App Service Environment is in an unhealthy or suspended state, you can't migrate at this time. See the [troubleshooting](side-by-side-migrate.md#troubleshooting) section for descriptions of the potential error messages that you can get. If your environment [isn't supported for migration using the side by side migration feature](side-by-side-migrate.md#supported-scenarios) or you want to migrate to App Service Environment v3 without using the side by side migration feature, see the [manual migration options](migration-alternatives.md).

-> The migration feature described in this article is used for in-place (same subnet) automated migration of App Service Environment v1 and v2 to App Service Environment v3. If you're looking for information on the side by side migration feature, see [Migrate to App Service Environment v3 by using the side by side migration feature](side-by-side-migrate.md). If you're looking for information on manual migration options, see [Manual migration options](migration-alternatives.md). For help deciding which migration option is right for you, see [Migration path decision tree](upgrade-to-asev3.md#migration-path-decision-tree). For more information on App Service Environment v3, see [App Service Environment v3 overview](overview.md).

-|Migrate cannot be called if IP SSL is enabled on any of the sites.|App Service Environments that have sites with IP SSL enabled can't be migrated using the migration feature at this time. |Migrate using one of the [manual migration options](migration-alternatives.md) if you want to migrate immediately. |

- - If you can't support downtime, see the [side by side migration feature](side-by-side-migrate.md) or the [migration-alternatives](migration-alternatives.md#migrate-manually).

-The [in-place migration feature](migrate.md) automates the migration to App Service Environment v3 and transfers all of your apps to the new environment. There's about one hour of downtime during this migration. If your apps can't have any downtime, we recommend that you use the [side by side migration feature](side-by-side-migrate.md), which is a zero-downtime migration option since the new environment is created in a different subnet. If you also choose not to use the side by side migration feature, you can use one of the manual options to re-create your apps in App Service Environment v3.

-description: Overview of the side by side migration feature for migration to App Service Environment v3.

-# Migration to App Service Environment v3 using the side by side migration feature (Preview)

-> The migration feature described in this article is used for side by side (different subnet) automated migration of App Service Environment v2 to App Service Environment v3 and is currently **in preview**.

-The side by side migration feature automates your migration to App Service Environment v3. The side by side migration feature creates a new App Service Environment v3 with all of your apps in a different subnet. Your existing App Service Environment isn't deleted until you initiate its deletion at the end of the migration process. Because of this process, there's a rollback option if you need to cancel your migration. This migration option is best for customers who want to migrate to App Service Environment v3 with zero downtime and can support using a different subnet for their new environment. If you need to use the same subnet and can support about one hour of application downtime, see the [in-place migration feature](migrate.md). For manual migration options that allow you to migrate at your own pace, see [manual migration options](migration-alternatives.md).

-At this time, the side by side migration feature supports migrations to App Service Environment v3 in the following regions:

-The following App Service Environment configurations can be migrated using the side by side migration feature. The table gives the App Service Environment v3 configuration when using the side by side migration feature based on your existing App Service Environment.

-## Side by side migration feature limitations

-The following are limitations when using the side by side migration feature:

-The side by side migration feature doesn't support the following scenarios. See the [manual migration options](migration-alternatives.md) if your App Service Environment falls into one of these categories.

-The App Service platform reviews your App Service Environment to confirm side by side migration support. If your scenario doesn't pass all validation checks, you can't migrate at this time using the side by side migration feature. If your environment is in an unhealthy or suspended state, you can't migrate until you make the needed updates.

-|Migrate can only be called on an ASE in ARM VNET and this ASE is in Classic VNET. |App Service Environments in Classic virtual networks can't migrate using the side by side migration feature. |Migrate using one of the [manual migration options](migration-alternatives.md). |

-|ASEv3 Migration is not yet ready. |The underlying infrastructure isn't ready to support App Service Environment v3. |Migrate using one of the [manual migration options](migration-alternatives.md) if you want to migrate immediately. Otherwise, wait for the side by side migration feature to be available in your region. |

-|Migrate cannot be called on ASEv2 that is zone-pinned. |App Service Environment v2 that's zone pinned can't be migrated using the side by side migration feature at this time. |Migrate using one of the [manual migration options](migration-alternatives.md) if you want to migrate immediately. |

-|This ASE cannot be migrated without downtime. |This error appears if you try to use the side by side migration feature on an App Service Environment v1. |The side by side migration feature doesn't support App Service Environment v1. Migrate using the [in-place migration feature](migrate.md) or one of the [manual migration options](migration-alternatives.md). |

-|Migrate cannot be called if IP SSL is enabled on any of the sites. |App Service Environments that have sites with IP SSL enabled can't be migrated using the side by side migration feature at this time. |Migrate using one of the [manual migration options](migration-alternatives.md) if you want to migrate immediately. Otherwise, you can disable the IP SSL on all sites in the App Service Environment and attempt migration again. |

-## Overview of the migration process using the side by side migration feature

-Side by side migration consists of a series of steps that must be followed in order. Key points are given for a subset of the steps. It's important to understand what happens during these steps and how your environment and apps are impacted. After reviewing the following information and when you're ready to migrate, follow the [step-by-step guide](how-to-side-by-side-migrate.md).

-Side by side migration requires a three to six hour service window for App Service Environment v2 to v3 migrations. During migration, scaling and environment configurations are blocked and the following events occur:

-> During the preview, in some cases there may be up to 20 minutes of downtime when you complete the final step of the migration. This downtime is due to the DNS change. The downtime is expected to be removed once the feature is generally available. If you have a requirement for zero downtime, you should wait until the side by side migration feature is generally available. During preview, however, you can still use the side by side migration feature to migrate your dev environments to App Service Environment v3 to learn about the migration process and see how it impacts your workloads.

- You can't migrate using the side by side migration feature at this time. If you have an unsupported environment and want to migrate immediately, see the [manual migration options](migration-alternatives.md).

- The side by side migration feature is best for customers who want to migrate to App Service Environment v3 but can't support application downtime. Since a new subnet is used for your new environment, there are networking considerations to be aware of, including new IPs. If you can support downtime, see the [in-place migration feature](migrate.md), which results in minimal configuration changes, or the [manual migration options](migration-alternatives.md). The in-place migration feature creates your App Service Environment v3 in the same subnet as your existing environment and uses the same networking infrastructure.

- No, there's no downtime during the side by side migration process. Your apps continue to run on your existing App Service Environment until you complete the final step of the migration where DNS changes are effective immediately. Once you complete the final step, your old App Service Environment is shut down and deleted. Your new App Service Environment v3 is now your production environment.

- The side by side migration feature supports this [migration scenario](#supported-scenarios).

- The side by side migration feature doesn't support this [migration scenario](#supported-scenarios) at this time. If you have a zone pinned App Service Environment and want to migrate immediately, see the [manual migration options](migration-alternatives.md).

- IP SSL isn't supported on App Service Environment v3. You must remove all IP SSL bindings before migrating using the migration feature or one of the manual options. If you intend to use the side by side migration feature, once you remove all IP SSL bindings, you pass that validation check and can proceed with the automated migration.

- You're on App Service Environment v3 so be sure to review the [features and feature differences](overview.md#feature-differences) compared to previous versions. Both your inbound and outbound IPs change when using the side by side migration feature. Note for ELB App Service Environment, previously there was a single IP for both inbound and outbound. For App Service Environment v3, they're separate. For more information, see [App Service Environment v3 networking](networking.md#addresses). For a full comparison of the App Service Environment versions, see [App Service Environment version comparison](version-comparison.md).

- If there's an unexpected issue, support teams are on hand. We recommend that you migrate dev environments before touching any production environments to learn about the migration process and see how it impacts your workloads. With the side by side migration feature, you can revert all changes if there's any issues.

- If you decide to migrate an App Service Environment using the side by side migration feature, your old environment is used up until the final step in the migration process. Once you complete the final step, the old environment and all of the apps hosted on it get shutdown and deleted. Your old environment is no longer accessible. A rollback to the old environment at this point isn't possible.

-> [Migrate your App Service Environment to App Service Environment v3 using the side by side migration feature](how-to-side-by-side-migrate.md)

-|**1**|**Pre-flight check**|Determine if your environment meets the prerequisites to automate your upgrade using one of the automated migration features. Decide whether an in-place or side by side migration is right for your use case.<br><br>- [Migration path decision tree](#migration-path-decision-tree)<br>- [Automated upgrade using the in-place migration feature](migrate.md)<br>- [Automated upgrade using the side by side migration feature](side-by-side-migrate.md)<br><br>If not, you can upgrade manually.<br><br>- [Manual migration](migration-alternatives.md)|

-|**2**|**Migrate**|Based on results of your review, either upgrade using one of the automated migration features or follow the manual steps.<br><br>- [Use the in-place automated migration feature](how-to-migrate.md)<br>- [Use the side by side automated migration feature](how-to-side-by-side-migrate.md)<br>- [Migrate manually](migration-alternatives.md)|

-|**3**|**Testing and troubleshooting**|Upgrading using one of the automated migration features requires a 3-6 hour service window. If you use the side by side migration feature, you have the opportunity to [test and validate your App Service Environment v3](side-by-side-migrate.md#redirect-customer-traffic-and-complete-migration) before completing the upgrade. Support teams are monitoring upgrades to ensure success. If you have a support plan and you need technical help, create a [support request](https://portal.azure.com/#blade/Microsoft_Azure_Support/HelpAndSupportBlade/newsupportrequest).|

-> [Migration to App Service Environment v3 using the side by side migration feature](side-by-side-migrate.md)

-## Application Gateway Layer 4 capabilities

-4. The same frontend TCP connection is used for subsequent requests from the client unless the TCP idle timeout closes that connection.

-[Configure Azure Application Gateway TCP/TLS proxy](how-to-tcp-tls-proxy.md)

-The following reference outlines the properties supported by the Azure App Configuration Kubernetes Provider.

-If the `spec.target.configMapData` property is not set, the generated ConfigMap will be populated with the list of key-values retrieved from Azure App Configuration, which allows the ConfigMap to be consumed as environment variables. Update this property if you wish to consume the ConfigMap as a mounted file. This property has the following child properties.

-The `spec.auth` property isn't required if the connection string of your App Configuration store is provided by setting the `spec.connectionStringReference` property. Otherwise, one of the identities, service principal, workload identity, or managed identity, will be used for authentication. The `spec.auth` has the following child properties. Only one of them should be specified. If none of them are set, the system-assigned managed identity of the virtual machine scale set will be used.

-|refresh|The settings for refreshing data from Azure App Configuration. If the property is absent, data from Azure App Configuration will not be refreshed.|false|object|

-If the `spec.configuration.selectors` property isn't set, all key-values with no label will be downloaded. It contains an array of *selector* objects, which have the following child properties.

-|enabled|The setting that determines whether data from Azure App Configuration is automatically refreshed. If the property is absent, a default value of `false` will be used.|false|bool|

-|monitoring|The key-values monitored for change detection, aka sentinel keys. The data from Azure App Configuration will be refreshed only if at least one of the monitored key-values is changed.|true|object|

-|interval|The interval at which the data will be refreshed from Azure App Configuration. It must be greater than or equal to 1 second. If the property is absent, a default value of 30 seconds will be used.|false|duration string|

-|refresh|The settings for refreshing data from Key Vaults. If the property is absent, data from Key Vaults will not be refreshed unless the corresponding Key Vault references are reloaded.|false|object|

-The `spec.secret.refresh` property has the following child property.

-|enabled|The setting that determines whether data from Key Vaults is automatically refreshed. If the property is absent, a default value of `false` will be used.|false|bool|

-|interval|The interval at which the data will be refreshed from Key Vault. It must be greater than or equal to 1 minute. The Key Vault refresh is independent of the App Configuration refresh configured via `spec.configuration.refresh`.|true|duration string|

-and the `configMapData.type` property is absent or set to `default`,

-the generated ConfigMap will be populated with the following data:

-and the `configMapData.type` property is set to `json`,

-the generated ConfigMap will be populated with the following data:

-and the `configMapData.type` property is set to `yaml`,

-the generated ConfigMap will be populated with the following data:

-and the `configMapData.type` property is set to `properties`,

-the generated ConfigMap will be populated with the following data:

-description: Learn how to use RedisListTrigger Azure Functions

-# RedisListTrigger Azure Function (preview)

-> Redis triggers aren't currently supported for functions running in the [Consumption plan](consumption-plan.md).

-The following sample polls the key `listTest` at a localhost Redis instance at `127.0.0.1:6379`:

-The isolated process examples aren't available in preview.

-[FunctionName(nameof(ListsTrigger))]

-public static void ListsTrigger(

- [RedisListTrigger("Redis", "listTest")] string entry,

- ILogger logger)

- logger.LogInformation($"The entry pushed to the list listTest: '{entry}'");

- @FunctionName("ListTrigger")

- public void ListTrigger(

- name = "entry",

- connectionStringSetting = "redisLocalhost",

- pollingIntervalInMs = 100,

- messagesPerWorker = 10,

- count = 1,

- listPopFromBeginning = false)

- String entry,

- context.getLogger().info(entry);

-### [v3](#tab/node-v3)

-```javascript

- "bindings": [

- {

- "type": "redisListTrigger",

- "listPopFromBeginning": true,

- "connectionStringSetting": "redisLocalhost",

- "key": "listTest",

- "pollingIntervalInMs": 1000,

- "messagesPerWorker": 100,

- "count": 10,

- "name": "entry",

- "direction": "in"

- }

- ],

- "scriptFile": "index.js"

-### [v4](#tab/node-v4)

-The JavaScript v4 programming model example isn't available in preview.

-```powershell

- "bindings": [

- {

- "type": "redisListTrigger",

- "listPopFromBeginning": true,

- "connectionStringSetting": "redisLocalhost",

- "key": "listTest",

- "pollingIntervalInMs": 1000,

- "messagesPerWorker": 100,

- "count": 10,

- "name": "entry",

- "direction": "in"

- }

- ],

- "scriptFile": "run.ps1"

- "bindings": [

- {

- "type": "redisListTrigger",

- "listPopFromBeginning": true,

- "connectionStringSetting": "redisLocalhost",

- "key": "listTest",

- "pollingIntervalInMs": 1000,

- "messagesPerWorker": 100,

- "count": 10,

- "name": "entry",

- "direction": "in"

- }

- ],

- "scriptFile": "__init__.py"

-The Python v2 programming model example isn't available in preview.

-| `ConnectionStringSetting` | Name of the setting in the `appsettings` that holds the cache connection string (for example, `<cacheName>.redis.cache.windows.net:6380,password=...`). | Yes | |

-| `Count` | Number of entries to pop from Redis at one time. These are processed in parallel. Only supported on Redis 6.2+ using the `COUNT` argument in [`LPOP`](https://redis.io/commands/lpop/) and [`RPOP`](https://redis.io/commands/rpop/). | Optional | `10` |

-| `connectionStringSetting` | The name of the setting in the `appsettings` that contains the cache connection string. For example: `<cacheName>.redis.cache.windows.net:6380,password...` | Yes | |

-| `connectionString` | The name of the setting in the `appsettings` that contains the cache connection string. For example: `<cacheName>.redis.cache.windows.net:6380,password...` | No | |

-| `count` | Number of entries to read from the cache at one time. These are processed in parallel. | Yes | `10` |

-### Output

-> [!NOTE]

-> Once the `RedisListTrigger` becomes generally available, the following information will be moved to a dedicated Output page.

-StackExchange.Redis.RedisValue

-| Output Type | Description |

-|||

-| [`StackExchange.Redis.RedisValue`](https://github.com/StackExchange/StackExchange.Redis/blob/main/src/StackExchange.Redis/RedisValue.cs) | `string`, `byte[]`, `ReadOnlyMemory<byte>`: The entry from the list. |

-| `Custom` | The trigger uses Json.NET serialization to map the message from the channel from a `string` to a custom type. |

-> [!NOTE]

-> Once the `RedisListTrigger` becomes generally available, the following information will be moved to a dedicated Output page.

-| Output Type | Description |

-description: Learn how to use RedisPubSubTrigger Azure Function

-# RedisPubSubTrigger Azure Function (preview)

-The isolated process examples aren't available in preview.

-//TBD

-[FunctionName(nameof(PubSubTrigger))]

-public static void PubSubTrigger(

- [RedisPubSubTrigger("redisConnectionString", "pubsubTest")] string message,

- ILogger logger)

- logger.LogInformation(message);

-This sample listens to any keyspace notifications for the key `myKey`.

-[FunctionName(nameof(KeyspaceTrigger))]

-public static void KeyspaceTrigger(

- [RedisPubSubTrigger("redisConnectionString", "__keyspace@0__:myKey")] string message,

- ILogger logger)

- logger.LogInformation(message);

-[FunctionName(nameof(KeyeventTrigger))]

-public static void KeyeventTrigger(

- [RedisPubSubTrigger("redisConnectionString", "__keyevent@0__:del")] string message,

- ILogger logger)

- logger.LogInformation(message);

-@FunctionName("PubSubTrigger")

- public void PubSubTrigger(

- name = "message",

- connectionStringSetting = "redisConnectionString",

-@FunctionName("KeyspaceTrigger")

- public void KeyspaceTrigger(

- name = "message",

- connectionStringSetting = "redisConnectionString",

- channel = "__keyspace@0__:myKey")

- @FunctionName("KeyeventTrigger")

- public void KeyeventTrigger(

- name = "message",

- connectionStringSetting = "redisConnectionString",

-### [v3](#tab/node-v3)

- "connectionStringSetting": "redisConnectionString",

-Here's binding data to listen to keyspace notifications for the key `myKey`.

- "connectionStringSetting": "redisConnectionString",

- "channel": "__keyspace@0__:myKey",

- "connectionStringSetting": "redisConnectionString",

-### [v4](#tab/node-v4)

-The JavaScript v4 programming model example isn't available in preview.

- "connectionStringSetting": "redisConnectionString",

-Here's binding data to listen to keyspace notifications for the key `myKey`.

- "connectionStringSetting": "redisConnectionString",

- "channel": "__keyspace@0__:myKey",

- "connectionStringSetting": "redisConnectionString",

- "connectionStringSetting": "redisConnectionString",

-Here's binding data to listen to keyspace notifications for the key `myKey`.

- "connectionStringSetting": "redisConnectionString",

- "channel": "__keyspace@0__:myKey",

- "connectionStringSetting": "redisConnectionString",

-The Python v2 programming model example isn't available in preview.

-| `ConnectionStringSetting` | Name of the setting in the `appsettings` that holds the cache connection string. For example,`<cacheName>.redis.cache.windows.net:6380,password=...`. | Yes | |

-| `connectionStringSetting` | Name of the setting in the `appsettings` that holds the cache connection string (for example, `<cacheName>.redis.cache.windows.net:6380,password=...`) | Yes | |

-| `type` | Trigger type. For the pub sub trigger, this is `redisPubSubTrigger`. | Yes | |

-| `connectionStringSetting` | Name of the setting in the `appsettings` that holds the cache connection string (for example, `<cacheName>.redis.cache.windows.net:6380,password=...`) | Yes | |

->The `connectionStringSetting` parameter does not hold the Redis cache connection string itself. Instead, it points to the name of the environment variable that holds the connection string. This makes the application more secure. For more information, see [Redis connection string](functions-bindings-cache.md#redis-connection-string).

-## Output

-> [!NOTE]

-> Once the `RedisPubSubTrigger` becomes generally available, the following information will be moved to a dedicated Output page.

-| Output Type | Description|

-|||

-| [`StackExchange.Redis.ChannelMessage`](https://github.com/StackExchange/StackExchange.Redis/blob/main/src/StackExchange.Redis/ChannelMessageQueue.cs)| The value returned by `StackExchange.Redis`. |

-| [`StackExchange.Redis.RedisValue`](https://github.com/StackExchange/StackExchange.Redis/blob/main/src/StackExchange.Redis/RedisValue.cs)| `string`, `byte[]`, `ReadOnlyMemory<byte>`: The message from the channel. |

-| `Custom`| The trigger uses Json.NET serialization to map the message from the channel from a `string` into a custom type. |

-> [!NOTE]

-> Once the `RedisPubSubTrigger` becomes generally available, the following information will be moved to a dedicated Output page.

-| Output Type | Description |

-| `byte[]` | The message from the channel. |

-| `string` | The message from the channel. |

-description: Learn how to use RedisStreamTrigger Azure Function

-# RedisStreamTrigger Azure Function (preview)

-> Redis triggers aren't currently supported for functions running in the [Consumption plan](consumption-plan.md).

-The isolated process examples aren't available in preview.

-//TBD

-[FunctionName(nameof(StreamsTrigger))]

-public static void StreamsTrigger(

- [RedisStreamTrigger("Redis", "streamTest")] string entry,

- ILogger logger)

- logger.LogInformation($"The entry pushed to the list listTest: '{entry}'");

- @FunctionName("StreamTrigger")

- public void StreamTrigger(

- name = "entry",

- connectionStringSetting = "redisLocalhost",

- pollingIntervalInMs = 100,

- messagesPerWorker = 10,

- count = 1,

- deleteAfterProcess = true)

- String entry,

- context.getLogger().info(entry);

-### [v3](#tab/node-v3)

- "deleteAfterProcess": false,

- "connectionStringSetting": "redisLocalhost",

- "messagesPerWorker": 100,

- "count": 10,

-### [v4](#tab/node-v4)

-The JavaScript v4 programming model example isn't available in preview.

-```powershell

- "deleteAfterProcess": false,

- "connectionStringSetting": "redisLocalhost",

- "messagesPerWorker": 100,

- "count": 10,

- "deleteAfterProcess": false,

- "connectionStringSetting": "redisLocalhost",

- "messagesPerWorker": 100,

- "count": 10,

-The Python v2 programming model example isn't available in preview.

-| `ConnectionStringSetting` | The name of the setting in the `appsettings` that contains cache connection string For example: `<cacheName>.redis.cache.windows.net:6380,password=...` | Yes | |

-| `connectionStringSetting` | The name of the setting in the `appsettings` that contains cache connection string For example: `<cacheName>.redis.cache.windows.net:6380,password=...` | Yes | |

-| `messagesPerWorker` | The number of messages each functions worker should process. It's used to determine how many workers the function should scale to | Optional | `100` |

-| `count` | Number of entries to read from Redis at one time. These are processed in parallel. | Optional | `10` |

-| `connectionStringSetting` | The name of the setting in the `appsettings` that contains cache connection string For example: `<cacheName>.redis.cache.windows.net:6380,password=...` | Yes | |

-The consumer group for all function instances is the `ID` of the function. For example, `Microsoft.Azure.WebJobs.Extensions.Redis.Samples.RedisSamples.StreamTrigger` for the `StreamTrigger` sample. Each function creates a new random GUID to use as its consumer name within the group to ensure that scaled out instances of the function don't read the same messages from the stream.

-### Output

-> [!NOTE]

-> Once the `RedisStreamTrigger` becomes generally available, the following information will be moved to a dedicated Output page.

-| Output Type | Description |

-> [!NOTE]

-> Once the `RedisStreamTrigger` becomes generally available, the following information will be moved to a dedicated Output page.

-| Output Type | Description |

-| Action | Direction | Type | Preview |

-||--|||

-| Triggers on Redis pub sub messages | N/A | [RedisPubSubTrigger](functions-bindings-cache-trigger-redispubsub.md) | Yes|

-| Triggers on Redis lists | N/A | [RedisListsTrigger](functions-bindings-cache-trigger-redislist.md) | Yes |

-| Triggers on Redis streams | N/A | [RedisStreamsTrigger](functions-bindings-cache-trigger-redisstream.md) | Yes |

-## Scope of availability for functions triggers

-> Redis triggers aren't currently supported for functions running in the [Consumption plan](consumption-plan.md).

->

-1. Add the extension bundle by adding or replacing the following code in your _host.json_ file:

- <!-- I don't see this in the samples. -->

- ```json

- >[!WARNING]

- >The Redis extension is currently only available in a preview bundle release.

- >

-Azure Cache for Redis triggers and bindings have a required property for the cache connection string. The connection string can be found on the [**Access keys**](/azure/azure-cache-for-redis/cache-configure#access-keys) menu in the Azure Cache for Redis portal. The Redis trigger or binding looks for an environmental variable holding the connection string with the name passed to the `ConnectionStringSetting` parameter. In local development, the `ConnectionStringSetting` can be defined using the [local.settings.json](/azure/azure-functions/functions-develop-local#local-settings-file) file. When deployed to Azure, [application settings](/azure/azure-functions/functions-how-to-use-azure-function-app-settings) can be used.

-You can find a full view of the [request parameters here](/rest/api/cognitiveservices/healthinsights/onco-phenotype/create-job).

-You can also find a full view of the [request parameters here](/rest/api/cognitiveservices/healthinsights/onco-phenotype/create-job).

-You can find a full view of the [response parameters here](/rest/api/cognitiveservices/healthinsights/onco-phenotype/get-job).

-This document describes details of all inferences generated by application of RI to a radiology document.

-To interact with the Radiology-Insights model, you can provide several model configuration parameters that modify the outcome of the responses. One of the configurations is ΓÇ£inferenceTypesΓÇ¥, which can be used if only part of the Radiology Insights inferences is required. If this list is omitted or empty, the model returns all the inference types.

-<details><summary>Examples request/response json</summary>

-</details>

-Example: ΓÇ£x-ray right footΓÇ¥, ΓÇ£left foot is normalΓÇ¥

-A mismatch with discrepancy type ΓÇ£textLaterityMissingΓÇ¥ has no token extensions.

-<details><summary>Examples request/response json</summary>

-</details>

-Field ΓÇ£sexIndicationΓÇ¥ contains one coding with a SNOMED concept for either MALE (FINDING) if the document refers to a male or FEMALE (FINDING) if the document refers to a female:

-<details><summary>Examples request/response json</summary>

-</details>

-Field ΓÇ£ordertypeΓÇ¥ contains one Coding, with one of the following Loinc codes:

-Fields ΓÇ£missingBodyPartsΓÇ¥ and/or ΓÇ£missingBodyPartsMeasurementsΓÇ¥ contain body parts (radlex codes) that are missing or whose measurements are missing. The token extensions refer to body parts or measurements that are present (or words that imply them).

-<details><summary>Examples request/response json</summary>

-</details>

-Field ΓÇ£ordertypeΓÇ¥ contains one Coding, with one of the following Loinc codes:

-Fields ΓÇ£presentBodyPartsΓÇ¥ and/or ΓÇ£presentBodyPartsMeasurementsΓÇ¥ contain body parts (radlex codes) that are present or whose measurements are present. The token extensions refer to body parts or measurements that are present (or words that imply them).

-<details><summary>Examples request/response json</summary>

-</details>

-This inference is created for a medical problem (for example ΓÇ£acute infection of the lungsΓÇ¥) or for a characteristic or a nonpathologic finding of a body part (for example ΓÇ£stomach normalΓÇ¥).

-Next to the token extensions, there can be an extension with url ΓÇ£sectionΓÇ¥. This extension has an inner extension with a display name that describes the section. The inner extension can also have a LOINC code.

-There can also be an extension with url ΓÇ£ci_sentenceΓÇ¥. This extension refers to the sentence containing the first token of the clinical indicator (that is, the medical problem), if any. The generation of such a sentence is switchable.

-Finding: fields within field ΓÇ£findingΓÇ¥

-list of fields within field ΓÇ£findingΓÇ¥, except ΓÇ£componentΓÇ¥:

-If the value is ΓÇ£NONE (QUALIFIER VALUE)ΓÇ¥, the finding is absent. This value is, for example, ΓÇ£no sepsisΓÇ¥.

-category: if filled, this field contains an array with one element. It contains one of the following SNOMED concepts:

-Finding: field ΓÇ£componentΓÇ¥

-Much relevant information is in the components. The componentΓÇÖs ΓÇ£codeΓÇ¥ field contains one CodeableConcept with one SNOMED code.

-Finding: component ΓÇ£subject of informationΓÇ¥

-This component has SNOMED code 131195008: SUBJECT OF INFORMATION (ATTRIBUTE). It also has the ΓÇ£valueCodeableConceptΓÇ¥ field filled. The value is a SNOMED code describing the medical problem that the finding pertains to.

-At least one ΓÇ£subject of informationΓÇ¥ component is present if and only if the ΓÇ£finding.codeΓÇ¥ field has 404684003: CLINICAL FINDING (FINDING). There can be several "subject of informationΓÇ¥ components, with different concepts in the ΓÇ£valueCodeableConceptΓÇ¥ field.

-Finding: component ΓÇ£anatomyΓÇ¥

-Zero or more components with SNOMED code ΓÇ£722871000000108: ANATOMY (QUALIFIER VALUE)ΓÇ¥. This component has field ΓÇ£valueCodeConceptΓÇ¥ filled with a SNOMED or radlex code. For example, for ΓÇ£lung infectionΓÇ¥ this component contains a code for the lungs.

-Finding: component ΓÇ£regionΓÇ¥

-Zero or more components with SNOMED code 45851105: REGION (ATTRIBUTE). Like anatomy, this component has field ΓÇ£valueCodeableConceptΓÇ¥ filled with a SNOMED or radlex code. Such a concept refers to the body region of the anatomy. For example, if the anatomy is a code for the vagina, the region may be a code for the female reproductive system.

-Finding: component ΓÇ£lateralityΓÇ¥

-Zero or more components with code 45651917: LATERALITY (ATTRIBUTE). Each has field ΓÇ£valueCodeableConceptΓÇ¥ set to a SNOMED concept pertaining to the laterality of the finding. For example, this component is filled for a finding pertaining to the right arm.

-Finding: component ΓÇ£change valuesΓÇ¥

-Zero or more components with code 288533004: CHANGE VALUES (QUALIFIER VALUE). Each has field ΓÇ£valueCodeableConceptΓÇ¥ set to a SNOMED concept pertaining to a size change in the finding (for example, a nodule that is growing or decreasing).

-Finding: component ΓÇ£percentageΓÇ¥

-At most one component with code 45606679: PERCENT (PROPERTY) (QUALIFIER VALUE). It has field ΓÇ£valueStringΓÇ¥ set with either a value or a range consisting of a lower and upper value, separated by ΓÇ£-ΓÇ£.

-Finding: component ΓÇ£severityΓÇ¥

-At most one component with code 272141005: SEVERITIES (QUALIFIER VALUE), indicating how severe the medical problem is. It has field ΓÇ£valueCodeableConceptΓÇ¥ set with a SNOMED code from the following list:

-Finding: component ΓÇ£chronicityΓÇ¥

-At most one component with code 246452003: CHRONICITY (ATTRIBUTE), indicating whether the medical problem is chronic or acute. It has field ΓÇ£valueCodeableConceptΓÇ¥ set with a SNOMED code from the following list:

-Finding: component ΓÇ£causeΓÇ¥

-At most one component with code 135650694: CAUSES OF HARM (QUALIFIER VALUE), indicating what the cause is of the medical problem. It has field ΓÇ£valueStringΓÇ¥ set to the strings of one or more tokens from the text, separated by ΓÇ£;;ΓÇ¥.

-Finding: component ΓÇ£qualifier valueΓÇ¥

-Finding: component ΓÇ£multipleΓÇ¥

-Exactly one component with code 46150521: MULTIPLE (QUALIFIER VALUE). It has field ΓÇ£valueBooleanΓÇ¥ set to true or false. This component indicates the difference between, for example, one nodule (multiple is false) or several nodules (multiple is true). This component has no token extensions.

-Finding: component ΓÇ£sizeΓÇ¥

-Zero or more components with code 246115007, "SIZE (ATTRIBUTE)". Even if there's just one size for a finding, there are several components if the size has two or three dimensions, for example, ΓÇ£2.1 x 3.3 cmΓÇ¥ or ΓÇ£1.2 x 2.2 x 1.5 cmΓÇ¥. There's a size component for every dimension.

-Every component has field ΓÇ£interpretationΓÇ¥ set to either SNOMED code 15240007: CURRENT or 9130008: PREVIOUS, depending on whether the size was measured during this visit or in the past.

-Every component has either field ΓÇ£valueQuantityΓÇ¥ or ΓÇ£valueRangeΓÇ¥ set.

-If ΓÇ£valueQuantityΓÇ¥ is set, then ΓÇ£valueQuantity.valueΓÇ¥ is always set. In most cases, ΓÇ£valueQuantity.unitΓÇ¥ is set. It's possible that ΓÇ£valueQuantity.comparatorΓÇ¥ is also set, to either ΓÇ£>ΓÇ¥, ΓÇ£<ΓÇ¥, ΓÇ£>=ΓÇ¥ or ΓÇ£<=ΓÇ¥. For example, the component is set to ΓÇ£<=ΓÇ¥ for ΓÇ£the tumor is up to 2 cmΓÇ¥.

-If ΓÇ£valueRangeΓÇ¥ is set, then ΓÇ£valueRange.lowΓÇ¥ and ΓÇ£valueRange.highΓÇ¥ are set to quantities with the same data as described in the previous paragraph. This field contains, for example, ΓÇ£The tumor is between 2.5 cm and 2.6 cm in size".

-<details><summary>Examples request/response json</summary>

-</details>

-Field ΓÇ£result.descriptionΓÇ¥ gives a description of the medical problem, for example ΓÇ£MALIGNANCYΓÇ¥.

-Field ΓÇ£result.findingΓÇ¥, if set, contains the same information as the ΓÇ£findingΓÇ¥ field in a finding inference.

-<details><summary>Examples request/response json</summary>

-</details>

-ΓÇ£isHedgingΓÇ¥ mean that the recommendation is uncertain, for example, ΓÇ£a follow-up could be doneΓÇ¥. ΓÇ£isConditionalΓÇ¥ is for input like ΓÇ£If the patient continues having pain, an MRI should be performed.ΓÇ¥

-ΓÇ£isOptionsΓÇ¥: is also for conditional input.

-ΓÇ£isGuidelineΓÇ¥ means that the recommendation is in a general guideline like the following:

-Field ΓÇ£effectiveDateTimeΓÇ¥ will be set when the procedure needs to be done (recommended) at a specific point in time. For example, ΓÇ£next WednesdayΓÇ¥. Field ΓÇ£effectivePeriodΓÇ¥ will be set if a specific period is mentioned, with a start and end datetime. For example, for ΓÇ£within six monthsΓÇ¥, the start datetime will be the date of service, and the end datetime will be the day six months after that.

-If set, field ΓÇ£findingsΓÇ¥ contains one or more findings that have to do with the recommendation. For example, a leg scan (procedure) can be recommended because of leg pain (finding).

-Every array element of field ΓÇ£findingsΓÇ¥ is a RecommendationFinding. Field RecommendationFinding.finding has the same information as a FindingInference.finding field.

-For field ΓÇ£RecommendationFinding.RecommendationFindingStatusΓÇ¥, see the OpenAPI specification for the possible values.

-Field ΓÇ£RecommendationFinding.criticalFindingΓÇ¥ is set if a critical result is associated with the finding. It then contains the same information as described for a critical result inference.

-Field ΓÇ£recommendedProcedureΓÇ¥ is either a GenericProcedureRecommendation, or an ImagingProcedureRecommendation. (Type ΓÇ£procedureRecommendationΓÇ¥ is a supertype for these two types.)

-<details><summary>Examples request/response json</summary>

-</details>

-Field ΓÇ£wasAcknowledgedΓÇ¥ is set to true if the communication was verbal (nonverbal communication might not have reached the recipient yet and cannot be considered acknowledged). Field ΓÇ£dateTimeΓÇ¥ is set if the date-time of the communication is known. Field ΓÇ£recipientΓÇ¥ is set if the recipient(s) are known. See the OpenAPI spec for its possible values.

-<details><summary>Examples request/response json</summary>

-</details>

-Field ΓÇ£imagingProceduresΓÇ¥ contains one or more instances of an imaging procedure, as documented for the follow up recommendations.

-Field ΓÇ£procedureCodesΓÇ¥, if set, contains LOINC codes.

-Field ΓÇ£orderedProcedureΓÇ¥ contains the description(s) and the code(s) of the ordered procedure(s) as given by the client. The descriptions are in field ΓÇ£orderedProcedure.descriptionΓÇ¥, separated by ΓÇ£;;ΓÇ¥. The codes are in ΓÇ£orderedProcedure.code.codingΓÇ¥. In every coding in the array, only field ΓÇ£codingΓÇ¥ is set.

-<details><summary>Examples request/response json</summary>

-</details>

- - description: Include/Exclude follow-up recommendations with no specific radiologic modality, default is false.

- - description: Include/Exclude follow-up recommendations in references to a guideline or article, default is false.

-When includeEvidence is false, no evidence is returned.

-This configuration overrules includeRecommendationsWithNoSpecifiedModality and provideFocusedSentenceEvidence and no evidence is shown.

-When includeEvidence is true, it depends on the value set on the two other configurations whether the evidence of the inference or a single focused sentence is given as evidence.

-The includeRecommendationsWithNoSpecifiedModality is true, includeRecommendationsInReferences is false, provideFocusedSentenceEvidence for recommendations is true and includeEvidence is true.

-<details><summary>Examples request/response json</summary>

-</details>

-The includeRecommendationsWithNoSpecifiedModality is false, includeRecommendationsInReferences is true, provideFocusedSentenceEvidence for findings is true and includeEvidence is true.

-<details><summary>Examples request/response json</summary>

-</details>

-| Data service (Deprecated<sup>1</sup>) | 50 | 50 | Not Available |

-| Data registry service | 50 | 50 | Not Available |

-| Render service - Traffic tiles and Static maps | 50 | 50 | 50 |

-| Spatial service | 50 | 50 | Not Available |

-This article describes the version details for the Azure Monitor agent virtual machine extension. This extension deploys the agent on virtual machines, scale sets, and Arc-enabled servers (on premise servers with Azure Arc agent installed).

-| [VM insights, Service Map, and Dependency agent](../vm/vminsights-overview.md) | Migrate to Azure Monitor Agent | Generally Available | [Enable VM Insights](../vm/vminsights-enable-overview.md) |

-| [Container insights](../containers/container-insights-overview.md) | Migrate to Azure Monitor Agent | **Linux**: Generally available<br>**Windows**:Public preview | [Enable Container Insights](../containers/container-insights-onboard.md) |

-| [Microsoft Sentinel](../../sentinel/overview.md) | Migrate to Azure Monitor Agent | Public Preview | See [AMA migration for Microsoft Sentinel](../../sentinel/ama-migrate.md). Only CEF and Firewall collection remain for GA status |

-| [Change Tracking and Inventory](../../automation/change-tracking/overview-monitoring-agent.md) | Migrate to Azure Monitor Agent | Generally Available | [Migration guidance from Change Tracking and inventory using Log Analytics to Change Tracking and inventory using Azure Monitoring Agent version](../../automation/change-tracking/guidance-migration-log-analytics-monitoring-agent.md) |

-| [Network Watcher](../../network-watcher/network-watcher-monitoring-overview.md) | Migrate to new service called Connection Monitor with Azure Monitor Agent | Generally Available | [Monitor network connectivity using Azure Monitor agent with connection monitor](../../network-watcher/azure-monitor-agent-with-connection-monitor.md) |

-| Azure Stack HCI Insights | Migrate to Azure Monitor Agent | Generally Available| [Monitor Azure Stack HCI with Insights](/azure-stack/hci/manage/monitor-hci-single) |

-| [Azure Virtual Desktop (AVD) Insights](../../virtual-desktop/insights.md) | Migrate to Azure Monitor Agent |Generally Available | [Use Azure Virtual Desktop Insights to monitor your deployment](../../virtual-desktop/insights.md#session-host-data-settings) |

-> [!NOTE]

-> Features and services listed above in preview **may not be available in Azure Government and China clouds**. They will be available typically within a month *after* the features/services become generally available.

-| [Microsoft Defender for Cloud, Servers, SQL, and Endpoint](../../security-center/security-center-introduction.md) | Migrate to Microsoft Defender for Cloud (No dependency on Log Analytics agents or Azure Monitor Agent) | Generally available | [Defender for Cloud plan and strategy for the Log Analytics agent deprecation](../../defender-for-cloud/upcoming-changes.md#defender-for-cloud-plan-and-strategy-for-the-log-analytics-agent-deprecation)|

-| [Automation Hybrid Runbook Worker overview](../../automation/automation-hybrid-runbook-worker.md) | Automation Hybrid Worker Extension (no dependency on Log Analytics agents or Azure Monitor Agent) | Generally available | [Migrate an existing Agent based to Extension based Hybrid Workers](../../automation/extension-based-hybrid-runbook-worker-install.md#migrate-an-existing-agent-based-to-extension-based-hybrid-workers) |

-* Although `ServerTelemetryChannel` is enabled by default, if the application is running in Linux or macOS, the channel doesn't automatically create a local storage folder to keep telemetry temporarily if there are network issues. Because of this limitation, telemetry is lost when there are temporary network or server issues. To work around this issue, configure a local folder for the channel.

-### [ASP.NET Core 6.0](#tab/netcore6)

-```csharp

-using Microsoft.ApplicationInsights.Channel;

-using Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel;

-var builder = WebApplication.CreateBuilder(args);

-// The following will configure the channel to use the given folder to temporarily

-// store telemetry items during network or Application Insights server issues.

-// User should ensure that the given folder already exists

-// and that the application has read/write permissions.

-builder.Services.AddSingleton(typeof(ITelemetryChannel),

- new ServerTelemetryChannel () {StorageFolder = "/tmp/myfolder"});

-builder.Services.AddApplicationInsightsTelemetry();

-var app = builder.Build();

-```

-### [ASP.NET Core 3.1](#tab/netcore3)

-```csharp

-using Microsoft.ApplicationInsights.Channel;

-using Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel;

-public void ConfigureServices(IServiceCollection services)

-{

- // The following will configure the channel to use the given folder to temporarily

- // store telemetry items during network or Application Insights server issues.

- // User should ensure that the given folder already exists

- // and that the application has read/write permissions.

- services.AddSingleton(typeof(ITelemetryChannel),

- new ServerTelemetryChannel () {StorageFolder = "/tmp/myfolder"});

- services.AddApplicationInsightsTelemetry();

-}

-```

-> [!NOTE]

-> This .NET version is no longer supported.

-This limitation isn't applicable from version [2.15.0](https://www.nuget.org/packages/Microsoft.ApplicationInsights.AspNetCore/2.15.0) and later.

-### Is this SDK supported for the new .NET Core 3.X Worker Service template applications?

-This SDK requires `HttpContext`. It doesn't work in any non-HTTP applications, including the .NET Core 3.X Worker Service applications. To enable Application Insights in such applications by using the newly released Microsoft.ApplicationInsights.WorkerService SDK, see [Application Insights for Worker Service applications (non-HTTP applications)](worker-service.md).

-|[Azure Cosmos DB](https://www.nuget.org/packages/Microsoft.Azure.Cosmos) | Tracked automatically if HTTP/HTTPS is used. TCP will also be captured automatically using preview package >= [3.33.0-preview](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/3.33.0-preview). |

-|Azure Spring Cloud | :x: | :x: | [ :white_check_mark: :link: ](azure-web-apps-java.md) | :x: | :x: |

-If you're using the *exec* form, add the parameter `-javaagent:"path/to/applicationinsights-agent-3.4.19.jar"` to the parameter list somewhere before the `"-jar"` parameter, for example:

-ENTRYPOINT ["java", "-javaagent:path/to/applicationinsights-agent-3.4.19.jar", "-jar", "<myapp.jar>"]

-If you're using the *shell* form, add the JVM arg `-javaagent:"path/to/applicationinsights-agent-3.4.19.jar"` somewhere before `-jar`, for example:

-ENTRYPOINT java -javaagent:"path/to/applicationinsights-agent-3.4.19.jar" -jar <myapp.jar>

-COPY agent/applicationinsights-agent-3.4.19.jar applicationinsights-agent-3.4.19.jar

-ENTRYPOINT["java", "-javaagent:applicationinsights-agent-3.4.19.jar", "-jar", "app.jar"]

-In this example we have copied the `applicationinsights-agent-3.4.19.jar` and `applicationinsights.json` files from an `agent` folder (you can choose any folder of your machine). These two files have to be in the same folder in the Docker container.

-JAVA_OPTS="$JAVA_OPTS -javaagent:path/to/applicationinsights-agent-3.4.19.jar"

-CATALINA_OPTS="$CATALINA_OPTS -javaagent:path/to/applicationinsights-agent-3.4.19.jar"

-If the file `<tomcat>/bin/setenv.sh` already exists, modify that file and add `-javaagent:path/to/applicationinsights-agent-3.4.19.jar` to `CATALINA_OPTS`.

-set CATALINA_OPTS=%CATALINA_OPTS% -javaagent:path/to/applicationinsights-agent-3.4.19.jar

-set "CATALINA_OPTS=%CATALINA_OPTS% -javaagent:path/to/applicationinsights-agent-3.4.19.jar"

-If the file `<tomcat>/bin/setenv.bat` already exists, modify that file and add `-javaagent:path/to/applicationinsights-agent-3.4.19.jar` to `CATALINA_OPTS`.

-Locate the file `<tomcat>/bin/tomcat8w.exe`. Run that executable and add `-javaagent:path/to/applicationinsights-agent-3.4.19.jar` to the `Java Options` under the `Java` tab.

-Add `-javaagent:path/to/applicationinsights-agent-3.4.19.jar` to the existing `JAVA_OPTS` environment variable in the file `JBOSS_HOME/bin/standalone.conf` (Linux) or `JBOSS_HOME/bin/standalone.conf.bat` (Windows):

- JAVA_OPTS="-javaagent:path/to/applicationinsights-agent-3.4.19.jar -Xms1303m -Xmx1303m ..."

-Add `-javaagent:path/to/applicationinsights-agent-3.4.19.jar` to the existing `jvm-options` in `JBOSS_HOME/domain/configuration/host.xml`:

- <option value="-javaagent:path/to/applicationinsights-agent-3.4.19.jar"/>

-Add `-javaagent:path/to/applicationinsights-agent-3.4.19.jar` to the existing `jvm-options` in `glassfish/domains/domain1/config/domain.xml`:

- -javaagent:path/to/applicationinsights-agent-3.4.19.jar>

- -javaagent:path/to/applicationinsights-agent-3.4.19.jar

-Add the JVM arg `-javaagent:"path/to/applicationinsights-agent-3.4.19.jar"` somewhere before `-jar`, for example:

-java -javaagent:"path/to/applicationinsights-agent-3.4.19.jar" -jar <myapp.jar>

- <version>3.4.19</version>

- <version>3.4.19</version>

-By default, Application Insights Java 3.x expects the configuration file to be named `applicationinsights.json`, and to be located in the same directory as `applicationinsights-agent-3.4.19.jar`.

-If you specify a relative path, it resolves relative to the directory where `applicationinsights-agent-3.4.19.jar` is located.

-If you specify a relative path, it resolves relative to the directory where `applicationinsights-agent-3.4.19.jar` is located.

- <version>3.4.19</version>

-`applicationinsights-agent-3.4.19.jar` is located.

-Download the [applicationinsights-agent-3.4.19.jar](https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.4.19/applicationinsights-agent-3.4.19.jar) file.

-Point the JVM to the jar file by adding `-javaagent:"path/to/applicationinsights-agent-3.4.19.jar"` to your application's JVM args.

- Create a configuration file named `applicationinsights.json`, and place it in the same directory as `applicationinsights-agent-3.4.19.jar` with the following content:

-Azure VMware Solution private cloud vCenter Server, NSX-T Data Center, and HCX Manager (if enabled) configurations are on a daily backup schedule. Open a [support request](https://rc.portal.azure.com/#create/Microsoft.Support) in the Azure portal to request restoration.

-> [!WARNING]

-> `$PreemptedNodeCount` is currently not available and returns `0` valued data.

-> We strongly recommend that you **avoid relying *only* on `GetSample(1)` in your autoscale formulas**. This is because `GetSample(1)` essentially says to the Batch service, "Give me the last sample you have, no matter how long ago you retrieved it." Since it's only a single sample, and it might be an older sample, it might not be representative of the larger picture of recent task or resource state. If you do use `GetSample(1)`, make sure that it's part of a larger statement and not the only data point that your formula relies on.

-description: Learn how to create an Azure Batch pool without public IP addresses.

-# Create a Batch pool without public IP addresses (preview)

-> [!WARNING]

-> This preview version will be retired on **31 March 2023**, and will be replaced by

-> [Simplified node communication pool without public IP addresses](simplified-node-communication-pool-no-public-ip.md).

-> For more information, see the [Retirement Migration Guide](batch-pools-without-public-ip-addresses-classic-retirement-migration-guide.md).

-> [!IMPORTANT]

-> - Support for pools without public IP addresses in Azure Batch is currently in public preview for the following regions: France Central, East Asia, West Central US, South Central US, West US 2, East US, North Europe, East US 2, Central US, West Europe, North Central US, West US, Australia East, Japan East, Japan West.

-> - This preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.

-> - For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).

-When you create an Azure Batch pool, you can provision the virtual machine configuration pool without a public IP address. This article explains how to set up a Batch pool without public IP addresses.

-## Why use a pool without public IP addresses?

-By default, all the compute nodes in an Azure Batch virtual machine configuration pool are assigned a public IP address. This address is used by the Batch service to schedule tasks and for communication with compute nodes, including outbound access to the internet.

-To restrict access to these nodes and reduce the discoverability of these nodes from the internet, you can provision the pool without public IP addresses.

-## Prerequisites

- - The VNet must be in the same subscription and region as the Batch account you use to create your pool.

- - The subnet specified for the pool must have enough unassigned IP addresses to accommodate the number of VMs targeted for the pool; that is, the sum of the `targetDedicatedNodes` and `targetLowPriorityNodes` properties of the pool. If the subnet doesn't have enough unassigned IP addresses, the pool partially allocates the compute nodes, and a resize error occurs.

- - You must disable private link service and endpoint network policies. This action can be done by using Azure CLI:

- `az network vnet subnet update --vnet-name <vnetname> -n <subnetname> --resource-group <resourcegroup> --disable-private-endpoint-network-policies --disable-private-link-service-network-policies`

-> [!IMPORTANT]

-> For each 100 dedicated or Spot nodes, Batch allocates one private link service and one load balancer. These resources are limited by the subscription's [resource quotas](../azure-resource-manager/management/azure-subscription-service-limits.md). For large pools, you might need to [request a quota increase](batch-quota-limit.md#increase-a-quota) for one or more of these resources. Additionally, no resource locks should be applied to any resource created by Batch, since this prevent cleanup of resources as a result of user-initiated actions such as deleting a pool or resizing to zero.

-## Current limitations

-1. Pools without public IP addresses must use Virtual Machine Configuration and not Cloud Services Configuration.

-1. [Custom endpoint configuration](pool-endpoint-configuration.md) to Batch compute nodes doesn't work with pools without public IP addresses.

-1. Because there are no public IP addresses, you can't [use your own specified public IP addresses](create-pool-public-ip.md) with this type of pool.

-1. [Basic VM size](../virtual-machines/sizes-previous-gen.md#basic-a) doesn't work with pools without public IP addresses.

-## Create a pool without public IP addresses in the Azure portal

-1. Navigate to your Batch account in the Azure portal.

-1. In the **Settings** window on the left, select **Pools**.

-1. In the **Pools** window, select **Add**.

-1. On the **Add Pool** window, select the option you intend to use from the **Image Type** dropdown.

-1. Select the correct **Publisher/Offer/Sku** of your image.

-1. Specify the remaining required settings, including the **Node size**, **Target dedicated nodes**, and **Target Spot/low-priority nodes**, and any desired optional settings.

-1. Optionally select a virtual network and subnet you wish to use. This virtual network must be in the same resource group as the pool you're creating.

-1. In **IP address provisioning type**, select **NoPublicIPAddresses**.

-

-## Use the Batch REST API to create a pool without public IP addresses

-The example below shows how to use the [Batch Service REST API](/rest/api/batchservice/pool/add) to create a pool that uses public IP addresses.

-### REST API URI

-```http

-POST {batchURL}/pools?api-version=2020-03-01.11.0

-client-request-id: 00000000-0000-0000-0000-000000000000

-```

-### Request body

-```json

-"pool": {

- "id": "pool2",

- "vmSize": "standard_a1",

- "virtualMachineConfiguration": {

- "imageReference": {

- "publisher": "Canonical",

- "offer": "UbuntuServer",

- "sku": "20.04-lts"

- },

- "nodeAgentSKUId": "batch.node.ubuntu 20.04"

- }

- "networkConfiguration": {

- "subnetId": "/subscriptions/<your_subscription_id>/resourceGroups/<your_resource_group>/providers/Microsoft.Network/virtualNetworks/<your_vnet_name>/subnets/<your_subnet_name>",

- "publicIPAddressConfiguration": {

- "provision": "NoPublicIPAddresses"

- }

- },

- "resizeTimeout": "PT15M",

- "targetDedicatedNodes": 5,

- "targetLowPriorityNodes": 0,

- "taskSlotsPerNode": 3,

- "taskSchedulingPolicy": {

- "nodeFillType": "spread"

- },

- "enableAutoScale": false,

- "enableInterNodeCommunication": true,

- "metadata": [

- {

- "name": "myproperty",

- "value": "myvalue"

- }

- ]

-}

-```

-> [!Important]

-> This document references a release version of Linux that is nearing or at, End of Life(EOL). Please consider updating to a more current version.

-## Outbound access to the internet

-In a pool without public IP addresses, your virtual machines won't be able to access the public internet unless you configure your network setup appropriately, such as by using [virtual network NAT](../virtual-network/nat-gateway/nat-overview.md). NAT only allows outbound access to the internet from the virtual machines in the virtual network. Batch-created compute nodes won't be publicly accessible, since they don't have public IP addresses associated.

-Another way to provide outbound connectivity is to use a user-defined route (UDR). This method lets you route traffic to a proxy machine that has public internet access.

-## Next steps

-> Batch **does not** support any VM SKU sizes that have only remote storage. A local temporary disk is required for Batch.

-> For example, Batch supports [ddv4 and ddsv4](../virtual-machines/ddv4-ddsv4-series.md), but does not support

-> [dv4 and dsv4](../virtual-machines/dv4-dsv4-series.md).

-It's recommended to avoid images with impending Batch support end of life (EOL) dates. These dates can be discovered via

-the [`ListSupportedImages` API](/rest/api/batchservice/account/listsupportedimages),

-[PowerShell](/powershell/module/az.batch/get-azbatchsupportedimage), or [Azure CLI](/cli/azure/batch/pool/supported-images).

-For more information, see the [Batch best practices guide](best-practices.md) regarding Batch pool VM image selection.

-description: How to use rendering applications with Azure Batch. This article provides a brief description of how to run each rendering application.

-# Rendering applications

-Rendering applications are used by creating Batch jobs and tasks. The task command line property specifies the appropriate command line and parameters. The easiest way to create the job tasks is to use the Batch Explorer templates as specified in [this article](./batch-rendering-using.md#using-batch-explorer). The templates can be viewed and modified versions created if necessary.

-This article provides a brief description of how to run each rendering application.

-## Rendering with Autodesk 3ds Max

-### Renderer support

-In addition to the renderers built into 3ds Max, the following renderers are available on the rendering VM images and can be referenced by the 3ds Max scene file:

-* Autodesk Arnold

-* Chaos Group V-Ray

-### Task command line

-Invoke the `3dsmaxcmdio.exe` application to perform command line rendering on a pool node. This application is on the path when the task is run. The `3dsmaxcmdio.exe` application has the same available parameters as the `3dsmaxcmd.exe` application, which is documented in the [3ds Max help documentation](https://help.autodesk.com/view/3DSMAX/2018/ENU/) (Rendering | Command-Line Rendering section).

-For example:

-```

-3dsmaxcmdio.exe -v:5 -rfw:0 -start:{0} -end:{0} -bitmapPath:"%AZ_BATCH_JOB_PREP_WORKING_DIR%\sceneassets\images" -outputName:dragon.jpg -w:1280 -h:720 "%AZ_BATCH_JOB_PREP_WORKING_DIR%\scenes\dragon.max"

-```

-Notes:

-* Great care must be taken to ensure the asset files are found. Ensure the paths are correct and relative using the **Asset Tracking** window, or use the `-bitmapPath` parameter on the command line.

-* See if there are issues with the render, such as inability to find assets, by checking the `stdout.txt` file written by 3ds Max when the task is run.

-### Batch Explorer templates

-Pool and job templates can be accessed from the **Gallery** in Batch Explorer. The template source files are available in the [Batch Explorer data repository on GitHub](https://github.com/Azure/BatchExplorer-data/tree/master/ncj/3dsmax).

-## Rendering with Autodesk Maya

-### Renderer support

-In addition to the renderers built into Maya, the following renderers are available on the rendering VM images and can be referenced by the 3ds Max scene file:

-* Autodesk Arnold

-* Chaos Group V-Ray

-### Task command line

-The `renderer.exe` command-line renderer is used in the task command line. The command-line renderer is documented in [Maya help](https://help.autodesk.com/view/MAYAUL/2018/ENU/?guid=GUID-EB558BC0-5C2B-439C-9B00-F97BCB9688E4).

-In the following example, a job preparation task is used to copy the scene files and assets to the job preparation working directory, an output folder is used to store the rendering image, and frame 10 is rendered.

-```

-render -renderer sw -proj "%AZ_BATCH_JOB_PREP_WORKING_DIR%" -verb -rd "%AZ_BATCH_TASK_WORKING_DIR%\output" -s 10 -e 10 -x 1920 -y 1080 "%AZ_BATCH_JOB_PREP_WORKING_DIR%\scene-file.ma"

-```

-For V-Ray rendering, the Maya scene file would normally specify V-Ray as the renderer. It can also be specified on the command line:

-```

-render -renderer vray -proj "%AZ_BATCH_JOB_PREP_WORKING_DIR%" -verb -rd "%AZ_BATCH_TASK_WORKING_DIR%\output" -s 10 -e 10 -x 1920 -y 1080 "%AZ_BATCH_JOB_PREP_WORKING_DIR%\scene-file.ma"

-```

-For Arnold rendering, the Maya scene file would normally specify Arnold as the renderer. It can also be specified on the command line:

-```

-render -renderer arnold -proj "%AZ_BATCH_JOB_PREP_WORKING_DIR%" -verb -rd "%AZ_BATCH_TASK_WORKING_DIR%\output" -s 10 -e 10 -x 1920 -y 1080 "%AZ_BATCH_JOB_PREP_WORKING_DIR%\scene-file.ma"

-```

-### Batch Explorer templates

-Pool and job templates can be accessed from the **Gallery** in Batch Explorer. The template source files are available in the [Batch Explorer data repository on GitHub](https://github.com/Azure/BatchExplorer-data/tree/master/ncj/maya).

-## Next steps

-Use the pool and job templates from the [data repository in GitHub](https://github.com/Azure/BatchExplorer-data/tree/master/ncj) using Batch Explorer. When required, create new templates or modify one of the supplied templates.

-description: It's possible to use any rendering applications with Azure Batch. However, Azure Marketplace VM images are available with common applications pre-installed.

-# Pre-installed applications on Batch rendering VM images

-> [!CAUTION]

-> This article references CentOS, a Linux distribution that is nearing End Of Life (EOL) status. Please consider your use and planning accordingly.

-It's possible to use any rendering applications with Azure Batch. However, Azure Marketplace VM images are available with common applications pre-installed.

-Where applicable, pay-for-use licensing is available for the pre-installed rendering applications. When a Batch pool is created, the required applications can be specified and both the cost of VM and applications will be billed per minute. Application prices are listed on the [Azure Batch pricing page](https://azure.microsoft.com/pricing/details/batch/#graphic-rendering).

-Some applications only support Windows, but most are supported on both Windows and Linux.

-> [!WARNING]

-> The rendering VM images and pay-for-use licensing have been [deprecated and will be retired on February 29, 2024](https://azure.microsoft.com/updates/azure-batch-rendering-vm-images-licensing-will-be-retired-on-29-february-2024/). To use Batch for rendering, [a custom VM image and standard application licensing should be used.](batch-rendering-functionality.md#batch-pools-using-custom-vm-images-and-standard-application-licensing)

-## Applications on latest CentOS 7 rendering image

-The following list applies to the CentOS rendering image, version 1.2.0.

-* Autodesk Maya I/O 2020 Update 4.6

-* Autodesk Arnold for Maya 2020 (Arnold version 6.2.0.0) MtoA-4.2.0-2020

-* Chaos Group V-Ray for Maya 2020 (version 5.00.21)

-* Blender (2.80)

-* AZ 10

-## Applications on latest Windows Server rendering image

-The following list applies to the Windows Server rendering image, version 1.5.0.

-* Autodesk Maya I/O 2020 Update 4.4

-* Autodesk 3ds Max I/O 2021 Update 3

-* Autodesk Arnold for Maya 2020 (Arnold version 6.1.0.1) MtoA-4.1.1.1-2020

-* Autodesk Arnold for 3ds Max 2021 (Arnold version 6.1.0.1) MAXtoA-4.2.2.20-2021

-* Chaos Group V-Ray for Maya 2020 (version 5.00.21)

-* Chaos Group V-Ray for 3ds Max 2021 (version 5.00.05)

-* Blender (2.79)

-* Blender (2.80)

-* AZ 10

-> [!IMPORTANT]

-> To run V-Ray with Maya outside of the [Azure Batch extension templates](https://github.com/Azure/batch-extension-templates), start `vrayses.exe` before running the render. To start the vrayses.exe outside of the templates you can use the following command `%MAYA_2020%\vray\bin\vrayses.exe"`.

->

-> For an example, see the start task of the [Maya and V-Ray template](https://github.com/Azure/batch-extension-templates/blob/master/templates/maya/render-vray-windows/pool.template.json) on GitHub.

-## Applications on previous Windows Server rendering images

-The following list applies to Windows Server 2016, version 1.3.8 rendering images.

-* Autodesk Maya I/O 2017 Update 5 (version 17.4.5459)

-* Autodesk Maya I/O 2018 Update 6 (version 18.4.0.7622)

-* Autodesk Maya I/O 2019

-* Autodesk 3ds Max I/O 2018 Update 4 (version 20.4.0.4254)

-* Autodesk 3ds Max I/O 2019 Update 1 (version 21.2.0.2219)

-* Autodesk 3ds Max I/O 2020 Update 2

-* Autodesk Arnold for Maya 2017 (Arnold version 5.3.0.2) MtoA-3.2.0.2-2017

-* Autodesk Arnold for Maya 2018 (Arnold version 5.3.0.2) MtoA-3.2.0.2-2018

-* Autodesk Arnold for Maya 2019 (Arnold version 5.3.0.2) MtoA-3.2.0.2-2019

-* Autodesk Arnold for 3ds Max 2018 (Arnold version 5.3.0.2)(version 1.2.926)

-* Autodesk Arnold for 3ds Max 2019 (Arnold version 5.3.0.2)(version 1.2.926)

-* Autodesk Arnold for 3ds Max 2020 (Arnold version 5.3.0.2)(version 1.2.926)

-* Chaos Group V-Ray for Maya 2017 (version 4.12.01)

-* Chaos Group V-Ray for Maya 2018 (version 4.12.01)

-* Chaos Group V-Ray for Maya 2019 (version 4.04.03)

-* Chaos Group V-Ray for 3ds Max 2018 (version 4.20.01)

-* Chaos Group V-Ray for 3ds Max 2019 (version 4.20.01)

-* Chaos Group V-Ray for 3ds Max 2020 (version 4.20.01)

-* Blender (2.79)

-* Blender (2.80)

-* AZ 10

-The following list applies to Windows Server 2016, version 1.3.7 rendering images.

-* Autodesk Maya I/O 2017 Update 5 (version 17.4.5459)

-* Autodesk Maya I/O 2018 Update 4 (version 18.4.0.7622)

-* Autodesk 3ds Max I/O 2019 Update 1 (version 21.2.0.2219)

-* Autodesk 3ds Max I/O 2018 Update 4 (version 20.4.0.4254)

-* Autodesk Arnold for Maya 2017 (Arnold version 5.2.0.1) MtoA-3.1.0.1-2017

-* Autodesk Arnold for Maya 2018 (Arnold version 5.2.0.1) MtoA-3.1.0.1-2018

-* Autodesk Arnold for 3ds Max 2018 (Arnold version 5.0.2.4)(version 1.2.926)

-* Autodesk Arnold for 3ds Max 2019 (Arnold version 5.0.2.4)(version 1.2.926)

-* Chaos Group V-Ray for Maya 2018 (version 3.52.03)

-* Chaos Group V-Ray for 3ds Max 2018 (version 3.60.02)

-* Chaos Group V-Ray for Maya 2019 (version 3.52.03)

-* Chaos Group V-Ray for 3ds Max 2019 (version 4.10.01)

-* Blender (2.79)

-> [!NOTE]

-> Chaos Group V-Ray for 3ds Max 2019 (version 4.10.01) introduces breaking changes to V-ray. To use the previous version (version 3.60.02), use Windows Server 2016, version 1.3.2 rendering nodes.

-## Applications on previous CentOS rendering images

-The following list applies to CentOS 7.6, version 1.1.6 rendering images.

-* Autodesk Maya I/O 2017 Update 5 (cut 201708032230)

-* Autodesk Maya I/O 2018 Update 2 (cut 201711281015)

-* Autodesk Maya I/O 2019 Update 1

-* Autodesk Arnold for Maya 2017 (Arnold version 5.3.1.1) MtoA-3.2.1.1-2017

-* Autodesk Arnold for Maya 2018 (Arnold version 5.3.1.1) MtoA-3.2.1.1-2018

-* Autodesk Arnold for Maya 2019 (Arnold version 5.3.1.1) MtoA-3.2.1.1-2019

-* Chaos Group V-Ray for Maya 2017 (version 3.60.04)

-* Chaos Group V-Ray for Maya 2018 (version 3.60.04)

-* Blender (2.68)

-* Blender (2.8)

-## Next steps

-To use the rendering VM images, they need to be specified in the pool configuration when a pool is created; see the [Batch pool capabilities for rendering](./batch-rendering-functionality.md).

-## Batch pools using rendering VM images

-> [!WARNING]

-> The rendering VM images and pay-for-use licensing have been [deprecated and will be retired on February 29, 2024](https://azure.microsoft.com/updates/azure-batch-rendering-vm-images-licensing-will-be-retired-on-29-february-2024/). To use Batch for rendering, [a custom VM image and standard application licensing should be used.](batch-rendering-functionality.md#batch-pools-using-custom-vm-images-and-standard-application-licensing)

-### Rendering application installation

-An Azure Marketplace rendering VM image can be specified in the pool configuration if only the pre-installed applications need to be used.

-There is a Windows image and a CentOS image. In the [Azure Marketplace](https://azuremarketplace.microsoft.com), the VM images can be found by searching for 'batch rendering'.

-The Azure portal and Batch Explorer provide GUI tools to select a rendering VM image when you create a pool. If using a Batch API, then specify the following property values for [ImageReference](/rest/api/batchservice/pool/add#imagereference) when creating a pool:

-| Publisher | Offer | Sku | Version |

-||||--|

-| batch | rendering-centos73 | rendering | latest |

-| batch | rendering-windows2016 | rendering | latest |

-Other options are available if additional applications are required on the pool VMs:

-### Pay-for-use licensing for pre-installed applications

-The applications that will be used and have a licensing fee need to be specified in the pool configuration.

-* Specify the `applicationLicenses` property when [creating a pool](/rest/api/batchservice/pool/add#request-body). The following values can be specified in the array of strings - "vray", "arnold", "3dsmax", "maya".

-* When you specify one or more applications, then the cost of those applications is added to the cost of the VMs. Application prices are listed on the [Azure Batch pricing page](https://azure.microsoft.com/pricing/details/batch/#graphic-rendering).

-> [!NOTE]

-> If instead you connect to a license server to use the rendering applications, do not specify the `applicationLicenses` property.

-You can use the Azure portal or Batch Explorer to select applications and show the application prices.

-If an attempt is made to use an application, but the application hasnΓÇÖt been specified in the `applicationLicenses` property of the pool configuration or does not reach a license server, then the application execution fails with a licensing error and non-zero exit code.

-### Environment variables for pre-installed applications

-To be able to create the command line for rendering tasks, the installation location of the rendering application executables must be specified. System environment variables have been created on the Azure Marketplace VM images, which can be used instead of having to specify actual paths. These environment variables are in addition to the [standard Batch environment variables](./batch-compute-node-environment-variables.md) created for each task.

-|Application|Application Executable|Environment Variable|

-||||

-|Autodesk 3ds Max 2021|3dsmaxcmdio.exe|3DSMAX_2021_EXEC|

-|Autodesk Maya 2020|render.exe|MAYA_2020_EXEC|

-|Chaos Group V-Ray Standalone|vray.exe|VRAY_4.10.03_EXEC|

-|Arnold 2020 command line|kick.exe|ARNOLD_2020_EXEC|

-|Blender|blender.exe|BLENDER_2018_EXEC|

-* Learn about [using rendering applications with Batch](batch-rendering-applications.md).

-Rendering is the process of taking 3D models and converting them into 2D images. 3D scene files are authored in applications such as Autodesk 3ds Max, Autodesk Maya, and Blender. Rendering applications such as Autodesk Maya, Autodesk Arnold, Chaos Group V-Ray, and Blender Cycles produce 2D images. Sometimes single images are created from the scene files. However, it's common to model and render multiple images, and then combine them in an animation.

-The process of rendering is computationally intensive; there can be many frames/images to produce and each image can take many hours to render. Rendering is therefore a perfect batch processing workload that can leverage Azure to run many renders in parallel and utilize a wide range of hardware, including GPUs.

- * Animations consist of many frames and each frame can be rendered in parallel. The more VMs available to process each frame, the faster all the frames and the animation can be produced.

- * Some rendering software allows single frames to be broken up into multiple pieces, such as tiles or slices. Each piece can be rendered separately, then combined into the final image when all pieces have finished. The more VMs that are available, the faster a frame can be rendered.

- * Individual frames can be complex and require many hours to render, even on high-end hardware; animations can consist of hundreds of thousands of frames. A huge amount of compute is required to render high-quality animations in a reasonable amount of time. In some cases, over 100,000 cores have been used to render thousands of frames in parallel.

- * Pay for capacity when allocated, but donΓÇÖt pay for it when there is no load, such as between projects.

- * Depending on the project, the requirement may be for the best price/performance or the best overall performance. Different scenes and/or rendering applications will have different memory requirements. Some rendering application can leverage GPUs for the best performance or certain features.

-The most common case is for there to be an existing on-premises render farm being managed by a render management application such as PipelineFX Qube, Royal Render, Thinkbox Deadline, or a custom application. The requirement is to extend the on-premises render farm capacity using Azure VMs.

-* Ensure the existing license server is on the virtual network and purchase the additional licenses required to cater for the extra Azure-based capacity.

-Client workstations may be performing rendering, but the rendering load is increasing and it is taking too long to solely use workstation capacity.

-* Deploy an on-premises render manager, such as Royal Render, and configure a hybrid environment to use Azure when further capacity or performance is required. A render manager is specifically tailored for rendering workloads and will include plug-ins for the popular client applications, enabling easy submission of rendering jobs.

-* A custom solution using Azure Batch to allocate and manage the compute capacity as well as providing the job scheduling to run the render jobs.

- Learn how to [use Azure infrastructure and services to extend an existing on-premises render farm](https://azure.microsoft.com/solutions/high-performance-computing/rendering/).

-description: How to use Azure Batch rendering capabilities. Try using the Batch Explorer application, either directly or invoked from a client application plug-in.

-# Using Azure Batch rendering

-> [!WARNING]

-> The rendering VM images and pay-for-use licensing have been [deprecated and will be retired on February 29, 2024](https://azure.microsoft.com/updates/azure-batch-rendering-vm-images-licensing-will-be-retired-on-29-february-2024/). To use Batch for rendering, [a custom VM image and standard application licensing should be used.](batch-rendering-functionality.md#batch-pools-using-custom-vm-images-and-standard-application-licensing)

-There are several ways to use Azure Batch rendering:

-* APIs:

- * Write code using any of the Batch APIs. Developers can integrate Azure Batch capabilities into their existing applications or workflow, whether cloud or based on-premises.

-* Command line tools:

- * The [Azure command line](/cli/azure/) or [PowerShell](/powershell/azure/) can be used to script Batch use.

- * In particular, the [Batch CLI template support](./batch-cli-templates.md) makes it much easier to create pools and submit jobs.

-* Batch Explorer UI:

- * [Batch Explorer](https://github.com/Azure/BatchLabs) is a cross-platform client tool that also allows Batch accounts to be managed and monitored.

- * For each of the rendering applications, a number of pool and job templates are provided that can be used to easily create pools and to submit jobs. A set of templates is listed in the application UI, with the template files being accessed from GitHub.

- * Custom templates can be authored from scratch or the supplied templates from GitHub can be copied and modified.

-* Client application plug-ins:

- * Plug-ins are available that allow Batch rendering to be used from directly within the client design and modeling applications. The plug-ins mainly invoke the Batch Explorer application with contextual information about the current 3D model and include features to help manage assets.

-The best way to try Azure Batch rendering and simplest way for end-users, who are not developers and not Azure experts, is to use the Batch Explorer application, either directly or invoked from a client application plug-in.

-## Using Batch Explorer

-Batch Explorer [downloads are available](https://azure.github.io/BatchExplorer/) for Windows, OSX, and Linux.

-### Using templates to create pools and run jobs

-A comprehensive set of templates is available for use with Batch Explorer that makes it easy to create pools and submit jobs for the various rendering applications without having to specify all the properties required to create pools, jobs, and tasks directly with Batch. The templates available in Batch Explorer are stored and visible in [a GitHub repository](https://github.com/Azure/BatchExplorer-data/tree/master/ncj).

-

-Templates are provided that cater for all the applications present on the Marketplace rendering VM images. For each application multiple templates exist, including pool templates to cater for CPU and GPU pools, Windows and Linux pools; job templates include full frame or tiled Blender rendering and V-Ray distributed rendering. The set of supplied templates will be expanded over time to cater for other Batch capabilities, such as pool auto-scaling.

-It's also possible for custom templates to be produced, from scratch or by modifying the supplied templates. Custom templates can be used by selecting the 'Local templates' item in the 'Gallery' section of Batch Explorer.

-### File system and data movement

-The 'Data' section in Batch Explorer allows files to be copied between a local file system and Azure Storage accounts.

-## Next steps

-* Learn about [using rendering applications with Batch](batch-rendering-applications.md).

-* Learn about [Storage and data movement options for rendering asset and output files](batch-rendering-storage-data-movement.md).

-When you use the Azure Compute Gallery for your custom image, you have control over the operating system type and configuration, as well as the type of data disks. Your Shared Image can include applications and reference data that become available on all the Batch pool nodes as soon as they are provisioned.

-> [!NOTE]

-> Currently, Azure Batch does not support the ΓÇÿTrustedLaunchΓÇÖ feature. You must use the standard security type to create a custom image instead.

->

-> You need to authenticate using Microsoft Entra ID. If you use shared-key-auth, you will get an authentication error.

-If you are creating a new VM for the image, use a first party Azure Marketplace image supported by Batch as the base image for your managed image. Only first party images can be used as a base image.

-> [!NOTE]

-> You need to authenticate using Microsoft Entra ID. If you use shared-key-auth, you will get an authentication error.

-1. Once the node is allocated, use **Connect** to generate user and the RDP file for Windows OR use SSH to for Linux to login to the allocated node and verify.

-Batch node agents aren't automatically upgraded for pools that have non-zero compute nodes. To ensure your Batch pools receive the latest security fixes and updates to the Batch node agent, you need to either resize the pool to zero compute nodes or recreate the pool. It's recommended to monitor the [Batch Node Agent release notes](https://github.com/Azure/Batch/blob/master/changelogs/nodeagent/CHANGELOG.md) to understand changes to new Batch node agent versions. Checking regularly for updates when they were released enables you to plan upgrades to the latest agent version.

-Using a job to run a single task is inefficient. For example, it's more efficient to use a single job containing 1000 tasks rather than creating 100 jobs that contain 10 tasks each. If you used 1000 jobs, each with a single task that would be the least efficient, slowest, and most expensive approach to take.

-> For tasks just submitted to Batch, the DeleteTask API call takes up to 10 minutes to take effect. Before it takes effect, other tasks might be prevented from being scheduled. It's because Batch Scheduler still tries to schedule the tasks just deleted. If you want to delete one task shortly after it's submitted, please terminate the task instead (since the terminate task will take effect immediately). And then delete the task 10 minutes later.

-Tasks can be automatically retried by Batch. There are two types of retries: user-controlled and internal. User-controlled retries are specified by the task's [maxTaskRetryCount](/dotnet/api/microsoft.azure.batch.taskconstraints.maxtaskretrycount). When a program specified in the task exits with a non-zero exit code, the task is retried up to the value of the `maxTaskRetryCount`.

-new data disks may be attached to Batch pools. These data disks attached to compute nodes aren't automatically partitioned, formatted or

-When provisioning [Batch pools in a virtual network](batch-virtual-network.md), ensure that you're closely following the guidelines regarding the use of the BatchNodeManagement.*region* service tag, ports, protocols and direction of the rule. Use of the service tag is highly recommended; don't use underlying Batch service IP addresses as they can change over time. Using Batch service IP addresses directly can cause instability, interruptions, or outages for your Batch pools.

-> [!IMPORTANT]

-> [!IMPORTANT]

-> Managed identities are not updated on existing VMs once a pool has been started. It is recommended to scale the pool down to zero before modifying the identity collection to ensure all VMs

-> have the same set of identities assigned.

-* [Python version 3.7 or later](https://www.python.org/downloads/)

-Azure Communications Gateway's per-region domain names might be as follows, where the `<deployment_id>` subdomain is autogenerated and unique to the deployment:

-* `r1.<deployment_id>.commsgw.azure.com`

-* `r2.<deployment_id>.commsgw.azure.com`

-Choose a DNS label to identify the test customer. The label can be up to 10 characters in length and can only contain letters, numbers, underscores, and dashes. You must not use wildcard subdomains or subdomains with multiple labels. For example, you could allocate the label `test`.

-You use this label to create a subdomain of each per-region domain name for your Azure Communications Gateway. Microsoft Phone System and Azure Communications Gateway use this subdomain to match calls to tenants.

-For example, the `test` label combined with the per-region domain names creates the following deployment-specific domain names:

-Make a note of the label you choose and the corresponding subdomains.

-1. Repeat the previous step for the second customer-specific per-region domain name.

-1. Finish verifying the two customer-specific per-region domain names by following [Add a subdomain to the customer tenant and verify it](/microsoftteams/direct-routing-sbc-multiple-tenants#add-a-subdomain-to-the-customer-tenant-and-verify-it).

-You must allocate six "service verification" test numbers for each of Operator Connect and Teams Phone Mobile. These numbers are used by the Operator Connect and Teams Phone Mobile programs for continuous call testing.

-1. In each **Service Location** section, find the **Hostname** field. This field provides the _per-region domain name_. Your deployment has two service regions and therefore two per-region domain names.

-1. Note down the base domain name and the per-region domain names. You'll need these values in the next steps.

-You most configure your Microsoft 365 tenant with two SIP trunks to Azure Communications Gateway. Each trunk connects to one of the per-region domain names that you found in [Find your Azure Communication Gateway's domain names](#find-your-azure-communication-gateways-domain-names).

-Follow [Connect your Session Border Controller (SBC) to Direct Routing](/microsoftteams/direct-routing-connect-the-sbc), using the following configuration settings.

-|Public internet |❌|✅|✅|- No extra setup<br>- Not recommended for production|

-Set up your network as in the following diagram and configure it in accordance with any network connectivity specifications for your chosen communications services. Your network must have two sites with cross-connect functionality. For more information on the reliability design for Azure Communications Gateway, see [Reliability in Azure Communications Gateway](reliability-communications-gateway.md).

- |The management Azure region: the region in which your monitoring and billing data is processed. We recommend that you select a region near or colocated with the two regions for handling call traffic. |**Instance details: Region**

- |The Azure regions to use for call traffic. |**Service Region One/Two: Region**|

- |The IPv4 address used by Azure Communications Gateway to contact your network from this region. |**Service Region One/Two: Operator IP address**|

- - Recommended for Operator Connect and Teams Phone Mobile because it enables flow-through API-based provisioning of your customers both on Azure Communications Gateway and in the Operator Connect environment. This enables additional functionality to be provided by Azure Communications Gateway, such as injecting custom SIP headers, while also fulfilling the requirement from the the Operator Connect and Teams Phone Mobile programs for you to use APIs for provisioning customers in the Operator Connect environment. For more information, see [Provisioning and Operator Connect APIs](interoperability-operator-connect.md#provisioning-and-operator-connect-apis).

-After you've started using Azure Communications Gateway, you can use Cost Management features to set budgets and monitor costs. You can also review forecasted costs and identify spending trends to identify areas where you might want to act.

-Costs for Azure Communications Gateway are only a portion of the monthly costs in your Azure bill. Although this article explains how to plan for and manage costs for Azure Communications Gateway, your Azure bill includes all services and resources used in your Azure subscription, including third-party Azure services.

-When you deploy Azure Communications Gateway, you're charged for how you use the voice features of the product. The charges are based on the number of users assigned to the platform by a series of Azure Communications Gateway meters. The meters include:

-For example, if you have 28,000 users assigned to the deployment each month, you're charged for:

-* The service availability fee for each hour in the month

-* 24,001 users in the 1000-25000 tier

-* 3000 users in the 25000+ tier

-If you've arranged any custom work with Microsoft, you might be charged an extra fee for that work. That fee isn't included in these meters.

-You can pay for Azure Communications Gateway charges with your Azure Prepayment credit. However, you can't use Azure Prepayment credit to pay for charges for third-party products and services including those from the Azure Marketplace.

-Access to Azure Communications Gateway is restricted. When you've completed the previous steps in this article, contact your onboarding team and ask them to enable your subscription. If you don't already have an onboarding team, contact azcog-enablement@microsoft.com with your Azure subscription ID and contact details.

-Wait for confirmation that Azure Communications Gateway is enabled before moving on to the next step.

-## Configure Microsoft Azure Peering Service Voice or ExpressRoute

-Connect your network to Azure:

-Each Azure Communications Gateway deployment consists of three separate regions: a Management Region and two Service Regions. This article describes the two different region types and their distinct redundancy models. It covers both regional reliability with availability zones and cross-region reliability with disaster recovery. For a more detailed overview of reliability in Azure, see [Azure reliability](/azure/architecture/framework/resiliency/overview).

-Service regions contain the voice and API infrastructure used for handling traffic between your network and your chosen communications services. Each instance of Azure Communications Gateway consists of two service regions that are deployed in an active-active mode (as required by the Operator Connect and Teams Phone Mobile programs). Fast failover between the service regions is provided at the infrastructure/IP level and at the application (SIP/RTP/HTTP) level.

-> You must always have two service regions, even if one of the service regions chosen is in a single-region Azure Geography (for example, Qatar). If you choose a single-region Azure Geography, choose a second Azure region in a different Azure Geography.

-These service regions are identical in operation and provide resiliency to both Zone and Regional failures. Each service region can carry 100% of the traffic using the Azure Communications Gateway instance. As such, end users should still be able to make and receive calls successfully during any Zone or Regional downtime.

-We expect your network to have two geographically redundant sites. Each site should be paired with an Azure Communications Gateway region. The redundancy model relies on cross-connectivity between your network and Azure Communications Gateway service regions.

-> - Detect when MCP in a region is unavailable, mark the targets for that region's SRV record as unavailable, and retry periodically to determine when the region is available again. MCP does not respond to SIP OPTIONS.

-A single deployment of Azure Communications Gateway is designed to handle your traffic within a geographic area. Deploy both service regions within the same geographic area (for example North America). This model ensures that latency on voice calls remains within the limits required by the Operator Connect and Teams Phone Mobile programs.

-|.NET v3 SDK |[>= `3.33.0-preview`](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/3.33.0-preview) |This feature is on by default if you're using a supported preview SDK version. You can disable tracing by setting `IsDistributedTracingEnabled = false` in `CosmosClientOptions`. |

-In addition to getting diagnostic logs for failed requests, point operations that take over 100 ms and query operations that take over 500 ms also generate diagnostics. You can configure the log level to control which diagnostics logs you receive.

-|Warning | Logs for errors and high latency requests. |

-The offer provides them with a discount up to 50% compared to pay-as-you-go pricing. The savings doesn't include operating system costs. Actual savings may vary based on instance type or usage.

-If you have purchased the one-year Azure Reserved Virtual Machine Instances for the qualified VMs in Sweden Central between September 1, 2023, and February 29, 2024 you'll continue to get the discount throughout the one-year term, even if the offer is canceled.

-description: Learn about the basics of the access control model of Azure Data Lake Storage Gen1, which derives from HDFS.

-# Access control in Azure Data Lake Storage Gen1

-Azure Data Lake Storage Gen1 implements an access control model that derives from HDFS, which in turn derives from the POSIX access control model. This article summarizes the basics of the access control model for Data Lake Storage Gen1.

-## Access control lists on files and folders

-There are two kinds of access control lists (ACLs), **Access ACLs** and **Default ACLs**.

-* **Access ACLs**: These control access to an object. Files and folders both have Access ACLs.

-* **Default ACLs**: A "template" of ACLs associated with a folder that determine the Access ACLs for any child items that are created under that folder. Files do not have Default ACLs.

-Both Access ACLs and Default ACLs have the same structure.

-> [!NOTE]

-> Changing the Default ACL on a parent does not affect the Access ACL or Default ACL of child items that already exist.

->

->

-## Permissions

-The permissions on a filesystem object are **Read**, **Write**, and **Execute**, and they can be used on files and folders as shown in the following table:

-| | File | Folder |

-||-|-|

-| **Read (R)** | Can read the contents of a file | Requires **Read** and **Execute** to list the contents of the folder|

-| **Write (W)** | Can write or append to a file | Requires **Write** and **Execute** to create child items in a folder |

-| **Execute (X)** | Does not mean anything in the context of Data Lake Storage Gen1 | Required to traverse the child items of a folder |

-### Short forms for permissions

-**RWX** is used to indicate **Read + Write + Execute**. A more condensed numeric form exists in which **Read=4**, **Write=2**, and **Execute=1**, the sum of which represents the permissions. Following are some examples.

-| Numeric form | Short form | What it means |

-|--|||

-| 7 | `RWX` | Read + Write + Execute |

-| 5 | `R-X` | Read + Execute |

-| 4 | `R--` | Read |

-| 0 | `` | No permissions |

-### Permissions do not inherit

-In the POSIX-style model that's used by Data Lake Storage Gen1, permissions for an item are stored on the item itself. In other words, permissions for an item cannot be inherited from the parent items.

-## Common scenarios related to permissions

-Following are some common scenarios to help you understand which permissions are needed to perform certain operations on a Data Lake Storage Gen1 account.

-| Operation | Object | / | Seattle/ | Portland/ | Data.txt |

-|--||--||-|-|

-| Read | Data.txt | `--X` | `--X` | `--X` | `R--` |

-| Append to | Data.txt | `--X` | `--X` | `--X` | `-W-` |

-| Delete | Data.txt | `--X` | `--X` | `-WX` | `` |

-| Create | Data.txt | `--X` | `--X` | `-WX` | `` |

-| List | / | `R-X` | `` | `` | `` |

-| List | /Seattle/ | `--X` | `R-X` | `` | `` |

-| List | /Seattle/Portland/ | `--X` | `--X` | `R-X` | `` |

-> [!NOTE]

-> Write permissions on the file are not required to delete it as long as the previous two conditions are true.

->

->

-## Users and identities

-Every file and folder has distinct permissions for these identities:

-* The owning user

-* The owning group

-* Named users

-* Named groups

-* All other users

-The identities of users and groups are Microsoft Entra identities. So unless otherwise noted, a "user," in the context of Data Lake Storage Gen1, can either mean a Microsoft Entra user or a Microsoft Entra security group.

-### The super-user

-A super-user has the most rights of all the users in the Data Lake Storage Gen1 account. A super-user:

-* Has RWX Permissions to **all** files and folders.

-* Can change the permissions on any file or folder.

-* Can change the owning user or owning group of any file or folder.

-All users that are part of the **Owners** role for a Data Lake Storage Gen1 account are automatically a super-user.

-### The owning user

-The user who created the item is automatically the owning user of the item. An owning user can:

-* Change the permissions of a file that is owned.

-* Change the owning group of a file that is owned, as long as the owning user is also a member of the target group.

-> [!NOTE]

-> The owning user *cannot* change the owning user of a file or folder. Only super-users can change the owning user of a file or folder.

->

->

-### The owning group

-**Background**

-In the POSIX ACLs, every user is associated with a "primary group." For example, user "alice" might belong to the "finance" group. Alice might also belong to multiple groups, but one group is always designated as her primary group. In POSIX, when Alice creates a file, the owning group of that file is set to her primary group, which in this case is "finance." The owning group otherwise behaves similarly to assigned permissions for other users/groups.

-Because there is no ΓÇ£primary groupΓÇ¥ associated to users in Data Lake Storage Gen1, the owning group is assigned as below.

-**Assigning the owning group for a new file or folder**

-* **Case 1**: The root folder "/". This folder is created when a Data Lake Storage Gen1 account is created. In this case, the owning group is set to an all-zero GUID. This value does not permit any access. It is a placeholder until such time a group is assigned.

-* **Case 2** (Every other case): When a new item is created, the owning group is copied from the parent folder.

-**Changing the owning group**

-The owning group can be changed by:

-* Any super-users.

-* The owning user, if the owning user is also a member of the target group.

-> [!NOTE]

-> The owning group *cannot* change the ACLs of a file or folder.

->

-> For accounts created on or before September 2018, the owning group was set to the user who created the account in the case of the root folder for **Case 1**, above. A single user account is not valid for providing permissions via the owning group, thus no permissions are granted by this default setting. You can assign this permission to a valid user group.

-## Access check algorithm

-The following pseudocode represents the access check algorithm for Data Lake Storage Gen1 accounts.

-```

-def access_check( user, desired_perms, path ) :

- # access_check returns true if user has the desired permissions on the path, false otherwise

- # user is the identity that wants to perform an operation on path

- # desired_perms is a simple integer with values from 0 to 7 ( R=4, W=2, X=1). User desires these permissions

- # path is the file or folder

- # Note: the "sticky bit" is not illustrated in this algorithm

-

-# Handle super users.

- if (is_superuser(user)) :

- return True

- # Handle the owning user. Note that mask IS NOT used.

- entry = get_acl_entry( path, OWNER )

- if (user == entry.identity)

- return ( (desired_perms & entry.permissions) == desired_perms )

- # Handle the named users. Note that mask IS used.

- entries = get_acl_entries( path, NAMED_USER )

- for entry in entries:

- if (user == entry.identity ) :

- mask = get_mask( path )

- return ( (desired_perms & entry.permmissions & mask) == desired_perms)

- # Handle named groups and owning group

- member_count = 0

- perms = 0

- entries = get_acl_entries( path, NAMED_GROUP | OWNING_GROUP )

- for entry in entries:

- if (user_is_member_of_group(user, entry.identity)) :

- member_count += 1

- perms | = entry.permissions

- if (member_count>0) :

- return ((desired_perms & perms & mask ) == desired_perms)

-

- # Handle other

- perms = get_perms_for_other(path)

- mask = get_mask( path )

- return ( (desired_perms & perms & mask ) == desired_perms)

-```

-### The mask

-As illustrated in the Access Check Algorithm, the mask limits access for **named users**, the **owning group**, and **named groups**.

-> [!NOTE]

-> For a new Data Lake Storage Gen1 account, the mask for the Access ACL of the root folder ("/") defaults to RWX.

->

->

-### The sticky bit

-The sticky bit is a more advanced feature of a POSIX filesystem. In the context of Data Lake Storage Gen1, it is unlikely that the sticky bit will be needed. In summary, if the sticky bit is enabled on a folder, a child item can only be deleted or renamed by the child item's owning user.

-The sticky bit is not shown in the Azure portal.

-## Default permissions on new files and folders

-When a new file or folder is created under an existing folder, the Default ACL on the parent folder determines:

-### umask

-When creating a file or folder, umask is used to modify how the default ACLs are set on the child item. umask is a 9-bit value on parent folders that contains an RWX value for **owning user**, **owning group**, and **other**.

-The umask for Azure Data Lake Storage Gen1 is a constant value set to 007. This value translates to

-| umask component | Numeric form | Short form | Meaning |

-||--|||

-| umask.owning_user | 0 | `` | For owning user, copy the parent's Default ACL to the child's Access ACL |

-| umask.owning_group | 0 | `` | For owning group, copy the parent's Default ACL to the child's Access ACL |

-| umask.other | 7 | `RWX` | For other, remove all permissions on the child's Access ACL |

-The umask value used by Azure Data Lake Storage Gen1 effectively means that the value for other is never transmitted by default on new children - regardless of what the Default ACL indicates.

-The following pseudocode shows how the umask is applied when creating the ACLs for a child item.

-```

-def set_default_acls_for_new_child(parent, child):

- child.acls = []

- for entry in parent.acls :

- new_entry = None

- if (entry.type == OWNING_USER) :

- new_entry = entry.clone(perms = entry.perms & (~umask.owning_user))

- elif (entry.type == OWNING_GROUP) :

- new_entry = entry.clone(perms = entry.perms & (~umask.owning_group))

- elif (entry.type == OTHER) :

- new_entry = entry.clone(perms = entry.perms & (~umask.other))

- else :

- new_entry = entry.clone(perms = entry.perms )

- child_acls.add( new_entry )

-```

-## Common questions about ACLs in Data Lake Storage Gen1

-### Do I have to enable support for ACLs?

-No. Access control via ACLs is always on for a Data Lake Storage Gen1 account.

-### Which permissions are required to recursively delete a folder and its contents?

-* The parent folder must have **Write + Execute** permissions.

-* The folder to be deleted, and every folder within it, requires **Read + Write + Execute** permissions.

-> [!NOTE]

-> You do not need Write permissions to delete files in folders. Also, the root folder "/" can **never** be deleted.

->

->

-### Who is the owner of a file or folder?

-The creator of a file or folder becomes the owner.

-### Which group is set as the owning group of a file or folder at creation?

-The owning group is copied from the owning group of the parent folder under which the new file or folder is created.

-### I am the owning user of a file but I donΓÇÖt have the RWX permissions I need. What do I do?

-The owning user can change the permissions of the file to give themselves any RWX permissions they need.

-### When I look at ACLs in the Azure portal I see user names but through APIs, I see GUIDs, why is that?

-Entries in the ACLs are stored as GUIDs that correspond to users in Microsoft Entra ID. The APIs return the GUIDs as is. The Azure portal tries to make ACLs easier to use by translating the GUIDs into friendly names when possible.

-### Why do I sometimes see GUIDs in the ACLs when I'm using the Azure portal?

-A GUID is shown when the user doesn't exist in Microsoft Entra anymore. Usually this happens when the user has left the company or if their account has been deleted in Microsoft Entra ID. Also, ensure that you're using the right ID for setting ACLs (details in question below).

-### When using service principal, what ID should I use to set ACLs?

-On the Azure Portal, go to **Microsoft Entra ID -> Enterprise applications** and select your application. The **Overview** tab should display an Object ID and this is what should be used when adding ACLs for data access (and not Application Id).

-### Does Data Lake Storage Gen1 support inheritance of ACLs?

-No, but Default ACLs can be used to set ACLs for child files and folder newly created under the parent folder.

-### What are the limits for ACL entries on files and folders?

-32 ACLs can be set per file and per directory. Access and default ACLs each have their own 32 ACL entry limit. Use security groups for ACL assignments if possible. By using groups, you're less likely to exceed the maximum number of ACL entries per file or directory.

-### Where can I learn more about POSIX access control model?

-* [POSIX Access Control Lists on Linux](https://www.linux.com/news/posix-acls-linux)

-* [HDFS permission guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html)

-* [POSIX FAQ](https://www.opengroup.org/austin/papers/posix_faq.html)

-* [POSIX 1003.1 2008](https://standards.ieee.org/wp-content/uploads/import/documents/interpretations/1003.1-2008_interp.pdf)

-* [POSIX 1003.1 2013](https://pubs.opengroup.org/onlinepubs/9699919799.2013edition/)

-* [POSIX 1003.1 2016](https://pubs.opengroup.org/onlinepubs/9699919799.2016edition/)

-* [POSIX ACL on Ubuntu](https://help.ubuntu.com/community/FilePermissionsACLs)

-## See also

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-description: Learn how to use Azure Data Lake Storage Gen1 to capture data received by Azure Event Hubs. Begin by verifying the prerequisites.

-# Use Azure Data Lake Storage Gen1 to capture data from Event Hubs

-Learn how to use Azure Data Lake Storage Gen1 to capture data received by Azure Event Hubs.

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **An Azure Data Lake Storage Gen1 account**. For instructions on how to create one, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md).

-* **An Event Hubs namespace**. For instructions, see [Create an Event Hubs namespace](../event-hubs/event-hubs-create.md#create-an-event-hubs-namespace). Make sure the Data Lake Storage Gen1 account and the Event Hubs namespace are in the same Azure subscription.

-## Assign permissions to Event Hubs

-In this section, you create a folder within the account where you want to capture the data from Event Hubs. You also assign permissions to Event Hubs so that it can write data into a Data Lake Storage Gen1 account.

-1. Open the Data Lake Storage Gen1 account where you want to capture data from Event Hubs and then click on **Data Explorer**.

- 

-1. Click **New Folder** and then enter a name for folder where you want to capture the data.

- 

-1. Assign permissions at the root of Data Lake Storage Gen1.

- a. Click **Data Explorer**, select the root of the Data Lake Storage Gen1 account, and then click **Access**.

- 

- b. Under **Access**, click **Add**, click **Select User or Group**, and then search for `Microsoft.EventHubs`.

- 

-

- Click **Select**.

- c. Under **Assign Permissions**, click **Select Permissions**. Set **Permissions** to **Execute**. Set **Add to** to **This folder and all children**. Set **Add as** to **An access permission entry and a default permission entry**.

- > [!IMPORTANT]

- > When creating a new folder hierarchy for capturing data received by Azure Event Hubs, this is an easy way to ensure access to the destination folder. However, adding permissions to all children of a top level folder with many child files and folders may take a long time. If your root folder contains a large number of files and folders, it may be faster to add **Execute** permissions for `Microsoft.EventHubs` individually to each folder in the path to your final destination folder.

- 

- Click **OK**.

-1. Assign permissions for the folder under the Data Lake Storage Gen1 account where you want to capture data.

- a. Click **Data Explorer**, select the folder in the Data Lake Storage Gen1 account, and then click **Access**.

- 

- b. Under **Access**, click **Add**, click **Select User or Group**, and then search for `Microsoft.EventHubs`.

- 

-

- Click **Select**.

- c. Under **Assign Permissions**, click **Select Permissions**. Set **Permissions** to **Read, Write,** and **Execute**. Set **Add to** to **This folder and all children**. Finally, set **Add as** to **An access permission entry and a default permission entry**.

- 

-

- Click **OK**.

-## Configure Event Hubs to capture data to Data Lake Storage Gen1

-In this section, you create an Event Hub within an Event Hubs namespace. You also configure the Event Hub to capture data to an Azure Data Lake Storage Gen1 account. This section assumes that you have already created an Event Hubs namespace.

-1. From the **Overview** pane of the Event Hubs namespace, click **+ Event Hub**.

- 

-1. Provide the following values to configure Event Hubs to capture data to Data Lake Storage Gen1.

- 

- a. Provide a name for the Event Hub.

-

- b. For this tutorial, set **Partition Count** and **Message Retention** to the default values.

-

- c. Set **Capture** to **On**. Set the **Time Window** (how frequently to capture) and **Size Window** (data size to capture).

-

- d. For **Capture Provider**, select **Azure Data Lake Store** and then select the Data Lake Storage Gen1 account you created earlier. For **Data Lake Path**, enter the name of the folder you created in the Data Lake Storage Gen1 account. You only need to provide the relative path to the folder.

- e. Leave the **Sample capture file name formats** to the default value. This option governs the folder structure that is created under the capture folder.

- f. Click **Create**.

-## Test the setup

-You can now test the solution by sending data to the Azure Event Hub. Follow the instructions at [Send events to Azure Event Hubs](../event-hubs/event-hubs-dotnet-framework-getstarted-send.md). Once you start sending the data, you see the data reflected in Data Lake Storage Gen1 using the folder structure you specified. For example, you see a folder structure, as shown in the following screenshot, in your Data Lake Storage Gen1 account.

-

-> [!NOTE]

-> Even if you do not have messages coming into Event Hubs, Event Hubs writes empty files with just the headers into the Data Lake Storage Gen1 account. The files are written at the same time interval that you provided while creating the Event Hubs.

->

->

-## Analyze data in Data Lake Storage Gen1

-Once the data is in Data Lake Storage Gen1, you can run analytical jobs to process and crunch the data. See [USQL Avro Example](https://github.com/Azure/usql/tree/master/Examples/AvroExamples) on how to do this using Azure Data Lake Analytics.

-

-## See also

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Copy data from Azure Storage Blobs to Data Lake Storage Gen1](data-lake-store-copy-data-azure-storage-blob.md)

-description: Learn the best practices about data ingestion, date security, and performance related to using Azure Data Lake Storage Gen1 (previously known as Azure Data Lake Store)

-# Best practices for using Azure Data Lake Storage Gen1

-In this article, you learn about best practices and considerations for working with Azure Data Lake Storage Gen1. This article provides information around security, performance, resiliency, and monitoring for Data Lake Storage Gen1. Before Data Lake Storage Gen1, working with truly big data in services like Azure HDInsight was complex. You had to shard data across multiple Blob storage accounts so that petabyte storage and optimal performance at that scale could be achieved. With Data Lake Storage Gen1, most of the hard limits for size and performance are removed. However, there are still some considerations that this article covers so that you can get the best performance with Data Lake Storage Gen1.

-## Security considerations

-Azure Data Lake Storage Gen1 offers POSIX access controls and detailed auditing for Microsoft Entra users, groups, and service principals. These access controls can be set to existing files and folders. The access controls can also be used to create defaults that can be applied to new files or folders. When permissions are set to existing folders and child objects, the permissions need to be propagated recursively on each object. If there are large number of files, propagating the permissions can take a long time. The time taken can range between 30-50 objects processed per second. Hence, plan the folder structure and user groups appropriately. Otherwise, it can cause unanticipated delays and issues when you work with your data.

-Assume you have a folder with 100,000 child objects. If you take the lower bound of 30 objects processed per second, to update the permission for the whole folder could take an hour. More details on Data Lake Storage Gen1 ACLs are available at [Access control in Azure Data Lake Storage Gen1](data-lake-store-access-control.md). For improved performance on assigning ACLs recursively, you can use the Azure Data Lake Command-Line Tool. The tool creates multiple threads and recursive navigation logic to quickly apply ACLs to millions of files. The tool is available for Linux and Windows, and the [documentation](https://github.com/Azure/data-lake-adlstool) and [downloads](https://aka.ms/adlstool-download) for this tool can be found on GitHub. These same performance improvements can be enabled by your own tools written with the Data Lake Storage Gen1 [.NET](data-lake-store-data-operations-net-sdk.md) and [Java](data-lake-store-get-started-java-sdk.md) SDKs.

-### Use security groups versus individual users

-When working with big data in Data Lake Storage Gen1, most likely a service principal is used to allow services such as Azure HDInsight to work with the data. However, there might be cases where individual users need access to the data as well. In such cases, you must use Microsoft Entra ID [security groups](data-lake-store-secure-data.md#create-security-groups-in-azure-active-directory) instead of assigning individual users to folders and files.

-Once a security group is assigned permissions, adding or removing users from the group doesnΓÇÖt require any updates to Data Lake Storage Gen1. This also helps ensure you don't exceed the limit of [32 Access and Default ACLs](../azure-resource-manager/management/azure-subscription-service-limits.md#data-lake-storage-limits) (this includes the four POSIX-style ACLs that are always associated with every file and folder: [the owning user](data-lake-store-access-control.md#the-owning-user), [the owning group](data-lake-store-access-control.md#the-owning-group), [the mask](data-lake-store-access-control.md#the-mask), and other).

-### Security for groups

-As discussed, when users need access to Data Lake Storage Gen1, itΓÇÖs best to use Microsoft Entra security groups. Some recommended groups to start with might be **ReadOnlyUsers**, **WriteAccessUsers**, and **FullAccessUsers** for the root of the account, and even separate ones for key subfolders. If there are any other anticipated groups of users that might be added later, but have not been identified yet, you might consider creating dummy security groups that have access to certain folders. Using security group ensures that later you do not need a long processing time for assigning new permissions to thousands of files.

-### Security for service principals

-Microsoft Entra service principals are typically used by services like Azure HDInsight to access data in Data Lake Storage Gen1. Depending on the access requirements across multiple workloads, there might be some considerations to ensure security inside and outside of the organization. For many customers, a single Microsoft Entra service principal might be adequate, and it can have full permissions at the root of the Data Lake Storage Gen1 account. Other customers might require multiple clusters with different service principals where one cluster has full access to the data, and another cluster with only read access. As with the security groups, you might consider making a service principal for each anticipated scenario (read, write, full) once a Data Lake Storage Gen1 account is created.

-### Enable the Data Lake Storage Gen1 firewall with Azure service access

-Data Lake Storage Gen1 supports the option of turning on a firewall and limiting access only to Azure services, which is recommended for a smaller attack vector from outside intrusions. Firewall can be enabled on the Data Lake Storage Gen1 account in the Azure portal via the **Firewall** > **Enable Firewall (ON)** > **Allow access to Azure services** options.

-

-Once firewall is enabled, only Azure services such as HDInsight, Data Factory, Azure Synapse Analytics, etc. have access to Data Lake Storage Gen1. Due to the internal network address translation used by Azure, the Data Lake Storage Gen1 firewall does not support restricting specific services by IP and is only intended for restrictions of endpoints outside of Azure, such as on-premises.

-## Performance and scale considerations

-One of the most powerful features of Data Lake Storage Gen1 is that it removes the hard limits on data throughput. Removing the limits enables customers to grow their data size and accompanied performance requirements without needing to shard the data. One of the most important considerations for optimizing Data Lake Storage Gen1 performance is that it performs the best when given parallelism.

-### Improve throughput with parallelism

-Consider giving 8-12 threads per core for the most optimal read/write throughput. This is due to blocking reads/writes on a single thread, and more threads can allow higher concurrency on the VM. To ensure that levels are healthy and parallelism can be increased, be sure to monitor the VMΓÇÖs CPU utilization.

-### Avoid small file sizes

-POSIX permissions and auditing in Data Lake Storage Gen1 comes with an overhead that becomes apparent when working with numerous small files. As a best practice, you must batch your data into larger files versus writing thousands or millions of small files to Data Lake Storage Gen1. Avoiding small file sizes can have multiple benefits, such as:

-* Lowering the authentication checks across multiple files

-* Reduced open file connections

-* Faster copying/replication

-* Fewer files to process when updating Data Lake Storage Gen1 POSIX permissions

-Depending on what services and workloads are using the data, a good size to consider for files is 256 MB or greater. If the file sizes cannot be batched when landing in Data Lake Storage Gen1, you can have a separate compaction job that combines these files into larger ones. For more information and recommendation on file sizes and organizing the data in Data Lake Storage Gen1, see [Structure your data set](data-lake-store-performance-tuning-guidance.md#structure-your-data-set).

-### Large file sizes and potential performance impact

-Although Data Lake Storage Gen1 supports large files up to petabytes in size, for optimal performance and depending on the process reading the data, it might not be ideal to go above 2 GB on average. For example, when using **Distcp** to copy data between locations or different storage accounts, files are the finest level of granularity used to determine map tasks. So, if you are copying 10 files that are 1 TB each, at most 10 mappers are allocated. Also, if you have lots of files with mappers assigned, initially the mappers work in parallel to move large files. However, as the job starts to wind down only a few mappers remain allocated and you can be stuck with a single mapper assigned to a large file. Microsoft has submitted improvements to Distcp to address this issue in future Hadoop versions.

-Another example to consider is when using Azure Data Lake Analytics with Data Lake Storage Gen1. Depending on the processing done by the extractor, some files that cannot be split (for example, XML, JSON) could suffer in performance when greater than 2 GB. In cases where files can be split by an extractor (for example, CSV), large files are preferred.

-### Capacity plan for your workload

-Azure Data Lake Storage Gen1 removes the hard IO throttling limits that are placed on Blob storage accounts. However, there are still soft limits that need to be considered. The default ingress/egress throttling limits meet the needs of most scenarios. If your workload needs to have the limits increased, work with Microsoft support. Also, look at the limits during the proof-of-concept stage so that IO throttling limits are not hit during production. If that happens, it might require waiting for a manual increase from the Microsoft engineering team. If IO throttling occurs, Azure Data Lake Storage Gen1 returns an error code of 429, and ideally should be retried with an appropriate exponential backoff policy.

-### Optimize ΓÇ£writesΓÇ¥ with the Data Lake Storage Gen1 driver buffer

-To optimize performance and reduce IOPS when writing to Data Lake Storage Gen1 from Hadoop, perform write operations as close to the Data Lake Storage Gen1 driver buffer size as possible. Try not to exceed the buffer size before flushing, such as when streaming using Apache Storm or Spark streaming workloads. When writing to Data Lake Storage Gen1 from HDInsight/Hadoop, it is important to know that Data Lake Storage Gen1 has a driver with a 4-MB buffer. Like many file system drivers, this buffer can be manually flushed before reaching the 4-MB size. If not, it is immediately flushed to storage if the next write exceeds the bufferΓÇÖs maximum size. Where possible, you must avoid an overrun or a significant underrun of the buffer when syncing/flushing policy by count or time window.

-## Resiliency considerations

-When architecting a system with Data Lake Storage Gen1 or any cloud service, you must consider your availability requirements and how to respond to potential interruptions in the service. An issue could be localized to the specific instance or even region-wide, so having a plan for both is important. Depending on the **recovery time objective** and the **recovery point objective** SLAs for your workload, you might choose a more or less aggressive strategy for high availability and disaster recovery.

-### High availability and disaster recovery

-High availability (HA) and disaster recovery (DR) can sometimes be combined together, although each has a slightly different strategy, especially when it comes to data. Data Lake Storage Gen1 already handles 3x replication under the hood to guard against localized hardware failures. However, since replication across regions is not built in, you must manage this yourself. When building a plan for HA, in the event of a service interruption the workload needs access to the latest data as quickly as possible by switching over to a separately replicated instance locally or in a new region.

-In a DR strategy, to prepare for the unlikely event of a catastrophic failure of a region, it is also important to have data replicated to a different region. This data might initially be the same as the replicated HA data. However, you must also consider your requirements for edge cases such as data corruption where you may want to create periodic snapshots to fall back to. Depending on the importance and size of the data, consider rolling delta snapshots of 1-, 6-, and 24-hour periods on the local and/or secondary store, according to risk tolerances.

-For data resiliency with Data Lake Storage Gen1, it is recommended to geo-replicate your data to a separate region with a frequency that satisfies your HA/DR requirements, ideally every hour. This frequency of replication minimizes massive data movements that can have competing throughput needs with the main system and a better recovery point objective (RPO). Additionally, you should consider ways for the application using Data Lake Storage Gen1 to automatically fail over to the secondary account through monitoring triggers or length of failed attempts, or at least send a notification to admins for manual intervention. Keep in mind that there is tradeoff of failing over versus waiting for a service to come back online. If the data hasn't finished replicating, a failover could cause potential data loss, inconsistency, or complex merging of the data.

-Below are the top three recommended options for orchestrating replication between Data Lake Storage Gen1 accounts, and key differences between each of them.

-| |Distcp |Azure Data Factory |AdlCopy |

-|||||

-|**Scale limits** | Bounded by worker nodes | Limited by Max Cloud Data Movement units | Bound by Analytics units |

-|**Supports copying deltas** | Yes | No | No |

-|**Built-in orchestration** | No (use Oozie Airflow or cron jobs) | Yes | No (Use Azure Automation or Windows Task Scheduler) |

-|**Supported file systems** | ADL, HDFS, WASB, S3, GS, CFS |Numerous, see [Connectors](../data-factory/connector-azure-blob-storage.md). | ADL to ADL, WASB to ADL (same region only) |

-|**OS support** |Any OS running Hadoop | N/A | Windows 10 |

-### Use Distcp for data movement between two locations

-Short for distributed copy, Distcp is a Linux command-line tool that comes with Hadoop and provides distributed data movement between two locations. The two locations can be Data Lake Storage Gen1, HDFS, WASB, or S3. This tool uses MapReduce jobs on a Hadoop cluster (for example, HDInsight) to scale out on all the nodes. Distcp is considered the fastest way to move big data without special network compression appliances. Distcp also provides an option to only update deltas between two locations, handles automatic retries, as well as dynamic scaling of compute. This approach is incredibly efficient when it comes to replicating things like Hive/Spark tables that can have many large files in a single directory and you only want to copy over the modified data. For these reasons, Distcp is the most recommended tool for copying data between big data stores.

-Copy jobs can be triggered by Apache Oozie workflows using frequency or data triggers, as well as Linux cron jobs. For intensive replication jobs, it is recommended to spin up a separate HDInsight Hadoop cluster that can be tuned and scaled specifically for the copy jobs. This ensures that copy jobs do not interfere with critical jobs. If running replication on a wide enough frequency, the cluster can even be taken down between each job. If failing over to secondary region, make sure that another cluster is also spun up in the secondary region to replicate new data back to the primary Data Lake Storage Gen1 account once it comes back up. For examples of using Distcp, see [Use Distcp to copy data between Azure Storage Blobs and Data Lake Storage Gen1](data-lake-store-copy-data-wasb-distcp.md).

-### Use Azure Data Factory to schedule copy jobs

-Azure Data Factory can also be used to schedule copy jobs using a **Copy Activity**, and can even be set up on a frequency via the **Copy Wizard**. Keep in mind that Azure Data Factory has a limit of cloud data movement units (DMUs), and eventually caps the throughput/compute for large data workloads. Additionally, Azure Data Factory currently does not offer delta updates between Data Lake Storage Gen1 accounts, so folders like Hive tables would require a complete copy to replicate. Refer to the [Copy Activity tuning guide](../data-factory/copy-activity-performance.md) for more information on copying with Data Factory.

-### AdlCopy

-AdlCopy is a Windows command-line tool that allows you to copy data between two Data Lake Storage Gen1 accounts only within the same region. The AdlCopy tool provides a standalone option or the option to use an Azure Data Lake Analytics account to run your copy job. Though it was originally built for on-demand copies as opposed to a robust replication, it provides another option to do distributed copying across Data Lake Storage Gen1 accounts within the same region. For reliability, itΓÇÖs recommended to use the premium Data Lake Analytics option for any production workload. The standalone version can return busy responses and has limited scale and monitoring.

-Like Distcp, the AdlCopy needs to be orchestrated by something like Azure Automation or Windows Task Scheduler. As with Data Factory, AdlCopy does not support copying only updated files, but recopies and overwrite existing files. For more information and examples of using AdlCopy, see [Copy data from Azure Storage Blobs to Data Lake Storage Gen1](data-lake-store-copy-data-azure-storage-blob.md).

-## Monitoring considerations

-Data Lake Storage Gen1 provides detailed diagnostic logs and auditing. Data Lake Storage Gen1 provides some basic metrics in the Azure portal under the Data Lake Storage Gen1 account and in Azure Monitor. Availability of Data Lake Storage Gen1 is displayed in the Azure portal. However, this metric is refreshed every seven minutes and cannot be queried through a publicly exposed API. To get the most up-to-date availability of a Data Lake Storage Gen1 account, you must run your own synthetic tests to validate availability. Other metrics such as total storage utilization, read/write requests, and ingress/egress can take up to 24 hours to refresh. So, more up-to-date metrics must be calculated manually through Hadoop command-line tools or aggregating log information. The quickest way to get the most recent storage utilization is running this HDFS command from a Hadoop cluster node (for example, head node):

-```console

-hdfs dfs -du -s -h adl://<adlsg1_account_name>.azuredatalakestore.net:443/

-```

-### Export Data Lake Storage Gen1 diagnostics

-One of the quickest ways to get access to searchable logs from Data Lake Storage Gen1 is to enable log shipping to **Log Analytics** under the **Diagnostics** blade for the Data Lake Storage Gen1 account. This provides immediate access to incoming logs with time and content filters, along with alerting options (email/webhook) triggered within 15-minute intervals. For instructions, see [Accessing diagnostic logs for Azure Data Lake Storage Gen1](data-lake-store-diagnostic-logs.md).

-For more real-time alerting and more control on where to land the logs, consider exporting logs to Azure EventHub where content can be analyzed individually or over a time window in order to submit real-time notifications to a queue. A separate application such as a [Logic App](../connectors/connectors-create-api-azure-event-hubs.md) can then consume and communicate the alerts to the appropriate channel, as well as submit metrics to monitoring tools like NewRelic, Datadog, or AppDynamics. Alternatively, if you are using a third-party tool such as ElasticSearch, you can export the logs to Blob Storage and use the [Azure Logstash plugin](https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash/logstash-input-azureblob) to consume the data into your Elasticsearch, Kibana, and Logstash (ELK) stack.

-### Turn on debug-level logging in HDInsight

-If Data Lake Storage Gen1 log shipping is not turned on, Azure HDInsight also provides a way to turn on [client-side logging for Data Lake Storage Gen1](data-lake-store-performance-tuning-mapreduce.md) via log4j. You must set the following property in **Ambari** > **YARN** > **Config** > **Advanced yarn-log4j configurations**:

-`log4j.logger.com.microsoft.azure.datalake.store=DEBUG`

-Once the property is set and the nodes are restarted, Data Lake Storage Gen1 diagnostics is written to the YARN logs on the nodes (/tmp/\<user\>/yarn.log), and important details like errors or throttling (HTTP 429 error code) can be monitored. This same information can also be monitored in Azure Monitor logs or wherever logs are shipped to in the [Diagnostics](data-lake-store-diagnostic-logs.md) blade of the Data Lake Storage Gen1 account. It is recommended to at least have client-side logging turned on or utilize the log shipping option with Data Lake Storage Gen1 for operational visibility and easier debugging.

-### Run synthetic transactions

-Currently, the service availability metric for Data Lake Storage Gen1 in the Azure portal has 7-minute refresh window. Also, it cannot be queried using a publicly exposed API. Hence, it is recommended to build a basic application that does synthetic transactions to Data Lake Storage Gen1 that can provide up to the minute availability. An example might be creating a WebJob, Logic App, or Azure Function App to perform a read, create, and update against Data Lake Storage Gen1 and send the results to your monitoring solution. The operations can be done in a temporary folder and then deleted after the test, which might be run every 30-60 seconds, depending on requirements.

-## Directory layout considerations

-When landing data into a data lake, itΓÇÖs important to pre-plan the structure of the data so that security, partitioning, and processing can be utilized effectively. Many of the following recommendations can be used whether itΓÇÖs with Azure Data Lake Storage Gen1, Blob Storage, or HDFS. Every workload has different requirements on how the data is consumed, but below are some common layouts to consider when working with IoT and batch scenarios.

-### IoT structure

-In IoT workloads, there can be a great deal of data being landed in the data store that spans across numerous products, devices, organizations, and customers. ItΓÇÖs important to pre-plan the directory layout for organization, security, and efficient processing of the data for down-stream consumers. A general template to consider might be the following layout:

-```console

-{Region}/{SubjectMatter(s)}/{yyyy}/{mm}/{dd}/{hh}/

-```

-For example, landing telemetry for an airplane engine within the UK might look like the following structure:

-```console

-UK/Planes/BA1293/Engine1/2017/08/11/12/

-```

-There's an important reason to put the date at the end of the folder structure. If you want to lock down certain regions or subject matters to users/groups, then you can easily do so with the POSIX permissions. Otherwise, if there was a need to restrict a certain security group to viewing just the UK data or certain planes, with the date structure in front a separate permission would be required for numerous folders under every hour folder. Additionally, having the date structure in front would exponentially increase the number of folders as time went on.

-### Batch jobs structure

-From a high-level, a commonly used approach in batch processing is to land data in an ΓÇ£inΓÇ¥ folder. Then, once the data is processed, put the new data into an ΓÇ£outΓÇ¥ folder for downstream processes to consume. This directory structure is seen sometimes for jobs that require processing on individual files and might not require massively parallel processing over large datasets. Like the IoT structure recommended above, a good directory structure has the parent-level folders for things such as region and subject matters (for example, organization, product/producer). This structure helps with securing the data across your organization and better management of the data in your workloads. Furthermore, consider date and time in the structure to allow better organization, filtered searches, security, and automation in the processing. The level of granularity for the date structure is determined by the interval on which the data is uploaded or processed, such as hourly, daily, or even monthly.

-Sometimes file processing is unsuccessful due to data corruption or unexpected formats. In such cases, directory structure might benefit from a **/bad** folder to move the files to for further inspection. The batch job might also handle the reporting or notification of these *bad* files for manual intervention. Consider the following template structure:

-```console

-{Region}/{SubjectMatter(s)}/In/{yyyy}/{mm}/{dd}/{hh}/

-{Region}/{SubjectMatter(s)}/Out/{yyyy}/{mm}/{dd}/{hh}/

-{Region}/{SubjectMatter(s)}/Bad/{yyyy}/{mm}/{dd}/{hh}/

-```

-For example, a marketing firm receives daily data extracts of customer updates from their clients in North America. It might look like the following snippet before and after being processed:

-```console

-NA/Extracts/ACMEPaperCo/In/2017/08/14/updates_08142017.csv

-NA/Extracts/ACMEPaperCo/Out/2017/08/14/processed_updates_08142017.csv

-```

-In the common case of batch data being processed directly into databases such as Hive or traditional SQL databases, there isnΓÇÖt a need for an **/in** or **/out** folder since the output already goes into a separate folder for the Hive table or external database. For example, daily extracts from customers would land into their respective folders, and orchestration by something like Azure Data Factory, Apache Oozie, or Apache Airflow would trigger a daily Hive or Spark job to process and write the data into a Hive table.

-## Next steps

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-* [Access Control in Azure Data Lake Storage Gen1](data-lake-store-access-control.md)

-* [Security in Azure Data Lake Storage Gen1](data-lake-store-security-overview.md)

-* [Tuning Azure Data Lake Storage Gen1 for performance](data-lake-store-performance-tuning-guidance.md)

-* [Performance tuning guidance for using HDInsight Spark with Azure Data Lake Storage Gen1](data-lake-store-performance-tuning-spark.md)

-* [Performance tuning guidance for using HDInsight Hive with Azure Data Lake Storage Gen1](data-lake-store-performance-tuning-hive.md)

-* [Create HDInsight clusters with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: Learn about the differences between Azure Data Lake Storage Gen1 and Azure Blob Storage regarding some key aspects of big data processing.

-# Comparing Azure Data Lake Storage Gen1 and Azure Blob Storage

-The table in this article summarizes the differences between Azure Data Lake Storage Gen1 and Azure Blob Storage along some key aspects of big data processing. Azure Blob Storage is a general purpose, scalable object store that is designed for a wide variety of storage scenarios. Azure Data Lake Storage Gen1 is a hyper-scale repository that is optimized for big data analytics workloads.

-| Category | Azure Data Lake Storage Gen1 | Azure Blob Storage |

-| -- | - | |

-| Purpose |Optimized storage for big data analytics workloads |General purpose object store for a wide variety of storage scenarios, including big data analytics |

-| Use Cases |Batch, interactive, streaming analytics and machine learning data such as log files, IoT data, click streams, large datasets |Any type of text or binary data, such as application back end, backup data, media storage for streaming and general purpose data. Additionally, full support for analytics workloads; batch, interactive, streaming analytics and machine learning data such as log files, IoT data, click streams, large datasets |

-| Key Concepts |Data Lake Storage Gen1 account contains folders, which in turn contains data stored as files |Storage account has containers, which in turn has data in the form of blobs |

-| Structure |Hierarchical file system |Object store with flat namespace |

-| API |REST API over HTTPS |REST API over HTTP/HTTPS |

-| Server-side API |[WebHDFS-compatible REST API](/rest/api/datalakestore/) |[Azure Blob Storage REST API](/rest/api/storageservices/Blob-Service-REST-API) |

-| Hadoop File System Client |Yes |Yes |

-| Data Operations - Authentication |Based on [Microsoft Entra identities](../active-directory/develop/authentication-vs-authorization.md) |Based on shared secrets - [Account Access Keys](../storage/common/storage-account-keys-manage.md) and [Shared Access Signature Keys](../storage/common/storage-sas-overview.md). |

-| Data Operations - Authentication Protocol |[OpenID Connect](https://openid.net/connect/). Calls must contain a valid JWT (JSON web token) issued by Microsoft Entra ID.|Hash-based Message Authentication Code (HMAC). Calls must contain a Base64-encoded SHA-256 hash over a part of the HTTP request. |

-| Data Operations - Authorization |POSIX Access Control Lists (ACLs). ACLs based on Microsoft Entra identities can be set at the file and folder level. |For account-level authorization ΓÇô Use [Account Access Keys](../storage/common/storage-account-keys-manage.md)<br>For account, container, or blob authorization - Use [Shared Access Signature Keys](../storage/common/storage-sas-overview.md) |

-| Data Operations - Auditing |Available. See [here](data-lake-store-diagnostic-logs.md) for information. |Available |

-| Encryption data at rest |<ul><li>Transparent, Server side</li> <ul><li>With service-managed keys</li><li>With customer-managed keys in Azure KeyVault</li></ul></ul> |<ul><li>Transparent, Server side</li> <ul><li>With service-managed keys</li><li>With customer-managed keys in Azure KeyVault (preview)</li></ul><li>Client-side encryption</li></ul> |

-| Management operations (for example, Account Create) |[Azure role-based access control (Azure RBAC)](../role-based-access-control/overview.md) for account management |[Azure role-based access control (Azure RBAC)](../role-based-access-control/overview.md) for account management |

-| Developer SDKs |.NET, Java, Python, Node.js |.NET, Java, Python, Node.js, C++, Ruby, PHP, Go, Android, iOS |

-| Analytics Workload Performance |Optimized performance for parallel analytics workloads. High Throughput and IOPS. |Optimized performance for parallel analytics workloads. |

-| Size limits |No limits on account sizes, file sizes, or number of files |For specific limits, see [Scalability targets for standard storage accounts](../storage/common/scalability-targets-standard-account.md) and [Scalability and performance targets for Blob storage](../storage/blobs/scalability-targets.md). Larger account limits available by contacting [Azure Support](https://azure.microsoft.com/support/faq/) |

-| Geo-redundancy |Locally redundant (multiple copies of data in one Azure region) |Locally redundant (LRS), zone redundant (ZRS), globally redundant (GRS), read-access globally redundant (RA-GRS). See [here](../storage/common/storage-redundancy.md) for more information |

-| Service state |Generally available |Generally available |

-| Regional availability |See [here](https://azure.microsoft.com/regions/#services) |Available in all Azure regions |

-| Price |See [Pricing](https://azure.microsoft.com/pricing/details/data-lake-store/) |See [Pricing](https://azure.microsoft.com/pricing/details/storage/) |

-description: List of open source applications that work with Azure Data Lake Storage Gen1 (previously known as Azure Data Lake Store)

-# Open Source Big Data applications that work with Azure Data Lake Storage Gen1

-This article lists the open source big data applications that work with Azure Data Lake Storage Gen1. For the applications in the table below, only the versions available with the listed distribution are supported. For information on what versions of these applications are available with HDInsight, see [HDInsight component versioning](../hdinsight/hdinsight-component-versioning.md).

-| Open Source Software | Distribution |

-| | |

-| [Apache Sqoop](https://sqoop.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [MapReduce](https://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Storm](https://storm.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Hive](https://hive.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [HCatalog](https://cwiki.apache.org/confluence/display/Hive/HCatalog) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Mahout](https://mahout.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Pig/Pig Latin](https://pig.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Oozie](https://oozie.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Zookeeper](https://zookeeper.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Tez](https://tez.apache.org/) |HDInsight 3.2, 3.4, 3.5, and 3.6 |

-| [Apache Spark](https://spark.apache.org/) |HDInsight 3.4, 3.5, and 3.6 |

-## See also

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-description: Learn how to enable access to Azure Data Lake Storage Gen1 from Azure virtual machines that have restricted access to resources.

-# Access Azure Data Lake Storage Gen1 from VMs within an Azure VNET

-Azure Data Lake Storage Gen1 is a PaaS service that runs on public Internet IP addresses. Any server that can connect to the public Internet can typically connect to Azure Data Lake Storage Gen1 endpoints as well. By default, all VMs that are in Azure VNETs can access the Internet and hence can access Azure Data Lake Storage Gen1. However, it is possible to configure VMs in a VNET to not have access to the Internet. For such VMs, access to Azure Data Lake Storage Gen1 is restricted as well. Blocking public Internet access for VMs in Azure VNETs can be done using any of the following approaches:

-* By configuring Network Security Groups (NSG)

-* By configuring User Defined Routes (UDR)

-* By exchanging routes via BGP (industry standard dynamic routing protocol), when ExpressRoute is used, that block access to the Internet

-In this article, you will learn how to enable access to Azure Data Lake Storage Gen1 from Azure VMs, which have been restricted to access resources using one of the three methods listed previously.

-## Enabling connectivity to Azure Data Lake Storage Gen1 from VMs with restricted connectivity

-To access Azure Data Lake Storage Gen1 from such VMs, you must configure them to access the IP address for the region where the Azure Data Lake Storage Gen1 account is available. You can identify the IP addresses for your Data Lake Storage Gen1 account regions by resolving the DNS names of your accounts (`<account>.azuredatalakestore.net`). To resolve DNS names of your accounts, you can use tools such as **nslookup**. Open a command prompt on your computer and run the following command:

-```console

-nslookup mydatastore.azuredatalakestore.net

-```

-The output resembles the following. The value against **Address** property is the IP address associated with your Data Lake Storage Gen1 account.

-```output

-Non-authoritative answer:

-Name: 1434ceb1-3a4b-4bc0-9c69-a0823fd69bba-mydatastore.projectcabostore.net

-Address: 104.44.88.112

-Aliases: mydatastore.azuredatalakestore.net

-```

-### Enabling connectivity from VMs restricted by using NSG

-When an NSG rule is used to block access to the Internet, then you can create another NSG that allows access to the Data Lake Storage Gen1 IP Address. For more information about NSG rules, see [Network security groups overview](../virtual-network/network-security-groups-overview.md). For instructions on how to create NSGs, see [How to create a network security group](../virtual-network/tutorial-filter-network-traffic.md).

-### Enabling connectivity from VMs restricted by using UDR or ExpressRoute

-When routes, either UDRs or BGP-exchanged routes, are used to block access to the Internet, a special route needs to be configured so that VMs in such subnets can access Data Lake Storage Gen1 endpoints. For more information, see [User-defined routes overview](../virtual-network/virtual-networks-udr-overview.md). For instructions on creating UDRs, see [Create UDRs in Resource Manager](../virtual-network/tutorial-create-route-table-powershell.md).

-### Enabling connectivity from VMs restricted by using ExpressRoute

-When an ExpressRoute circuit is configured, the on-premises servers can access Data Lake Storage Gen1 through public peering. More details on configuring ExpressRoute for public peering is available at [ExpressRoute FAQs](../expressroute/expressroute-faqs.md).

-## See also

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-* [Securing data stored in Azure Data Lake Storage Gen1](data-lake-store-security-overview.md)

-description: Use AdlCopy tool to copy data from Azure Storage Blobs to Azure Data Lake Storage Gen1

-# Copy data from Azure Storage Blobs to Azure Data Lake Storage Gen1

-> [!div class="op_single_selector"]

-> * [Using DistCp](data-lake-store-copy-data-wasb-distcp.md)

-> * [Using AdlCopy](data-lake-store-copy-data-azure-storage-blob.md)

->

->

-Data Lake Storage Gen1 provides a command-line tool, AdlCopy, to copy data from the following sources:

-* From Azure Storage blobs into Data Lake Storage Gen1. You can't use AdlCopy to copy data from Data Lake Storage Gen1 to Azure Storage blobs.

-* Between two Data Lake Storage Gen1 accounts.

-Also, you can use the AdlCopy tool in two different modes:

-* **Standalone**, where the tool uses Data Lake Storage Gen1 resources to perform the task.

-* **Using a Data Lake Analytics account**, where the units assigned to your Data Lake Analytics account are used to perform the copy operation. You might want to use this option when you are looking to perform the copy tasks in a predictable manner.

-## Prerequisites

-Before you begin this article, you must have the following:

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure Storage blobs** container with some data.

-* **A Data Lake Storage Gen1 account**. For instructions on how to create one, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md)

-* **Data Lake Analytics account (optional)** - See [Get started with Azure Data Lake Analytics](../data-lake-analytics/data-lake-analytics-get-started-portal.md) for instructions on how to create a Data Lake Analytics account.

-* **AdlCopy tool**. Install the AdlCopy tool.

-## Syntax of the AdlCopy tool

-Use the following syntax to work with the AdlCopy tool

-```console

-AdlCopy /Source <Blob or Data Lake Storage Gen1 source> /Dest <Data Lake Storage Gen1 destination> /SourceKey <Key for Blob account> /Account <Data Lake Analytics account> /Units <Number of Analytics units> /Pattern

-```

-The parameters in the syntax are described below:

-| Option | Description |

-| | |

-| Source |Specifies the location of the source data in the Azure storage blob. The source can be a blob container, a blob, or another Data Lake Storage Gen1 account. |

-| Dest |Specifies the Data Lake Storage Gen1 destination to copy to. |

-| SourceKey |Specifies the storage access key for the Azure storage blob source. This is required only if the source is a blob container or a blob. |

-| Account |**Optional**. Use this if you want to use Azure Data Lake Analytics account to run the copy job. If you use the /Account option in the syntax but do not specify a Data Lake Analytics account, AdlCopy uses a default account to run the job. Also, if you use this option, you must add the source (Azure Storage Blob) and destination (Azure Data Lake Storage Gen1) as data sources for your Data Lake Analytics account. |

-| Units |Specifies the number of Data Lake Analytics units that will be used for the copy job. This option is mandatory if you use the **/Account** option to specify the Data Lake Analytics account. |

-| Pattern |Specifies a regex pattern that indicates which blobs or files to copy. AdlCopy uses case-sensitive matching. The default pattern when no pattern is specified is to copy all items. Specifying multiple file patterns is not supported. |

-## Use AdlCopy (as standalone) to copy data from an Azure Storage blob

-1. Open a command prompt and navigate to the directory where AdlCopy is installed, typically `%HOMEPATH%\Documents\adlcopy`.

-1. Run the following command to copy a specific blob from the source container to a Data Lake Storage Gen1 folder:

- ```console

- AdlCopy /source https://<source_account>.blob.core.windows.net/<source_container>/<blob name> /dest swebhdfs://<dest_adlsg1_account>.azuredatalakestore.net/<dest_folder>/ /sourcekey <storage_account_key_for_storage_container>

- ```

- For example:

- ```console

- AdlCopy /source https://mystorage.blob.core.windows.net/mycluster/HdiSamples/HdiSamples/WebsiteLogSampleData/SampleLog/909f2b.log /dest swebhdfs://mydatalakestorage.azuredatalakestore.net/mynewfolder/ /sourcekey uJUfvD6cEvhfLoBae2yyQf8t9/BpbWZ4XoYj4kAS5Jf40pZaMNf0q6a8yqTxktwVgRED4vPHeh/50iS9atS5LQ==

- ```

- >[!NOTE]

- >The syntax above specifies the file to be copied to a folder in the Data Lake Storage Gen1 account. AdlCopy tool creates a folder if the specified folder name does not exist.

- You will be prompted to enter the credentials for the Azure subscription under which you have your Data Lake Storage Gen1 account. You will see an output similar to the following:

- ```output

- Initializing Copy.

- Copy Started.

- 100% data copied.

- Finishing Copy.

- Copy Completed. 1 file copied.

- ```

-1. You can also copy all the blobs from one container to the Data Lake Storage Gen1 account using the following command:

- ```console

- AdlCopy /source https://<source_account>.blob.core.windows.net/<source_container>/ /dest swebhdfs://<dest_adlsg1_account>.azuredatalakestore.net/<dest_folder>/ /sourcekey <storage_account_key_for_storage_container>

- ```

- For example:

- ```console

- AdlCopy /Source https://mystorage.blob.core.windows.net/mycluster/example/data/gutenberg/ /dest adl://mydatalakestorage.azuredatalakestore.net/mynewfolder/ /sourcekey uJUfvD6cEvhfLoBae2yyQf8t9/BpbWZ4XoYj4kAS5Jf40pZaMNf0q6a8yqTxktwVgRED4vPHeh/50iS9atS5LQ==

- ```

-### Performance considerations

-If you are copying from an Azure Blob Storage account, you may be throttled during copy on the blob storage side. This will degrade the performance of your copy job. To learn more about the limits of Azure Blob Storage, see Azure Storage limits at [Azure subscription and service limits](../azure-resource-manager/management/azure-subscription-service-limits.md).

-## Use AdlCopy (as standalone) to copy data from another Data Lake Storage Gen1 account

-You can also use AdlCopy to copy data between two Data Lake Storage Gen1 accounts.

-1. Open a command prompt and navigate to the directory where AdlCopy is installed, typically `%HOMEPATH%\Documents\adlcopy`.

-1. Run the following command to copy a specific file from one Data Lake Storage Gen1 account to another.

- ```console

- AdlCopy /Source adl://<source_adlsg1_account>.azuredatalakestore.net/<path_to_file> /dest adl://<dest_adlsg1_account>.azuredatalakestore.net/<path>/

- ```

- For example:

- ```console

- AdlCopy /Source adl://mydatastorage.azuredatalakestore.net/mynewfolder/909f2b.log /dest adl://mynewdatalakestorage.azuredatalakestore.net/mynewfolder/

- ```

- > [!NOTE]

- > The syntax above specifies the file to be copied to a folder in the destination Data Lake Storage Gen1 account. AdlCopy tool creates a folder if the specified folder name does not exist.

- >

- >

- You will be prompted to enter the credentials for the Azure subscription under which you have your Data Lake Storage Gen1 account. You will see an output similar to the following:

- ```output

- Initializing Copy.

- Copy Started.|

- 100% data copied.

- Finishing Copy.

- Copy Completed. 1 file copied.

- ```

-1. The following command copies all files from a specific folder in the source Data Lake Storage Gen1 account to a folder in the destination Data Lake Storage Gen1 account.

- ```console

- AdlCopy /Source adl://mydatastorage.azuredatalakestore.net/mynewfolder/ /dest adl://mynewdatalakestorage.azuredatalakestore.net/mynewfolder/

- ```

-### Performance considerations

-When using AdlCopy as a standalone tool, the copy is run on shared, Azure-managed resources. The performance you may get in this environment depends on system load and available resources. This mode is best used for small transfers on an ad hoc basis. No parameters need to be tuned when using AdlCopy as a standalone tool.

-## Use AdlCopy (with Data Lake Analytics account) to copy data

-You can also use your Data Lake Analytics account to run the AdlCopy job to copy data from Azure storage blobs to Data Lake Storage Gen1. You would typically use this option when the data to be moved is in the range of gigabytes and terabytes, and you want better and predictable performance throughput.

-To use your Data Lake Analytics account with AdlCopy to copy from an Azure Storage Blob, the source (Azure Storage Blob) must be added as a data source for your Data Lake Analytics account. For instructions on adding additional data sources to your Data Lake Analytics account, see [Manage Data Lake Analytics account data sources](../data-lake-analytics/data-lake-analytics-manage-use-portal.md#manage-data-sources).

-> [!NOTE]

-> If you are copying from an Azure Data Lake Storage Gen1 account as the source using a Data Lake Analytics account, you do not need to associate the Data Lake Storage Gen1 account with the Data Lake Analytics account. The requirement to associate the source store with the Data Lake Analytics account is only when the source is an Azure Storage account.

->

->

-Run the following command to copy from an Azure Storage blob to a Data Lake Storage Gen1 account using Data Lake Analytics account:

-```console

-AdlCopy /source https://<source_account>.blob.core.windows.net/<source_container>/<blob name> /dest swebhdfs://<dest_adlsg1_account>.azuredatalakestore.net/<dest_folder>/ /sourcekey <storage_account_key_for_storage_container> /Account <data_lake_analytics_account> /Units <number_of_data_lake_analytics_units_to_be_used>

-```

-For example:

-```console

-AdlCopy /Source https://mystorage.blob.core.windows.net/mycluster/example/data/gutenberg/ /dest swebhdfs://mydatalakestorage.azuredatalakestore.net/mynewfolder/ /sourcekey uJUfvD6cEvhfLoBae2yyQf8t9/BpbWZ4XoYj4kAS5Jf40pZaMNf0q6a8yqTxktwVgRED4vPHeh/50iS9atS5LQ== /Account mydatalakeanalyticaccount /Units 2

-```

-Similarly, run the following command to copy all files from a specific folder in the source Data Lake Storage Gen1 account to a folder in the destination Data Lake Storage Gen1 account using Data Lake Analytics account:

-```console

-AdlCopy /Source adl://mysourcedatalakestorage.azuredatalakestore.net/mynewfolder/ /dest adl://mydestdatastorage.azuredatalakestore.net/mynewfolder/ /Account mydatalakeanalyticaccount /Units 2

-```

-### Performance considerations

-When copying data in the range of terabytes, using AdlCopy with your own Azure Data Lake Analytics account provides better and more predictable performance. The parameter that should be tuned is the number of Azure Data Lake Analytics Units to use for the copy job. Increasing the number of units will increase the performance of your copy job. Each file to be copied can use maximum one unit. Specifying more units than the number of files being copied will not increase performance.

-## Use AdlCopy to copy data using pattern matching

-In this section, you learn how to use AdlCopy to copy data from a source (in our example below we use Azure Storage Blob) to a destination Data Lake Storage Gen1 account using pattern matching. For example, you can use the steps below to copy all files with .csv extension from the source blob to the destination.

-1. Open a command prompt and navigate to the directory where AdlCopy is installed, typically `%HOMEPATH%\Documents\adlcopy`.

-1. Run the following command to copy all files with *.csv extension from a specific blob from the source container to a Data Lake Storage Gen1 folder:

- ```console

- AdlCopy /source https://<source_account>.blob.core.windows.net/<source_container>/<blob name> /dest swebhdfs://<dest_adlsg1_account>.azuredatalakestore.net/<dest_folder>/ /sourcekey <storage_account_key_for_storage_container> /Pattern *.csv

- ```

- For example:

- ```console

- AdlCopy /source https://mystorage.blob.core.windows.net/mycluster/HdiSamples/HdiSamples/FoodInspectionData/ /dest adl://mydatalakestorage.azuredatalakestore.net/mynewfolder/ /sourcekey uJUfvD6cEvhfLoBae2yyQf8t9/BpbWZ4XoYj4kAS5Jf40pZaMNf0q6a8yqTxktwVgRED4vPHeh/50iS9atS5LQ== /Pattern *.csv

- ```

-## Billing

-* If you use the AdlCopy tool as standalone you will be billed for egress costs for moving data, if the source Azure Storage account is not in the same region as the Data Lake Storage Gen1 account.

-* If you use the AdlCopy tool with your Data Lake Analytics account, standard [Data Lake Analytics billing rates](https://azure.microsoft.com/pricing/details/data-lake-analytics/) will apply.

-## Considerations for using AdlCopy

-* AdlCopy (for version 1.0.5), supports copying data from sources that collectively have more than thousands of files and folders. However, if you encounter issues copying a large dataset, you can distribute the files/folders into different subfolders and use the path to those subfolders as the source instead.

-## Performance considerations for using AdlCopy

-AdlCopy supports copying data containing thousands of files and folders. However, if you encounter issues copying a large dataset, you can distribute the files/folders into smaller subfolders. AdlCopy was built for ad hoc copies. If you are trying to copy data on a recurring basis, you should consider using [Azure Data Factory](../data-factory/connector-azure-data-lake-store.md) that provides full management around the copy operations.

-## Release notes

-* 1.0.13 - If you are copying data to the same Azure Data Lake Storage Gen1 account across multiple adlcopy commands, you do not need to reenter your credentials for each run anymore. Adlcopy will now cache that information across multiple runs.

-## Next steps

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-* [Use Azure HDInsight with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: Use the DistCp tool to copy data to and from Azure Storage blobs to Azure Data Lake Storage Gen1

-# Use DistCp to copy data between Azure Storage blobs and Azure Data Lake Storage Gen1

-> [!div class="op_single_selector"]

-> * [Using DistCp](data-lake-store-copy-data-wasb-distcp.md)

-> * [Using AdlCopy](data-lake-store-copy-data-azure-storage-blob.md)

->

->

-If you have an HDInsight cluster with access to Azure Data Lake Storage Gen1, you can use Hadoop ecosystem tools like DistCp to copy data to and from an HDInsight cluster storage (WASB) into a Data Lake Storage Gen1 account. This article shows how to use the DistCp tool.

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **An Azure Data Lake Storage Gen1 account**. For instructions on how to create one, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md).

-* **Azure HDInsight cluster** with access to a Data Lake Storage Gen1 account. See [Create an HDInsight cluster with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md). Make sure you enable Remote Desktop for the cluster.

-## Use DistCp from an HDInsight Linux cluster

-An HDInsight cluster comes with the DistCp tool, which can be used to copy data from different sources into an HDInsight cluster. If you've configured the HDInsight cluster to use Data Lake Storage Gen1 as additional storage, you can use DistCp out-of-the-box to copy data to and from a Data Lake Storage Gen1 account. In this section, we look at how to use the DistCp tool.

-1. From your desktop, use SSH to connect to the cluster. See [Connect to a Linux-based HDInsight cluster](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md). Run the commands from the SSH prompt.

-1. Verify whether you can access the Azure Storage blobs (WASB). Run the following command:

- ```

- hdfs dfs ΓÇôls wasb://<container_name>@<storage_account_name>.blob.core.windows.net/

- ```

- The output provides a list of contents in the storage blob.

-1. Similarly, verify whether you can access the Data Lake Storage Gen1 account from the cluster. Run the following command:

- ```

- hdfs dfs -ls adl://<data_lake_storage_gen1_account>.azuredatalakestore.net:443/

- ```

- The output provides a list of files and folders in the Data Lake Storage Gen1 account.

-1. Use DistCp to copy data from WASB to a Data Lake Storage Gen1 account.

- ```

- hadoop distcp wasb://<container_name>@<storage_account_name>.blob.core.windows.net/example/data/gutenberg adl://<data_lake_storage_gen1_account>.azuredatalakestore.net:443/myfolder

- ```

- The command copies the contents of the **/example/data/gutenberg/** folder in WASB to **/myfolder** in the Data Lake Storage Gen1 account.

-1. Similarly, use DistCp to copy data from a Data Lake Storage Gen1 account to WASB.

- ```

- hadoop distcp adl://<data_lake_storage_gen1_account>.azuredatalakestore.net:443/myfolder wasb://<container_name>@<storage_account_name>.blob.core.windows.net/example/data/gutenberg

- ```

- The command copies the contents of **/myfolder** in the Data Lake Storage Gen1 account to **/example/data/gutenberg/** folder in WASB.

-## Performance considerations while using DistCp

-Because the DistCp toolΓÇÖs lowest granularity is a single file, setting the maximum number of simultaneous copies is the most important parameter to optimize it against Data Lake Storage Gen1. You can control the number of simultaneous copies by setting the number of mappers (ΓÇÿmΓÇÖ) parameter on the command line. This parameter specifies the maximum number of mappers that are used to copy data. The default value is 20.

-Example:

-```

- hadoop distcp wasb://<container_name>@<storage_account_name>.blob.core.windows.net/example/data/gutenberg adl://<data_lake_storage_gen1_account>.azuredatalakestore.net:443/myfolder -m 100

-```

-### How to determine the number of mappers to use

-Here's some guidance that you can use.

-* **Step 1: Determine total YARN memory** - The first step is to determine the YARN memory available to the cluster where you run the DistCp job. This information is available in the Ambari portal associated with the cluster. Navigate to YARN and view the **Configs** tab to see the YARN memory. To get the total YARN memory, multiply the YARN memory per node with the number of nodes you have in your cluster.

-* **Step 2: Calculate the number of mappers** - The value of **m** is equal to the quotient of total YARN memory divided by the YARN container size. The YARN container size information is also available in the Ambari portal. Navigate to YARN and view the **Configs** tab. The YARN container size is displayed in this window. The equation to arrive at the number of mappers (**m**) is:

- `m = (number of nodes * YARN memory for each node) / YARN container size`

-Example:

-LetΓÇÖs assume that you have four D14v2s nodes in the cluster and you want to transfer 10 TB of data from 10 different folders. Each of the folders contains varying amounts of data and the file sizes within each folder are different.

-* Total YARN memory - From the Ambari portal you determine that the YARN memory is 96 GB for a D14 node. So, total YARN memory for four node cluster is:

- `YARN memory = 4 * 96GB = 384GB`

-* Number of mappers - From the Ambari portal you determine that the YARN container size is 3072 for a D14 cluster node. So, the number of mappers is:

- `m = (4 nodes * 96GB) / 3072MB = 128 mappers`

-If other applications are using memory, you can choose to only use a portion of your clusterΓÇÖs YARN memory for DistCp.

-### Copying large datasets

-When the size of the dataset to be moved is large (for example, > 1 TB) or if you have many different folders, consider using multiple DistCp jobs. There's likely no performance gain, but it spreads out the jobs so that if any job fails, you need to only restart that specific job instead of the entire job.

-### Limitations

-* DistCp tries to create mappers that are similar in size to optimize performance. Increasing the number of mappers may not always increase performance.

-* DistCp is limited to only one mapper per file. Therefore, you shouldn't have more mappers than you have files. Because DistCp can assign only one mapper to a file, this limits the amount of concurrency that can be used to copy large files.

-* If you have a small number of large files, split them into 256-MB file chunks to give you more potential concurrency.

-* If you're copying from an Azure Blob storage account, your copy job may be throttled on the Blob storage side. This degrades the performance of your copy job. To learn more about the limits of Azure Blob storage, see Azure Storage limits at [Azure subscription and service limits](../azure-resource-manager/management/azure-subscription-service-limits.md).

-## See also

-* [Copy data from Azure Storage blobs to Data Lake Storage Gen1](data-lake-store-copy-data-azure-storage-blob.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-* [Use Azure HDInsight with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: Use the Azure Data Lake Storage Gen1 .NET SDK for filesystem operations on Data Lake Storage Gen1 such as create folders, etc.

-# Filesystem operations on Data Lake Storage Gen1 using the .NET SDK

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-data-operations-net-sdk.md)

-> * [Java SDK](data-lake-store-get-started-java-sdk.md)

-> * [REST API](data-lake-store-data-operations-rest-api.md)

-> * [Python](data-lake-store-data-operations-python.md)

->

->

-In this article, you learn how to perform filesystem operations on Data Lake Storage Gen1 using the .NET SDK. Filesystem operations include creating folders in a Data Lake Storage Gen1 account, uploading files, downloading files, etc.

-For instructions on how to do account management operations on Data Lake Storage Gen1 using the .NET SDK, see [Account management operations on Data Lake Storage Gen1 using .NET SDK](data-lake-store-get-started-net-sdk.md).

-## Prerequisites

-* **Visual Studio 2013 or above**. The instructions in this article use Visual Studio 2019.

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure Data Lake Storage Gen1 account**. For instructions on how to create an account, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md).

-## Create a .NET application

-The code sample available [on GitHub](https://github.com/Azure-Samples/data-lake-store-adls-dot-net-get-started/tree/master/AdlsSDKGettingStarted) walks you through the process of creating files in the store, concatenating files, downloading a file, and deleting some files in the store. This section of the article walks you through the main parts of the code.

-1. In Visual Studio, select the **File** menu, **New**, and then **Project**.

-1. Choose **Console App (.NET Framework)**, and then select **Next**.

-1. In **Project name**, enter `CreateADLApplication`, and then select **Create**.

-1. Add the NuGet packages to your project.

- 1. Right-click the project name in the Solution Explorer and click **Manage NuGet Packages**.

- 1. In the **NuGet Package Manager** tab, make sure that **Package source** is set to **nuget.org**. Also, make sure the **Include prerelease** check box is selected.

- 1. Search for and install the following NuGet packages:

- * `Microsoft.Azure.DataLake.Store` - This article uses v1.0.0.

- * `Microsoft.Rest.ClientRuntime.Azure.Authentication` - This article uses v2.3.1.

- Close the **NuGet Package Manager**.

-1. Open **Program.cs**, delete the existing code, and then include the following statements to add references to namespaces.

- ```

- using System;

- using System.IO;using System.Threading;

- using System.Linq;

- using System.Text;

- using System.Collections.Generic;

- using System.Security.Cryptography.X509Certificates; // Required only if you're using an Azure AD application created with certificates

-

- using Microsoft.Rest;

- using Microsoft.Rest.Azure.Authentication;

- using Microsoft.Azure.DataLake.Store;

- using Microsoft.IdentityModel.Clients.ActiveDirectory;

- ```

-1. Declare the variables as shown below, and provide the values for the placeholders. Also, make sure the local path and file name you provide here exist on the computer.

- ```

- namespace SdkSample

- {

- class Program

- {

- private static string _adlsg1AccountName = "<DATA-LAKE-STORAGE-GEN1-NAME>.azuredatalakestore.net";

- }

- }

- ```

-In the remaining sections of the article, you can see how to use the available .NET methods to do operations such as authentication, file upload, etc.

-## Authentication

-* For end-user authentication for your application, see [End-user authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md).

-* For service-to-service authentication for your application, see [Service-to-service authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-service-to-service-authenticate-net-sdk.md).

-## Create client object

-The following snippet creates the Data Lake Storage Gen1 filesystem client object, which is used to issue requests to the service.

-```

-// Create client objects

-AdlsClient client = AdlsClient.CreateClient(_adlsg1AccountName, adlCreds);

-```

-## Create a file and directory

-Add the following snippet to your application. This snippet adds a file and any parent directory that does not exist.

-```

-// Create a file - automatically creates any parent directories that don't exist

-// The AdlsOutputStream preserves record boundaries - it does not break records while writing to the store

-using (var stream = client.CreateFile(fileName, IfExists.Overwrite))

-{

- byte[] textByteArray = Encoding.UTF8.GetBytes("This is test data to write.\r\n");

- stream.Write(textByteArray, 0, textByteArray.Length);

- textByteArray = Encoding.UTF8.GetBytes("This is the second line.\r\n");

- stream.Write(textByteArray, 0, textByteArray.Length);

-}

-```

-## Append to a file

-The following snippet appends data to an existing file in Data Lake Storage Gen1 account.

-```

-// Append to existing file

-using (var stream = client.GetAppendStream(fileName))

-{

- byte[] textByteArray = Encoding.UTF8.GetBytes("This is the added line.\r\n");

- stream.Write(textByteArray, 0, textByteArray.Length);

-}

-```

-## Read a file

-The following snippet reads the contents of a file in Data Lake Storage Gen1.

-```

-//Read file contents

-using (var readStream = new StreamReader(client.GetReadStream(fileName)))

-{

- string line;

- while ((line = readStream.ReadLine()) != null)

- {

- Console.WriteLine(line);

- }

-}

-```

-## Get file properties

-The following snippet returns the properties associated with a file or a directory.

-```

-// Get file properties

-var directoryEntry = client.GetDirectoryEntry(fileName);

-PrintDirectoryEntry(directoryEntry);

-```

-The definition of the `PrintDirectoryEntry` method is available as part of the sample [on GitHub](https://github.com/Azure-Samples/data-lake-store-adls-dot-net-get-started/tree/master/AdlsSDKGettingStarted).

-## Rename a file

-The following snippet renames an existing file in a Data Lake Storage Gen1 account.

-```

-// Rename a file

-string destFilePath = "/Test/testRenameDest3.txt";

-client.Rename(fileName, destFilePath, true);

-```

-## Enumerate a directory

-The following snippet enumerates directories in a Data Lake Storage Gen1 account.

-```

-// Enumerate directory

-foreach (var entry in client.EnumerateDirectory("/Test"))

-{

- PrintDirectoryEntry(entry);

-}

-```

-The definition of the `PrintDirectoryEntry` method is available as part of the sample [on GitHub](https://github.com/Azure-Samples/data-lake-store-adls-dot-net-get-started/tree/master/AdlsSDKGettingStarted).

-## Delete directories recursively

-The following snippet deletes a directory, and all its subdirectories, recursively.

-```

-// Delete a directory and all its subdirectories and files

-client.DeleteRecursive("/Test");

-```

-## Samples

-Here are a few samples that show how to use the Data Lake Storage Gen1 Filesystem SDK.

-* [Basic sample on GitHub](https://github.com/Azure-Samples/data-lake-store-adls-dot-net-get-started/tree/master/AdlsSDKGettingStarted)

-* [Advanced sample on GitHub](https://github.com/Azure-Samples/data-lake-store-adls-dot-net-samples)

-## See also

-* [Account management operations on Data Lake Storage Gen1 using .NET SDK](data-lake-store-get-started-net-sdk.md)

-* [Data Lake Storage Gen1 .NET SDK Reference](/dotnet/api/overview/azure/data-lake-store)

-## Next steps

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-description: Learn how to use Python SDK to work with the Data Lake Storage Gen1 file system.

-# Filesystem operations on Azure Data Lake Storage Gen1 using Python

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-data-operations-net-sdk.md)

-> * [Java SDK](data-lake-store-get-started-java-sdk.md)

-> * [REST API](data-lake-store-data-operations-rest-api.md)

-> * [Python](data-lake-store-data-operations-python.md)

->

->

-In this article, you learn how to use Python SDK to perform filesystem operations on Azure Data Lake Storage Gen1. For instructions on how to perform account management operations on Data Lake Storage Gen1 using Python, see [Account management operations on Data Lake Storage Gen1 using Python](data-lake-store-get-started-python.md).

-## Prerequisites

-* **Python**. You can download Python from [here](https://www.python.org/downloads/). This article uses Python 3.6.2.

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure Data Lake Storage Gen1 account**. Follow the instructions at [Get started with Azure Data Lake Storage Gen1 using the Azure portal](data-lake-store-get-started-portal.md).

-## Install the modules

-To work with Data Lake Storage Gen1 using Python, you need to install three modules.

-* The `azure-mgmt-resource` module, which includes Azure modules for Active Directory, etc.

-* The `azure-mgmt-datalake-store` module, which includes the Azure Data Lake Storage Gen1 account management operations. For more information on this module, see the [azure-mgmt-datalake-store module reference](/python/api/azure-mgmt-datalake-store/).

-* The `azure-datalake-store` module, which includes the Azure Data Lake Storage Gen1 filesystem operations. For more information on this module, see the [azure-datalake-store file-system module reference](/python/api/azure-datalake-store/azure.datalake.store.core/).

-Use the following commands to install the modules.

-```console

-pip install azure-mgmt-resource

-pip install azure-mgmt-datalake-store

-pip install azure-datalake-store

-```

-## Create a new Python application

-1. In the IDE of your choice create a new Python application, for example, **mysample.py**.

-2. Add the following lines to import the required modules

- ```python

- ## Use this only for Azure AD service-to-service authentication

- from azure.common.credentials import ServicePrincipalCredentials

- ## Use this only for Azure AD end-user authentication

- from azure.common.credentials import UserPassCredentials

- ## Use this only for Azure AD multi-factor authentication

- from msrestazure.azure_active_directory import AADTokenCredentials

- ## Required for Azure Data Lake Storage Gen1 account management

- from azure.mgmt.datalake.store import DataLakeStoreAccountManagementClient

- from azure.mgmt.datalake.store.models import DataLakeStoreAccount

- ## Required for Azure Data Lake Storage Gen1 filesystem management

- from azure.datalake.store import core, lib, multithread

- ## Common Azure imports

- from azure.mgmt.resource.resources import ResourceManagementClient

- from azure.mgmt.resource.resources.models import ResourceGroup

- ## Use these as needed for your application

- import logging, getpass, pprint, uuid, time

- ```

-3. Save changes to mysample.py.

-## Authentication

-In this section, we talk about the different ways to authenticate with Microsoft Entra ID. The options available are:

-* For end-user authentication for your application, see [End-user authentication with Data Lake Storage Gen1 using Python](data-lake-store-end-user-authenticate-python.md).

-* For service-to-service authentication for your application, see [Service-to-service authentication with Data Lake Storage Gen1 using Python](data-lake-store-service-to-service-authenticate-python.md).

-## Create filesystem client

-The following snippet first creates the Data Lake Storage Gen1 account client. It uses the client object to create a Data Lake Storage Gen1 account. Finally, the snippet creates a filesystem client object.

-```python

-## Declare variables

-subscriptionId = 'FILL-IN-HERE'

-adlsAccountName = 'FILL-IN-HERE'

-## Create a filesystem client object

-adlsFileSystemClient = core.AzureDLFileSystem(adlCreds, store_name=adlsAccountName)

-```

-## Create a directory

-```python

-## Create a directory

-adlsFileSystemClient.mkdir('/mysampledirectory')

-```

-## Upload a file

-```python

-## Upload a file

-multithread.ADLUploader(adlsFileSystemClient, lpath='C:\\data\\mysamplefile.txt', rpath='/mysampledirectory/mysamplefile.txt', nthreads=64, overwrite=True, buffersize=4194304, blocksize=4194304)

-```

-## Download a file

-```python

-## Download a file

-multithread.ADLDownloader(adlsFileSystemClient, lpath='C:\\data\\mysamplefile.txt.out', rpath='/mysampledirectory/mysamplefile.txt', nthreads=64, overwrite=True, buffersize=4194304, blocksize=4194304)

-```

-## Delete a directory

-```python

-## Delete a directory

-adlsFileSystemClient.rm('/mysampledirectory', recursive=True)

-```

-## Next steps

-* [Account management operations on Data Lake Storage Gen1 using Python](data-lake-store-get-started-python.md).

-## See also

-* [Azure Data Lake Storage Gen1 Python (Filesystem) Reference](/python/api/azure-datalake-store/azure.datalake.store.core)

-* [Open Source Big Data applications compatible with Azure Data Lake Storage Gen1](data-lake-store-compatible-oss-other-applications.md)

-description: Use WebHDFS REST APIs to perform filesystem operations on Azure Data Lake Storage Gen1

-# Filesystem operations on Azure Data Lake Storage Gen1 using REST API

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-data-operations-net-sdk.md)

-> * [Java SDK](data-lake-store-get-started-java-sdk.md)

-> * [REST API](data-lake-store-data-operations-rest-api.md)

-> * [Python](data-lake-store-data-operations-python.md)

->

->

-In this article, you learn how to use WebHDFS REST APIs and Data Lake Storage Gen1 REST APIs to perform filesystem operations on Azure Data Lake Storage Gen1. For instructions on how to perform account management operations on Data Lake Storage Gen1 using REST API, see [Account management operations on Data Lake Storage Gen1 using REST API](data-lake-store-get-started-rest-api.md).

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure Data Lake Storage Gen1 account**. Follow the instructions at [Get started with Azure Data Lake Storage Gen1 using the Azure portal](data-lake-store-get-started-portal.md).

-* **[cURL](https://curl.haxx.se/)**. This article uses cURL to demonstrate how to make REST API calls against a Data Lake Storage Gen1 account.

-<a name='how-do-i-authenticate-using-azure-active-directory'></a>

-## How do I authenticate using Microsoft Entra ID?

-You can use two approaches to authenticate using Microsoft Entra ID.

-* For end-user authentication for your application (interactive), see [End-user authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-end-user-authenticate-rest-api.md).

-* For service-to-service authentication for your application (non-interactive), see [Service-to-service authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-service-to-service-authenticate-rest-api.md).

-## Create folders

-This operation is based on the WebHDFS REST API call defined [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Make_a_Directory).

-Use the following cURL command. Replace **\<yourstorename>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -X PUT -H "Authorization: Bearer <REDACTED>" -d "" 'https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/?op=MKDIRS'

-```

-In the preceding command, replace \<`REDACTED`\> with the authorization token you retrieved earlier. This command creates a directory called **mytempdir** under the root folder of your Data Lake Storage Gen1 account.

-If the operation completes successfully, you should see a response like the following snippet:

-```output

-{"boolean":true}

-```

-## List folders

-This operation is based on the WebHDFS REST API call defined [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#List_a_Directory).

-Use the following cURL command. Replace **\<yourstorename>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -X GET -H "Authorization: Bearer <REDACTED>" 'https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/?op=LISTSTATUS'

-```

-In the preceding command, replace \<`REDACTED`\> with the authorization token you retrieved earlier.

-If the operation completes successfully, you should see a response like the following snippet:

-```output

-{

-"FileStatuses": {

- "FileStatus": [{

- "length": 0,

- "pathSuffix": "mytempdir",

- "type": "DIRECTORY",

- "blockSize": 268435456,

- "accessTime": 1458324719512,

- "modificationTime": 1458324719512,

- "replication": 0,

- "permission": "777",

- "owner": "<GUID>",

- "group": "<GUID>"

- }]

-}

-}

-```

-## Upload data

-This operation is based on the WebHDFS REST API call defined [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File).

-Use the following cURL command. Replace **\<yourstorename>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -X PUT -L -T 'C:\temp\list.txt' -H "Authorization: Bearer <REDACTED>" 'https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/list.txt?op=CREATE'

-```

-In the preceding syntax **-T** parameter is the location of the file you are uploading.

-The output is similar to the following snippet:

-

-```output

-HTTP/1.1 307 Temporary Redirect

-...

-Location: https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/list.txt?op=CREATE&write=true

-...

-Content-Length: 0

-HTTP/1.1 100 Continue

-HTTP/1.1 201 Created

-...

-```

-## Read data

-This operation is based on the WebHDFS REST API call defined [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Open_and_Read_a_File).

-Reading data from a Data Lake Storage Gen1 account is a two-step process.

-* You first submit a GET request against the endpoint `https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/myinputfile.txt?op=OPEN`. This call returns a location to submit the next GET request to.

-* You then submit the GET request against the endpoint `https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/myinputfile.txt?op=OPEN&read=true`. This call displays the contents of the file.

-However, because there is no difference in the input parameters between the first and the second step, you can use the `-L` parameter to submit the first request. `-L` option essentially combines two requests into one and makes cURL redo the request on the new location. Finally, the output from all the request calls is displayed, like shown in the following snippet. Replace **\<yourstorename>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -L GET -H "Authorization: Bearer <REDACTED>" 'https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/myinputfile.txt?op=OPEN'

-```

-You should see an output similar to the following snippet:

-```output

-HTTP/1.1 307 Temporary Redirect

-...

-Location: https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/somerandomfile.txt?op=OPEN&read=true

-...

-HTTP/1.1 200 OK

-...

-Hello, Data Lake Store user!

-```

-## Rename a file

-This operation is based on the WebHDFS REST API call defined [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Rename_a_FileDirectory).

-Use the following cURL command to rename a file. Replace **\<yourstorename>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -X PUT -H "Authorization: Bearer <REDACTED>" -d "" 'https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/myinputfile.txt?op=RENAME&destination=/mytempdir/myinputfile1.txt'

-```

-You should see an output similar to the following snippet:

-```output

-HTTP/1.1 200 OK

-...

-{"boolean":true}

-```

-## Delete a file

-This operation is based on the WebHDFS REST API call defined [here](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Delete_a_FileDirectory).

-Use the following cURL command to delete a file. Replace **\<yourstorename>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -X DELETE -H "Authorization: Bearer <REDACTED>" 'https://<yourstorename>.azuredatalakestore.net/webhdfs/v1/mytempdir/myinputfile1.txt?op=DELETE'

-```

-You should see an output like the following:

-```output

-HTTP/1.1 200 OK

-...

-{"boolean":true}

-```

-## Next steps

-* [Account management operations on Data Lake Storage Gen1 using REST API](data-lake-store-get-started-rest-api.md).

-## See also

-* [Azure Data Lake Storage Gen1 REST API Reference](/rest/api/datalakestore/)

-* [Open Source Big Data applications compatible with Azure Data Lake Storage Gen1](data-lake-store-compatible-oss-other-applications.md)

-description: Understand the different scenarios and tools using which data can ingested, processed, downloaded, and visualized in Data Lake Storage Gen1 (previously known as Azure Data Lake Store)

-# Using Azure Data Lake Storage Gen1 for big data requirements

-There are four key stages in big data processing:

-* Ingesting large amounts of data into a data store, at real-time or in batches

-* Processing the data

-* Downloading the data

-* Visualizing the data

-In this article, we look at these stages with respect to Azure Data Lake Storage Gen1 to understand the options and tools available to meet your big data needs.

-## Ingest data into Data Lake Storage Gen1

-This section highlights the different sources of data and the different ways in which that data can be ingested into a Data Lake Storage Gen1 account.

-

-### Ad hoc data

-This represents smaller data sets that are used for prototyping a big data application. There are different ways of ingesting ad hoc data depending on the source of the data.

-| Data Source | Ingest it using |

-| | |

-| Local computer |<ul> <li>[Azure portal](data-lake-store-get-started-portal.md)</li> <li>[Azure PowerShell](data-lake-store-get-started-powershell.md)</li> <li>[Azure CLI](data-lake-store-get-started-cli-2.0.md)</li> <li>[Using Data Lake Tools for Visual Studio](../data-lake-analytics/data-lake-analytics-data-lake-tools-get-started.md) </li></ul> |

-| Azure Storage Blob |<ul> <li>[Azure Data Factory](../data-factory/connector-azure-data-lake-store.md)</li> <li>[AdlCopy tool](data-lake-store-copy-data-azure-storage-blob.md)</li><li>[DistCp running on HDInsight cluster](data-lake-store-copy-data-wasb-distcp.md)</li> </ul> |

-### Streamed data

-This represents data that can be generated by various sources such as applications, devices, sensors, etc. This data can be ingested into Data Lake Storage Gen1 by a variety of tools. These tools will usually capture and process the data on an event-by-event basis in real-time, and then write the events in batches into Data Lake Storage Gen1 so that they can be further processed.

-Following are tools that you can use:

-* [Azure Stream Analytics](../stream-analytics/stream-analytics-define-outputs.md) - Events ingested into Event Hubs can be written to Azure Data Lake Storage Gen1 using an Azure Data Lake Storage Gen1 output.

-* [EventProcessorHost](../event-hubs/event-hubs-dotnet-standard-getstarted-send.md) ΓÇô You can receive events from Event Hubs and then write it to Data Lake Storage Gen1 using the [Data Lake Storage Gen1 .NET SDK](data-lake-store-get-started-net-sdk.md).

-### Relational data

-You can also source data from relational databases. Over a period of time, relational databases collect huge amounts of data which can provide key insights if processed through a big data pipeline. You can use the following tools to move such data into Data Lake Storage Gen1.

-* [Apache Sqoop](data-lake-store-data-transfer-sql-sqoop.md)

-* [Azure Data Factory](../data-factory/copy-activity-overview.md)

-### Web server log data (upload using custom applications)

-This type of dataset is specifically called out because analysis of web server log data is a common use case for big data applications and requires large volumes of log files to be uploaded to Data Lake Storage Gen1. You can use any of the following tools to write your own scripts or applications to upload such data.

-* [Azure CLI](data-lake-store-get-started-cli-2.0.md)

-* [Azure PowerShell](data-lake-store-get-started-powershell.md)

-* [Azure Data Lake Storage Gen1 .NET SDK](data-lake-store-get-started-net-sdk.md)

-* [Azure Data Factory](../data-factory/copy-activity-overview.md)

-For uploading web server log data, and also for uploading other kinds of data (e.g. social sentiments data), it is a good approach to write your own custom scripts/applications because it gives you the flexibility to include your data uploading component as part of your larger big data application. In some cases this code may take the form of a script or simple command line utility. In other cases, the code may be used to integrate big data processing into a business application or solution.

-### Data associated with Azure HDInsight clusters

-Most HDInsight cluster types (Hadoop, HBase, Storm) support Data Lake Storage Gen1 as a data storage repository. HDInsight clusters access data from Azure Storage Blobs (WASB). For better performance, you can copy the data from WASB into a Data Lake Storage Gen1 account associated with the cluster. You can use the following tools to copy the data.

-* [Apache DistCp](data-lake-store-copy-data-wasb-distcp.md)

-* [AdlCopy Service](data-lake-store-copy-data-azure-storage-blob.md)

-* [Azure Data Factory](../data-factory/connector-azure-data-lake-store.md)

-### Data stored in on-premises or IaaS Hadoop clusters

-Large amounts of data may be stored in existing Hadoop clusters, locally on machines using HDFS. The Hadoop clusters may be in an on-premises deployment or may be within an IaaS cluster on Azure. There could be requirements to copy such data to Azure Data Lake Storage Gen1 for a one-off approach or in a recurring fashion. There are various options that you can use to achieve this. Below is a list of alternatives and the associated trade-offs.

-| Approach | Details | Advantages | Considerations |

-| | | | |

-| Use Azure Data Factory (ADF) to copy data directly from Hadoop clusters to Azure Data Lake Storage Gen1 |[ADF supports HDFS as a data source](../data-factory/connector-hdfs.md) |ADF provides out-of-the-box support for HDFS and first class end-to-end management and monitoring |Requires Data Management Gateway to be deployed on-premises or in the IaaS cluster |

-| Export data from Hadoop as files. Then copy the files to Azure Data Lake Storage Gen1 using appropriate mechanism. |You can copy files to Azure Data Lake Storage Gen1 using: <ul><li>[Azure PowerShell for Windows OS](data-lake-store-get-started-powershell.md)</li><li>[Azure CLI](data-lake-store-get-started-cli-2.0.md)</li><li>Custom app using any Data Lake Storage Gen1 SDK</li></ul> |Quick to get started. Can do customized uploads |Multi-step process that involves multiple technologies. Management and monitoring will grow to be a challenge over time given the customized nature of the tools |

-| Use Distcp to copy data from Hadoop to Azure Storage. Then copy data from Azure Storage to Data Lake Storage Gen1 using appropriate mechanism. |You can copy data from Azure Storage to Data Lake Storage Gen1 using: <ul><li>[Azure Data Factory](../data-factory/copy-activity-overview.md)</li><li>[AdlCopy tool](data-lake-store-copy-data-azure-storage-blob.md)</li><li>[Apache DistCp running on HDInsight clusters](data-lake-store-copy-data-wasb-distcp.md)</li></ul> |You can use open-source tools. |Multi-step process that involves multiple technologies |

-### Really large datasets

-For uploading datasets that range in several terabytes, using the methods described above can sometimes be slow and costly. In such cases, you can use the options below.

-* **Using Azure ExpressRoute**. Azure ExpressRoute lets you create private connections between Azure datacenters and infrastructure on your premises. This provides a reliable option for transferring large amounts of data. For more information, see [Azure ExpressRoute documentation](../expressroute/expressroute-introduction.md).

-* **"Offline" upload of data**. If using Azure ExpressRoute is not feasible for any reason, you can use [Azure Import/Export service](../import-export/storage-import-export-service.md) to ship hard disk drives with your data to an Azure data center. Your data is first uploaded to Azure Storage Blobs. You can then use [Azure Data Factory](../data-factory/connector-azure-data-lake-store.md) or [AdlCopy tool](data-lake-store-copy-data-azure-storage-blob.md) to copy data from Azure Storage Blobs to Data Lake Storage Gen1.

- > [!NOTE]

- > While using the Import/Export service, the file sizes on the disks that you ship to Azure data center should not be greater than 195 GB.

- >

- >

-## Process data stored in Data Lake Storage Gen1

-Once the data is available in Data Lake Storage Gen1 you can run analysis on that data using the supported big data applications. Currently, you can use Azure HDInsight and Azure Data Lake Analytics to run data analysis jobs on the data stored in Data Lake Storage Gen1.

-

-You can look at the following examples.

-* [Create an HDInsight cluster with Data Lake Storage Gen1 as storage](data-lake-store-hdinsight-hadoop-use-portal.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-## Download data from Data Lake Storage Gen1

-You might also want to download or move data from Azure Data Lake Storage Gen1 for scenarios such as:

-* Move data to other repositories to interface with your existing data processing pipelines. For example, you might want to move data from Data Lake Storage Gen1 to Azure SQL Database or SQL Server.

-* Download data to your local computer for processing in IDE environments while building application prototypes.

-

-In such cases, you can use any of the following options:

-* [Apache Sqoop](data-lake-store-data-transfer-sql-sqoop.md)

-* [Azure Data Factory](../data-factory/copy-activity-overview.md)

-* [Apache DistCp](data-lake-store-copy-data-wasb-distcp.md)

-You can also use the following methods to write your own script/application to download data from Data Lake Storage Gen1.

-* [Azure CLI](data-lake-store-get-started-cli-2.0.md)

-* [Azure PowerShell](data-lake-store-get-started-powershell.md)

-* [Azure Data Lake Storage Gen1 .NET SDK](data-lake-store-get-started-net-sdk.md)

-## Visualize data in Data Lake Storage Gen1

-You can use a mix of services to create visual representations of data stored in Data Lake Storage Gen1.

-

-* You can start by using [Azure Data Factory to move data from Data Lake Storage Gen1 to Azure Synapse Analytics](../data-factory/copy-activity-overview.md)

-* After that, you can [integrate Power BI with Azure Synapse Analytics](/power-bi/connect-data/service-azure-sql-data-warehouse-with-direct-connect) to create visual representation of the data.

-description: Use Sqoop to copy data between Azure SQL Database and Azure Data Lake Storage Gen1

-# Copy data between Data Lake Storage Gen1 and Azure SQL Database using Sqoop

-Learn how to use Apache Sqoop to import and export data between Azure SQL Database and Azure Data Lake Storage Gen1.

-## What is Sqoop?

-Big data applications are a natural choice for processing unstructured and semi-structured data, such as logs and files. However, you may also have a need to process structured data that's stored in relational databases.

-[Apache Sqoop](https://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html) is a tool designed to transfer data between relational databases and a big data repository, such as Data Lake Storage Gen1. You can use it to import data from a relational database management system (RDBMS) such as Azure SQL Database into Data Lake Storage Gen1. You can then transform and analyze the data using big data workloads, and then export the data back into an RDBMS. In this article, you use a database in Azure SQL Database as your relational database to import/export from.

-## Prerequisites

-Before you begin, you must have the following:

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **An Azure Data Lake Storage Gen1 account**. For instructions on how to create the account, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md)

-* **Azure HDInsight cluster** with access to a Data Lake Storage Gen1 account. See [Create an HDInsight cluster with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md). This article assumes you have an HDInsight Linux cluster with Data Lake Storage Gen1 access.

-* **Azure SQL Database**. For instructions on how to create a database in Azure SQL Database, see [Create a database in Azure SQL Database](/azure/azure-sql/database/single-database-create-quickstart)

-## Create sample tables in the database

-1. To start, create two sample tables in the database. Use [SQL Server Management Studio](/azure/azure-sql/database/connect-query-ssms) or Visual Studio to connect to the database and then run the following queries.

- **Create Table1**

- ```sql

- CREATE TABLE [dbo].[Table1](

- [ID] [int] NOT NULL,

- [FName] [nvarchar](50) NOT NULL,

- [LName] [nvarchar](50) NOT NULL,

- CONSTRAINT [PK_Table_1] PRIMARY KEY CLUSTERED

- (

- [ID] ASC

- )

- ) ON [PRIMARY]

- GO

- ```

- **Create Table2**

- ```sql

- CREATE TABLE [dbo].[Table2](

- [ID] [int] NOT NULL,

- [FName] [nvarchar](50) NOT NULL,

- [LName] [nvarchar](50) NOT NULL,

- CONSTRAINT [PK_Table_2] PRIMARY KEY CLUSTERED

- (

- [ID] ASC

- )

- ) ON [PRIMARY]

- GO

- ```

-1. Run the following command to add some sample data to **Table1**. Leave **Table2** empty. Later, you'll import data from **Table1** into Data Lake Storage Gen1. Then, you'll export data from Data Lake Storage Gen1 into **Table2**.

- ```sql

- INSERT INTO [dbo].[Table1] VALUES (1,'Neal','Kell'), (2,'Lila','Fulton'), (3, 'Erna','Myers'), (4,'Annette','Simpson');

- ```

-## Use Sqoop from an HDInsight cluster with access to Data Lake Storage Gen1

-An HDInsight cluster already has the Sqoop packages available. If you've configured the HDInsight cluster to use Data Lake Storage Gen1 as additional storage, you can use Sqoop (without any configuration changes) to import/export data between a relational database such as Azure SQL Database, and a Data Lake Storage Gen1 account.

-1. For this article, we assume you created a Linux cluster so you should use SSH to connect to the cluster. See [Connect to a Linux-based HDInsight cluster](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-1. Verify whether you can access the Data Lake Storage Gen1 account from the cluster. Run the following command from the SSH prompt:

- ```console

- hdfs dfs -ls adl://<data_lake_storage_gen1_account>.azuredatalakestore.net/

- ```

- This command provides a list of files/folders in the Data Lake Storage Gen1 account.

-### Import data from Azure SQL Database into Data Lake Storage Gen1

-1. Navigate to the directory where Sqoop packages are available. Typically, this location is `/usr/hdp/<version>/sqoop/bin`.

-1. Import the data from **Table1** into the Data Lake Storage Gen1 account. Use the following syntax:

- ```console

- sqoop-import --connect "jdbc:sqlserver://<sql-database-server-name>.database.windows.net:1433;username=<username>@<sql-database-server-name>;password=<password>;database=<sql-database-name>" --table Table1 --target-dir adl://<data-lake-storage-gen1-name>.azuredatalakestore.net/Sqoop/SqoopImportTable1

- ```

- The **sql-database-server-name** placeholder represents the name of the server where the database is running. **sql-database-name** placeholder represents the actual database name.

- For example,

- ```console

- sqoop-import --connect "jdbc:sqlserver://mysqoopserver.database.windows.net:1433;username=user1@mysqoopserver;password=<password>;database=mysqoopdatabase" --table Table1 --target-dir adl://myadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1

- ```

-1. Verify that the data has been transferred to the Data Lake Storage Gen1 account. Run the following command:

- ```console

- hdfs dfs -ls adl://hdiadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1/

- ```

- You should see the following output.

- ```console

- -rwxrwxrwx 0 sshuser hdfs 0 2016-02-26 21:09 adl://hdiadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1/_SUCCESS

- -rwxrwxrwx 0 sshuser hdfs 12 2016-02-26 21:09 adl://hdiadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1/part-m-00000

- -rwxrwxrwx 0 sshuser hdfs 14 2016-02-26 21:09 adl://hdiadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1/part-m-00001

- -rwxrwxrwx 0 sshuser hdfs 13 2016-02-26 21:09 adl://hdiadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1/part-m-00002

- -rwxrwxrwx 0 sshuser hdfs 18 2016-02-26 21:09 adl://hdiadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1/part-m-00003

- ```

- Each **part-m-*** file corresponds to a row in the source table, **Table1**. You can view the contents of the part-m-* files to verify.

-### Export data from Data Lake Storage Gen1 into Azure SQL Database

-1. Export the data from the Data Lake Storage Gen1 account to the empty table, **Table2**, in the Azure SQL Database. Use the following syntax.

- ```console

- sqoop-export --connect "jdbc:sqlserver://<sql-database-server-name>.database.windows.net:1433;username=<username>@<sql-database-server-name>;password=<password>;database=<sql-database-name>" --table Table2 --export-dir adl://<data-lake-storage-gen1-name>.azuredatalakestore.net/Sqoop/SqoopImportTable1 --input-fields-terminated-by ","

- ```

- For example,

- ```console

- sqoop-export --connect "jdbc:sqlserver://mysqoopserver.database.windows.net:1433;username=user1@mysqoopserver;password=<password>;database=mysqoopdatabase" --table Table2 --export-dir adl://myadlsg1store.azuredatalakestore.net/Sqoop/SqoopImportTable1 --input-fields-terminated-by ","

- ```

-1. Verify that the data was uploaded to the SQL Database table. Use [SQL Server Management Studio](/azure/azure-sql/database/connect-query-ssms) or Visual Studio to connect to the Azure SQL Database and then run the following query.

- ```sql

- SELECT * FROM TABLE2

- ```

- This command should have the following output.

- ```output

- ID FName LName

- -

- 1 Neal Kell

- 2 Lila Fulton

- 3 Erna Myers

- 4 Annette Simpson

- ```

-## Performance considerations while using Sqoop

-For information about performance tuning your Sqoop job to copy data to Data Lake Storage Gen1, see the [Sqoop performance blog post](/archive/blogs/shanyu/performance-tuning-for-hdinsight-storm-and-microsoft-azure-eventhubs).

-## Next steps

-* [Copy data from Azure Storage Blobs to Data Lake Storage Gen1](data-lake-store-copy-data-azure-storage-blob.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-* [Use Azure HDInsight with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: 'Understand how to setup and access diagnostic logs for Azure Data Lake Storage Gen1 '

-# Accessing diagnostic logs for Azure Data Lake Storage Gen1

-Learn to enable diagnostic logging for your Azure Data Lake Storage Gen1 account and how to view the logs collected for your account.

-Organizations can enable diagnostic logging for their Azure Data Lake Storage Gen1 account to collect data access audit trails that provides information such as list of users accessing the data, how frequently the data is accessed, how much data is stored in the account, etc. When enabled, the diagnostics and/or requests are logged on a best-effort basis. Both Requests and Diagnostics log entries are created only if there are requests made against the service endpoint.

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure Data Lake Storage Gen1 account**. Follow the instructions at [Get started with Azure Data Lake Storage Gen1 using the Azure portal](data-lake-store-get-started-portal.md).

-## Enable diagnostic logging for your Data Lake Storage Gen1 account

-1. Sign on to the new [Azure portal](https://portal.azure.com).

-2. Open your Data Lake Storage Gen1 account, and from your Data Lake Storage Gen1 account blade, click **Diagnostic settings**.

-3. In the **Diagnostics settings** blade, click **Turn on diagnostics**.

- 

-3. In the **Diagnostics settings** blade, make the following changes to configure diagnostic logging.

-

- 

-

- * For **Name**, enter a value for the diagnostic log configuration.

- * You can choose to store/process the data in different ways.

-

- * Select the option to **Archive to a storage account** to store logs to an Azure Storage account. You use this option if you want to archive the data that will be batch-processed at a later date. If you select this option you must provide an Azure Storage account to save the logs to.

-

- * Select the option to **Stream to an event hub** to stream log data to an Azure Event Hub. Most likely you will use this option if you have a downstream processing pipeline to analyze incoming logs at real time. If you select this option, you must provide the details for the Azure Event Hub you want to use.

- * Select the option to **Send to Log Analytics** to use the Azure Monitor service to analyze the generated log data. If you select this option, you must provide the details for the Log Analytics workspace that you would use the perform log analysis. See [View or analyze data collected with Azure Monitor logs search](../azure-monitor/logs/log-analytics-tutorial.md) for details on using Azure Monitor logs.

-

- * Specify whether you want to get audit logs or request logs or both.

- * Specify the number of days for which the data must be retained. Retention is only applicable if you are using Azure storage account to archive log data.

- * Click **Save**.

-Once you have enabled diagnostic settings, you can watch the logs in the **Diagnostic Logs** tab.

-## View diagnostic logs for your Data Lake Storage Gen1 account

-There are two ways to view the log data for your Data Lake Storage Gen1 account.

-* From the Data Lake Storage Gen1 account settings view

-* From the Azure Storage account where the data is stored

-### Using the Data Lake Storage Gen1 Settings view

-1. From your Data Lake Storage Gen1 account **Settings** blade, click **Diagnostic Logs**.

-

- 

-2. In the **Diagnostics Logs** blade, you should see the logs categorized by **Audit Logs** and **Request Logs**.

-

- * Request logs capture every API request made on the Data Lake Storage Gen1 account.

- * Audit Logs are similar to request Logs but provide a much more detailed breakdown of the operations being performed on the Data Lake Storage Gen1 account. For example, a single upload API call in request logs might result in multiple "Append" operations in the audit logs.

-3. To download the logs, click the **Download** link against each log entry.

-### From the Azure Storage account that contains log data

-1. Open the Azure Storage account blade associated with Data Lake Storage Gen1 for logging, and then click Blobs. The **Blob service** blade lists two containers.

-

- 

-

- * The container **insights-logs-audit** contains the audit logs.

- * The container **insights-logs-requests** contains the request logs.

-2. Within these containers, the logs are stored under the following structure.

-

- 

-

- As an example, the complete path to an audit log could be `https://adllogs.blob.core.windows.net/insights-logs-audit/resourceId=/SUBSCRIPTIONS/<sub-id>/RESOURCEGROUPS/myresourcegroup/PROVIDERS/MICROSOFT.DATALAKESTORE/ACCOUNTS/mydatalakestorage/y=2016/m=07/d=18/h=04/m=00/PT1H.json`

-

- Similarly, the complete path to a request log could be `https://adllogs.blob.core.windows.net/insights-logs-requests/resourceId=/SUBSCRIPTIONS/<sub-id>/RESOURCEGROUPS/myresourcegroup/PROVIDERS/MICROSOFT.DATALAKESTORE/ACCOUNTS/mydatalakestorage/y=2016/m=07/d=18/h=14/m=00/PT1H.json`

-## Understand the structure of the log data

-The audit and request logs are in a JSON format. In this section, we look at the structure of JSON for request and audit logs.

-### Request logs

-Here's a sample entry in the JSON-formatted request log. Each blob has one root object called **records** that contains an array of log objects.

-```json

-{

-"records":

- [

- . . . .

- ,

- {

- "time": "2016-07-07T21:02:53.456Z",

- "resourceId": "/SUBSCRIPTIONS/<subscription_id>/RESOURCEGROUPS/<resource_group_name>/PROVIDERS/MICROSOFT.DATALAKESTORE/ACCOUNTS/<data_lake_storage_gen1_account_name>",

- "category": "Requests",

- "operationName": "GETCustomerIngressEgress",

- "resultType": "200",

- "callerIpAddress": "::ffff:1.1.1.1",

- "correlationId": "4a11c709-05f5-417c-a98d-6e81b3e29c58",

- "identity": "1808bd5f-62af-45f4-89d8-03c5e81bac30",

- "properties": {"HttpMethod":"GET","Path":"/webhdfs/v1/Samples/Outputs/Drivers.csv","RequestContentLength":0,"StoreIngressSize":0 ,"StoreEgressSize":4096,"ClientRequestId":"3b7adbd9-3519-4f28-a61c-bd89506163b8","StartTime":"2016-07-07T21:02:52.472Z","EndTime":"2016-07-07T21:02:53.456Z","QueryParameters":"api-version=<version>&op=<operationName>"}

- }

- ,

- . . . .

- ]

-}

-```

-#### Request log schema

-| Name | Type | Description |

-| | | |

-| time |String |The timestamp (in UTC) of the log |

-| resourceId |String |The ID of the resource that operation took place on |

-| category |String |The log category. For example, **Requests**. |

-| operationName |String |Name of the operation that is logged. For example, getfilestatus. |

-| resultType |String |The status of the operation, For example, 200. |

-| callerIpAddress |String |The IP address of the client making the request |

-| correlationId |String |The ID of the log that can used to group together a set of related log entries |

-| identity |Object |The identity that generated the log |

-| properties |JSON |See below for details |

-#### Request log properties schema

-| Name | Type | Description |

-| | | |

-| HttpMethod |String |The HTTP Method used for the operation. For example, GET. |

-| Path |String |The path the operation was performed on |

-| RequestContentLength |int |The content length of the HTTP request |

-| ClientRequestId |String |The ID that uniquely identifies this request |

-| StartTime |String |The time at which the server received the request |

-| EndTime |String |The time at which the server sent a response |

-| StoreIngressSize |Long |Size in bytes ingressed to Data Lake Store |

-| StoreEgressSize |Long |Size in bytes egressed from Data Lake Store |

-| QueryParameters |String |Description: These are the http query parameters. Example 1: api-version=2014-01-01&op=getfilestatus Example 2: op=APPEND&append=true&syncFlag=DATA&filesessionid=bee3355a-4925-4435-bb4d-ceea52811aeb&leaseid=bee3355a-4925-4435-bb4d-ceea52811aeb&offset=28313319&api-version=2017-08-01 |

-### Audit logs

-Here's a sample entry in the JSON-formatted audit log. Each blob has one root object called **records** that contains an array of log objects

-```json

-{

-"records":

- [

- . . . .

- ,

- {

- "time": "2016-07-08T19:08:59.359Z",

- "resourceId": "/SUBSCRIPTIONS/<subscription_id>/RESOURCEGROUPS/<resource_group_name>/PROVIDERS/MICROSOFT.DATALAKESTORE/ACCOUNTS/<data_lake_storage_gen1_account_name>",

- "category": "Audit",

- "operationName": "SeOpenStream",

- "resultType": "0",

- "resultSignature": "0",

- "correlationId": "381110fc03534e1cb99ec52376ceebdf;Append_BrEKAmg;25.66.9.145",

- "identity": "A9DAFFAF-FFEE-4BB5-A4A0-1B6CBBF24355",

- "properties": {"StreamName":"adl://<data_lake_storage_gen1_account_name>.azuredatalakestore.net/logs.csv"}

- }

- ,

- . . . .

- ]

-}

-```

-#### Audit log schema

-| Name | Type | Description |

-| | | |

-| time |String |The timestamp (in UTC) of the log |

-| resourceId |String |The ID of the resource that operation took place on |

-| category |String |The log category. For example, **Audit**. |

-| operationName |String |Name of the operation that is logged. For example, getfilestatus. |

-| resultType |String |The status of the operation, For example, 200. |

-| resultSignature |String |Additional details on the operation. |

-| correlationId |String |The ID of the log that can used to group together a set of related log entries |

-| identity |Object |The identity that generated the log |

-| properties |JSON |See below for details |

-#### Audit log properties schema

-| Name | Type | Description |

-| | | |

-| StreamName |String |The path the operation was performed on |

-## Samples to process the log data

-When sending logs from Azure Data Lake Storage Gen1 to Azure Monitor logs (see [View or analyze data collected with Azure Monitor logs search](../azure-monitor/logs/log-analytics-tutorial.md) for details on using Azure Monitor logs), the following query will return a table containing a list of user display names, the time of the events, and the count of events for the time of the event along with a visual chart. It can easily be modified to show user GUID or other attributes:

-```

-search *

-| where ( Type == "AzureDiagnostics" )

-| summarize count(TimeGenerated) by identity_s, TimeGenerated

-```

-Azure Data Lake Storage Gen1 provides a sample on how to process and analyze the log data. You can find the sample at [https://github.com/Azure/AzureDataLake/tree/master/Samples/AzureDiagnosticsSample](https://github.com/Azure/AzureDataLake/tree/master/Samples/AzureDiagnosticsSample).

-## See also

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-description: Learn how to further protect your data from region-wide outages or accidental deletions beyond the locally redundant storage of Azure Data Lake Storage Gen1.

-# High availability and disaster recovery guidance for Data Lake Storage Gen1

-Data Lake Storage Gen1 provides locally redundant storage (LRS). Therefore, the data in your Data Lake Storage Gen1 account is resilient to transient hardware failures within a datacenter through automated replicas. This ensures durability and high availability, meeting the Data Lake Storage Gen1 SLA. This article provides guidance on how to further protect your data from rare region-wide outages or accidental deletions.

-## Disaster recovery guidance

-It's critical for you to prepare a disaster recovery plan. Review the information in this article and these additional resources to help you create your own plan.

-* [Disaster recovery and high availability for Azure applications](/azure/architecture/framework/resiliency/backup-and-recovery)

-* [Azure resiliency technical guidance](/azure/architecture/framework/resiliency/app-design)

-### Best practice recommendations

-We recommend that you copy your critical data to another Data Lake Storage Gen1 account in another region with a frequency aligned to the needs of your disaster recovery plan. There are a variety of methods to copy data including [ADLCopy](data-lake-store-copy-data-azure-storage-blob.md), [Azure PowerShell](data-lake-store-get-started-powershell.md), or [Azure Data Factory](../data-factory/connector-azure-data-lake-store.md). Azure Data Factory is a useful service for creating and deploying data movement pipelines on a recurring basis.

-If a regional outage occurs, you can then access your data in the region where the data was copied. You can monitor the [Azure Service Health Dashboard](https://azure.microsoft.com/status/) to determine the Azure service status across the globe.

-## Data corruption or accidental deletion recovery guidance

-While Data Lake Storage Gen1 provides data resiliency through automated replicas, this does not prevent your application (or developers/users) from corrupting data or accidentally deleting it.

-To prevent accidental deletion, we recommend that you first set the correct access policies for your Data Lake Storage Gen1 account. This includes applying [Azure resource locks](../azure-resource-manager/management/lock-resources.md) to lock down important resources and applying account and file level access control using the available [Data Lake Storage Gen1 security features](data-lake-store-security-overview.md). We also recommend that you routinely create copies of your critical data using [ADLCopy](data-lake-store-copy-data-azure-storage-blob.md), [Azure PowerShell](data-lake-store-get-started-powershell.md) or [Azure Data Factory](../data-factory/connector-azure-data-lake-store.md) in another Data Lake Storage Gen1 account, folder, or Azure subscription. This can be used to recover from a data corruption or deletion incident. Azure Data Factory is a useful service for creating and deploying data movement pipelines on a recurring basis.

-You can also enable [diagnostic logging](data-lake-store-diagnostic-logs.md) for a Data Lake Storage Gen1 account to collect data access audit trails. The audit trails provide information about who might have deleted or updated a file.

-## Next steps

-* [Get started with Data Lake Storage Gen1](data-lake-store-get-started-portal.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-description: Encryption in Azure Data Lake Storage Gen1 helps you protect your data, implement enterprise security policies, and meet regulatory compliance requirements. This article provides an overview of the design, and discusses some of the technical aspects of implementation.

-# Encryption of data in Azure Data Lake Storage Gen1

-Encryption in Azure Data Lake Storage Gen1 helps you protect your data, implement enterprise security policies, and meet regulatory compliance requirements. This article provides an overview of the design, and discusses some of the technical aspects of implementation.

-Data Lake Storage Gen1 supports encryption of data both at rest and in transit. For data at rest, Data Lake Storage Gen1 supports "on by default," transparent encryption. Here is what these terms mean in a bit more detail:

-* **On by default**: When you create a new Data Lake Storage Gen1 account, the default setting enables encryption. Thereafter, data that is stored in Data Lake Storage Gen1 is always encrypted prior to storing on persistent media. This is the behavior for all data, and it cannot be changed after an account is created.

-* **Transparent**: Data Lake Storage Gen1 automatically encrypts data prior to persisting, and decrypts data prior to retrieval. The encryption is configured and managed at the Data Lake Storage Gen1 account level by an administrator. No changes are made to the data access APIs. Thus, no changes are required in applications and services that interact with Data Lake Storage Gen1 because of encryption.

-Data in transit (also known as data in motion) is also always encrypted in Data Lake Storage Gen1. In addition to encrypting data prior to storing to persistent media, the data is also always secured in transit by using HTTPS. HTTPS is the only protocol that is supported for the Data Lake Storage Gen1 REST interfaces. The following diagram shows how data becomes encrypted in Data Lake Storage Gen1:

-

-## Set up encryption with Data Lake Storage Gen1

-Encryption for Data Lake Storage Gen1 is set up during account creation, and it is always enabled by default. You can either manage the keys yourself, or allow Data Lake Storage Gen1 to manage them for you (this is the default).

-For more information, see [Getting started](./data-lake-store-get-started-portal.md).

-## How encryption works in Data Lake Storage Gen1

-The following information covers how to manage master encryption keys, and it explains the three different types of keys you can use in data encryption for Data Lake Storage Gen1.

-### Master encryption keys

-Data Lake Storage Gen1 provides two modes for management of master encryption keys (MEKs). For now, assume that the master encryption key is the top-level key. Access to the master encryption key is required to decrypt any data stored in Data Lake Storage Gen1.

-The two modes for managing the master encryption key are as follows:

-* Service managed keys

-* Customer managed keys

-In both modes, the master encryption key is secured by storing it in Azure Key Vault. Key Vault is a fully managed, highly secure service on Azure that can be used to safeguard cryptographic keys. For more information, see [Key Vault](https://azure.microsoft.com/services/key-vault).

-Here is a brief comparison of capabilities provided by the two modes of managing the MEKs.

-| Question | Service managed keys | Customer managed keys |

-| -- | -- | |

-|How is data stored?|Always encrypted prior to being stored.|Always encrypted prior to being stored.|

-|Where is the Master Encryption Key stored?|Key Vault|Key Vault|

-|Are any encryption keys stored in the clear outside of Key Vault? |No|No|

-|Can the MEK be retrieved by Key Vault?|No. After the MEK is stored in Key Vault, it can only be used for encryption and decryption.|No. After the MEK is stored in Key Vault, it can only be used for encryption and decryption.|

-|Who owns the Key Vault instance and the MEK?|The Data Lake Storage Gen1 service|You own the Key Vault instance, which belongs in your own Azure subscription. The MEK in Key Vault can be managed by software or hardware.|

-|Can you revoke access to the MEK for the Data Lake Storage Gen1 service?|No|Yes. You can manage access control lists in Key Vault, and remove access control entries to the service identity for the Data Lake Storage Gen1 service.|

-|Can you permanently delete the MEK?|No|Yes. If you delete the MEK from Key Vault, the data in the Data Lake Storage Gen1 account cannot be decrypted by anyone, including the Data Lake Storage Gen1 service. <br><br> If you have explicitly backed up the MEK prior to deleting it from Key Vault, the MEK can be restored, and the data can then be recovered. However, if you have not backed up the MEK prior to deleting it from Key Vault, the data in the Data Lake Storage Gen1 account can never be decrypted thereafter.|

-Aside from this difference of who manages the MEK and the Key Vault instance in which it resides, the rest of the design is the same for both modes.

-It's important to remember the following when you choose the mode for the master encryption keys:

-* You can choose whether to use customer managed keys or service managed keys when you provision a Data Lake Storage Gen1 account.

-* After a Data Lake Storage Gen1 account is provisioned, the mode cannot be changed.

-### Encryption and decryption of data

-There are three types of keys that are used in the design of data encryption. The following table provides a summary:

-| Key | Abbreviation | Associated with | Storage location | Type | Notes |

-|--|--|--|-|||

-| Master Encryption Key | MEK | A Data Lake Storage Gen1 account | Key Vault | Asymmetric | It can be managed by Data Lake Storage Gen1 or you. |

-| Data Encryption Key | DEK | A Data Lake Storage Gen1 account | Persistent storage, managed by the Data Lake Storage Gen1 service | Symmetric | The DEK is encrypted by the MEK. The encrypted DEK is what is stored on persistent media. |

-| Block Encryption Key | BEK | A block of data | None | Symmetric | The BEK is derived from the DEK and the data block. |

-The following diagram illustrates these concepts:

-

-#### Pseudo algorithm when a file is to be decrypted:

-1. Check if the DEK for the Data Lake Storage Gen1 account is cached and ready for use.

- - If not, then read the encrypted DEK from persistent storage, and send it to Key Vault to be decrypted. Cache the decrypted DEK in memory. It is now ready to use.

-2. For every block of data in the file:

- - Read the encrypted block of data from persistent storage.

- - Generate the BEK from the DEK and the encrypted block of data.

- - Use the BEK to decrypt data.

-#### Pseudo algorithm when a block of data is to be encrypted:

-1. Check if the DEK for the Data Lake Storage Gen1 account is cached and ready for use.

- - If not, then read the encrypted DEK from persistent storage, and send it to Key Vault to be decrypted. Cache the decrypted DEK in memory. It is now ready to use.

-2. Generate a unique BEK for the block of data from the DEK.

-3. Encrypt the data block with the BEK, by using AES-256 encryption.

-4. Store the encrypted data block of data on persistent storage.

-> [!NOTE]

-> The DEK is always stored encrypted by the MEK, whether on persistent media or cached in memory.

-## Key rotation

-When you are using customer-managed keys, you can rotate the MEK. To learn how to set up a Data Lake Storage Gen1 account with customer-managed keys, see [Getting started](./data-lake-store-get-started-portal.md).

-### Prerequisites

-When you set up the Data Lake Storage Gen1 account, you have chosen to use your own keys. This option cannot be changed after the account has been created. The following steps assume that you are using customer-managed keys (that is, you have chosen your own keys from Key Vault).

-Note that if you use the default options for encryption, your data is always encrypted by using keys managed by Data Lake Storage Gen1. In this option, you don't have the ability to rotate keys, as they are managed by Data Lake Storage Gen1.

-### How to rotate the MEK in Data Lake Storage Gen1

-1. Sign in to the [Azure portal](https://portal.azure.com/).

-2. Browse to the Key Vault instance that stores your keys associated with your Data Lake Storage Gen1 account. Select **Keys**.

- 

-3. Select the key associated with your Data Lake Storage Gen1 account, and create a new version of this key. Note that Data Lake Storage Gen1 currently only supports key rotation to a new version of a key. It doesn't support rotating to a different key.

- 

-4. Browse to the Data Lake Storage Gen1 account, and select **Encryption**.

- 

-5. A message notifies you that a new key version of the key is available. Click **Rotate Key** to update the key to the new version.

- 

-This operation should take less than two minutes, and there is no expected downtime due to key rotation. After the operation is complete, the new version of the key is in use.

-> [!IMPORTANT]

-> After the key rotation operation is complete, the old version of the key is no longer actively used for encrypting new data. There may be cases however where accessing older data may need the old key. To allow for reading of such older data, do not delete the old key

-description: Learn how to achieve end-user authentication with Azure Data Lake Storage Gen1 using Microsoft Entra ID with Java

-# End-user authentication with Azure Data Lake Storage Gen1 using Java

-> [!div class="op_single_selector"]

-> * [Using Java](data-lake-store-end-user-authenticate-java-sdk.md)

-> * [Using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md)

-> * [Using Python](data-lake-store-end-user-authenticate-python.md)

-> * [Using REST API](data-lake-store-end-user-authenticate-rest-api.md)

->

->

-In this article, you learn about how to use the Java SDK to do end-user authentication with Azure Data Lake Storage Gen1. For service-to-service authentication with Data Lake Storage Gen1 using Java SDK, see [Service-to-service authentication with Data Lake Storage Gen1 using Java](data-lake-store-service-to-service-authenticate-java.md).

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Create a Microsoft Entra ID "Native" Application**. You must have completed the steps in [End-user authentication with Data Lake Storage Gen1 using Microsoft Entra ID](data-lake-store-end-user-authenticate-using-active-directory.md).

-* [Maven](https://maven.apache.org/install.html). This tutorial uses Maven for build and project dependencies. Although it is possible to build without using a build system like Maven or Gradle, these systems make is much easier to manage dependencies.

-* (Optional) And IDE like [IntelliJ IDEA](https://www.jetbrains.com/idea/download/) or [Eclipse](https://www.eclipse.org/downloads/) or similar.

-## End-user authentication

-1. Create a Maven project using [mvn archetype](https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html) from the command line or using an IDE. For instructions on how to create a Java project using IntelliJ, see [here](https://www.jetbrains.com/help/idea/2016.1/creating-and-running-your-first-java-application.html). For instructions on how to create a project using Eclipse, see [here](https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2FgettingStarted%2Fqs-3.htm).

-2. Add the following dependencies to your Maven **pom.xml** file. Add the following snippet before the **\</project>** tag:

-

- ```xml

- <dependencies>

- <dependency>

- <groupId>com.microsoft.azure</groupId>

- <artifactId>azure-data-lake-store-sdk</artifactId>

- <version>2.2.3</version>

- </dependency>

- <dependency>

- <groupId>org.slf4j</groupId>

- <artifactId>slf4j-nop</artifactId>

- <version>1.7.21</version>

- </dependency>

- </dependencies>

- ```

-

- The first dependency is to use the Data Lake Storage Gen1 SDK (`azure-data-lake-store-sdk`) from the maven repository. The second dependency is to specify the logging framework (`slf4j-nop`) to use for this application. The Data Lake Storage Gen1 SDK uses [SLF4J](https://www.slf4j.org/) logging façade, which lets you choose from a number of popular logging frameworks, like Log4j, Java logging, Logback, etc., or no logging. For this example, we disable logging, hence we use the **slf4j-nop** binding. To use other logging options in your app, see [here](https://www.slf4j.org/manual.html#projectDep).

-3. Add the following import statements to your application.

- ```java

- import com.microsoft.azure.datalake.store.ADLException;

- import com.microsoft.azure.datalake.store.ADLStoreClient;

- import com.microsoft.azure.datalake.store.DirectoryEntry;

- import com.microsoft.azure.datalake.store.IfExists;

- import com.microsoft.azure.datalake.store.oauth2.AccessTokenProvider;

- import com.microsoft.azure.datalake.store.oauth2.DeviceCodeTokenProvider;

- ```

-4. Use the following snippet in your Java application to obtain token for the Active Directory native application you created earlier using the `DeviceCodeTokenProvider`. Replace **FILL-IN-HERE** with the actual values for the Microsoft Entra native application.

- ```java

- private static String nativeAppId = "FILL-IN-HERE";

-

- AccessTokenProvider provider = new DeviceCodeTokenProvider(nativeAppId);

- ```

-The Data Lake Storage Gen1 SDK provides convenient methods that let you manage the security tokens needed to talk to the Data Lake Storage Gen1 account. However, the SDK does not mandate that only these methods be used. You can use any other means of obtaining token as well, like using the [Azure AD SDK](https://github.com/AzureAD/azure-activedirectory-library-for-java), or your own custom code.

-## Next steps

-In this article, you learned how to use end-user authentication to authenticate with Azure Data Lake Storage Gen1 using Java SDK. You can now look at the following articles that talk about how to use the Java SDK to work with Azure Data Lake Storage Gen1.

-* [Data operations on Data Lake Storage Gen1 using Java SDK](data-lake-store-get-started-java-sdk.md)

-description: Learn how to achieve end-user authentication with Azure Data Lake Storage Gen1 using Microsoft Entra ID with .NET SDK

-# End-user authentication with Azure Data Lake Storage Gen1 using .NET SDK

-> [!div class="op_single_selector"]

-> * [Using Java](data-lake-store-end-user-authenticate-java-sdk.md)

-> * [Using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md)

-> * [Using Python](data-lake-store-end-user-authenticate-python.md)

-> * [Using REST API](data-lake-store-end-user-authenticate-rest-api.md)

->

->

-In this article, you learn about how to use the .NET SDK to do end-user authentication with Azure Data Lake Storage Gen1. For service-to-service authentication with Data Lake Storage Gen1 using .NET SDK, see [Service-to-service authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-service-to-service-authenticate-net-sdk.md).

-## Prerequisites

-* **Visual Studio 2013 or above**. The instructions below use Visual Studio 2019.

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Create a Microsoft Entra ID "Native" Application**. You must have completed the steps in [End-user authentication with Data Lake Storage Gen1 using Microsoft Entra ID](data-lake-store-end-user-authenticate-using-active-directory.md).

-## Create a .NET application

-1. In Visual Studio, select the **File** menu, **New**, and then **Project**.

-2. Choose **Console App (.NET Framework)**, and then select **Next**.

-3. In **Project name**, enter `CreateADLApplication`, and then select **Create**.

-4. Add the NuGet packages to your project.

- 1. Right-click the project name in the Solution Explorer and click **Manage NuGet Packages**.

- 2. In the **NuGet Package Manager** tab, make sure that **Package source** is set to **nuget.org** and that **Include prerelease** check box is selected.

- 3. Search for and install the following NuGet packages:

- * `Microsoft.Azure.Management.DataLake.Store` - This tutorial uses v2.1.3-preview.

- * `Microsoft.Rest.ClientRuntime.Azure.Authentication` - This tutorial uses v2.2.12.

- 

- 4. Close the **NuGet Package Manager**.

-5. Open **Program.cs**

-6. Replace the using statements with the following lines:

- ```csharp

- using System;

- using System.IO;

- using System.Linq;

- using System.Text;

- using System.Threading;

- using System.Collections.Generic;

-

- using Microsoft.Rest;

- using Microsoft.Rest.Azure.Authentication;

- using Microsoft.Azure.Management.DataLake.Store;

- using Microsoft.Azure.Management.DataLake.Store.Models;

- using Microsoft.IdentityModel.Clients.ActiveDirectory;

- ```

-## End-user authentication

-Add this snippet in your .NET client application. Replace the placeholder values with the values retrieved from a Microsoft Entra native application (listed as prerequisite). This snippet lets you authenticate your application **interactively** with Data Lake Storage Gen1, which means you are prompted to enter your Azure credentials.

-For ease of use, the following snippet uses default values for client ID and redirect URI that are valid for any Azure subscription. In the following snippet, you only need to provide the value for your tenant ID. You can retrieve the Tenant ID using the instructions provided at [Get the tenant ID](../active-directory/develop/howto-create-service-principal-portal.md#sign-in-to-the-application).

-

- ```csharp

- private static void Main(string[] args)

- {

- //User login via interactive popup

- string TENANT = "<AAD-directory-domain>";

- string CLIENTID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

- System.Uri ARM_TOKEN_AUDIENCE = new System.Uri(@"https://management.core.windows.net/");

- System.Uri ADL_TOKEN_AUDIENCE = new System.Uri(@"https://datalake.azure.net/");

- string MY_DOCUMENTS = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyDocuments);

- string TOKEN_CACHE_PATH = System.IO.Path.Combine(MY_DOCUMENTS, "my.tokencache");

- var tokenCache = GetTokenCache(TOKEN_CACHE_PATH);

- var armCreds = GetCreds_User_Popup(TENANT, ARM_TOKEN_AUDIENCE, CLIENTID, tokenCache);

- var adlCreds = GetCreds_User_Popup(TENANT, ADL_TOKEN_AUDIENCE, CLIENTID, tokenCache);

- }

- ```

-A couple of things to know about the preceding snippet:

-* The preceding snippet uses a helper functions `GetTokenCache` and `GetCreds_User_Popup`. The code for these helper functions is available [here on GitHub](https://github.com/Azure-Samples/data-lake-analytics-dotnet-auth-options#gettokencache).

-* To help you complete the tutorial faster, the snippet uses a native application client ID that is available by default for all Azure subscriptions. So, you can **use this snippet as-is in your application**.

-* However, if you do want to use your own Microsoft Entra domain and application client ID, you must create a Microsoft Entra native application and then use the Microsoft Entra tenant ID, client ID, and redirect URI for the application you created. See [Create an Active Directory Application for end-user authentication with Data Lake Storage Gen1](data-lake-store-end-user-authenticate-using-active-directory.md) for instructions.

-

-## Next steps

-In this article, you learned how to use end-user authentication to authenticate with Azure Data Lake Storage Gen1 using .NET SDK. You can now look at the following articles that talk about how to use the .NET SDK to work with Azure Data Lake Storage Gen1.

-* [Account management operations on Data Lake Storage Gen1 using .NET SDK](data-lake-store-get-started-net-sdk.md)

-* [Data operations on Data Lake Storage Gen1 using .NET SDK](data-lake-store-data-operations-net-sdk.md)

-description: Learn how to achieve end-user authentication with Azure Data Lake Storage Gen1 using Microsoft Entra ID with Python

-# End-user authentication with Azure Data Lake Storage Gen1 using Python

-> [!div class="op_single_selector"]

-> * [Using Java](data-lake-store-end-user-authenticate-java-sdk.md)

-> * [Using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md)

-> * [Using Python](data-lake-store-end-user-authenticate-python.md)

-> * [Using REST API](data-lake-store-end-user-authenticate-rest-api.md)

->

->

-In this article, you learn about how to use the Python SDK to do end-user authentication with Azure Data Lake Storage Gen1. End-user authentication can further be split into two categories:

-* End-user authentication without multi-factor authentication

-* End-user authentication with multi-factor authentication

-Both these options are discussed in this article. For service-to-service authentication with Data Lake Storage Gen1 using Python, see [Service-to-service authentication with Data Lake Storage Gen1 using Python](data-lake-store-service-to-service-authenticate-python.md).

-## Prerequisites

-* **Python**. You can download Python from [here](https://www.python.org/downloads/). This article uses Python 3.6.2.

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Create a Microsoft Entra ID "Native" Application**. You must have completed the steps in [End-user authentication with Data Lake Storage Gen1 using Microsoft Entra ID](data-lake-store-end-user-authenticate-using-active-directory.md).

-## Install the modules

-To work with Data Lake Storage Gen1 using Python, you need to install three modules.

-* The `azure-mgmt-resource` module, which includes Azure modules for Active Directory, etc.

-* The `azure-mgmt-datalake-store` module, which includes the Azure Data Lake Storage Gen1 account management operations. For more information on this module, see [Azure Data Lake Storage Gen1 Management module reference](/python/api/azure-mgmt-datalake-store/).

-* The `azure-datalake-store` module, which includes the Azure Data Lake Storage Gen1 filesystem operations. For more information on this module, see [azure-datalake-store Filesystem module reference](/python/api/azure-datalake-store/azure.datalake.store.core/).

-Use the following commands to install the modules.

-```console

-pip install azure-mgmt-resource

-pip install azure-mgmt-datalake-store

-pip install azure-datalake-store

-```

-## Create a new Python application

-1. In the IDE of your choice, create a new Python application, for example, `mysample.py`.

-2. Add the following snippet to import the required modules

- ```python

- ## Use this for Azure AD authentication

- from msrestazure.azure_active_directory import AADTokenCredentials

- ## Required for Azure Data Lake Storage Gen1 account management

- from azure.mgmt.datalake.store import DataLakeStoreAccountManagementClient

- from azure.mgmt.datalake.store.models import DataLakeStoreAccount

- ## Required for Azure Data Lake Storage Gen1 filesystem management

- from azure.datalake.store import core, lib, multithread

- # Common Azure imports

- import adal

- from azure.mgmt.resource.resources import ResourceManagementClient

- from azure.mgmt.resource.resources.models import ResourceGroup

- ## Use these as needed for your application

- import logging, pprint, uuid, time

- ```

-3. Save changes to `mysample.py`.

-## End-user authentication with multi-factor authentication

-### For account management

-Use the following snippet to authenticate with Microsoft Entra ID for account management operations on a Data Lake Storage Gen1 account. The following snippet can be used to authenticate your application using multi-factor authentication. Provide the values below for an existing Microsoft Entra ID **native** application.

-```python

-authority_host_url = "https://login.microsoftonline.com"

-tenant = "FILL-IN-HERE"

-authority_url = authority_host_url + '/' + tenant

-client_id = 'FILL-IN-HERE'

-redirect = 'urn:ietf:wg:oauth:2.0:oob'

-RESOURCE = 'https://management.core.windows.net/'

-context = adal.AuthenticationContext(authority_url)

-code = context.acquire_user_code(RESOURCE, client_id)

-print(code['message'])

-mgmt_token = context.acquire_token_with_device_code(RESOURCE, code, client_id)

-armCreds = AADTokenCredentials(mgmt_token, client_id, resource = RESOURCE)

-```

-### For filesystem operations

-Use this to authenticate with Microsoft Entra ID for filesystem operations on a Data Lake Storage Gen1 account. The following snippet can be used to authenticate your application using multi-factor authentication. Provide the values below for an existing Microsoft Entra ID **native** application.

-```console

-adlCreds = lib.auth(tenant_id='FILL-IN-HERE', resource = 'https://datalake.azure.net/')

-```

-## End-user authentication without multi-factor authentication

-This is deprecated. For more information, see [Azure Authentication using Python SDK](/azure/developer/python/sdk/authentication-overview).

-## Next steps

-In this article, you learned how to use end-user authentication to authenticate with Azure Data Lake Storage Gen1 using Python. You can now look at the following articles that talk about how to use Python to work with Azure Data Lake Storage Gen1.

-* [Account management operations on Data Lake Storage Gen1 using Python](data-lake-store-get-started-python.md)

-* [Data operations on Data Lake Storage Gen1 using Python](data-lake-store-data-operations-python.md)

-description: Learn how to achieve end-user authentication with Azure Data Lake Storage Gen1 using Microsoft Entra ID using REST API

-# End-user authentication with Azure Data Lake Storage Gen1 using REST API

-> [!div class="op_single_selector"]

-> * [Using Java](data-lake-store-end-user-authenticate-java-sdk.md)

-> * [Using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md)

-> * [Using Python](data-lake-store-end-user-authenticate-python.md)

-> * [Using REST API](data-lake-store-end-user-authenticate-rest-api.md)

->

->

-In this article, you learn about how to use the REST API to do end-user authentication with Azure Data Lake Storage Gen1. For service-to-service authentication with Data Lake Storage Gen1 using REST API, see [Service-to-service authentication with Data Lake Storage Gen1 using REST API](data-lake-store-service-to-service-authenticate-rest-api.md).

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Create a Microsoft Entra ID "Native" Application**. You must have completed the steps in [End-user authentication with Data Lake Storage Gen1 using Microsoft Entra ID](data-lake-store-end-user-authenticate-using-active-directory.md).

-* **[cURL](https://curl.haxx.se/)**. This article uses cURL to demonstrate how to make REST API calls against a Data Lake Storage Gen1 account.

-## End-user authentication

-End-user authentication is the recommended approach if you want a user to log in to your application using Microsoft Entra ID. Your application is able to access Azure resources with the same level of access as the logged-in user. The user needs to provide their credentials periodically in order for your application to maintain access.

-The result of having the end-user login is that your application is given an access token and a refresh token. The access token gets attached to each request made to Data Lake Storage Gen1 or Data Lake Analytics, and it is valid for one hour by default. The refresh token can be used to obtain a new access token, and it is valid for up to two weeks by default, if used regularly. You can use two different approaches for end-user login.

-In this scenario, the application prompts the user to log in and all the operations are performed in the context of the user. Perform the following steps:

-1. Through your application, redirect the user to the following URL:

- `https://login.microsoftonline.com/<TENANT-ID>/oauth2/authorize?client_id=<APPLICATION-ID>&response_type=code&redirect_uri=<REDIRECT-URI>`

- > [!NOTE]

- > \<REDIRECT-URI> needs to be encoded for use in a URL. So, for https://localhost, use `https%3A%2F%2Flocalhost`)

- For the purpose of this tutorial, you can replace the placeholder values in the URL above and paste it in a web browser's address bar. You will be redirected to authenticate using your Azure login. Once you successfully log in, the response is displayed in the browser's address bar. The response will be in the following format:

- `http://localhost/?code=<AUTHORIZATION-CODE>&session_state=<GUID>`

-2. Capture the authorization code from the response. For this tutorial, you can copy the authorization code from the address bar of the web browser and pass it in the POST request to the token endpoint, as shown in the following snippet:

- ```console

- curl -X POST https://login.microsoftonline.com/<TENANT-ID>/oauth2/token \

- -F redirect_uri=<REDIRECT-URI> \

- -F grant_type=authorization_code \

- -F resource=https://management.core.windows.net/ \

- -F client_id=<APPLICATION-ID> \

- -F code=<AUTHORIZATION-CODE>

- ```

- > [!NOTE]

- > In this case, the \<REDIRECT-URI> need not be encoded.

- >

- >

-3. The response is a JSON object that contains an access token (for example, `"access_token": "<ACCESS_TOKEN>"`) and a refresh token (for example, `"refresh_token": "<REFRESH_TOKEN>"`). Your application uses the access token when accessing Azure Data Lake Storage Gen1 and the refresh token to get another access token when an access token expires.

- ```json

- {"token_type":"Bearer","scope":"user_impersonation","expires_in":"3599","expires_on":"1461865782","not_before": "1461861882","resource":"https://management.core.windows.net/","access_token":"<REDACTED>","refresh_token":"<REDACTED>","id_token":"<REDACTED>"}

- ```

-4. When the access token expires, you can request a new access token using the refresh token, as shown in the following snippet:

- ```console

- curl -X POST https://login.microsoftonline.com/<TENANT-ID>/oauth2/token \

- -F grant_type=refresh_token \

- -F resource=https://management.core.windows.net/ \

- -F client_id=<APPLICATION-ID> \

- -F refresh_token=<REFRESH-TOKEN>

- ```

-For more information on interactive user authentication, see [Authorization code grant flow](/previous-versions/azure/dn645542(v=azure.100)).

-## Next steps

-In this article, you learned how to use service-to-service authentication to authenticate with Azure Data Lake Storage Gen1 using REST API. You can now look at the following articles that talk about how to use the REST API to work with Azure Data Lake Storage Gen1.

-* [Account management operations on Data Lake Storage Gen1 using REST API](data-lake-store-get-started-rest-api.md)

-* [Data operations on Data Lake Storage Gen1 using REST API](data-lake-store-data-operations-rest-api.md)

-description: Learn how to achieve end-user authentication with Azure Data Lake Storage Gen1 using Microsoft Entra ID

-# End-user authentication with Azure Data Lake Storage Gen1 using Microsoft Entra ID

-> [!div class="op_single_selector"]

-> * [End-user authentication](data-lake-store-end-user-authenticate-using-active-directory.md)

-> * [Service-to-service authentication](data-lake-store-service-to-service-authenticate-using-active-directory.md)

->

->

-Azure Data Lake Storage Gen1 uses Microsoft Entra ID for authentication. Before authoring an application that works with Data Lake Storage Gen1 or Azure Data Lake Analytics, you must decide how to authenticate your application with Microsoft Entra ID. The two main options available are:

-* End-user authentication (this article)

-* Service-to-service authentication (pick this option from the drop-down above)

-Both these options result in your application being provided with an OAuth 2.0 token, which gets attached to each request made to Data Lake Storage Gen1 or Azure Data Lake Analytics.

-This article talks about how to create an **Microsoft Entra native application for end-user authentication**. For instructions on Microsoft Entra application configuration for service-to-service authentication, see [Service-to-service authentication with Data Lake Storage Gen1 using Microsoft Entra ID](./data-lake-store-service-to-service-authenticate-using-active-directory.md).

-## Prerequisites

-* An Azure subscription. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* Your subscription ID. You can retrieve it from the Azure portal. For example, it's available from the Data Lake Storage Gen1 account blade.

- 

-* Your Microsoft Entra domain name. You can retrieve it by hovering the mouse in the top-right corner of the Azure portal. From the screenshot below, the domain name is **contoso.onmicrosoft.com**, and the GUID within brackets is the tenant ID.

- 

-* Your Azure tenant ID. For instructions on how to retrieve the tenant ID, see [Get the tenant ID](../active-directory/develop/howto-create-service-principal-portal.md#sign-in-to-the-application).

-## End-user authentication

-This authentication mechanism is the recommended approach if you want an end user to sign in to your application via Microsoft Entra ID. Your application is then able to access Azure resources with the same level of access as the end user that logged in. Your end user needs to provide their credentials periodically in order for your application to maintain access.

-The result of having the end-user sign-in is that your application is given an access token and a refresh token. The access token gets attached to each request made to Data Lake Storage Gen1 or Data Lake Analytics, and it's valid for one hour by default. The refresh token can be used to obtain a new access token, and it's valid for up to two weeks by default. You can use two different approaches for end-user sign-in.

-### Using the OAuth 2.0 pop-up

-Your application can trigger an OAuth 2.0 authorization pop-up, in which the end user can enter their credentials. This pop-up also works with the Microsoft Entra Two-factor Authentication (2FA) process, if necessary.

-> [!NOTE]

-> This method is not yet supported in the Azure AD Authentication Library (ADAL) for Python or Java.

->

->

-### Directly passing in user credentials

-Your application can directly provide user credentials to Microsoft Entra ID. This method only works with organizational ID user accounts; it isn't compatible with personal / ΓÇ£live IDΓÇ¥ user accounts, including the accounts ending in @outlook.com or @live.com. Furthermore, this method isn't compatible with user accounts that require Microsoft Entra Two-factor Authentication (2FA).

-### What do I need for this approach?

-* Microsoft Entra domain name. This requirement is already listed in the prerequisite of this article.

-* Microsoft Entra tenant ID. This requirement is already listed in the prerequisite of this article.

-* Microsoft Entra ID **native application**

-* Application ID for the Microsoft Entra native application

-* Redirect URI for the Microsoft Entra native application

-* Set delegated permissions

-## Step 1: Create an Active Directory native application

-Create and configure a Microsoft Entra native application for end-user authentication with Data Lake Storage Gen1 using Microsoft Entra ID. For instructions, see [Create a Microsoft Entra application](../active-directory/develop/howto-create-service-principal-portal.md).

-While following the instructions in the link, make sure you select **Native** for application type, as shown in the following screenshot:

-

-## Step 2: Get application ID and redirect URI

-See [Get the application ID](../active-directory/develop/howto-create-service-principal-portal.md#sign-in-to-the-application) to retrieve the application ID.

-To retrieve the redirect URI, do the following steps.

-1. From the Azure portal, select **Microsoft Entra ID**, select **App registrations**, and then find and select the Microsoft Entra native application that you created.

-2. From the **Settings** blade for the application, select **Redirect URIs**.

- 

-3. Copy the value displayed.

-## Step 3: Set permissions

-1. From the Azure portal, select **Microsoft Entra ID**, select **App registrations**, and then find and select the Microsoft Entra native application that you created.

-2. From the **Settings** blade for the application, select **Required permissions**, and then select **Add**.

- 

-3. In the **Add API Access** blade, select **Select an API**, select **Azure Data Lake**, and then select **Select**.

- 

-4. In the **Add API Access** blade, select **Select permissions**, select the check box to give **Full access to Data Lake Store**, and then select **Select**.

- 

- Select **Done**.

-5. Repeat the last two steps to grant permissions for **Windows Azure Service Management API** as well.

-## Next steps

-In this article, you created a Microsoft Entra native application and gathered the information you need in your client applications that you author using .NET SDK, Java SDK, REST API, etc. You can now proceed to the following articles that talk about how to use the Microsoft Entra web application to first authenticate with Data Lake Storage Gen1 and then perform other operations on the store.

-* [End-user-authentication with Data Lake Storage Gen1 using Java SDK](data-lake-store-end-user-authenticate-java-sdk.md)

-* [End-user authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md)

-* [End-user authentication with Data Lake Storage Gen1 using Python](data-lake-store-end-user-authenticate-python.md)

-* [End-user authentication with Data Lake Storage Gen1 using REST API](data-lake-store-end-user-authenticate-rest-api.md)

-description: Use the Azure CLI to create a Data Lake Storage Gen1 account and perform basic operations.

-# Get started with Azure Data Lake Storage Gen1 using the Azure CLI

-> [!div class="op_single_selector"]

-> * [Portal](data-lake-store-get-started-portal.md)

-> * [PowerShell](data-lake-store-get-started-powershell.md)

-> * [Azure CLI](data-lake-store-get-started-cli-2.0.md)

->

->

-Learn how to use the Azure CLI to create an Azure Data Lake Storage Gen1 account and perform basic operations such as create folders, upload and download data files, delete your account, etc. For more information about Data Lake Storage Gen1, see [Overview of Data Lake Storage Gen1](data-lake-store-overview.md).

-The Azure CLI is Azure's command-line experience for managing Azure resources. It can be used on macOS, Linux, and Windows. For more information, see [Overview of Azure CLI](/cli/azure). You can also look at the [Azure Data Lake Storage Gen1 CLI reference](/cli/azure/dls) for a complete list of commands and syntax.

-## Prerequisites

-Before you begin this article, you must have the following:

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure CLI** - See [Install Azure CLI](/cli/azure/install-azure-cli) for instructions.

-## Authentication

-This article uses a simpler authentication approach with Data Lake Storage Gen1 where you log in as an end-user user. The access level to the Data Lake Storage Gen1 account and file system is then governed by the access level of the logged in user. However, there are other approaches as well to authenticate with Data Lake Storage Gen1, which are **end-user authentication** or **service-to-service authentication**. For instructions and more information on how to authenticate, see [End-user authentication](data-lake-store-end-user-authenticate-using-active-directory.md) or [Service-to-service authentication](./data-lake-store-service-to-service-authenticate-using-active-directory.md).

-## Log in to your Azure subscription

-1. Log into your Azure subscription.

- ```azurecli

- az login

- ```

- You get a code to use in the next step. Use a web browser to open the page https://aka.ms/devicelogin and enter the code to authenticate. You are prompted to log in using your credentials.

-2. Once you log in, the window lists all the Azure subscriptions that are associated with your account. Use the following command to use a specific subscription.

-

- ```azurecli

- az account set --subscription <subscription id>

- ```

-## Create an Azure Data Lake Storage Gen1 account

-1. Create a new resource group. In the following command, provide the parameter values you want to use. If the location name contains spaces, put it in quotes. For example "East US 2".

-

- ```azurecli

- az group create --location "East US 2" --name myresourcegroup

- ```

-2. Create the Data Lake Storage Gen1 account.

-

- ```azurecli

- az dls account create --account mydatalakestoragegen1 --resource-group myresourcegroup

- ```

-## Create folders in a Data Lake Storage Gen1 account

-You can create folders under your Azure Data Lake Storage Gen1 account to manage and store data. Use the following command to create a folder called **mynewfolder** at the root of the Data Lake Storage Gen1 account.

-```azurecli

-az dls fs create --account mydatalakestoragegen1 --path /mynewfolder --folder

-```

-> [!NOTE]

-> The `--folder` parameter ensures that the command creates a folder. If this parameter is not present, the command creates an empty file called mynewfolder at the root of the Data Lake Storage Gen1 account.

->

->

-## Upload data to a Data Lake Storage Gen1 account

-You can upload data to Data Lake Storage Gen1 directly at the root level or to a folder that you created within the account. The snippets below demonstrate how to upload some sample data to the folder (**mynewfolder**) you created in the previous section.

-If you are looking for some sample data to upload, you can get the **Ambulance Data** folder from the [Azure Data Lake Git Repository](https://github.com/MicrosoftBigData/usql/tree/master/Examples/Samples/Data/AmbulanceData). Download the file and store it in a local directory on your computer, such as C:\sampledata\.

-```azurecli

-az dls fs upload --account mydatalakestoragegen1 --source-path "C:\SampleData\AmbulanceData\vehicle1_09142014.csv" --destination-path "/mynewfolder/vehicle1_09142014.csv"

-```

-> [!NOTE]

-> For the destination, you must specify the complete path including the file name.

->

->

-## List files in a Data Lake Storage Gen1 account

-Use the following command to list the files in a Data Lake Storage Gen1 account.

-```azurecli

-az dls fs list --account mydatalakestoragegen1 --path /mynewfolder

-```

-The output of this should be similar to the following:

-```json

-[

- {

- "accessTime": 1491323529542,

- "aclBit": false,

- "blockSize": 268435456,

- "group": "1808bd5f-62af-45f4-89d8-03c5e81bac20",

- "length": 1589881,

- "modificationTime": 1491323531638,

- "msExpirationTime": 0,

- "name": "mynewfolder/vehicle1_09142014.csv",

- "owner": "1808bd5f-62af-45f4-89d8-03c5e81bac20",

- "pathSuffix": "vehicle1_09142014.csv",

- "permission": "770",

- "replication": 1,

- "type": "FILE"

- }

-]

-```

-## Rename, download, and delete data from a Data Lake Storage Gen1 account

-* **To rename a file**, use the following command:

-

- ```azurecli

- az dls fs move --account mydatalakestoragegen1 --source-path /mynewfolder/vehicle1_09142014.csv --destination-path /mynewfolder/vehicle1_09142014_copy.csv

- ```

-* **To download a file**, use the following command. Make sure the destination path you specify already exists.

-

- ```azurecli

- az dls fs download --account mydatalakestoragegen1 --source-path /mynewfolder/vehicle1_09142014_copy.csv --destination-path "C:\mysampledata\vehicle1_09142014_copy.csv"

- ```

- > [!NOTE]

- > The command creates the destination folder if it does not exist.

- >

- >

-* **To delete a file**, use the following command:

-

- ```azurecli

- az dls fs delete --account mydatalakestoragegen1 --path /mynewfolder/vehicle1_09142014_copy.csv

- ```

- If you want to delete the folder **mynewfolder** and the file **vehicle1_09142014_copy.csv** together in one command, use the --recurse parameter

- ```azurecli

- az dls fs delete --account mydatalakestoragegen1 --path /mynewfolder --recurse

- ```

-## Work with permissions and ACLs for a Data Lake Storage Gen1 account

-In this section you learn about how to manage ACLs and permissions using the Azure CLI. For a detailed discussion on how ACLs are implemented in Azure Data Lake Storage Gen1, see [Access control in Azure Data Lake Storage Gen1](data-lake-store-access-control.md).

-* **To update the owner of a file/folder**, use the following command:

- ```azurecli

- az dls fs access set-owner --account mydatalakestoragegen1 --path /mynewfolder/vehicle1_09142014.csv --group 80a3ed5f-959e-4696-ba3c-d3c8b2db6766 --owner 6361e05d-c381-4275-a932-5535806bb323

- ```

-* **To update the permissions for a file/folder**, use the following command:

- ```azurecli

- az dls fs access set-permission --account mydatalakestoragegen1 --path /mynewfolder/vehicle1_09142014.csv --permission 777

- ```

-

-* **To get the ACLs for a given path**, use the following command:

- ```azurecli

- az dls fs access show --account mydatalakestoragegen1 --path /mynewfolder/vehicle1_09142014.csv

- ```

- The output should be similar to the following:

- ```output

- {

- "entries": [

- "user::rwx",

- "group::rwx",

- "other::"

- ],

- "group": "1808bd5f-62af-45f4-89d8-03c5e81bac20",

- "owner": "1808bd5f-62af-45f4-89d8-03c5e81bac20",

- "permission": "770",

- "stickyBit": false

- }

- ```

-* **To set an entry for an ACL**, use the following command:

- ```azurecli

- az dls fs access set-entry --account mydatalakestoragegen1 --path /mynewfolder --acl-spec user:6360e05d-c381-4275-a932-5535806bb323:-w-

- ```

-* **To remove an entry for an ACL**, use the following command:

- ```azurecli

- az dls fs access remove-entry --account mydatalakestoragegen1 --path /mynewfolder --acl-spec user:6360e05d-c381-4275-a932-5535806bb323

- ```

-* **To remove an entire default ACL**, use the following command:

- ```azurecli

- az dls fs access remove-all --account mydatalakestoragegen1 --path /mynewfolder --default-acl

- ```

-* **To remove an entire non-default ACL**, use the following command:

- ```azurecli

- az dls fs access remove-all --account mydatalakestoragegen1 --path /mynewfolder

- ```

-

-## Delete a Data Lake Storage Gen1 account

-Use the following command to delete a Data Lake Storage Gen1 account.

-```azurecli

-az dls account delete --account mydatalakestoragegen1

-```

-When prompted, enter **Y** to delete the account.

-## Next steps

-* [Use Azure Data Lake Storage Gen1 for big data requirements](data-lake-store-data-scenarios.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-* [Use Azure HDInsight with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: Use the Java SDK for Azure Data Lake Storage Gen1 to perform filesystem operations on Data Lake Storage Gen1 such as creating folders, and uploading and downloading data files.

-# Filesystem operations on Azure Data Lake Storage Gen1 using Java SDK

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-data-operations-net-sdk.md)

-> * [Java SDK](data-lake-store-get-started-java-sdk.md)

-> * [REST API](data-lake-store-data-operations-rest-api.md)

-> * [Python](data-lake-store-data-operations-python.md)

->

->

-Learn how to use the Azure Data Lake Storage Gen1 Java SDK to perform basic operations such as create folders, upload and download data files, etc. For more information about Data Lake Storage Gen1, see [Azure Data Lake Storage Gen1](data-lake-store-overview.md).

-You can access the Java SDK API docs for Data Lake Storage Gen1 at [Azure Data Lake Storage Gen1 Java API docs](https://azure.github.io/azure-data-lake-store-java/javadoc/).

-## Prerequisites

-* Java Development Kit (JDK 7 or higher, using Java version 1.7 or higher)

-* Data Lake Storage Gen1 account. Follow the instructions at [Get started with Azure Data Lake Storage Gen1 using the Azure portal](data-lake-store-get-started-portal.md).

-* [Maven](https://maven.apache.org/install.html). This tutorial uses Maven for build and project dependencies. Although it is possible to build without using a build system like Maven or Gradle, these systems make is much easier to manage dependencies.

-* (Optional) And IDE like [IntelliJ IDEA](https://www.jetbrains.com/idea/download/) or [Eclipse](https://www.eclipse.org/downloads/) or similar.

-## Create a Java application

-The code sample available [on GitHub](https://azure.microsoft.com/documentation/samples/data-lake-store-java-upload-download-get-started/) walks you through the process of creating files in the store, concatenating files, downloading a file, and deleting some files in the store. This section of the article walks you through the main parts of the code.

-1. Create a Maven project using [mvn archetype](https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html) from the command line or using an IDE. For instructions on how to create a Java project using IntelliJ, see [here](https://www.jetbrains.com/help/idea/2016.1/creating-and-running-your-first-java-application.html). For instructions on how to create a project using Eclipse, see [here](https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2FgettingStarted%2Fqs-3.htm).

-2. Add the following dependencies to your Maven **pom.xml** file. Add the following snippet before the **\</project>** tag:

-

- ```xml

- <dependencies>

- <dependency>

- <groupId>com.microsoft.azure</groupId>

- <artifactId>azure-data-lake-store-sdk</artifactId>

- <version>2.1.5</version>

- </dependency>

- <dependency>

- <groupId>org.slf4j</groupId>

- <artifactId>slf4j-nop</artifactId>

- <version>1.7.21</version>

- </dependency>

- </dependencies>

- ```

-

- The first dependency is to use the Data Lake Storage Gen1 SDK (`azure-data-lake-store-sdk`) from the maven repository. The second dependency is to specify the logging framework (`slf4j-nop`) to use for this application. The Data Lake Storage Gen1 SDK uses [SLF4J](https://www.slf4j.org/) logging façade, which lets you choose from a number of popular logging frameworks, like Log4j, Java logging, Logback, etc., or no logging. For this example, we disable logging, hence we use the **slf4j-nop** binding. To use other logging options in your app, see [here](https://www.slf4j.org/manual.html#projectDep).

-3. Add the following import statements to your application.

- ```java

- import com.microsoft.azure.datalake.store.ADLException;

- import com.microsoft.azure.datalake.store.ADLStoreClient;

- import com.microsoft.azure.datalake.store.DirectoryEntry;

- import com.microsoft.azure.datalake.store.IfExists;

- import com.microsoft.azure.datalake.store.oauth2.AccessTokenProvider;

- import com.microsoft.azure.datalake.store.oauth2.ClientCredsTokenProvider;

- import java.io.*;

- import java.util.Arrays;

- import java.util.List;

- ```

-## Authentication

-* For end-user authentication for your application, see [End-user-authentication with Data Lake Storage Gen1 using Java](data-lake-store-end-user-authenticate-java-sdk.md).

-* For service-to-service authentication for your application, see [Service-to-service authentication with Data Lake Storage Gen1 using Java](data-lake-store-service-to-service-authenticate-java.md).

-## Create a Data Lake Storage Gen1 client

-Creating an [ADLStoreClient](https://azure.github.io/azure-data-lake-store-java/javadoc/) object requires you to specify the Data Lake Storage Gen1 account name and the token provider you generated when you authenticated with Data Lake Storage Gen1 (see [Authentication](#authentication) section). The Data Lake Storage Gen1 account name needs to be a fully qualified domain name. For example, replace **FILL-IN-HERE** with something like **mydatalakestoragegen1.azuredatalakestore.net**.

-```java

-private static String accountFQDN = "FILL-IN-HERE"; // full account FQDN, not just the account name

-ADLStoreClient client = ADLStoreClient.createClient(accountFQDN, provider);

-```

-The code snippets in the following sections contain examples of some common filesystem operations. You can look at the full [Data Lake Storage Gen1 Java SDK API docs](https://azure.github.io/azure-data-lake-store-java/javadoc/) of the **ADLStoreClient** object to see other operations.

-## Create a directory

-The following snippet creates a directory structure in the root of the Data Lake Storage Gen1 account you specified.

-```java

-// create directory

-client.createDirectory("/a/b/w");

-System.out.println("Directory created.");

-```

-## Create a file

-The following snippet creates a file (c.txt) in the directory structure and writes some data to the file.

-```java

-// create file and write some content

-String filename = "/a/b/c.txt";

-OutputStream stream = client.createFile(filename, IfExists.OVERWRITE );

-PrintStream out = new PrintStream(stream);

-for (int i = 1; i <= 10; i++) {

- out.println("This is line #" + i);

- out.format("This is the same line (%d), but using formatted output. %n", i);

-}

-out.close();

-System.out.println("File created.");

-```

-You can also create a file (d.txt) using byte arrays.

-```java

-// create file using byte arrays

-stream = client.createFile("/a/b/d.txt", IfExists.OVERWRITE);

-byte[] buf = getSampleContent();

-stream.write(buf);

-stream.close();

-System.out.println("File created using byte array.");

-```

-The definition for `getSampleContent` function used in the preceding snippet is available as part of the sample [on GitHub](https://azure.microsoft.com/documentation/samples/data-lake-store-java-upload-download-get-started/).

-## Append to a file

-The following snippet appends content to an existing file.

-```java

-// append to file

-stream = client.getAppendStream(filename);

-stream.write(getSampleContent());

-stream.close();

-System.out.println("File appended.");

-```

-The definition for `getSampleContent` function used in the preceding snippet is available as part of the sample [on GitHub](https://azure.microsoft.com/documentation/samples/data-lake-store-java-upload-download-get-started/).

-## Read a file

-The following snippet reads content from a file in a Data Lake Storage Gen1 account.

-```java

-// Read File

-InputStream in = client.getReadStream(filename);

-BufferedReader reader = new BufferedReader(new InputStreamReader(in));

-String line;

-while ( (line = reader.readLine()) != null) {

- System.out.println(line);

-}

-reader.close();

-System.out.println();

-System.out.println("File contents read.");

-```

-## Concatenate files

-The following snippet concatenates two files in a Data Lake Storage Gen1 account. If successful, the concatenated file replaces the two existing files.

-```java

-// concatenate the two files into one

-List<String> fileList = Arrays.asList("/a/b/c.txt", "/a/b/d.txt");

-client.concatenateFiles("/a/b/f.txt", fileList);

-System.out.println("Two files concatenated into a new file.");

-```

-## Rename a file

-The following snippet renames a file in a Data Lake Storage Gen1 account.

-```java

-//rename the file

-client.rename("/a/b/f.txt", "/a/b/g.txt");

-System.out.println("New file renamed.");

-```

-## Get metadata for a file

-The following snippet retrieves the metadata for a file in a Data Lake Storage Gen1 account.

-```java

-// get file metadata

-DirectoryEntry ent = client.getDirectoryEntry(filename);

-printDirectoryInfo(ent);

-System.out.println("File metadata retrieved.");

-```

-## Set permissions on a file

-The following snippet sets permissions on the file that you created in the previous section.

-```java

-// set file permission

-client.setPermission(filename, "744");

-System.out.println("File permission set.");

-```

-## List directory contents

-The following snippet lists the contents of a directory, recursively.

-```java

-// list directory contents

-List<DirectoryEntry> list = client.enumerateDirectory("/a/b", 2000);

-System.out.println("Directory listing for directory /a/b:");

-for (DirectoryEntry entry : list) {

- printDirectoryInfo(entry);

-}

-System.out.println("Directory contents listed.");

-```

-The definition for `printDirectoryInfo` function used in the preceding snippet is available as part of the sample [on GitHub](https://azure.microsoft.com/documentation/samples/data-lake-store-java-upload-download-get-started/).

-## Delete files and folders

-The following snippet deletes the specified files and folders in a Data Lake Storage Gen1 account, recursively.

-```java

-// delete directory along with all the subdirectories and files in it

-client.deleteRecursive("/a");

-System.out.println("All files and folders deleted recursively");

-promptEnterKey();

-```

-## Build and run the application

-1. To run from within an IDE, locate and press the **Run** button. To run from Maven, use [exec:exec](https://www.mojohaus.org/exec-maven-plugin/exec-mojo.html).

-2. To produce a standalone jar that you can run from command-line build the jar with all dependencies included, using the [Maven assembly plugin](https://maven.apache.org/plugins/maven-assembly-plugin/usage.html). The pom.xml in the [example source code on GitHub](https://github.com/Azure-Samples/data-lake-store-java-upload-download-get-started/blob/master/pom.xml) has an example.

-## Next steps

-* [Explore JavaDoc for the Java SDK](https://azure.github.io/azure-data-lake-store-java/javadoc/)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-description: Learn how to use the .NET SDK for Azure Data Lake Storage Gen1 account management operations.

-# Account management operations on Azure Data Lake Storage Gen1 using .NET SDK

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-get-started-net-sdk.md)

-> * [REST API](data-lake-store-get-started-rest-api.md)

-> * [Python](data-lake-store-get-started-python.md)

->

->

-In this article, you learn how to perform account management operations on Azure Data Lake Storage Gen1 using .NET SDK. Account management operations include creating a Data Lake Storage Gen1 account, listing the accounts in an Azure subscription, deleting the accounts, etc.

-For instructions on how to perform data management operations on Data Lake Storage Gen1 using .NET SDK, see [Filesystem operations on Data Lake Storage Gen1 using .NET SDK](data-lake-store-data-operations-net-sdk.md).

-## Prerequisites

-* **Visual Studio 2013 or above**. The instructions below use Visual Studio 2019.

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-## Create a .NET application

-1. In Visual Studio, select the **File** menu, **New**, and then **Project**.

-2. Choose **Console App (.NET Framework)**, and then select **Next**.

-3. In **Project name**, enter `CreateADLApplication`, and then select **Create**.

-4. Add the NuGet packages to your project.

- 1. Right-click the project name in the Solution Explorer and click **Manage NuGet Packages**.

- 2. In the **NuGet Package Manager** tab, make sure that **Package source** is set to **nuget.org** and that **Include prerelease** check box is selected.

- 3. Search for and install the following NuGet packages:

- * `Microsoft.Azure.Management.DataLake.Store` - This tutorial uses v2.1.3-preview.

- * `Microsoft.Rest.ClientRuntime.Azure.Authentication` - This tutorial uses v2.2.12.

- 

- 4. Close the **NuGet Package Manager**.

-5. Open **Program.cs**, delete the existing code, and then include the following statements to add references to namespaces.

- ```csharp

- using System;

- using System.IO;

- using System.Linq;

- using System.Text;

- using System.Threading;

- using System.Collections.Generic;

- using System.Security.Cryptography.X509Certificates; // Required only if you are using an Azure AD application created with certificates

-

- using Microsoft.Rest;

- using Microsoft.Rest.Azure.Authentication;

- using Microsoft.Azure.Management.DataLake.Store;

- using Microsoft.Azure.Management.DataLake.Store.Models;

- using Microsoft.IdentityModel.Clients.ActiveDirectory;

- ```

-6. Declare the variables and provide the values for placeholders. Also, make sure the local path and file name you provide exist on the computer.

- ```csharp

- namespace SdkSample

- {

- class Program

- {

- private static DataLakeStoreAccountManagementClient _adlsClient;

-

- private static string _adlsAccountName;

- private static string _resourceGroupName;

- private static string _location;

- private static string _subId;

- private static void Main(string[] args)

- {

- _adlsAccountName = "<DATA-LAKE-STORAGE-GEN1-NAME>.azuredatalakestore.net";

- _resourceGroupName = "<RESOURCE-GROUP-NAME>";

- _location = "East US 2";

- _subId = "<SUBSCRIPTION-ID>";

- }

- }

- }

- ```

-In the remaining sections of the article, you can see how to use the available .NET methods to perform operations such as authentication, file upload, etc.

-## Authentication

-* For end-user authentication for your application, see [End-user authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-end-user-authenticate-net-sdk.md).

-* For service-to-service authentication for your application, see [Service-to-service authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-service-to-service-authenticate-net-sdk.md).

-## Create client object

-The following snippet creates the Data Lake Storage Gen1 account client object, which is used to issue account management requests to the service, such as create account, delete account, etc.

-```csharp

-// Create client objects and set the subscription ID

-_adlsClient = new DataLakeStoreAccountManagementClient(armCreds) { SubscriptionId = _subId };

-```

-

-## Create a Data Lake Storage Gen1 account

-The following snippet creates a Data Lake Storage Gen1 account in the Azure subscription you provided while creating the Data Lake Storage Gen1 account client object.

-```csharp

-// Create Data Lake Storage Gen1 account

-var adlsParameters = new DataLakeStoreAccount(location: _location);

-_adlsClient.Account.Create(_resourceGroupName, _adlsAccountName, adlsParameters);

-```

-## List all Data Lake Storage Gen1 accounts within a subscription

-Add the following method to your class definition. The following snippet lists all Data Lake Storage Gen1 accounts within a given Azure subscription.

-```csharp

-// List all Data Lake Storage Gen1 accounts within the subscription

-public static List<DataLakeStoreAccountBasic> ListAdlStoreAccounts()

-{

- var response = _adlsClient.Account.List(_adlsAccountName);

- var accounts = new List<DataLakeStoreAccountBasic>(response);

- while (response.NextPageLink != null)

- {

- response = _adlsClient.Account.ListNext(response.NextPageLink);

- accounts.AddRange(response);

- }

- return accounts;

-}

-```

-## Delete a Data Lake Storage Gen1 account

-The following snippet deletes the Data Lake Storage Gen1 account you created earlier.

-```csharp

-// Delete Data Lake Storage Gen1 account

-_adlsClient.Account.Delete(_resourceGroupName, _adlsAccountName);

-```

-## See also

-* [Filesystem operations on Data Lake Storage Gen1 using .NET SDK](data-lake-store-data-operations-net-sdk.md)

-* [Data Lake Storage Gen1 .NET SDK Reference](/dotnet/api/overview/azure/data-lake-store)

-## Next steps

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-description: Use the Azure portal to create a Data Lake Storage Gen1 account and perform basic operations in the account.

-# Get started with Azure Data Lake Storage Gen1 using the Azure portal

-> [!div class="op_single_selector"]

-> * [Portal](data-lake-store-get-started-portal.md)

-> * [PowerShell](data-lake-store-get-started-powershell.md)

-> * [Azure CLI](data-lake-store-get-started-cli-2.0.md)

->

->

-Learn how to use the Azure portal to create a Data Lake Storage Gen1 account and perform basic operations such as create folders, upload, and download data files, delete your account, etc. For more information, see [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md).

-## Prerequisites

-Before you begin this tutorial, you must have the following items:

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-## Create a Data Lake Storage Gen1 account

-1. Sign on to the new [Azure portal](https://portal.azure.com).

-2. Click **Create a resource > Storage > Data Lake Storage Gen1**.

-3. In the **New Data Lake Storage Gen1** blade, provide the values as shown in the following screenshot:

- 

- * **Name**. Enter a unique name for the Data Lake Storage Gen1 account.

- * **Subscription**. Select the subscription under which you want to create a new Data Lake Storage Gen1 account.

- * **Resource Group**. Select an existing resource group, or select the **Create new** option to create one. A resource group is a container that holds related resources for an application. For more information, see [Resource Groups in Azure](../azure-resource-manager/management/overview.md#resource-groups).

- * **Location**: Select a location where you want to create the Data Lake Storage Gen1 account.

- * **Encryption Settings**. There are three options:

- * **Do not enable encryption**.

- * **Use keys managed by Data Lake Storage Gen1**, if you want Data Lake Storage Gen1 to manage your encryption keys.

- * **Use keys from your own Key Vault**. You can select an existing Azure Key Vault or create a new Key Vault. To use the keys from a Key Vault, you must assign permissions for the Data Lake Storage Gen1 account to access the Azure Key Vault. For the instructions, see [Assign permissions to Azure Key Vault](#assign-permissions-to-azure-key-vault).

- 

- Click **OK** in the **Encryption Settings** blade.

- For more information, see [Encryption of data in Azure Data Lake Storage Gen1](./data-lake-store-encryption.md).

-4. Click **Create**. If you chose to pin the account to the dashboard, you are taken back to the dashboard and you can see the progress of your Data Lake Storage Gen1 account provisioning. Once the Data Lake Storage Gen1 account is provisioned, the account blade shows up.

-## <a name="assign-permissions-to-azure-key-vault"></a>Assign permissions to Azure Key Vault

-If you used keys from an Azure Key Vault to configure encryption on the Data Lake Storage Gen1 account, you must configure access between the Data Lake Storage Gen1 account and the Azure Key Vault account. Perform the following steps to do so.

-1. If you used keys from the Azure Key Vault, the blade for the Data Lake Storage Gen1 account displays a warning at the top. Click the warning to open **Encryption**.

- 

-2. The blade shows two options to configure access.

- 

- * In the first option, click **Grant Permissions** to configure access. The first option is enabled only when the user that created the Data Lake Storage Gen1 account is also an admin for the Azure Key Vault.

- * The other option is to run the PowerShell cmdlet displayed on the blade. You need to be the owner of the Azure Key Vault or have the ability to grant permissions on the Azure Key Vault. After you have run the cmdlet, come back to the blade and click **Enable** to configure access.

-> [!NOTE]

-> You can also create a Data Lake Storage Gen1 account using Azure Resource Manager templates. These templates are accessible from [Azure QuickStart Templates](https://azure.microsoft.com/resources/templates/?term=data+lake+store):

-> * Without data encryption: [Deploy Azure Data Lake Storage Gen1 account with no data encryption](https://azure.microsoft.com/resources/templates/data-lake-store-no-encryption/).

-> * With data encryption using Data Lake Storage Gen1: [Deploy Data Lake Storage Gen1 account with encryption(Data Lake)](https://azure.microsoft.com/resources/templates/data-lake-store-encryption-adls/).

-> * With data encryption using Azure Key Vault: [Deploy Data Lake Storage Gen1 account with encryption(Key Vault)](https://azure.microsoft.com/resources/templates/data-lake-store-encryption-key-vault/).

->

->

-## <a name="createfolder"></a>Create folders

-You can create folders under your Data Lake Storage Gen1 account to manage and store data.

-1. Open the Data Lake Storage Gen1 account that you created. From the left pane, click **All resources**, and then from the **All resources** blade, click the account name under which you want to create folders. If you pinned the account to the startboard, click that account tile.

-2. In your Data Lake Storage Gen1 account blade, click **Data Explorer**.

- 

-3. From Data Explorer blade, click **New Folder**, enter a name for the new folder, and then click **OK**.

- 

- The newly created folder is listed in the **Data Explorer** blade. You can create nested folders up to any level.

- 

-## <a name="uploaddata"></a>Upload data

-You can upload your data to a Data Lake Storage Gen1 account directly at the root level or to a folder that you created within the account.

-1. From the **Data Explorer** blade, click **Upload**.

-2. In the **Upload files** blade, navigate to the files you want to upload, and then click **Add selected files**. You can also select more than one file to upload.

- 

-If you are looking for some sample data to upload, you can get the **Ambulance Data** folder from the [Azure Data Lake Git Repository](https://github.com/MicrosoftBigData/usql/tree/master/Examples/Samples/Data/AmbulanceData).

-## <a name="properties"></a>Actions available on the stored data

-Click the ellipsis icon against a file, and from the pop-up menu, click the action you want to perform on the data.

-

-## Secure your data

-You can secure the data stored in your Data Lake Storage Gen1 account using Microsoft Entra ID and access control (ACLs). For instructions on how to do that, see [Securing data in Azure Data Lake Storage Gen1](data-lake-store-secure-data.md).

-## Delete your account

-To delete a Data Lake Storage Gen1 account, from your Data Lake Storage Gen1 blade, click **Delete**. To confirm the action, you'll be prompted to enter the name of the account you wish to delete. Enter the name of the account, and then click **Delete**.

-

-## Next steps

-* [Use Azure Data Lake Storage Gen1 for big data requirements](data-lake-store-data-scenarios.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-* [Use Azure HDInsight with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: Use Azure PowerShell to create an Azure Data Lake Storage Gen1 account and perform basic operations.

-# Get started with Azure Data Lake Storage Gen1 using Azure PowerShell

-> [!div class="op_single_selector"]

-> * [Portal](data-lake-store-get-started-portal.md)

-> * [PowerShell](data-lake-store-get-started-powershell.md)

-> * [Azure CLI](data-lake-store-get-started-cli-2.0.md)

->

->

-Learn how to use Azure PowerShell to create an Azure Data Lake Storage Gen1 account and perform basic operations such as create folders, upload and download data files, delete your account, etc. For more information about Data Lake Storage Gen1, see [Overview of Data Lake Storage Gen1](data-lake-store-overview.md).

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure PowerShell 1.0 or greater**. See [How to install and configure Azure PowerShell](/powershell/azure/).

-## Authentication

-This article uses a simpler authentication approach with Data Lake Storage Gen1 where you're prompted to enter your Azure account credentials. The access level to Data Lake Storage Gen1 account and file system is then governed by the access level of the logged in user. However, there are other approaches to authenticate with Data Lake Storage Gen1, which are end-user authentication or service-to-service authentication. For instructions and more information on how to authenticate, see [End-user authentication](data-lake-store-end-user-authenticate-using-active-directory.md) or [Service-to-service authentication](./data-lake-store-service-to-service-authenticate-using-active-directory.md).

-## Create a Data Lake Storage Gen1 account

-1. From your desktop, open a new Windows PowerShell window. Enter the following snippet to log in to your Azure account, set the subscription, and register the Data Lake Storage Gen1 provider. When prompted to log in, make sure you log in as one of the subscription administrators/owner:

- ```PowerShell

- # Log in to your Azure account

- Connect-AzAccount

- # List all the subscriptions associated to your account

- Get-AzSubscription

- # Select a subscription

- Set-AzContext -SubscriptionId <subscription ID>

- # Register for Azure Data Lake Storage Gen1

- Register-AzResourceProvider -ProviderNamespace "Microsoft.DataLakeStore"

- ```

-1. A Data Lake Storage Gen1 account is associated with an Azure resource group. Start by creating a resource group.

- ```PowerShell

- $resourceGroupName = "<your new resource group name>"

- New-AzResourceGroup -Name $resourceGroupName -Location "East US 2"

- ```

- 

-1. Create a Data Lake Storage Gen1 account. The name you specify must only contain lowercase letters and numbers.

- ```PowerShell

- $dataLakeStorageGen1Name = "<your new Data Lake Storage Gen1 account name>"

- New-AzDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeStorageGen1Name -Location "East US 2"

- ```

- 

-1. Verify that the account is successfully created.

- ```PowerShell

- Test-AzDataLakeStoreAccount -Name $dataLakeStorageGen1Name

- ```

- The output for the cmdlet should be **True**.

-## Create directory structures

-You can create directories under your Data Lake Storage Gen1 account to manage and store data.

-1. Specify a root directory.

- ```PowerShell

- $myrootdir = "/"

- ```

-1. Create a new directory called **mynewdirectory** under the specified root.

- ```PowerShell

- New-AzDataLakeStoreItem -Folder -AccountName $dataLakeStorageGen1Name -Path $myrootdir/mynewdirectory

- ```

-1. Verify that the new directory is successfully created.

- ```PowerShell

- Get-AzDataLakeStoreChildItem -AccountName $dataLakeStorageGen1Name -Path $myrootdir

- ```

- It should show an output as shown in the following screenshot:

- 

-## Upload data

-You can upload your data to Data Lake Storage Gen1 directly at the root level, or to a directory that you created within the account. The snippets in this section demonstrate how to upload some sample data to the directory (**mynewdirectory**) you created in the previous section.

-If you are looking for some sample data to upload, you can get the **Ambulance Data** folder from the [Azure Data Lake Git Repository](https://github.com/MicrosoftBigData/usql/tree/master/Examples/Samples/Data/AmbulanceData). Download the file and store it in a local directory on your computer, such as C:\sampledata\.

-```PowerShell

-Import-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name `

- -Path "C:\sampledata\vehicle1_09142014.csv" `

- -Destination $myrootdir\mynewdirectory\vehicle1_09142014.csv

-```

-## Rename, download, and delete data

-To rename a file, use the following command:

-```PowerShell

-Move-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name `

- -Path $myrootdir\mynewdirectory\vehicle1_09142014.csv `

- -Destination $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv

-```

-To download a file, use the following command:

-```PowerShell

-Export-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name `

- -Path $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv `

- -Destination "C:\sampledata\vehicle1_09142014_Copy.csv"

-```

-To delete a file, use the following command:

-```PowerShell

-Remove-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name `

- -Paths $myrootdir\mynewdirectory\vehicle1_09142014_Copy.csv

-```

-When prompted, enter **Y** to delete the item. If you have more than one file to delete, you can provide all the paths separated by comma.

-```PowerShell

-Remove-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name `

- -Paths $myrootdir\mynewdirectory\vehicle1_09142014.csv, $myrootdir\mynewdirectoryvehicle1_09142014_Copy.csv

-```

-## Delete your account

-Use the following command to delete your Data Lake Storage Gen1 account.

-```PowerShell

-Remove-AzDataLakeStoreAccount -Name $dataLakeStorageGen1Name

-```

-When prompted, enter **Y** to delete the account.

-## Next steps

-* [Performance tuning guidance for using PowerShell with Azure Data Lake Storage Gen1](data-lake-store-performance-tuning-powershell.md)

-* [Use Azure Data Lake Storage Gen1 for big data requirements](data-lake-store-data-scenarios.md)

-* [Secure data in Data Lake Storage Gen1](data-lake-store-secure-data.md)

-* [Use Azure Data Lake Analytics with Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md)

-* [Use Azure HDInsight with Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-description: Learn how to use the Python SDK for Azure Data Lake Storage Gen1 account management operations.

-# Account management operations on Azure Data Lake Storage Gen1 using Python

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-get-started-net-sdk.md)

-> * [REST API](data-lake-store-get-started-rest-api.md)

-> * [Python](data-lake-store-get-started-python.md)

->

->

-Learn how to use the Python SDK for Azure Data Lake Storage Gen1 to perform basic account management operations such as create a Data Lake Storage Gen1 account, list the Data Lake Storage Gen1 accounts, etc. For instructions on how to perform filesystem operations on Data Lake Storage Gen1 using Python, see [Filesystem operations on Data Lake Storage Gen1 using Python](data-lake-store-data-operations-python.md).

-## Prerequisites

-* **Python**. You can download Python from [here](https://www.python.org/downloads/). This article uses Python 3.6.2.

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **An Azure resource group**. For instructions, see [Create an Azure resource group](../azure-resource-manager/management/manage-resource-groups-portal.md).

-## Install the modules

-To work with Data Lake Storage Gen1 using Python, you need to install three modules.

-* The `azure-mgmt-resource` module, which includes Azure modules for Active Directory, etc.

-* The `azure-mgmt-datalake-store` module, which includes the Azure Data Lake Storage Gen1 account management operations. For more information on this module, see [Azure Data Lake Storage Gen1 Management module reference](/python/api/azure-mgmt-datalake-store/).

-* The `azure-datalake-store` module, which includes the Azure Data Lake Storage Gen1 filesystem operations. For more information on this module, see [azure-datalake-store filesystem module reference](/python/api/azure-datalake-store/azure.datalake.store.core/).

-Use the following commands to install the modules.

-```console

-pip install azure-identity

-pip install azure-mgmt-resource

-pip install azure-mgmt-datalake-store

-pip install azure-datalake-store

-```

-## Create a new Python application

-1. In the IDE of your choice create a new Python application, for example, **mysample.py**.

-2. Add the following snippet to import the required modules:

- ```python

- # Acquire a credential object for the app identity. When running in the cloud,

- # DefaultAzureCredential uses the app's managed identity (MSI) or user-assigned service principal.

- # When run locally, DefaultAzureCredential relies on environment variables named

- # AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_TENANT_ID.

- from azure.identity import DefaultAzureCredential

- ## Required for Data Lake Storage Gen1 account management

- from azure.mgmt.datalake.store import DataLakeStoreAccountManagementClient

- from azure.mgmt.datalake.store.models import CreateDataLakeStoreAccountParameters

- ## Required for Data Lake Storage Gen1 filesystem management

- from azure.datalake.store import core, lib, multithread

- # Common Azure imports

- import adal

- from azure.mgmt.resource.resources import ResourceManagementClient

- from azure.mgmt.resource.resources.models import ResourceGroup

- # Use these as needed for your application

- import logging, getpass, pprint, uuid, time

- ```

-3. Save changes to mysample.py.

-## Authentication

-In this section, we talk about the different ways to authenticate with Microsoft Entra ID. The options available are:

-* For end-user authentication for your application, see [End-user authentication with Data Lake Storage Gen1 using Python](data-lake-store-end-user-authenticate-python.md).

-* For service-to-service authentication for your application, see [Service-to-service authentication with Data Lake Storage Gen1 using Python](data-lake-store-service-to-service-authenticate-python.md).

-## Create client and Data Lake Storage Gen1 account

-The following snippet first creates the Data Lake Storage Gen1 account client. It uses the client object to create a Data Lake Storage Gen1 account. Finally, the snippet creates a filesystem client object.

-```python

-## Declare variables

-subscriptionId = 'FILL-IN-HERE'

-adlsAccountName = 'FILL-IN-HERE'

-resourceGroup = 'FILL-IN-HERE'

-location = 'eastus2'

-credential = DefaultAzureCredential()

-## Create Data Lake Storage Gen1 account management client object

-adlsAcctClient = DataLakeStoreAccountManagementClient(credential, subscription_id=subscriptionId)

-## Create a Data Lake Storage Gen1 account

-adlsAcctResult = adlsAcctClient.accounts.begin_create(

- resourceGroup,

- adlsAccountName,

- CreateDataLakeStoreAccountParameters(

- location=location

- )

-)

-```

-

-## List the Data Lake Storage Gen1 accounts

-```python

-## List the existing Data Lake Storage Gen1 accounts

-result_list_response = adlsAcctClient.accounts.list()

-result_list = list(result_list_response)

-for items in result_list:

- print(items)

-```

-## Delete the Data Lake Storage Gen1 account

-```python

-## Delete an existing Data Lake Storage Gen1 account

-adlsAcctClient.accounts.begin_delete(resourceGroup, adlsAccountName)

-```

-

-## Next steps

-* [Filesystem operations on Data Lake Storage Gen1 using Python](data-lake-store-data-operations-python.md).

-## See also

-* [azure-datalake-store Python (Filesystem) reference](/python/api/azure-datalake-store/azure.datalake.store.core)

-* [Open Source Big Data applications compatible with Azure Data Lake Storage Gen1](data-lake-store-compatible-oss-other-applications.md)

-description: Use the WebHDFS REST API to perform account management operations on an Azure Data Lake Storage Gen1 account.

-# Account management operations on Azure Data Lake Storage Gen1 using REST API

-> [!div class="op_single_selector"]

-> * [.NET SDK](data-lake-store-get-started-net-sdk.md)

-> * [REST API](data-lake-store-get-started-rest-api.md)

-> * [Python](data-lake-store-get-started-python.md)

->

->

-In this article, you learn how to perform account management operations on Azure Data Lake Storage Gen1 using the REST API. Account management operations include creating a Data Lake Storage Gen1 account, deleting a Data Lake Storage Gen1 account, etc. For instructions on how to perform filesystem operations on Data Lake Storage Gen1 using REST API, see [Filesystem operations on Data Lake Storage Gen1 using REST API](data-lake-store-data-operations-rest-api.md).

-## Prerequisites

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **[cURL](https://curl.haxx.se/)**. This article uses cURL to demonstrate how to make REST API calls against a Data Lake Storage Gen1 account.

-<a name='how-do-i-authenticate-using-azure-active-directory'></a>

-## How do I authenticate using Microsoft Entra ID?

-You can use two approaches to authenticate using Microsoft Entra ID.

-* For end-user authentication for your application (interactive), see [End-user authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-end-user-authenticate-rest-api.md).

-* For service-to-service authentication for your application (non-interactive), see [Service-to-service authentication with Data Lake Storage Gen1 using .NET SDK](data-lake-store-service-to-service-authenticate-rest-api.md).

-## Create a Data Lake Storage Gen1 account

-This operation is based on the REST API call defined [here](/rest/api/datalakestore/accounts/create).

-Use the following cURL command. Replace **\<yourstoragegen1name>** with your Data Lake Storage Gen1 name.

-```console

-curl -i -X PUT -H "Authorization: Bearer <REDACTED>" -H "Content-Type: application/json" https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.DataLakeStore/accounts/<yourstoragegen1name>?api-version=2015-10-01-preview -d@"C:\temp\input.json"

-```

-In the above command, replace \<`REDACTED`\> with the authorization token you retrieved earlier. The request payload for this command is contained in the **input.json** file that is provided for the `-d` parameter above. The contents of the input.json file resemble the following snippet:

-```json

-{

-"location": "eastus2",

-"tags": {

- "department": "finance"

- },

-"properties": {}

-}

-```

-## Delete a Data Lake Storage Gen1 account

-This operation is based on the REST API call defined [here](/rest/api/datalakestore/accounts/delete).

-Use the following cURL command to delete a Data Lake Storage Gen1 account. Replace **\<yourstoragegen1name>** with your Data Lake Storage Gen1 account name.

-```console

-curl -i -X DELETE -H "Authorization: Bearer <REDACTED>" https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.DataLakeStore/accounts/<yourstoragegen1name>?api-version=2015-10-01-preview

-```

-You should see an output like the following snippet:

-```output

-HTTP/1.1 200 OK

-...

-...

-```

-## Next steps

-* [Filesystem operations on Data Lake Storage Gen1 using REST API](data-lake-store-data-operations-rest-api.md).

-## See also

-* [Azure Data Lake Storage Gen1 REST API Reference](/rest/api/datalakestore/)

-* [Open Source Big Data applications compatible with Azure Data Lake Storage Gen1](data-lake-store-compatible-oss-other-applications.md)

-description: Use the Azure portal to create and use HDInsight clusters with Azure Data Lake Storage Gen1

-# Create HDInsight clusters with Azure Data Lake Storage Gen1 by using the Azure portal

-> [!div class="op_single_selector"]

-> * [Use the Azure portal](data-lake-store-hdinsight-hadoop-use-portal.md)

-> * [Use PowerShell (for default storage)](data-lake-store-hdinsight-hadoop-use-powershell-for-default-storage.md)

-> * [Use PowerShell (for additional storage)](data-lake-store-hdinsight-hadoop-use-powershell.md)

-> * [Use Resource Manager](data-lake-store-hdinsight-hadoop-use-resource-manager-template.md)

->

->

-Learn how to use the Azure portal to create a HDInsight cluster with Azure Data Lake Storage Gen1 as the default storage or an additional storage. Even though additional storage is optional for a HDInsight cluster, it's recommended to store your business data in the additional storage accounts.

-## Prerequisites

-Before you begin, ensure that you've met the following requirements:

-* **An Azure subscription**. Go to [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **An Azure Data Lake Storage Gen1 account**. Follow the instructions from [Get started with Azure Data Lake Storage Gen1 by using the Azure portal](data-lake-store-get-started-portal.md). You must also create a root folder on the account. In this article, a root folder called __/clusters__ is used.

-* **a Microsoft Entra service principal**. This how-to guide provides instructions on how to create a service principal in Microsoft Entra ID. However, to create a service principal, you must be a Microsoft Entra administrator. If you're an administrator, you can skip this prerequisite and continue.

->[!NOTE]

->You can create a service principal only if you're a Microsoft Entra administrator. Your Microsoft Entra administrator must create a service principal before you can create an HDInsight cluster with Data Lake Storage Gen1. Also, the service principal must be created with a certificate, as described at [Create a service principal with certificate](../active-directory/develop/howto-authenticate-service-principal-powershell.md#create-service-principal-with-self-signed-certificate).

->

-## Create an HDInsight cluster

-In this section, you create a HDInsight cluster with Data Lake Storage Gen1 as the default or the additional storage. This article focuses only on the part of configuring Data Lake Storage Gen1. For the general cluster creation information and procedures, see [Create Hadoop clusters in HDInsight](../hdinsight/hdinsight-hadoop-provision-linux-clusters.md).

-### Create a cluster with Data Lake Storage Gen1 as default storage

-To create an HDInsight cluster with a Data Lake Storage Gen1 as the default storage account:

-1. Sign in to the [Azure portal](https://portal.azure.com).

-2. Follow [Create clusters](../hdinsight/hdinsight-hadoop-create-linux-clusters-portal.md#create-clusters) for the general information on creating HDInsight clusters.

-3. On the **Storage** blade, under **Primary storage type**, select **Azure Data Lake Storage Gen1**, and then enter the following information:

- 

- * **Select Data Lake Store account**: Select an existing Data Lake Storage Gen1 account. An existing Data Lake Storage Gen1 account is required. See [Prerequisites](#prerequisites).

- * **Root path**: Enter a path where the cluster-specific files are to be stored. On the screenshot, it is __/clusters/myhdiadlcluster/__, in which the __/clusters__ folder must exist, and the Portal creates *myhdicluster* folder. The *myhdicluster* is the cluster name.

- * **Data Lake Store access**: Configure access between the Data Lake Storage Gen1 account and HDInsight cluster. For instructions, see [Configure Data Lake Storage Gen1 access](#configure-data-lake-storage-gen1-access).

- * **Additional storage accounts**: Add Azure storage accounts as additional storage accounts for the cluster. To add additional Data Lake Storage Gen1 accounts is done by giving the cluster permissions on data in more Data Lake Storage Gen1 accounts while configuring a Data Lake Storage Gen1 account as the primary storage type. See [Configure Data Lake Storage Gen1 access](#configure-data-lake-storage-gen1-access).

-4. On the **Data Lake Store access**, click **Select**, and then continue with cluster creation as described in [Create Hadoop clusters in HDInsight](../hdinsight/hdinsight-hadoop-create-linux-clusters-portal.md).

-### Create a cluster with Data Lake Storage Gen1 as additional storage

-The following instructions create a HDInsight cluster with an Azure Blob storage account as the default storage, and a storage account with Data Lake Storage Gen1 as an additional storage.

-To create a HDInsight cluster with Data Lake Storage Gen1 as an additional storage account:

-1. Sign in to the [Azure portal](https://portal.azure.com).

-2. Follow [Create clusters](../hdinsight/hdinsight-hadoop-create-linux-clusters-portal.md#create-clusters) for the general information on creating HDInsight clusters.

-3. On the **Storage** blade, under **Primary storage type**, select **Azure Storage**, and then enter the following information:

- 

- * **Selection method** - To specify a storage account that is part of your Azure subscription, select **My subscriptions**, and then select the storage account. To specify a storage account that is outside your Azure subscription, select **Access key**, and then provide the information for the outside storage account.

- * **Default container** - Use either the default value or specify your own name.

- * **Additional storage accounts** - Add more Azure storage accounts as the additional storage.

- * **Data Lake Store access** - Configure access between the Data Lake Storage Gen1 account and HDInsight cluster. For instructions see [Configure Data Lake Storage Gen1 access](#configure-data-lake-storage-gen1-access).

-## Configure Data Lake Storage Gen1 access

-In this section, you configure Data Lake Storage Gen1 access from HDInsight clusters using a Microsoft Entra service principal.

-### Specify a service principal

-From the Azure portal, you can either use an existing service principal or create a new one.

-To create a service principal from the Azure portal:

-1. See [Create Service Principal and Certificates](../active-directory/develop/howto-create-service-principal-portal.md) using Microsoft Entra ID.

-To use an existing service principal from the Azure portal:

-1. Service Principal should have owner permissions on the Storage account. See [Set up permissions for the Service Principal to be owner on the storage account](#configure-serviceprincipal-permissions).

-1. Select **Data Lake Store access**.

-1. On the **Data Lake Storage Gen1 access** blade, select **Use existing**.

-1. Select **Service principal**, and then select a service principal.

-1. Upload the certificate (.pfx file) that's associated with your selected service principal, and then enter the certificate password.

- 

-1. Select **Access** to configure the folder access. See [Configure file permissions](#configure-file-permissions).

-### <a name="configure-serviceprincipal-permissions"></a>Set up permissions for the Service Principal to be owner on the storage account

-1. On the Access Control(IAM) blade of storage account click Add a role assignment.

-2. On the Add a role assignment blade select Role as ΓÇÿownerΓÇÖ, and select the SPN and click save.

-### <a name="configure-file-permissions"></a>Configure file permissions

-The configuration is different depending on whether the account is used as the default storage or an additional storage account:

-* Used as default storage

- * permission at the root level of the Data Lake Storage Gen1 account

- * permission at the root level of the HDInsight cluster storage. For example, the __/clusters__ folder used earlier in the tutorial.

-* Use as an additional storage

- * Permission at the folders where you need file access.

-To assign permission at the storage account with Data Lake Storage Gen1 at the root level:

-1. On the **Data Lake Storage Gen1 access** blade, select **Access**. The **Select file permissions** blade is opened. It lists all the storage accounts in your subscription.

-1. Hover (do not click) the mouse over the name of the account with Data Lake Storage Gen1 to make the check box visible, then select the check box.

- 

- By default, __READ__, __WRITE__, AND __EXECUTE__ are all selected.

-1. Click **Select** on the bottom of the page.

-1. Select **Run** to assign permission.

-1. Select **Done**.

-To assign permission at the HDInsight cluster root level:

-1. On the **Data Lake Storage Gen1 access** blade, select **Access**. The **Select file permissions** blade is opened. It lists all the storage accounts with Data Lake Storage Gen1 in your subscription.

-1. From the **Select file permissions** blade, select the storage account with Data Lake Storage Gen1 name to show its content.

-1. Select the HDInsight cluster storage root by selecting the checkbox on the left of the folder. According to the screenshot earlier, the cluster storage root is __/clusters__ folder that you specified while selecting Data Lake Storage Gen1 as default storage.

-1. Set the permissions on the folder. By default, read, write, and execute are all selected.

-1. Click **Select** on the bottom of the page.

-1. Select **Run**.

-1. Select **Done**.

-If you are using Data Lake Storage Gen1 as additional storage, you must assign permission only for the folders that you want to access from the HDInsight cluster. For example, in the screenshot below, you provide access only to the **mynewfolder** folder in a storage account with Data Lake Storage Gen1.

-

-## <a name="verify-cluster-set-up"></a>Verify cluster setup

-After the cluster setup is complete, on the cluster blade, verify your results by doing either or both of the following steps:

-* To verify that the associated storage for the cluster is the account with Data Lake Storage Gen1 that you specified, select **Storage accounts** in the left pane.

- 

-* To verify that the service principal is correctly associated with the HDInsight cluster, select **Data Lake Storage Gen1 access** in the left pane.

- 

-## Examples

-After you set up the cluster with Data Lake Storage Gen1 as your storage, see these examples of how to use HDInsight cluster to analyze the data that's stored in Data Lake Storage Gen1.

-### Run a Hive query against data in a Data Lake Storage Gen1 (as primary storage)

-To run a Hive query, use the Hive views interface in the Ambari portal. For instructions on how to use Ambari Hive views, see [Use the Hive View with Hadoop in HDInsight](../hdinsight/hadoop/apache-hadoop-use-hive-ambari-view.md).

-When you work with data in a Data Lake Storage Gen1, there are a few strings to change.

-If you use, for example, the cluster that you created with Data Lake Storage Gen1 as primary storage, the path to the data is: *adl://<data_lake_storage_gen1_account_name>/azuredatalakestore.net/path/to/file*. A Hive query to create a table from sample data that's stored in the Data Lake Storage Gen1 looks like the following statement:

-```console

-CREATE EXTERNAL TABLE websitelog (str string) LOCATION 'adl://hdiadlsg1storage.azuredatalakestore.net/clusters/myhdiadlcluster/HdiSamples/HdiSamples/WebsiteLogSampleData/SampleLog/'

-```

-Descriptions:

-* `adl://hdiadlsg1storage.azuredatalakestore.net/` is the root of the account with Data Lake Storage Gen1.

-* `/clusters/myhdiadlcluster` is the root of the cluster data that you specified while creating the cluster.

-* `/HdiSamples/HdiSamples/WebsiteLogSampleData/SampleLog/` is the location of the sample file that you used in the query.

-### Run a Hive query against data in a Data Lake Storage Gen1 (as additional storage)

-If the cluster that you created uses Blob storage as default storage, the sample data is not contained in the storage account with Data Lake Storage Gen1 that's used as additional storage. In such a case, first transfer the data from Blob storage to the storage account with Data Lake Storage Gen1, and then run the queries as shown in the preceding example.

-For information on how to copy data from Blob storage to a storage account with Data Lake Storage Gen1, see the following articles:

-* [Use Distcp to copy data between Azure Blob storage and Data Lake Storage Gen1](data-lake-store-copy-data-wasb-distcp.md)

-* [Use AdlCopy to copy data from Azure Blob storage to Data Lake Storage Gen1](data-lake-store-copy-data-azure-storage-blob.md)

-### Use Data Lake Storage Gen1 with a Spark cluster

-You can use a Spark cluster to run Spark jobs on data that is stored in a Data Lake Storage Gen1. For more information, see [Use HDInsight Spark cluster to analyze data in Data Lake Storage Gen1](../hdinsight/spark/apache-spark-use-with-data-lake-store.md).

-### Use Data Lake Storage Gen1 in a Storm topology

-## See also

-* [Use Data Lake Storage Gen1 with Azure HDInsight clusters](../hdinsight/hdinsight-hadoop-use-data-lake-storage-gen1.md)

-* [PowerShell: Create an HDInsight cluster to use Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-powershell.md)

-[makecert]: /windows-hardware/drivers/devtest/makecert

-[pvk2pfx]: /windows-hardware/drivers/devtest/pvk2pfx

-description: Use Azure PowerShell to create and use Azure HDInsight clusters with Azure Data Lake Storage Gen1.

-# Create HDInsight clusters with Azure Data Lake Storage Gen1 as default storage by using PowerShell

-> [!div class="op_single_selector"]

-> * [Use the Azure portal](data-lake-store-hdinsight-hadoop-use-portal.md)

-> * [Use PowerShell (for default storage)](data-lake-store-hdinsight-hadoop-use-powershell-for-default-storage.md)

-> * [Use PowerShell (for additional storage)](data-lake-store-hdinsight-hadoop-use-powershell.md)

-> * [Use Resource Manager](data-lake-store-hdinsight-hadoop-use-resource-manager-template.md)

-Learn how to use Azure PowerShell to configure Azure HDInsight clusters with Azure Data Lake Storage Gen1, as default storage. For instructions on creating an HDInsight cluster with Data Lake Storage Gen1 as additional storage, see [Create an HDInsight cluster with Data Lake Storage Gen1 as additional storage](data-lake-store-hdinsight-hadoop-use-powershell.md).

-Here are some important considerations for using HDInsight with Data Lake Storage Gen1:

-* The option to create HDInsight clusters with access to Data Lake Storage Gen1 as default storage is available for HDInsight version 3.5 and 3.6.

-* The option to create HDInsight clusters with access to Data Lake Storage Gen1 as default storage is *not available* for HDInsight Premium clusters.

-To configure HDInsight to work with Data Lake Storage Gen1 by using PowerShell, follow the instructions in the next five sections.

-## Prerequisites

-Before you begin this tutorial, make sure that you meet the following requirements:

-* **An Azure subscription**: Go to [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure PowerShell 1.0 or greater**: See [How to install and configure PowerShell](/powershell/azure/).

-* **Windows Software Development Kit (SDK)**: To install Windows SDK, go to [Downloads and tools for Windows 10](https://dev.windows.com/downloads). The SDK is used to create a security certificate.

-* **Microsoft Entra service principal**: This tutorial describes how to create a service principal in Microsoft Entra ID. However, to create a service principal, you must be a Microsoft Entra administrator. If you are an administrator, you can skip this prerequisite and proceed with the tutorial.

- >[!NOTE]

- >You can create a service principal only if you are a Microsoft Entra administrator. Your Microsoft Entra administrator must create a service principal before you can create an HDInsight cluster with Data Lake Storage Gen1. The service principal must be created with a certificate, as described in [Create a service principal with certificate](../active-directory/develop/howto-authenticate-service-principal-powershell.md#create-service-principal-with-certificate-from-certificate-authority).

- >

-## Create an Azure Data Lake Storage Gen1 account

-To create a Data Lake Storage Gen1 account, do the following:

-1. From your desktop, open a PowerShell window, and then enter the snippets below. When you are prompted to sign in, sign in as one of the subscription administrators or owners.

- ```azurepowershell

- # Sign in to your Azure account

- Connect-AzAccount

- # List all the subscriptions associated to your account

- Get-AzSubscription

- # Select a subscription

- Set-AzContext -SubscriptionId <subscription ID>

- # Register for Data Lake Storage Gen1

- Register-AzResourceProvider -ProviderNamespace "Microsoft.DataLakeStore"

- ```

- > [!NOTE]

- > If you register the Data Lake Storage Gen1 resource provider and receive an error similar to `Register-AzResourceProvider : InvalidResourceNamespace: The resource namespace 'Microsoft.DataLakeStore' is invalid`, your subscription might not be approved for Data Lake Storage Gen1. To enable your Azure subscription for Data Lake Storage Gen1, follow the instructions in [Get started with Azure Data Lake Storage Gen1 by using the Azure portal](data-lake-store-get-started-portal.md).

- >

-2. A Data Lake Storage Gen1 account is associated with an Azure resource group. Start by creating a resource group.

- ```azurepowershell

- $resourceGroupName = "<your new resource group name>"

- New-AzResourceGroup -Name $resourceGroupName -Location "East US 2"

- ```

- You should see an output like this:

- ```output

- ResourceGroupName : hdiadlgrp

- Location : eastus2

- ProvisioningState : Succeeded

- Tags :

- ResourceId : /subscriptions/<subscription-id>/resourceGroups/hdiadlgrp

- ```

-3. Create a Data Lake Storage Gen1 account. The account name you specify must contain only lowercase letters and numbers.

- ```azurepowershell

- $dataLakeStorageGen1Name = "<your new Data Lake Storage Gen1 name>"

- New-AzDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeStorageGen1Name -Location "East US 2"

- ```

- You should see an output like the following:

- ```output

- ...

- ProvisioningState : Succeeded

- State : Active

- CreationTime : 5/5/2017 10:53:56 PM

- EncryptionState : Enabled

- ...

- LastModifiedTime : 5/5/2017 10:53:56 PM

- Endpoint : hdiadlstore.azuredatalakestore.net

- DefaultGroup :

- Id : /subscriptions/<subscription-id>/resourceGroups/hdiadlgrp/providers/Microsoft.DataLakeStore/accounts/hdiadlstore

- Name : hdiadlstore

- Type : Microsoft.DataLakeStore/accounts

- Location : East US 2

- Tags : {}

- ```

-4. Using Data Lake Storage Gen1 as default storage requires you to specify a root path to which the cluster-specific files are copied during cluster creation. To create a root path, which is **/clusters/hdiadlcluster** in the snippet, use the following cmdlets:

- ```azurepowershell

- $myrootdir = "/"

- New-AzDataLakeStoreItem -Folder -AccountName $dataLakeStorageGen1Name -Path $myrootdir/clusters/hdiadlcluster

- ````

-## Set up authentication for role-based access to Data Lake Storage Gen1

-Every Azure subscription is associated with a Microsoft Entra entity. Users and services that access subscription resources by using the Azure portal or the Azure Resource Manager API must first authenticate with Microsoft Entra ID. Access is granted to Azure subscriptions and services by assigning them the appropriate role on an Azure resource. For services, a service principal identifies the service in Microsoft Entra ID.

-This section illustrates how to grant an application service, such as HDInsight, access to an Azure resource (the Data Lake Storage Gen1 account that you created earlier). You do so by creating a service principal for the application and assigning roles to it via PowerShell.

-To set up Active Directory authentication for Data Lake Storage Gen1, perform the tasks in the following two sections.

-### Create a self-signed certificate

-Make sure you have [Windows SDK](https://dev.windows.com/en-us/downloads) installed before proceeding with the steps in this section. You must have also created a directory, such as *C:\mycertdir*, where you create the certificate.

-1. From the PowerShell window, go to the location where you installed Windows SDK (typically, *C:\Program Files (x86)\Windows Kits\10\bin\x86*) and use the [MakeCert][makecert] utility to create a self-signed certificate and a private key. Use the following commands:

- ```azurepowershell

- $certificateFileDir = "<my certificate directory>"

- cd $certificateFileDir

- makecert -sv mykey.pvk -n "cn=HDI-ADL-SP" CertFile.cer -r -len 2048

- ```

- You will be prompted to enter the private key password. After the command is successfully executed, you should see **CertFile.cer** and **mykey.pvk** in the certificate directory that you specified.

-2. Use the [Pvk2Pfx][pvk2pfx] utility to convert the .pvk and .cer files that MakeCert created to a .pfx file. Run the following command:

- ```azurepowershell

- pvk2pfx -pvk mykey.pvk -spc CertFile.cer -pfx CertFile.pfx -po <password>

- ```

- When you are prompted, enter the private key password that you specified earlier. The value you specify for the **-po** parameter is the password that's associated with the .pfx file. After the command has been completed successfully, you should also see a **CertFile.pfx** in the certificate directory that you specified.

-<a name='create-an-azure-ad-and-a-service-principal'></a>

-### Create a Microsoft Entra ID and a service principal

-In this section, you create a service principal for a Microsoft Entra application, assign a role to the service principal, and authenticate as the service principal by providing a certificate. To create an application in Microsoft Entra ID, run the following commands:

-1. Paste the following cmdlets in the PowerShell console window. Make sure that the value you specify for the **-DisplayName** property is unique. The values for **-HomePage** and **-IdentiferUris** are placeholder values and are not verified.

- ```azurepowershell

- $certificateFilePath = "$certificateFileDir\CertFile.pfx"

- $password = Read-Host -Prompt "Enter the password" # This is the password you specified for the .pfx file

- $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certificateFilePath, $password)

- $rawCertificateData = $certificatePFX.GetRawCertData()

- $credential = [System.Convert]::ToBase64String($rawCertificateData)

- $application = New-AzADApplication `

- -DisplayName "HDIADL" `

- -HomePage "https://contoso.com" `

- -IdentifierUris "https://contoso.com" `

- -CertValue $credential `

- -StartDate $certificatePFX.NotBefore `

- -EndDate $certificatePFX.NotAfter

- $applicationId = $application.ApplicationId

- ```

-2. Create a service principal by using the application ID.

- ```azurepowershell

- $servicePrincipal = New-AzADServicePrincipal -ApplicationId $applicationId -Role Contributor

- $objectId = $servicePrincipal.Id

- ```

-3. Grant the service principal access to the Data Lake Storage Gen1 root and all the folders in the root path that you specified earlier. Use the following cmdlets:

- ```azurepowershell

- Set-AzDataLakeStoreItemAclEntry -AccountName $dataLakeStorageGen1Name -Path / -AceType User -Id $objectId -Permissions All

- Set-AzDataLakeStoreItemAclEntry -AccountName $dataLakeStorageGen1Name -Path /clusters -AceType User -Id $objectId -Permissions All

- Set-AzDataLakeStoreItemAclEntry -AccountName $dataLakeStorageGen1Name -Path /clusters/hdiadlcluster -AceType User -Id $objectId -Permissions All

- ```

-## Create an HDInsight Linux cluster with Data Lake Storage Gen1 as the default storage

-In this section, you create an HDInsight Hadoop Linux cluster with Data Lake Storage Gen1 as the default storage. For this release, the HDInsight cluster and Data Lake Storage Gen1 must be in the same location.

-1. Retrieve the subscription tenant ID, and store it to use later.

- ```azurepowershell

- $tenantID = (Get-AzContext).Tenant.TenantId

- ```

-2. Create the HDInsight cluster by using the following cmdlets:

- ```azurepowershell

- # Set these variables

- $location = "East US 2"

- $storageAccountName = $dataLakeStorageGen1Name # Data Lake Storage Gen1 account name

- $storageRootPath = "<Storage root path you specified earlier>" # e.g. /clusters/hdiadlcluster

- $clusterName = "<unique cluster name>"

- $clusterNodes = <ClusterSizeInNodes> # The number of nodes in the HDInsight cluster

- $httpCredentials = Get-Credential

- $sshCredentials = Get-Credential

- New-AzHDInsightCluster `

- -ClusterType Hadoop `

- -OSType Linux `

- -ClusterSizeInNodes $clusterNodes `

- -ResourceGroupName $resourceGroupName `

- -ClusterName $clusterName `

- -HttpCredential $httpCredentials `

- -Location $location `

- -DefaultStorageAccountType AzureDataLakeStore `

- -DefaultStorageAccountName "$storageAccountName.azuredatalakestore.net" `

- -DefaultStorageRootPath $storageRootPath `

- -Version "3.6" `

- -SshCredential $sshCredentials `

- -AadTenantId $tenantId `

- -ObjectId $objectId `

- -CertificateFilePath $certificateFilePath `

- -CertificatePassword $password

- ```

- After the cmdlet has been successfully completed, you should see an output that lists the cluster details.

-## Run test jobs on the HDInsight cluster to use Data Lake Storage Gen1

-After you have configured an HDInsight cluster, you can run test jobs on it to ensure that it can access Data Lake Storage Gen1. To do so, run a sample Hive job to create a table that uses the sample data that's already available in Data Lake Storage Gen1 at *\<cluster root>/example/data/sample.log*.

-In this section, you make a Secure Shell (SSH) connection into the HDInsight Linux cluster that you created, and then you run a sample Hive query.

-* If you are using a Windows client to make an SSH connection into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Windows](../hdinsight/hdinsight-hadoop-linux-use-ssh-windows.md).

-* If you are using a Linux client to make an SSH connection into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Linux](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-1. After you have made the connection, start the Hive command-line interface (CLI) by using the following command:

- ```powershell

- hive

- ```

-2. Use the CLI to enter the following statements to create a new table named **vehicles** by using the sample data in Data Lake Storage Gen1:

- ```azurepowershell

- DROP TABLE log4jLogs;

- CREATE EXTERNAL TABLE log4jLogs (t1 string, t2 string, t3 string, t4 string, t5 string, t6 string, t7 string)

- ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '

- STORED AS TEXTFILE LOCATION 'adl:///example/data/';

- SELECT t4 AS sev, COUNT(*) AS count FROM log4jLogs WHERE t4 = '[ERROR]' AND INPUT__FILE__NAME LIKE '%.log' GROUP BY t4;

- ```

-You should see the query output on the SSH console.

->[!NOTE]

->The path to the sample data in the preceding CREATE TABLE command is `adl:///example/data/`, where `adl:///` is the cluster root. Following the example of the cluster root that's specified in this tutorial, the command is `adl://hdiadlstore.azuredatalakestore.net/clusters/hdiadlcluster`. You can either use the shorter alternative or provide the complete path to the cluster root.

->

-## Access Data Lake Storage Gen1 by using HDFS commands

-After you have configured the HDInsight cluster to use Data Lake Storage Gen1, you can use Hadoop Distributed File System (HDFS) shell commands to access the store.

-In this section, you make an SSH connection into the HDInsight Linux cluster that you created, and then you run the HDFS commands.

-* If you are using a Windows client to make an SSH connection into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Windows](../hdinsight/hdinsight-hadoop-linux-use-ssh-windows.md).

-* If you are using a Linux client to make an SSH connection into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Linux](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-After you've made the connection, list the files in Data Lake Storage Gen1 by using the following HDFS file system command.

-```azurepowershell

-hdfs dfs -ls adl:///

-```

-You can also use the `hdfs dfs -put` command to upload some files to Data Lake Storage Gen1, and then use `hdfs dfs -ls` to verify whether the files were successfully uploaded.

-## See also

-* [Use Data Lake Storage Gen1 with Azure HDInsight clusters](../hdinsight/hdinsight-hadoop-use-data-lake-storage-gen1.md)

-* [Azure portal: Create an HDInsight cluster to use Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-[makecert]: /windows-hardware/drivers/devtest/makecert

-[pvk2pfx]: /windows-hardware/drivers/devtest/pvk2pfx

-description: Learn how to use Azure PowerShell to configure an HDInsight cluster with Azure Data Lake Storage Gen1 as additional storage.

-# Use Azure PowerShell to create an HDInsight cluster with Azure Data Lake Storage Gen1 (as additional storage)

-> [!div class="op_single_selector"]

-> * [Using Portal](data-lake-store-hdinsight-hadoop-use-portal.md)

-> * [Using PowerShell (for default storage)](data-lake-store-hdinsight-hadoop-use-powershell-for-default-storage.md)

-> * [Using PowerShell (for additional storage)](data-lake-store-hdinsight-hadoop-use-powershell.md)

-> * [Using Resource Manager](data-lake-store-hdinsight-hadoop-use-resource-manager-template.md)

->

->

-Learn how to use Azure PowerShell to configure an HDInsight cluster with Azure Data Lake Storage Gen1, **as additional storage**. For instructions on how to create an HDInsight cluster with Data Lake Storage Gen1 as default storage, see [Create an HDInsight cluster with Data Lake Storage Gen1 as default storage](data-lake-store-hdinsight-hadoop-use-powershell-for-default-storage.md).

-> [!NOTE]

-> If you are going to use Data Lake Storage Gen1 as additional storage for HDInsight cluster, we strongly recommend that you do this while you create the cluster as described in this article. Adding Data Lake Storage Gen1 as additional storage to an existing HDInsight cluster is a complicated process and prone to errors.

->

-For supported cluster types, Data Lake Storage Gen1 can be used as a default storage or additional storage account. When Data Lake Storage Gen1 is used as additional storage, the default storage account for the clusters will still be Azure Blob storage (WASB) and the cluster-related files (such as logs, etc.) are still written to the default storage, while the data that you want to process can be stored in a Data Lake Storage Gen1. Using Data Lake Storage Gen1 as an additional storage account does not impact performance or the ability to read/write to the storage from the cluster.

-## Using Data Lake Storage Gen1 for HDInsight cluster storage

-Here are some important considerations for using HDInsight with Data Lake Storage Gen1:

-* Option to create HDInsight clusters with access to Data Lake Storage Gen1 as additional storage is available for HDInsight versions 3.2, 3.4, 3.5, and 3.6.

-Configuring HDInsight to work with Data Lake Storage Gen1 using PowerShell involves the following steps:

-* Create a Data Lake Storage Gen1 account

-* Set up authentication for role-based access to Data Lake Storage Gen1

-* Create HDInsight cluster with authentication to Data Lake Storage Gen1

-* Run a test job on the cluster

-## Prerequisites

-Before you begin this tutorial, you must have the following:

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure PowerShell 1.0 or greater**. See [How to install and configure Azure PowerShell](/powershell/azure/).

-* **Windows SDK**. You can install it from [here](https://dev.windows.com/en-us/downloads). You use this to create a security certificate.

-* **Microsoft Entra service principal**. Steps in this tutorial provide instructions on how to create a service principal in Microsoft Entra ID. However, you must be a Microsoft Entra administrator to be able to create a service principal. If you are a Microsoft Entra administrator, you can skip this prerequisite and proceed with the tutorial.

- **If you are not a Microsoft Entra administrator**, you will not be able to perform the steps required to create a service principal. In such a case, your Microsoft Entra administrator must first create a service principal before you can create an HDInsight cluster with Data Lake Storage Gen1. Also, the service principal must be created using a certificate, as described at [Create a service principal with certificate](../active-directory/develop/howto-authenticate-service-principal-powershell.md#create-service-principal-with-certificate-from-certificate-authority).

-## Create a Data Lake Storage Gen1 account

-Follow these steps to create a Data Lake Storage Gen1 account.

-1. From your desktop, open a new Azure PowerShell window, and enter the following snippet. When prompted to log in, make sure you log in as one of the subscription administrator/owner:

- ```azurepowershell

- # Log in to your Azure account

- Connect-AzAccount

- # List all the subscriptions associated to your account

- Get-AzSubscription

- # Select a subscription

- Set-AzContext -SubscriptionId <subscription ID>

- # Register for Data Lake Storage Gen1

- Register-AzResourceProvider -ProviderNamespace "Microsoft.DataLakeStore"

- ```

- > [!NOTE]

- > If you receive an error similar to `Register-AzResourceProvider : InvalidResourceNamespace: The resource namespace 'Microsoft.DataLakeStore' is invalid` when registering the Data Lake Storage Gen1 resource provider, it is possible that your subscription is not approved for Data Lake Storage Gen1. Make sure you enable your Azure subscription for Data Lake Storage Gen1 by following these [instructions](data-lake-store-get-started-portal.md).

- >

- >

-2. A storage account with Data Lake Storage Gen1 is associated with an Azure Resource Group. Start by creating an Azure Resource Group.

- ```azurepowershell

- $resourceGroupName = "<your new resource group name>"

- New-AzResourceGroup -Name $resourceGroupName -Location "East US 2"

- ```

- You should see an output like this:

- ```output

- ResourceGroupName : hdiadlgrp

- Location : eastus2

- ProvisioningState : Succeeded

- Tags :

- ResourceId : /subscriptions/<subscription-id>/resourceGroups/hdiadlgrp

- ```

-3. Create a storage account with Data Lake Storage Gen1. The account name you specify must only contain lowercase letters and numbers.

- ```azurepowershell

- $dataLakeStorageGen1Name = "<your new storage account with Data Lake Storage Gen1 name>"

- New-AzDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeStorageGen1Name -Location "East US 2"

- ```

- You should see an output like the following:

- ```output

- ...

- ProvisioningState : Succeeded

- State : Active

- CreationTime : 5/5/2017 10:53:56 PM

- EncryptionState : Enabled

- ...

- LastModifiedTime : 5/5/2017 10:53:56 PM

- Endpoint : hdiadlstore.azuredatalakestore.net

- DefaultGroup :

- Id : /subscriptions/<subscription-id>/resourceGroups/hdiadlgrp/providers/Microsoft.DataLakeStore/accounts/hdiadlstore

- Name : hdiadlstore

- Type : Microsoft.DataLakeStore/accounts

- Location : East US 2

- Tags : {}

- ```

-5. Upload some sample data to Data Lake Storage Gen1. We'll use this later in this article to verify that the data is accessible from an HDInsight cluster. If you are looking for some sample data to upload, you can get the **Ambulance Data** folder from the [Azure Data Lake Git Repository](https://github.com/MicrosoftBigData/usql/tree/master/Examples/Samples/Data/AmbulanceData).

- ```azurepowershell

- $myrootdir = "/"

- Import-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name -Path "C:\<path to data>\vehicle1_09142014.csv" -Destination $myrootdir\vehicle1_09142014.csv

- ```

-## Set up authentication for role-based access to Data Lake Storage Gen1

-Every Azure subscription is associated with a Microsoft Entra ID. Users and services that access resources of the subscription using the Azure portal or Azure Resource Manager API must first authenticate with that Microsoft Entra ID. Access is granted to Azure subscriptions and services by assigning them the appropriate role on an Azure resource. For services, a service principal identifies the service in the Microsoft Entra ID. This section illustrates how to grant an application service, like HDInsight, access to an Azure resource (the storage account with Data Lake Storage Gen1 you created earlier) by creating a service principal for the application and assigning roles to that via Azure PowerShell.

-To set up Active Directory authentication for Data Lake Storage Gen1, you must perform the following tasks.

-* Create a self-signed certificate

-* Create an application in Microsoft Entra ID and a Service Principal

-### Create a self-signed certificate

-Make sure you have [Windows SDK](https://dev.windows.com/en-us/downloads) installed before proceeding with the steps in this section. You must have also created a directory, such as **C:\mycertdir**, where the certificate will be created.

-1. From the PowerShell window, navigate to the location where you installed Windows SDK (typically, `C:\Program Files (x86)\Windows Kits\10\bin\x86` and use the [MakeCert][makecert] utility to create a self-signed certificate and a private key. Use the following commands.

- ```azurepowershell

- $certificateFileDir = "<my certificate directory>"

- cd $certificateFileDir

- makecert -sv mykey.pvk -n "cn=HDI-ADL-SP" CertFile.cer -r -len 2048

- ```

- You will be prompted to enter the private key password. After the command successfully executes, you should see a **CertFile.cer** and **mykey.pvk** in the certificate directory you specified.

-2. Use the [Pvk2Pfx][pvk2pfx] utility to convert the .pvk and .cer files that MakeCert created to a .pfx file. Run the following command.

- ```azurepowershell

- pvk2pfx -pvk mykey.pvk -spc CertFile.cer -pfx CertFile.pfx -po <password>

- ```

- When prompted enter the private key password you specified earlier. The value you specify for the **-po** parameter is the password that is associated with the .pfx file. After the command successfully completes, you should also see a CertFile.pfx in the certificate directory you specified.

-<a name='create-an-azure-active-directory-and-a-service-principal'></a>

-### Create a Microsoft Entra ID and a service principal

-In this section, you perform the steps to create a service principal for a Microsoft Entra application, assign a role to the service principal, and authenticate as the service principal by providing a certificate. Run the following commands to create an application in Microsoft Entra ID.

-1. Paste the following cmdlets in the PowerShell console window. Make sure the value you specify for the **-DisplayName** property is unique. Also, the values for **-HomePage** and **-IdentiferUris** are placeholder values and are not verified.

- ```azurepowershell

- $certificateFilePath = "$certificateFileDir\CertFile.pfx"

- $password = Read-Host -Prompt "Enter the password" # This is the password you specified for the .pfx file

- $certificatePFX = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certificateFilePath, $password)

- $rawCertificateData = $certificatePFX.GetRawCertData()

- $credential = [System.Convert]::ToBase64String($rawCertificateData)

- $application = New-AzADApplication `

- -DisplayName "HDIADL" `

- -HomePage "https://contoso.com" `

- -IdentifierUris "https://contoso.com" `

- -CertValue $credential `

- -StartDate $certificatePFX.NotBefore `

- -EndDate $certificatePFX.NotAfter

- $applicationId = $application.ApplicationId

- ```

-2. Create a service principal using the application ID.

- ```azurepowershell

- $servicePrincipal = New-AzADServicePrincipal -ApplicationId $applicationId -Role Contributor

- $objectId = $servicePrincipal.Id

- ```

-3. Grant the service principal access to the Data Lake Storage Gen1 folder and the file that you will access from the HDInsight cluster. The snippet below provides access to the root of the storage account with Data Lake Storage Gen1 (where you copied the sample data file), and the file itself.

- ```azurepowershell

- Set-AzDataLakeStoreItemAclEntry -AccountName $dataLakeStorageGen1Name -Path / -AceType User -Id $objectId -Permissions All

- Set-AzDataLakeStoreItemAclEntry -AccountName $dataLakeStorageGen1Name -Path /vehicle1_09142014.csv -AceType User -Id $objectId -Permissions All

- ```

-## Create an HDInsight Linux cluster with Data Lake Storage Gen1 as additional storage

-In this section, we create an HDInsight Hadoop Linux cluster with Data Lake Storage Gen1 as additional storage. For this release, the HDInsight cluster and storage account with Data Lake Storage Gen1 must be in the same location.

-1. Start with retrieving the subscription tenant ID. You will need that later.

- ```azurepowershell

- $tenantID = (Get-AzContext).Tenant.TenantId

- ```

-2. For this release, for a Hadoop cluster, Data Lake Storage Gen1 can only be used as an additional storage for the cluster. The default storage will still be the Azure Blob storage (WASB). So, we'll first create the storage account and storage containers required for the cluster.

- ```azurepowershell

- # Create an Azure storage account

- $location = "East US 2"

- $storageAccountName = "<StorageAccountName>" # Provide a Storage account name

- New-AzStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $storageAccountName -Location $location -Type Standard_GRS

- # Create an Azure Blob Storage container

- $containerName = "<ContainerName>" # Provide a container name

- $storageAccountKey = (Get-AzStorageAccountKey -Name $storageAccountName -ResourceGroupName $resourceGroupName)[0].Value

- $destContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey

- New-AzStorageContainer -Name $containerName -Context $destContext

- ```

-3. Create the HDInsight cluster. Use the following cmdlets.

- ```azurepowershell

- # Set these variables

- $clusterName = $containerName # As a best practice, have the same name for the cluster and container

- $clusterNodes = <ClusterSizeInNodes> # The number of nodes in the HDInsight cluster

- $httpCredentials = Get-Credential

- $sshCredentials = Get-Credential

- New-AzHDInsightCluster -ClusterName $clusterName -ResourceGroupName $resourceGroupName -HttpCredential $httpCredentials -Location $location -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $containerName -ClusterSizeInNodes $clusterNodes -ClusterType Hadoop -Version "3.4" -OSType Linux -SshCredential $sshCredentials -ObjectID $objectId -AadTenantId $tenantID -CertificateFilePath $certificateFilePath -CertificatePassword $password

- ```

- After the cmdlet successfully completes, you should see an output listing the cluster details.

-## Run test jobs on the HDInsight cluster to use the Data Lake Storage Gen1

-After you have configured an HDInsight cluster, you can run test jobs on the cluster to test that the HDInsight cluster can access Data Lake Storage Gen1. To do so, we will run a sample Hive job that creates a table using the sample data that you uploaded earlier to your storage account with Data Lake Storage Gen1.

-In this section you will SSH into the HDInsight Linux cluster you created and run the a sample Hive query.

-* If you are using a Windows client to SSH into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Windows](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-* If you are using a Linux client to SSH into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Linux](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md)

-1. Once connected, start the Hive CLI by using the following command:

- ```azurepowershell

- hive

- ```

-2. Using the CLI, enter the following statements to create a new table named **vehicles** by using the sample data in Data Lake Storage Gen1:

- ```azurepowershell

- DROP TABLE vehicles;

- CREATE EXTERNAL TABLE vehicles (str string) LOCATION 'adl://<mydatalakestoragegen1>.azuredatalakestore.net:443/';

- SELECT * FROM vehicles LIMIT 10;

- ```

- You should see an output similar to the following:

- ```output

- 1,1,2014-09-14 00:00:03,46.81006,-92.08174,51,S,1

- 1,2,2014-09-14 00:00:06,46.81006,-92.08174,13,NE,1

- 1,3,2014-09-14 00:00:09,46.81006,-92.08174,48,NE,1

- 1,4,2014-09-14 00:00:12,46.81006,-92.08174,30,W,1

- 1,5,2014-09-14 00:00:15,46.81006,-92.08174,47,S,1

- 1,6,2014-09-14 00:00:18,46.81006,-92.08174,9,S,1

- 1,7,2014-09-14 00:00:21,46.81006,-92.08174,53,N,1

- 1,8,2014-09-14 00:00:24,46.81006,-92.08174,63,SW,1

- 1,9,2014-09-14 00:00:27,46.81006,-92.08174,4,NE,1

- 1,10,2014-09-14 00:00:30,46.81006,-92.08174,31,N,1

- ```

-## Access Data Lake Storage Gen1 using HDFS commands

-Once you have configured the HDInsight cluster to use Data Lake Storage Gen1, you can use the HDFS shell commands to access the store.

-In this section you will SSH into the HDInsight Linux cluster you created and run the HDFS commands.

-* If you are using a Windows client to SSH into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Windows](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-* If you are using a Linux client to SSH into the cluster, see [Use SSH with Linux-based Hadoop on HDInsight from Linux](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md)

-Once connected, use the following HDFS filesystem command to list the files in the storage account with Data Lake Storage Gen1.

-```azurepowershell

-hdfs dfs -ls adl://<storage account with Data Lake Storage Gen1 name>.azuredatalakestore.net:443/

-```

-This should list the file that you uploaded earlier to Data Lake Storage Gen1.

-```output

-15/09/17 21:41:15 INFO web.CaboWebHdfsFileSystem: Replacing original urlConnectionFactory with org.apache.hadoop.hdfs.web.URLConnectionFactory@21a728d6

-Found 1 items

-```

-You can also use the `hdfs dfs -put` command to upload some files to Data Lake Storage Gen1, and then use `hdfs dfs -ls` to verify whether the files were successfully uploaded.

-## See Also

-* [Use Data Lake Storage Gen1 with Azure HDInsight clusters](../hdinsight/hdinsight-hadoop-use-data-lake-storage-gen1.md)

-* [Portal: Create an HDInsight cluster to use Data Lake Storage Gen1](data-lake-store-hdinsight-hadoop-use-portal.md)

-[makecert]: /windows-hardware/drivers/devtest/makecert

-[pvk2pfx]: /windows-hardware/drivers/devtest/pvk2pfx

-description: Use Azure Resource Manager templates to create and use Azure HDInsight clusters with Azure Data Lake Storage Gen1.

-# Create an HDInsight cluster with Azure Data Lake Storage Gen1 using Azure Resource Manager template

-> [!div class="op_single_selector"]

-> * [Using Portal](data-lake-store-hdinsight-hadoop-use-portal.md)

-> * [Using PowerShell (for default storage)](data-lake-store-hdinsight-hadoop-use-powershell-for-default-storage.md)

-> * [Using PowerShell (for additional storage)](data-lake-store-hdinsight-hadoop-use-powershell.md)

-> * [Using Resource Manager](data-lake-store-hdinsight-hadoop-use-resource-manager-template.md)

->

->

-Learn how to use Azure PowerShell to configure an HDInsight cluster with Azure Data Lake Storage Gen1, **as additional storage**.

-For supported cluster types, Data Lake Storage Gen1 can be used as a default storage or as an additional storage account. When Data Lake Storage Gen1 is used as additional storage, the default storage account for the clusters will still be Azure Blob storage (WASB) and the cluster-related files (such as logs, etc.) are still written to the default storage, while the data that you want to process can be stored in a Data Lake Storage Gen1 account. Using Data Lake Storage Gen1 as an additional storage account does not impact performance or the ability to read/write to the storage from the cluster.

-## Using Data Lake Storage Gen1 for HDInsight cluster storage

-Here are some important considerations for using HDInsight with Data Lake Storage Gen1:

-* Option to create HDInsight clusters with access to Data Lake Storage Gen1 as default storage is available for HDInsight version 3.5 and 3.6.

-* Option to create HDInsight clusters with access to Data Lake Storage Gen1 as additional storage is available for HDInsight versions 3.2, 3.4, 3.5, and 3.6.

-In this article, we provision a Hadoop cluster with Data Lake Storage Gen1 as additional storage. For instructions on how to create a Hadoop cluster with Data Lake Storage Gen1 as default storage, see [Create an HDInsight cluster with Data Lake Storage Gen1 using Azure portal](data-lake-store-hdinsight-hadoop-use-portal.md).

-## Prerequisites

-Before you begin this tutorial, you must have the following:

-* **An Azure subscription**. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial/).

-* **Azure PowerShell 1.0 or greater**. See [How to install and configure Azure PowerShell](/powershell/azure/).

-* **Microsoft Entra service principal**. Steps in this tutorial provide instructions on how to create a service principal in Microsoft Entra ID. However, you must be a Microsoft Entra administrator to be able to create a service principal. If you are a Microsoft Entra administrator, you can skip this prerequisite and proceed with the tutorial.

- **If you are not a Microsoft Entra administrator**, you will not be able to perform the steps required to create a service principal. In such a case, your Microsoft Entra administrator must first create a service principal before you can create an HDInsight cluster with Data Lake Storage Gen1. Also, the service principal must be created using a certificate, as described at [Create a service principal with certificate](../active-directory/develop/howto-authenticate-service-principal-powershell.md#create-service-principal-with-certificate-from-certificate-authority).

-## Create an HDInsight cluster with Data Lake Storage Gen1

-The Resource Manager template, and the prerequisites for using the template, are available on GitHub at [Deploy a HDInsight Linux cluster with new Data Lake Storage Gen1](https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.hdinsight/hdinsight-datalake-store-azure-storage). Follow the instructions provided at this link to create an HDInsight cluster with Data Lake Storage Gen1 as the additional storage.

-The instructions at the link mentioned above require PowerShell. Before you start with those instructions, make sure you log in to your Azure account. From your desktop, open a new Azure PowerShell window, and enter the following snippets. When prompted to log in, make sure you log in as one of the subscription administrators/owner:

-```

-# Log in to your Azure account

-Connect-AzAccount

-# List all the subscriptions associated to your account

-Get-AzSubscription

-# Select a subscription

-Set-AzContext -SubscriptionId <subscription ID>

-```

-The template deploys these resource types:

-* [Microsoft.DataLakeStore/accounts](/azure/templates/microsoft.datalakestore/accounts)

-* [Microsoft.Storage/storageAccounts](/azure/templates/microsoft.storage/storageaccounts)

-* [Microsoft.HDInsight/clusters](/azure/templates/microsoft.hdinsight/clusters)

-## Upload sample data to Data Lake Storage Gen1

-The Resource Manager template creates a new storage account with Data Lake Storage Gen1 and associates it with the HDInsight cluster. You must now upload some sample data to Data Lake Storage Gen1. You'll need this data later in the tutorial to run jobs from an HDInsight cluster that access data in the storage account with Data Lake Storage Gen1. For instructions on how to upload data, see [Upload a file to Data Lake Storage Gen1](data-lake-store-get-started-portal.md#uploaddata). If you are looking for some sample data to upload, you can get the **Ambulance Data** folder from the [Azure Data Lake Git Repository](https://github.com/Azure/usql/tree/master/Examples/Samples/Data/AmbulanceData).

-## Set relevant ACLs on the sample data

-To make sure the sample data you upload is accessible from the HDInsight cluster, you must ensure that the Microsoft Entra application that is used to establish identity between the HDInsight cluster and Data Lake Storage Gen1 has access to the file/folder you are trying to access. To do this, perform the following steps.

-1. Find the name of the Microsoft Entra application that is associated with HDInsight cluster and the storage account with Data Lake Storage Gen1. One way to look for the name is to open the HDInsight cluster blade that you created using the Resource Manager template, click the **Cluster Microsoft Entra identity** tab, and look for the value of **Service Principal Display Name**.

-2. Now, provide access to this Microsoft Entra application on the file/folder that you want to access from the HDInsight cluster. To set the right ACLs on the file/folder in Data Lake Storage Gen1, see [Securing data in Data Lake Storage Gen1](data-lake-store-secure-data.md#filepermissions).

-## Run test jobs on the HDInsight cluster to use Data Lake Storage Gen1

-After you have configured an HDInsight cluster, you can run test jobs on the cluster to test that the HDInsight cluster can access Data Lake Storage Gen1. To do so, we will run a sample Hive job that creates a table using the sample data that you uploaded earlier to your storage account with Data Lake Storage Gen1.

-In this section, you SSH into an HDInsight Linux cluster and run the sample Hive query. If you are using a Windows client, we recommend using **PuTTY**, which can be downloaded from [https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html](https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html).

-For more information on using PuTTY, see [Use SSH with Linux-based Hadoop on HDInsight from Windows](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-1. Once connected, start the Hive CLI by using the following command:

- ```

- hive

- ```

-2. Using the CLI, enter the following statements to create a new table named **vehicles** by using the sample data in Data Lake Storage Gen1:

- ```

- DROP TABLE vehicles;

- CREATE EXTERNAL TABLE vehicles (str string) LOCATION 'adl://<mydatalakestoragegen1>.azuredatalakestore.net:443/';

- SELECT * FROM vehicles LIMIT 10;

- ```

- You should see output similar to the following:

- ```

- 1,1,2014-09-14 00:00:03,46.81006,-92.08174,51,S,1

- 1,2,2014-09-14 00:00:06,46.81006,-92.08174,13,NE,1

- 1,3,2014-09-14 00:00:09,46.81006,-92.08174,48,NE,1

- 1,4,2014-09-14 00:00:12,46.81006,-92.08174,30,W,1

- 1,5,2014-09-14 00:00:15,46.81006,-92.08174,47,S,1

- 1,6,2014-09-14 00:00:18,46.81006,-92.08174,9,S,1

- 1,7,2014-09-14 00:00:21,46.81006,-92.08174,53,N,1

- 1,8,2014-09-14 00:00:24,46.81006,-92.08174,63,SW,1

- 1,9,2014-09-14 00:00:27,46.81006,-92.08174,4,NE,1

- 1,10,2014-09-14 00:00:30,46.81006,-92.08174,31,N,1

- ```

-## Access Data Lake Storage Gen1 using HDFS commands

-Once you have configured the HDInsight cluster to use Data Lake Storage Gen1, you can use the HDFS shell commands to access the store.

-In this section, you SSH into an HDInsight Linux cluster and run the HDFS commands. If you are using a Windows client, we recommend using **PuTTY**, which can be downloaded from [https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html](https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html).

-For more information on using PuTTY, see [Use SSH with Linux-based Hadoop on HDInsight from Windows](../hdinsight/hdinsight-hadoop-linux-use-ssh-unix.md).

-Once connected, use the following HDFS filesystem command to list the files in the storage account with Data Lake Storage Gen1.

-```

-hdfs dfs -ls adl://<storage account with Data Lake Storage Gen1 name>.azuredatalakestore.net:443/

-```

-This should list the file that you uploaded earlier to Data Lake Storage Gen1.

-```

-15/09/17 21:41:15 INFO web.CaboWebHdfsFileSystem: Replacing original urlConnectionFactory with org.apache.hadoop.hdfs.web.URLConnectionFactory@21a728d6

-Found 1 items

-```

-You can also use the `hdfs dfs -put` command to upload some files to Data Lake Storage Gen1, and then use `hdfs dfs -ls` to verify whether the files were successfully uploaded.

-## Next steps

-* [Copy data from Azure Storage Blobs to Data Lake Storage Gen1](data-lake-store-copy-data-wasb-distcp.md)

-* [Use Data Lake Storage Gen1 with Azure HDInsight clusters](../hdinsight/hdinsight-hadoop-use-data-lake-storage-gen1.md)

-

-description: Learn how to access and manage your Azure Data Lake Storage Gen1 data and resources in Azure Storage Explorer

-# Manage Data Lake Storage Gen1 resources by using Storage Explorer

-[Azure Data Lake Storage Gen1](./data-lake-store-overview.md) is a service for storing large amounts of unstructured data, such as text or binary data. You can get access to the data from anywhere via HTTP or HTTPS. Data Lake Storage Gen1 in Azure Storage Explorer enables you to access and manage Data Lake Storage Gen1 data and resources, along with other Azure entities like blobs and queues. Now you can use the same tool to manage your different Azure entities in one place.

-Another advantage is that you don't need to have subscription permission to manage Data Lake Storage Gen1 data. In Storage Explorer, you can attach the Data Lake Storage Gen1 path to the **Local & Attached** node as long as someone grants the permission.

-## Prerequisites

-To complete the steps in this article, you need the following prerequisites:

-* An Azure subscription. See [Get Azure free trial](https://azure.microsoft.com/pricing/free-trial).

-* A Data Lake Storage Gen1 account. For instructions on how to create one, see [Get started with Azure Data Lake Storage Gen1](./data-lake-store-get-started-portal.md).

-## Install Storage Explorer

-Install the latest Azure Storage Explorer bits from the [product webpage](https://azure.microsoft.com/features/storage-explorer/). The installation supports Windows, Linux, and Mac versions.

-## Connect to an Azure subscription

-1. In Storage Explorer, select the plug-in icon.

- 

- This opens the **Connect to Azure Storage** dialog box.

-1. On the **Select Resource** page, select **Subscription**.

-1. On the **Select Azure Environment** page, select the Azure environment to sign in to, and then select **Next**.

-1. In the **Sign in** dialog box, enter your Azure credentials, and then select **Next**.

-1. In Storage Explorer, in the **ACCOUNT MANAGEMENT** pane, select the subscription that contains the Data Lake Storage Gen1 account that you want to manage, and then select **Open Explorer**.

-1. In the **EXPLORER** pane, expand your subscription. The pane updates and displays the accounts in the selected subscription. This includes any Data Lake Storage Gen1 accounts, for example:

- 

-## Connect to Data Lake Storage Gen1

-You can access resources that don't exist in your subscription if someone gives you the URI for the resources. You can then connect to Data Lake Storage Gen1 by using the URI after you sign in.

-1. Open Storage Explorer.

-1. Expand **Local & Attached**.

-1. Right-click **Data Lake Storage Gen1 (Preview)**, and then select **Connect to Data Lake Storage Gen1**.

-1. Enter the URI, for example:

- 

- The tool browses to the location of the URL that you just entered.

- 

-## View the contents of a Data Lake Storage Gen1 account

-A Data Lake Storage Gen1 account's resources contain folders and files. The following steps show how to view the contents of a Data Lake Storage Gen1 account within Storage Explorer.

-1. Open Storage Explorer.

-1. Expand the subscription that contains the Data Lake Storage Gen1 account that you want to view.

-1. Expand **Data Lake Storage Gen1 (Preview)**.

-1. Select the Data Lake Storage Gen1 account that you want to view.

- The main pane displays the contents of the Data Lake Storage Gen1 account.

- 

-## Manage resources in Data Lake Storage Gen1

-You can manage Data Lake Storage Gen1 resources by doing following operations:

-* Browse through Data Lake Storage Gen1 resources across multiple Data Lake Storage Gen1 accounts.

-* Use a connection string to connect to and manage Data Lake Storage Gen1 directly.

-* View Data Lake Storage Gen1 resources shared by others through an ACL under **Local & Attached**.

-* Perform file and folder CRUD operations: support recursive folders and multi-selected files.

-* Drag, drop, and add a folder to quickly access recent locations. This operation mirrors the desktop File Explorer experience.

-* Copy and open a Data Lake Storage Gen1 hyperlink in Storage Explorer with one click.

-* Display the **Activities** log in the lower pane to view activity status.

-* Display folder statistics and file properties.

-## Manage resources in Azure Storage Explorer

-After you create a Data Lake Storage Gen1 account, you can:

-* Upload folders and files, download folders and files, and open resources on your local computer.

-* Pin to **Quick Access**, create a new folder, copy a URL, and select all.

-* Copy and paste, rename, delete, get folder statistics, and refresh.

-The following items show how to manage resources in a Data Lake Storage Gen1 account. Follow the steps for the task that you want to do.

-### Upload files

-1. On the main pane's toolbar, select **Upload**, and then select **Upload Files**.

-1. In the **Select files to upload** dialog box, select the files that you want to upload.

-1. Select **Open** to begin the upload.

-> [!NOTE]

-> You can also directly drag the files on a local computer to start uploading.

-### Upload a folder

-1. On the main pane's toolbar, select **Upload**, and then select **Upload Folder**.

-1. In the **Select folder to upload** dialog box, select the folder that you want to upload.

-1. Select **Select Folder** to begin the upload.

-> [!NOTE]

-> You can also directly drag a folder on a local computer to start uploading.

-### Download folders or files to your local computer

-1. Select the folders or files that you want to download.

-1. On the main pane's toolbar, select **Download**.

-1. In the **Select a folder to save the downloaded files into** dialog box, specify the location and the name.

-1. Select **Save**.

-### Open a folder or file from your local computer

-1. Select the folder or file that you want to open.

-1. On the main pane's toolbar, select **Open**. Or, right-click the selected folder or file, and then select **Open** on the shortcut menu.

-The file is downloaded and opened through the application that's associated with the underlying file type. Or, the folder is opened in the main pane.

-### Copy folders or files to the clipboard

-You can copy Data Lake Storage Gen1 folders or files and paste them in another Data Lake Storage Gen1 account. Copy and paste operations across storage types aren't supported. For example, you can't copy Data Lake Storage Gen1 folders or files and paste them to Azure Blob storage or the other way around.

-1. Select the folders or files that you want to copy.

-1. On the main pane's toolbar, select **Copy**. Or, right-click the selected folders or files, and then select **Copy** on the shortcut menu.

-1. In the navigation pane, browse to another Data Lake Storage Gen1 account, and select it to view it in the main pane.

-1. On the main pane's toolbar, select **Paste** to create a copy. Or, select **Paste** on the destination's shortcut menu.

-> [!NOTE]

-> The copy/paste operation works by downloading the folders or files to the local computer and then uploading them to the destination. The tool doesn't perform the action in the back end. The copy/paste operation on large files is slow.

-### Delete folders or files

-1. Select the folders or files that you want to delete.

-1. On the main pane's toolbar, select **Delete**. Or, right-click the selected folders or files, and then select **Delete** on the shortcut menu.

-1. Select **Yes** in the confirmation dialog box.

-### Pin to Quick Access

-1. Select the folder that you want to pin so that you can easily access the resources.

-1. On the main pane's toolbar, select **Pin to Quick access**.

- In the navigation pane, the selected folder is added to the **Quick Access** node.

- 

-### Use deep links

-If you have a URL, you can enter the URL into the address path in File Explorer or a browser. Then Storage Explorer opens automatically and goes to the location of the URL.

-

-## Next steps

-* View the [latest Storage Explorer release notes and videos](https://www.storageexplorer.com).

-* [Get started with Storage Explorer](../vs-azure-tools-storage-manage-with-storage-explorer.md)

-* [Get started with Azure Data Lake Storage Gen1](./data-lake-store-overview.md)

-description: Understand how you can integrate Azure Data Lake Storage Gen1 with other Azure services.

-# Integrating Azure Data Lake Storage Gen1 with other Azure services

-Azure Data Lake Storage Gen1 can be used in conjunction with other Azure services to enable a wider range of scenarios. The following article lists the services that Data Lake Storage Gen1 can be integrated with.

-## Use Data Lake Storage Gen1 with Azure HDInsight

-You can provision an [Azure HDInsight](https://azure.microsoft.com/documentation/learning-paths/hdinsight-self-guided-hadoop-training/) cluster that uses Data Lake Storage Gen1 as the HDFS-compliant storage. For this release, for Hadoop and Storm clusters on Windows and Linux, you can use Data Lake Storage Gen1 only as an additional storage. Such clusters still use Azure Storage (WASB) as the default storage. However, for HBase clusters on Windows and Linux, you can use Data Lake Storage Gen1 as the default storage, or additional storage, or both.

-For instructions on how to provision an HDInsight cluster with Data Lake Storage Gen1, see:

-* [Provision an HDInsight cluster with Data Lake Storage Gen1 using Azure portal](data-lake-store-hdinsight-hadoop-use-portal.md)

-* [Provision an HDInsight cluster with Data Lake Storage Gen1 as default storage using Azure PowerShell](data-lake-store-hdinsight-hadoop-use-powershell-for-default-storage.md)

-* [Provision an HDInsight cluster with Data Lake Storage Gen1 as additional storage using Azure PowerShell](data-lake-store-hdinsight-hadoop-use-powershell.md)

-## Use Data Lake Storage Gen1 with Azure Data Lake Analytics

-[Azure Data Lake Analytics](../data-lake-analytics/data-lake-analytics-overview.md) enables you to work with Big Data at cloud scale. It dynamically provisions resources and lets you do analytics on terabytes or even exabytes of data that can be stored in a number of supported data sources, one of them being Data Lake Storage Gen1. Data Lake Analytics is specially optimized to work with Data Lake Storage Gen1 - providing the highest level of performance, throughput, and parallelization for you big data workloads.

-For instructions on how to use Data Lake Analytics with Data Lake Storage Gen1, see [Get Started with Data Lake Analytics using Data Lake Storage Gen1](../data-lake-analytics/data-lake-analytics-get-started-portal.md).

-## Use Data Lake Storage Gen1 with Azure Data Factory

-You can use [Azure Data Factory](https://azure.microsoft.com/services/data-factory/) to ingest data from Azure tables, Azure SQL Database, Azure SQL DataWarehouse, Azure Storage Blobs, and on-premises databases. Being a first class citizen in the Azure ecosystem, Azure Data Factory can be used to orchestrate the ingestion of data from these source to Data Lake Storage Gen1.

-For instructions on how to use Azure Data Factory with Data Lake Storage Gen1, see [Move data to and from Data Lake Storage Gen1 using Data Factory](../data-factory/connector-azure-data-lake-store.md).

-## Copy data from Azure Storage Blobs into Data Lake Storage Gen1

-Azure Data Lake Storage Gen1 provides a command-line tool, AdlCopy, that enables you to copy data from Azure Blob Storage into a Data Lake Storage Gen1 account. For more information, see [Copy data from Azure Storage Blobs to Data Lake Storage Gen1](data-lake-store-copy-data-azure-storage-blob.md).

-## Copy data between Azure SQL Database and Data Lake Storage Gen1

-You can use Apache Sqoop to import and export data between Azure SQL Database and Data Lake Storage Gen1. For more information, see [Copy data between Data Lake Storage Gen1 and Azure SQL Database using Sqoop](data-lake-store-data-transfer-sql-sqoop.md).

-## Use Data Lake Storage Gen1 with Stream Analytics

-You can use Data Lake Storage Gen1 as one of the outputs to store data streamed using Azure Stream Analytics. For more information, see [Stream data from Azure Storage Blob into Data Lake Storage Gen1 using Azure Stream Analytics](data-lake-store-stream-analytics.md).

-## Use Data Lake Storage Gen1 with Power BI

-You can use Power BI to import data from a Data Lake Storage Gen1 account to analyze and visualize the data. For more information, see [Analyze data in Data Lake Storage Gen1 using Power BI](data-lake-store-power-bi.md).

-## Use Data Lake Storage Gen1 with Data Catalog

-You can register data from Data Lake Storage Gen1 into the Azure Data Catalog to make the data discoverable throughout the organization. For more information see [Register data from Data Lake Storage Gen1 in Azure Data Catalog](data-lake-store-with-data-catalog.md).

-## Use Data Lake Storage Gen1 with SQL Server Integration Services (SSIS)

-You can use the Data Lake Storage Gen1 connection manager in SSIS to connect an SSIS package with Data Lake Storage Gen1. For more information, see [Use Data Lake Storage Gen1 with SSIS](/sql/integration-services/connection-manager/azure-data-lake-store-connection-manager).

-## Use Data Lake Storage Gen1 with Azure Event Hubs

-You can use Azure Data Lake Storage Gen1 to archive and capture data received by Azure Event Hubs. For more information see [Use Data Lake Storage Gen1 with Azure Event Hubs](data-lake-store-archive-eventhub-capture.md).

-## See also

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-* [Get Started with Data Lake Storage Gen1 using Portal](data-lake-store-get-started-portal.md)

-* [Get started with Data Lake Storage Gen1 using PowerShell](data-lake-store-get-started-powershell.md)

-description: Learn what to consider as you plan and complete a migration to Azure Data Lake Storage Gen1 as it becomes available in new regions.

-# Migrate Azure Data Lake Storage Gen1 across regions

-As Azure Data Lake Storage Gen1 becomes available in new regions, you might choose to do a one-time migration, to take advantage of the new region. Learn what to consider as you plan and complete the migration.

-## Prerequisites

-* **An Azure subscription**. For more information, see [Create your free Azure account today](https://azure.microsoft.com/pricing/free-trial/).

-* **A Data Lake Storage Gen1 account in two different regions**. For more information, see [Get started with Azure Data Lake Storage Gen1](data-lake-store-get-started-portal.md).

-* **Azure Data Factory**. For more information, see [Introduction to Azure Data Factory](../data-factory/introduction.md).

-## Migration considerations

-First, identify the migration strategy that works best for your application that writes, reads, or processes data in Data Lake Storage Gen1. When you choose a strategy, consider your application's availability requirements, and the downtime that occurs during a migration. For example, your simplest approach might be to use the "lift-and-shift" cloud migration model. In this approach, you pause the application in your existing region while all your data is copied to the new region. When the copy process is finished, you resume your application in the new region, and then delete the old Data Lake Storage Gen1 account. Downtime during the migration is required.

-To reduce downtime, you might immediately start ingesting new data in the new region. When you have the minimum data needed, run your application in the new region. In the background, continue to copy older data from the existing Data Lake Storage Gen1 account to the new Data Lake Storage Gen1 account in the new region. By using this approach, you can make the switch to the new region with little downtime. When all the older data has been copied, delete the old Data Lake Storage Gen1 account.

-Other important details to consider when planning your migration are:

-* **Data volume**. The volume of data (in gigabytes, the number of files and folders, and so on) affects the time and resources you need for the migration.

-* **Data Lake Storage Gen1 account name**. The new account name in the new region must be globally unique. For example, the name of your old Data Lake Storage Gen1 account in East US 2 might be contosoeastus2.azuredatalakestore.net. You might name your new Data Lake Storage Gen1 account in North EU contosonortheu.azuredatalakestore.net.

-* **Tools**. We recommend that you use the [Azure Data Factory Copy Activity](../data-factory/connector-azure-data-lake-store.md) to copy Data Lake Storage Gen1 files. Data Factory supports data movement with high performance and reliability. Keep in mind that Data Factory copies only the folder hierarchy and content of the files. You need to manually apply any access control lists (ACLs) that you use in the old account to the new account. For more information, including performance targets for best-case scenarios, see the [Copy Activity performance and tuning guide](../data-factory/copy-activity-performance.md). If you want data copied more quickly, you might need to use additional Cloud Data Movement Units. Some other tools, like AdlCopy, don't support copying data between regions.

-* **Bandwidth charges**. [Bandwidth charges](https://azure.microsoft.com/pricing/details/bandwidth/) apply because data is transferred out of an Azure region.

-* **ACLs on your data**. Secure your data in the new region by applying ACLs to files and folders. For more information, see [Securing data stored in Azure Data Lake Storage Gen1](data-lake-store-secure-data.md). We recommend that you use the migration to update and adjust your ACLs. You might want to use settings similar to your current settings. You can view the ACLs that are applied to any file by using the Azure portal, [PowerShell cmdlets](/powershell/module/az.datalakestore/get-azdatalakestoreitempermission), or SDKs.

-* **Location of analytics services**. For best performance, your analytics services, like Azure Data Lake Analytics or Azure HDInsight, should be in the same region as your data.

-## Next steps

-* [Overview of Azure Data Lake Storage Gen1](data-lake-store-overview.md)

-description: Understand how virtual network integration works in Azure Data Lake Storage Gen1

-# Virtual network integration for Azure Data Lake Storage Gen1

-This article introduces virtual network integration for Azure Data Lake Storage Gen1. With virtual network integration, you can configure your accounts to accept traffic only from specific virtual networks and subnets.

-This feature helps to secure your Data Lake Storage account from external threats.

-Virtual network integration for Data Lake Storage Gen1 makes use of the virtual network service endpoint security between your virtual network and Microsoft Entra ID to generate additional security claims in the access token. These claims are then used to authenticate your virtual network to your Data Lake Storage Gen1 account and allow access.

-> [!NOTE]

-> There's no additional charge associated with using these capabilities. Your account is billed at the standard rate for Data Lake Storage Gen1. For more information, see [pricing](https://azure.microsoft.com/pricing/details/data-lake-store/?cdn=disable). For all other Azure services that you use, see [pricing](https://azure.microsoft.com/pricing/#product-picker).

-## Scenarios for virtual network integration for Data Lake Storage Gen1

-With Data Lake Storage Gen1 virtual network integration, you can restrict access to your Data Lake Storage Gen1 account from specific virtual networks and subnets. After your account is locked to the specified virtual network subnet, other virtual networks/VMs in Azure aren't allowed access. Functionally, Data Lake Storage Gen1 virtual network integration enables the same scenario as [virtual network service endpoints](../virtual-network/virtual-network-service-endpoints-overview.md). A few key differences are detailed in the following sections.

-

-> [!NOTE]

-> The existing IP firewall rules can be used in addition to virtual network rules to allow access from on-premises networks too.

-## Optimal routing with Data Lake Storage Gen1 virtual network integration

-A key benefit of virtual network service endpoints is [optimal routing](../virtual-network/virtual-network-service-endpoints-overview.md#key-benefits) from your virtual network. You can perform the same route optimization to Data Lake Storage Gen1 accounts. Use the following [user-defined routes](../virtual-network/virtual-networks-udr-overview.md#user-defined) from your virtual network to your Data Lake Storage Gen1 account.

-**Data Lake Storage public IP address** ΓÇô Use the public IP address for your target Data Lake Storage Gen1 accounts. To identify the IP addresses for your Data Lake Storage Gen1 account, [resolve the DNS names](./data-lake-store-connectivity-from-vnets.md#enabling-connectivity-to-azure-data-lake-storage-gen1-from-vms-with-restricted-connectivity) of your accounts. Create a separate entry for each address.

-```azurecli

-# Create a route table for your resource group.

-az network route-table create --resource-group $RgName --name $RouteTableName

-# Create route table rules for Data Lake Storage public IP addresses.

-# There's one rule per Data Lake Storage public IP address.

-az network route-table route create --name toADLSregion1 --resource-group $RgName --route-table-name $RouteTableName --address-prefix <ADLS Public IP Address> --next-hop-type Internet

-# Update the virtual network, and apply the newly created route table to it.

-az network vnet subnet update --vnet-name $VnetName --name $SubnetName --resource-group $RgName --route-table $RouteTableName

-```

-## Data exfiltration from the customer virtual network

-In addition to securing the Data Lake Storage accounts for access from the virtual network, you also might be interested in making sure there's no exfiltration to an unauthorized account.

-Use a firewall solution in your virtual network to filter the outbound traffic based on the destination account URL. Allow access to only approved Data Lake Storage Gen1 accounts.

-Some available options are:

-> [!NOTE]

-> Using firewalls in the data path introduces an additional hop in the data path. It might affect the network performance for end-to-end data exchange. Throughput availability and connection latency might be affected.

-## Limitations

-

-

-## Configuration

-<a name='step-1-configure-your-virtual-network-to-use-an-azure-ad-service-endpoint'></a>

-### Step 1: Configure your virtual network to use a Microsoft Entra service endpoint

-1. Go to the Azure portal, and sign in to your account.

-

-2. [Create a new virtual network](../virtual-network/quick-create-portal.md)in your subscription. Or you can go to an existing virtual network. The virtual network must be in the same region as the Data Lake Storage Gen 1 account.

-

-3. On the **Virtual network** blade, select **Service endpoints**.

-

-4. Select **Add** to add a new service endpoint.