Polybase configuration on SQL Server 2017 Part II

Image: Microsoft

You know I am all about linked servers. Nowadays your precious data can be stored everywhere, not just on several servers with different SQL versions, your data probably is wide spread in the cloud. It’s also a good idea to store data in the cloud with stretch database to release your local discs from excessive data and still be able to query it, but also use it in your SSIS and BI environment and keep an acceptable ETL. With Microsoft’s polybase you can access, import and export any data structured, semi, or non structured on the Hadoop platform and azure blob storage using T- SQL language.

In my last post, we had a first look and troubleshoot of a polybase installation. This time we are going to configure and use the polybase in SQL server 2017. I’m going to use the Blob storage on Azure to demonstrate how you can implement this solution in your (local) SQL database.

First things first, now you’ve got your polybase installation ready, check if the services for polybase exist and are running.

Services: ‘SQL Server PolyBase Data Movement’ and ‘SQL Server PolyBase Engine’

You need to configure Polybase in order to start using it. Fire up SSMS and open a new query window. Type

Option 4 is Azure blob storage (WASB[S]). For more info on availability of the Polybase connectivity configuration, take a look here. Run the query and make sure you restart both Polybase services on the machine to finish the configuration.


In order to start using the blob storage make sure you have an Azure storage account if you don’t have an Azure account yet, create one here.

Login to Azure and on the left side select and create a new storage account

Give it some time, once the storage is created, you also need to create a container on the Azure storage.

To connect your local db to azure  storage, you need to get the azure storage key from your Azure storage account you just created and put it in the configuration file of your SQL installation.
Look for the core-site.xml file in the installation path of SQL Server.
The path looks simular to this: “C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf”
This will open the config directory  with the core-site.xml file.

Open the file in notepad and add code before the block of code mentioning Kerebos.

Fill in the storage name, in my case polybasedemo  and the storagekey and save the file.

Now we have to create an external data source in SSMS. Replace containername@storagename with the names you created on Azure.

Next up, we create the external file format to define external data on Azure blob storage, this needs to be done in order to create the external table

This creates 2 new server objects in SSMS and now all is left to create the external table itself. In this demo I use an excel sheet with some irrelevant data to have some test data available.

USE [DemoPolybase] CREATE EXTERNAL TABLE [dbo].[Customers] ( [Name] VARCHAR(255) NULL, [adres] VARCHAR(255) NULL, [postalcode] VARCHAR(6) NULL ) WITH (LOCATION = N'/Customer_Export.csv', DATA_SOURCE = PolyBaseDemo, FILE_FORMAT = PolybaseFormat, REJECT_TYPE = Value, REJECT_VALUE = 10) GO

And to see if this worked, just query the data 🙂

SELECT * FROM [Customers]

Now put this knowledge into action yourself with some real data!



Setting up Ubuntu 17.10 for .NET Core Development (including SQL Server, Visual Studio Code, PowerShell and SQL Operations Studio)

In this post I will show you how to set up an Ubuntu desktop that you can use for .NET Core Development.


Maybe because you want to do Microsoft development on older hardware or because you think Windows is too bloated.

Install the OS

In this example I use the latest Ubuntu version. Mind you, in April another LTS version will be out but I can’t wait and will probably update this article.

Update and essentials

Once installed, open up your terminal and enter:

This the essentials you will want, build tools and zsh.

Oh My Zsh

I prefer the zsh and Oh My Zsh over Bash because of its auto complete features and eye candy.

And the Powerline fonts for a lovely prompt:

Set the default shell to zsh:

Now log out and log back in.
You can then set your favorite theme by editing ~/.zshrc.


Next we will install Node.js and fix npm so we can run it without sudo:

Trust the Microsoft sources

Add the repo’s for the .Net Core SDK, Powershell, Visual Studio Code, SQL Server at once

Now install the software:

Configure Sql Server

You will need to run setup to choose the correct Sql Server version and to set the SA password.

Install Sql Server Tools

Next we need to instal sqlcmd:

And we’re done

We now have an Ubuntu desktop for .NET Core development, including Sql Server. Now go ahead and do

Next, install the Entity Framework Core package, create a full fledged backend and enjoy the fact that Microsoft has gone out of its way to make this all possible!


For your convenience: here is a Gist that contains this script.