Browsing:

Tag: Azure

Artificial Intelligence prepping data – Back to basic part 1

What street legal cars taught me about Machine learning? It’s all about the right data being available and the RDW data can’t be trusted!

I impulsively signed up for an Artificial Intelligence certification track 2 months ago, So I’ve been experimenting with Artificial Intelligence for a while now and in the beginning of the course it was one though cookie! Those formula’s to interpreted the data predictability really freaked me out!

But once I got past the formula’s and I saw the resemblance of the workspace with Microsoft products like SSIS and BI I see endless possibilities. This takes the data to a whole new level.

Preparing the data:

I did a test on all cars that are currently on the road in the Netherlands and combined it with performance data. I wanted to find the fastest street legal car. I guess I just wanted to find out what kind of cars I should fancy these days according to the performance stats.

I used an open data set from the dutch RWD (Driver and Vehicle Standards Agency). It contained 14m rows and it’s 7GB in size. So I had to prepare the data in order to keep the experiment basic and performance high. I imported it into my SQL server and I filtered out the the stationwagons, campers, scooters and trailers, So I was left with a 900000  rows data set.

I use a SQL Server 2017 and the Microsoft Azure machine learning studio to create a new experiment.

In order to make a prediction I needed to combine the brand data with the engine displacement data, because horsepower data was not available, to see which models are high performance based on the engine capacity. So sadly the smaller engines which are supercharged are not correctly represented in the prediction.

The calculation based on above rules, took a local SQL server on an i5 laptop about 15 minutes. I needed more data preparation.

Based on engine displacement, a top 3 came up. But I didn’t like the results at all. Sure, the engine displacement was high, but the cars are heavy and their performance isn’t the best. Super charged Turbo’s and gearing make all the difference, but aren’t properly represented in this data result.

I had to filter out a lot of data, next up I added the weight of the car, but it wasn’t trust worthy either. I found a data set which contained the Kw of the cars and top speed and joined the data with my current results and added a calculation in SQL on the Kw row * 1,362 to calculate the Hp of the car. The Hp outcome looks pretty accurate.  After 4 hours of combining data and filtering the queries I gave up. Based on this data there is no way you can truly point out the fastest cars. I had to change my plans. Too many uncertain variables to make a decent prediction and not even close to the start of an IA project 🙁

Lot’s of NULL data
This No. 1 car can’t be trusted!

After more data crunching, The results are still not really worth to display. So here is a TOP 21 of “fastest” cars…based on…well the obvious HP and Weight sorting:

btw, did you know there is only one Koenigsegg on the dutch roads.

Ok, I got a little bit carried away with data prepping.

Now let’s import it into an IA experiment: First you need to create a resource in the Azure Portal for your workspace. I won’t get into details, we did this before!

Verify that you created the following new resources: A Machine Learning Workspace, A Machine Learning Plan and A Storage Account.

Browse to the Machine learning workspace you created and launch Machine Learning Studio. This opens a new browser page.

Go to experiments and down in the left  corner click NEW.

Rename the experiment and add a dataset. Upload a new dataset. Datasets –> NEW–> Select data to upload. Now that you have the dataset ready, you can drag it into your experiment and start running tests and variables on the data.

In my next post we will dive deeper into Artificial Intelligence

 

 

 


PaaS taking over the world! Are dba’s a dying breed?

PaaS vs IaaS

 

It’s easter weekend, or as the dutch say ‘PaaS weekend’. So, it might be a good idea to talk a bit about PaaS. What is it and what does this mean for you as a Database administrator? Are Dba’s a dying breed. Will they just shift over to more complex or broader tasks or are they here to stay. It all depends on the company and the software they are running.

 

 

 

What is #Iaas? in short, it’s a VM. Your database is running in a data center. As a Dba you have the same job requirements as when running a database on-premise. Update, backup, patches, tuning, security and account control.

But #PaaS is a different story, it runs the database as a service, so there is no need for a dba. At least that’s what they tell you. But does PaaS solve all your performance and tuning needs, is that faulty query when moved to PaaS suddely solved? Nope. Machine learning is doing a great job so far, but it isn’t the magical quick fix, yet.

In all honesty most companies don’t care about tuning a database, not all applications have complex queries and tasks running on their SQL server, most are fine running an express edition. They don’t even bother having a Dba. The database is taken care of by a system engineer, if being looked at, at all.

Where does this put you as a Dba? Don’t sob, we still need you! It’s a big relief that the market, which is still flooded with old school high maintenance MS SQL driven apps. How nice if these could be taken care of with PaaS. The only good thing coming out of these high maintenance, splintered databases, is the data itself. Do you really want to spend your time updating, patching and granting rights to users and saying no to SA account requests? No, you don’t!

But second, in the real world, companies don’t evolve as fast as the IT world itself. Most applications, lots outdated or not are not being replaced overnight and not all the vendors are quite ready for PaaS environments with their applications. So you just have to decide on which side you want to be, fast IT or slow IT. There is still a big playground available for both for the coming years.


Polybase configuration on SQL Server 2017 Part II

Image: Microsoft

Nowadays your precious data can be stored everywhere, not just on several servers with different SQL versions, your data probably is wide spread in the cloud. It’s also a good idea to store data in the cloud with stretch database to release your local discs from excessive data and still be able to query it, but also use it in your SSIS and BI environment and keep an acceptable ETL. With Microsoft’s polybase you can access, import and export any data structured, semi, or non structured on the Hadoop platform and azure blob storage using T- SQL language.

The best business knowledge comes from the data you collect. So it might be a good idea to put the data you collect into some good use. Businesses collect lot’s of data, but in most cases this is also where it ends. Those who read my posts before,  know I am all about combining various sources with linked servers, since SQL 2014 lot’s of new features are available for using all your data on business intelligence platforms.

In my last post, we had a first look and troubleshoot of a polybase installation. This time we are going to configure and use the polybase in SQL server 2017. I’m going to use the Blob storage on Azure to demonstrate how you can implement this solution in your (local) SQL database.

First things first, now you’ve got your polybase installation ready, check if the services for polybase exist and are running.

Services: ‘SQL Server PolyBase Data Movement’ and ‘SQL Server PolyBase Engine’

You need to configure Polybase in order to start using it. Fire up SSMS and open a new query window. Type

sp_configure ' hadoop connectivity', 4;

reconfigure

Option 4 is Azure blob storage (WASB[S]). For more info on availability of the Polybase connectivity configuration, take a look here. Run the query and make sure you restart both Polybase services on the machine to finish the configuration.

In order to start using the blob storage make sure you have an Azure storage account if you don’t have an Azure account yet, create one here.

Login to Azure and on the left side select and create a new storage account

Give it some time, once the storage is created, you also need to create a container on the Azure storage.

To connect your local db to azure  storage, you need to get the azure storage key from your Azure storage account you just created and put it in the configuration file of your SQL installation.
Look for the core-site.xml file in the installation path of SQL Server.
The path looks simular to this: “C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf”
This will open the config directory  with the core-site.xml file.

Open the file in notepad and add code before the block of code mentioning Kerebos.

Fill in the storage name, in my case polybasedemo  and the storagekey and save the file.

 fs.azure.account.key..blob.core.windows.net

Now we have to create an external data source in SSMS. Replace containername@storagename with the names you created on Azure.

CREATE EXTERNAL DATA SOURCE PolyBaseDemo WITH
( TYPE = HADOOP, -- wasbs:// containername@storagename.blob.core.windows.net/ 
LOCATION = 'wasbs://containername@storagename.blob.core.windows.net/' );

Next up, we create the external file format to define external data on Azure blob storage, this needs to be done in order to create the external table

CREATE EXTERNAL FILE FORMAT PolybaseFormat 
WITH ( FORMAT_TYPE = DELIMITEDTEXT , FORMAT_OPTIONS ( FIELD_TERMINATOR = ',' ) );

This creates 2 new server objects in SSMS and now all is left to create the external table itself. In this demo I use an excel sheet with some irrelevant data to have some test data available.

USE [DemoPolybase] CREATE EXTERNAL TABLE [dbo].[Customers] ( [Name] VARCHAR(255) NULL, [adres] VARCHAR(255) NULL, [postalcode] VARCHAR(6) NULL ) WITH (LOCATION = N’/Customer_Export.csv’, DATA_SOURCE = PolyBaseDemo, FILE_FORMAT = PolybaseFormat, REJECT_TYPE = Value, REJECT_VALUE = 10) GO

And to see if this worked, just query the data 🙂

SELECT * FROM [Customers]

Now put this knowledge into action yourself with some real data!


SQL2016 Stretch database. Yes, it’s another linked server

Your manager always wants to keep all the data…now with bigdata being a thing and algoritmes are used more efficient (IOT), people want to hoard data even more, but it needs to be online to be valuable. If you have to restore the data first, your data is not very valuable. But all this extra data also slows down your database performance…and gives the dba’ers extra challenges….but not anymore.! Well, at least when your boss got a big wallet and don’t mind spending it on a SQL buget.

In SQL2016 Microsoft introduced the stretch database. which give you live access to ‘archived’ data and makes it feel like it’s on premise. And  it’s a secretly just a linked server! A linked server between mssql and Azure.LS+azure

How does this work, and is it safe?

Yes! since it works on secure linked server technology and it stores an encryption key on the on-premises sqlserver, it’s suitable for all your cold data, not hot, because it does slow down the query speed, but it will not clogg your network and your data will be available at all times, even when the data is being stretched over to Azure the data still is fully query-able.

First ask yourself, Do you really need to keep this data? If yes, How do you create a stretchdatabase? Well, this is easy!

Hesitations? use this link for info on the MS Stretch database advisor.

I use a simple test database, the adventureworks2016 ctp3.

Before we can start, check if the data archive function is enabled, if not (0) enable it.

####Check status of Remote data archive####
SELECT @@VERSION AS 'SQL Server Version'; 
EXEC sp_configure 'remote data archive'; 
GO 
####Activate Remote data archive####
EXEC sp_configure 'remote data archive' , '1'; 
GO 
RECONFIGURE; 
GO

Next we can create a master key to encrypt all your data locally, this way, the data can’t be read in Azure.
Note: You can also create a masterkey from the stretchdb wizard, it’s up to you. I’ll show both options.

USE AdventureWorks2016CTP3; 
GO 

CREATE MASTER KEY ENCRYPTION BY PASSWORD='stretchdb2016'; 
GO

Next I will create a simple table  and populate it with some data, to keep the demo quick and simple :

CREATE TABLE dbo.Stretchtable
(
FirstName VARCHAR (50),
LastName VARCHAR (50)
);
GO


Use [AdventureWorks2016CTP3]
GO
INSERT INTO dbo.Stretchtable (FirstName, lastName)
VALUES ('Reed', 'Richards'), ('Benjamin', 'Grimm'), ('Sue', 'Storm'), 
 ('Johnny', 'Storm'), ('Victor', 'Von Doom'), ('Willie','Lumpkin');
 GO

Keep your Azure account ready! I use the SQL logical server resource, you need to create a resource before you can stretch the database.

Azure resource

 

Select the table you will be stretching to Azure. Now enable the table for stretching and the wizard will open up.

Enable stretch database

It’s a Hybride archive – You can stretch the whole table or filter out the (c)old data from a massive table.
4_EnableStretchTable

 

Next thing you create a master key, if you didn’t make it before, if you made it already, It will ask you to fill in the master password.Masterkey_StretchTable
10_LoginStretchTable

Fill in the IP range for a firewall rule.IP_Azure

12_EnableStretchTable

Hey Ho, let’s go, that was easy!

Now the table will start to stretch, if you want to see if it stretched yet, try a select query and check the executionplan.
14_EnableStretchTable

Want your data back? No problem, You can bring back the data to your on-premise server, if you don’t want to use Azure. Because, when you delete your azure account, the data will be gone too!

bringbackmydatabringbackmydata2bringbackmydata3

 

Note: Data storage can be expensive on Azure, but this function gives us a positive view on the ever growing databases and it’s possibilities.


Linked server to Microsoft Azure #Error 405150

When Azure was launched, we missed one important feature, the linked server, It seemed impossible to do cross database queries from your local database to the Azure database, but Microsoft listened and made it possible to do database queries that spans multiple databases across platforms.

There is plenty of hits when googled on the errors, but none of these are complete! I will write down the  process of linking your server to Azure.

when I first created a linked server to Azure, I got this error  “reference to database and/or server name is not supported in this version error: 405150”, When trying to add a linked server from SSMS, you are not given option to set a default database. So I used Tsql instead, because without a specific catalog, your linked server will not work.LS_Azure

Azure doesn’t allow you to alter the master db, so you have to connect directly on the database you are going to use.

What people never tell LS newbies and make you google this error “Msg 2812, Level 16, State 62, Line 1 Could not find stored procedure ‘sp_addlinkedserver’ ” and read tons of blogs and tutorials all providing different queries, but still no work, follow these steps”.

The Tutorial, the Solution:

Open up ODBC, or simply run odbcad32.exe. and start by making a new system DNS with the Native client.

ls_SYSDNS

In the next step, you provide your datasource name and the complete server name as you can find on the Azure server.

LS_makedns

Also provide the database you want to connect to, don’t skip this one!

LS_Choosedb

 

Test the connection, see it succeed! if not, go through the above printscreens you probably missed a step.

LS_Testsuccesf

Now we can create the linked server on your database, by creating a query on your SSMS and can run your linked server query, it should look similar to this:

EXEC master.dbo.sp_addlinkedserver @server = N’Azure_server’, @srvproduct=N”, @provider=N’sqlncli’, @datasrc=N’t20ko02v18.database.windows.net’, @catalog=N’Teletraan IV’ EXEC master.dbo.sp_addlinkedsrvlogin @rmtsrvname=N’Azure_server’, @useself=N’False’,@locallogin=NULL,@rmtuser=N’skyscream’,@rmtpassword=’password’

To test if the LS is added, you could run exec sp_helpserveror just test the connection.LS_SPhelpserver

These are all the steps to a Linked Azure server. Congratulations, you are now connected to the Azure mothership of data.

Data manipulation

Next step is to insert data from the Azure linked server into our local database. Btw, this query works both ways, you could also insert data into the Azure database.

Create an INSERT statement

BEGIN TRAN INSERT INTO [Azure_demo].[dbo].[Demo] (Initials,Prefix,LastName,Gender,Street,HouseNumber ,Zipcode,City,PhoneNumber,MobileNumber,EmailAddress) SELECT Initials,Prefix,LastName,Gender,Street,HouseNumber ,Zipcode,City,PhoneNumber,MobileNumber,EmailAddress FROM [Azure_server].[Teletraan IV].[dbo].[AzureData] WHERE InsertDate BETWEEN ‘2008-08-22 00:00:00.000’ AND ‘2009-02-06 00:00:00.000’ ROLLBACK TRAN –COMMIT TRAN

Ls_Qresult

Now all rows have been added through our linked server into our local database. LS_Check
If this blog saved your ass or just made your day, please let me know 🙂