SQL2016 Stretch database. Yes, it’s another linked server

Your manager always wants to keep all the data…now with bigdata being a thing and algoritmes are used more efficient (IOT), people want to hoard data even more, but it needs to be online to be valuable. If you have to restore the data first, your data is not very valuable. But all this extra data also slows down your database performance…and gives the dba’ers extra challenges….but not anymore.! Well, at least when your boss got a big wallet and don’t mind spending it on a SQL buget.

In SQL2016 Microsoft introduced the stretch database. which give you live access to ‘archived’ data and makes it feel like it’s on premise. And  it’s a secretly just a linked server! A linked server between mssql and Azure.LS+azure

How does this work, and is it safe?

Yes! since it works on secure linked server technology and it stores an encryption key on the on-premises sqlserver, it’s suitable for all your cold data, not hot, because it does slow down the query speed, but it will not clogg your network and your data will be available at all times, even when the data is being stretched over to Azure the data still is fully query-able.

First ask yourself, Do you really need to keep this data? If yes, How do you create a stretchdatabase? Well, this is easy!

Hesitations? use this link for info on the MS Stretch database advisor.

I use a simple test database, the adventureworks2016 ctp3.

Before we can start, check if the data archive function is enabled, if not (0) enable it.

####Check status of Remote data archive####
SELECT @@VERSION AS 'SQL Server Version'; 
EXEC sp_configure 'remote data archive'; 
GO 
####Activate Remote data archive####
EXEC sp_configure 'remote data archive' , '1'; 
GO 
RECONFIGURE; 
GO

Next we can create a master key to encrypt all your data locally, this way, the data can’t be read in Azure.
Note: You can also create a masterkey from the stretchdb wizard, it’s up to you. I’ll show both options.

USE AdventureWorks2016CTP3; 
GO 

CREATE MASTER KEY ENCRYPTION BY PASSWORD='stretchdb2016'; 
GO

Next I will create a simple table  and populate it with some data, to keep the demo quick and simple :

CREATE TABLE dbo.Stretchtable
(
FirstName VARCHAR (50),
LastName VARCHAR (50)
);
GO


Use [AdventureWorks2016CTP3]
GO
INSERT INTO dbo.Stretchtable (FirstName, lastName)
VALUES ('Reed', 'Richards'), ('Benjamin', 'Grimm'), ('Sue', 'Storm'), 
 ('Johnny', 'Storm'), ('Victor', 'Von Doom'), ('Willie','Lumpkin');
 GO

Keep your Azure account ready! I use the SQL logical server resource, you need to create a resource before you can stretch the database.

Azure resource

 

Select the table you will be stretching to Azure. Now enable the table for stretching and the wizard will open up.

Enable stretch database

It’s a Hybride archive – You can stretch the whole table or filter out the (c)old data from a massive table.
4_EnableStretchTable

 

Next thing you create a master key, if you didn’t make it before, if you made it already, It will ask you to fill in the master password.Masterkey_StretchTable
10_LoginStretchTable

Fill in the IP range for a firewall rule.IP_Azure

12_EnableStretchTable

Hey Ho, let’s go, that was easy!

Now the table will start to stretch, if you want to see if it stretched yet, try a select query and check the executionplan.
14_EnableStretchTable

Want your data back? No problem, You can bring back the data to your on-premise server, if you don’t want to use Azure. Because, when you delete your azure account, the data will be gone too!

bringbackmydatabringbackmydata2bringbackmydata3

 

Note: Data storage can be expensive on Azure, but this function gives us a positive view on the ever growing databases and it’s possibilities.


Meet the Buttonfactory – September 15

  • tbf_powershellkopieThursday, September 15, 2016

    to

    Beercoo

    Weg en Bos 51, 2661 DL Bergschenhoek

    You are invited to join us at our first meetup!
    There will be 2 presentations. One is about Powershell and the other about SQL Server 2016.

    Planning:

    19:00 A drink and some food

    19:30 Powershell Objects by Jacqueline
    Powershell is an object oriented scripting language, but what does that even mean? In this presentation you’ll get to learn all about objects and how you can supercharge your scripts leveraging them. No Powershell skills required!

    20:15 What is new in SQL Server 2016? By Melanie
    SQL Server 2016 is the biggest leap forward in Microsoft data platform history!
    OK, I copied that line from the Microsoft site. But the coolest new feature is the stretched database. We are going to tell you all about it.

    21:00 Another drink.

    The meetup is free of charge and will be dutch spoken.
    No prior knowledge is required!

    Dutch PowerShell Meetup

    Rotterdam, NL
    26 Powershellerines

    Know Ada Lovelace? Grace Hopper? This group firmly believes that programming, scripting and system administration is for women! But where are you? Let’s get together and learn…

    Check out this Meetup Group →


Docker for impatient newbies part 1: Getting started

If you are one of the few that have not yet taken the plunge into Docker don’t feel ashamed. I haven’t either. I’ve been too busy doing other stuff lately. But yesterday I actually started using Docker because I had a real life use case. I have a Node application that I wanted to ship in a Docker container. And I wanted it quickly.

So here is what you need to package a Node application in a Docker container.

Docker for Mac and Docker for Windows

Step 1 is to install the Docker engine for Mac or Windows. Head over to https://docs.docker.com/engine/installation/mac/ and install Docker for Mac or Docker for Windows.

What happens? You just transformed your machine into a Docker engine. With this Docker engine you can host containers, from images containing your applications.
This is how you get started and start developing. Production environments are better installed in another Docker Engine, somewhere in the cloud or in a datacenter.

Docker on Linux

The Docker Engine on Linux is in its natural habitat and installing it (on Ubuntu Xenial) is as easy as:

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
apt-cache policy docker-engine
sudo apt-get install -y docker-engine

sudo usermod -aG docker $USER

First steps

First, check the installation:

docker version

1.12 is the latest version:

Let’s go ahead and create our first image for a Node Express app.

Package a Node.js Express app

Cd into a Node application you want to package, or clone mine. If you clone mine, install Node.js on your platform if you haven’t already. and check if it runs.

git clone https://github.com/jacqinthebox/node-express-starter
cd node-express-starter
npm install
gulp serve

If all is well you will be presented with a static website. The theme is from Black Tie.

First step is to create a Dockerfile.
Assuming you are still in the node-express-starter dir, run:

touch Dockerfile

This will be the folder structure from node-express-starter:

.
├── Dockerfile
├── app
├── bower.json
├── gulpfile.js
├── main.js
├── node_modules
└── package.json

This is how the Dockerfile should look like:

FROM mhart/alpine-node:6.3.0

MAINTAINER Jacqueline

# copy all the files from the Node app to /var/www in the container:
COPY  . /var/www

#set it as workdir
WORKDIR /var/www

#install dependencies from package.json
RUN npm install

# Expose port 3000 from the container to the host
EXPOSE 3000

ENTRYPOINT ["node","app.js"]

What does this mean?

  • FROM: here we need to put the base image for our image. Lots of these images yoy can find on the [Docker Hub](https://hub.docker.com/explore/). I want the image to be as small as possible. So that’s why I’m using an [Alpine](https://github.com/mhart/alpine-node) based image.
  • COPY: I copy all the files from my node project to the /var/www folder on my Docker image
  • WORKDIR: /var/www is the workdir (pwd)
  • RUN: while in /var/www, the npm install is run to install all dependencies of the Node app.
  • EXPOSE: here you can set the port for communication with the outside world. My Node app runs on port 3000
  • ENTRYPOINT: the command ‘node app.js’ will run upon starting the container (not when creating the image of course)

Now create the container image. You should still be in the node-express-starter dir. Run the following command (don’t forget the dot):

docker build -t jacqueline/impatient .

This has created a Docker image with the tag (-t) ‘jacqueline/impatient’.

Now run it

docker run -d -p 8080:3000 jacqueline/impatient

With -d the container instance is ran as a daemon and the -p is the port redirection (3000 we exposed in our image will be forwarded to port 8080)

Now head over to http://localhost :8080 and you should be presented with the Node web application:

Recap:

  • We installed the Docker Engine on our computer
  • We created a Docker image containing our Node app
  • We started an instance of this image

container

Stopping and cleaning up

If you work with containers, lots of harddrive space will be consumed. This is how you clean up stuff.

First, check which containers are currently running with ‘docker ps’:

docker ps

Obviously, the one we just ran is active:
docker
Notice the container ID. You can stop this instance with:

docker stop 45

Next, delete the container. First list all the containers:

docker ps -a

Again, notice the ID:

docker rm 45

Now that the container is deleted, we can go ahead and remove the image. First check which images are present:

docker images

There are 2. I’m only going to remove my own image, not the Alpine one.

docker rmi 78

In the screenshot above notice how I forgot to delete the container, but I could still delete the image with the -f flag.

The End. But will be continued.

Of course now that we took the Docker plunge, we now want to host our container in the cloud. At Digital Ocean, AWS or even Azure. This will be covered in part 2.


Build a C# REST API and consume it with Powershell (and write Pester tests!) (part 2)

This is the sequel to part 1.
Now we are going to query the AdventureWorks Database.

Install the AdventureWorks database

The script below downloads the AdventureWorks database for SQL Server 2014, extracts it in c:\temp and restores it. As you can see I had to change the data and the log locations because the original AdventureWorksDb is created in another version of SQL Server.

With the SQL Powershell commandlets you have to create 2 ‘Microsoft.SqlServer.Management.Smo.RelocateFile’ objects to do so. Now, if you have both SQL Server 2014 Express and Visual Studio 2015 Community Edition installed, the SQL Server Management dlls get messed up, because both version 12 and 13 are loaded in the app domain. You can check that with

[appdomain]::CurrentDomain.GetAssemblies() | ? { $_.Location -like "*smo*" } 

This knowledge results in this script. See also this question from StackOverflow.


New-Item 'c:\temp' -ItemType Directory -Force 
Set-Location C:\temp
choco install 7zip.commandline --yes --force
wget https://msftdbprodsamples.codeplex.com/downloads/get/880661# -OutFile adventureworksdb.zip
7z e .\adventureworksdb.zip

Import-Module SQLPS

$RelocateData = New-Object 'Microsoft.SqlServer.Management.Smo.RelocateFile, Microsoft.SqlServer.SmoExtended, Version=12.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' -ArgumentList "AdventureWorks2014_Data", "C:\Program Files\Microsoft SQL Server\MSSQL12.SQLEXPRESS\MSSQL\DATA\AdventureWorks2014.mdf"

$RelocateLog = New-Object 'Microsoft.SqlServer.Management.Smo.RelocateFile, Microsoft.SqlServer.SmoExtended, Version=12.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' -ArgumentList "AdventureWorks2014_Log", "C:\Program Files\Microsoft SQL Server\MSSQL12.SQLEXPRESS\MSSQL\Log\AdventureWorks2014.ldf"

#Invoke-Sqlcmd -ServerInstance '.\sqlexpress' -Query "DROP DATABASE AdventureWorks2014"
Restore-SqlDatabase -ServerInstance localhost\sqlexpress -BackupFile C:\temp\AdventureWorks2014.bak -Database AdventureWorks2014 -RelocateFile @($RelocateData,$RelocateLog)

Add Linq to SQL classes

Now it’s time to add an Object Relation Mapper. If the app is using Microsoft SQL and is relatively easy (only a few tables, not too much relationships) then why not go ahead and use good old Linq2SQL. It’s incredibly easy.

So type CTRL+Shift+A
Add Linq to SQL Classes and call it AdventureWorks.dbml.

20160515linq

Next, open Server Explorer and drag the Person table to the canvas like this:

20160515serverexpl

Now build the Project. And we are done. This simple drag and drop action created a Person class and added the ConnectionString to the Web.Config file.

Create a Person Controller

In the Controllers folder, add a new Class named PersonController.

Change the first Get method as follows:

// GET: api/Person
public IQueryable<Person> Get()
   {
       var db = new AdventureWorksDataContext();
       var people = db.Persons.Take(10);
       return people;
   }

So here we changed the return type in ‘Queryable’, because that is the return type of a AdventureWorksDataContext collection. You see that we only pick 10 records because the Person table has almost 20.000 records and it takes a long time to load them all.

Now, hit build and start debugging. Fire up Powershell and type:

curl http://localhost:49491/api/person

You may need to change the url to match yours. The result should be:

ice_screenshot_20160515-134347

Consume the api with Powershell and test with Pester

A tip: read http://mikefrobbins.com/2014/10/09/using-pester-for-test-driven-development-in-powershell/ because it explains in a very concise way how to use Pester. It’s great to be able to write unit tests for Powershell. If you are not sure why you should be writing tests, read this.

Shouldn’t I test the api as well? Yes, I definitely should and I will, but it’s not in scope of this article. So let’s get started with Powershell.

First, add a new Test:

New-Fixture -Name Get-AWPeople -Path .\AWcmdlets -Verbose

This creates 2 files in the AWcmdlets folder:

ice_screenshot_20160515-135708

This is the contents of the Get-AWPeople.Tests.ps1 file:

So let’s invoke a test:

20160515pester-fail

Of course, it fails. Now, let’s write code in the Get-AWPeople.ps1 file:

function Get-AWPeople 
    [CmdletBinding()]
    param (
        [Parameter(Mandatory, 
                   ValueFromPipeline)]
        [string]$Uri
    )
 
    PROCESS {
       $data = ((Invoke-WebRequest $Uri).content) | ` 
                  ConvertFrom-Json
       Write-Output $data.count
    }

}

We would expect the output to be 10 of course, because our api returns only 10 entries.

Next, write the test:

$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$sut = (Split-Path -Leaf $MyInvocation.MyCommand.Path).Replace(".Tests.", ".")
. "$here\$sut"

Describe "Get-AWPeople" {
    It "returns 10 entries from the Person table" {
        $result = Get-AWPeople -Uri "http://localhost:49491/api/person"
        $result | Should Be 0
    }
}

Let’s run the test again:

20160515pester2-fail

Now change the 0 in the file to 10 and run Invoke-Pester:

20160515pester2-OK

And we have a success. Now on to write some more functionality!

agile


Query a database through a C# REST API with Powershell (part 1)

It is probably known that you can query an SQL database in Powershell relatively easy, but wouldn’t it be great to quickly write a REST API in front of the database? So that you can add business logic if you wish? And use Powershell as a REST client? And then be able to code a decent frontend for the API for whatever device?

Let’s get started!
In this series I will first create a WebApi from scratch. Of course, you can also use the templates in Visual Studio, but I prefer to have a bit of knowledge of the code that’s in my project. It’s not that hard and you will end up with a clean code base.

Step 1. Get your dev environment ready

You can use a Vagrant box. If you use this Vagrantfile a install.ps1 script will be copied to your desktop. Run it, grab a coffee or go shopping because we are on Windows and Windows apps can be huge.

Step 2. Getting the VS Project in place

Start Visual Studio
Create a new empty solution:

ice_screenshot_20160508-093224

I named the empty solution BusinessApp (I’m lacking inspiration for a better name).

Then right click the newly made solution in the Solution Explorer (the pane on the right) and click Add and the New Project:

20150508-context

 

 

 

 

 

I named the new Project BusinessApp.Api. If you set your solution up like this you can add more projects as you continue extending the app, for example for an Angular (or whatever framework) frontend, or if you want to separate your datalayer. You can also put your Powershell client modules in a separate project if you wish.

Then open up the Nuget Package Manager Console and install the WebApi dll’s:

Install-Package Microsoft.AspNet.WebApi

Make sure to choose the correct Package source (Microsoft and .NET).

Step 3. Add routing

Add a new folder and name it App_Start.
Create a new class in the folder and name it WebApiConfig.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web.Http;


namespace BusinessApp.Api
{
    public static class WebApiConfig
    {
        public static void Register(HttpConfiguration config)
        {
            config.MapHttpAttributeRoutes();
            GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;
            GlobalConfiguration.Configuration.Formatters.Remove(GlobalConfiguration.Configuration.Formatters.XmlFormatter);

            config.Routes.MapHttpRoute(
                name: "DefaultApi",
                routeTemplate: "api/{controller}/{id}",
                defaults: new { id = RouteParameter.Optional }
            );
        }
    }
}

In this class we configure that we want our api to return and consume json. Also, we configure our routes to match the controller name, followed by id, wich is optional. E.g http://example.com/api/employees/1 would match a controllername Employees, and it would return employee with id 1.

Step 4. Enable CORS

We need to enable CORS else we won’t be able to consume the api from from another domain outside the domain from which the resource originated. In a production web environment you should configure this very carefully. I will CORS very permissive because I want my code to work.

Install CORS with in Nuget console:

Install-Package Microsoft.AspNet.WebApi.Cors

Then modify the WebApiConfig.cs class as follows:

using System.Web;
using System.Web.Http;
using System.Web.Http.Cors;

namespace BusinessApp.Api
{
    public static class WebApiConfig
    {
        public static void Register(HttpConfiguration config)
        {
            var cors = new EnableCorsAttribute("*", "*", "*");
            config.EnableCors(cors);
            config.MapHttpAttributeRoutes();
            GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;
            GlobalConfiguration.Configuration.Formatters.Remove(GlobalConfiguration.Configuration.Formatters.XmlFormatter);

Step 5. Add a Controller

  • Create a folder named ‘Controllers’
  • Right click the Controllers folder and click Add and then Controller
  • Click Web API 2 Controller with read/write actions.

ice_screenshot_20160508-092302

I named the Controller Test Controller.

Step 5. Add a Global.asax file

We need to add a Global.asax file to call the WebApiConfig.cs methods at startup.

Right click the solution, click Add, click New Item and search for Global.asax, then Add it.

ice_screenshot_20160508-095951

Modify Global.asax (see the highlighted lines):

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Http;
using System.Web.Security;
using System.Web.SessionState;

namespace BusinessApp.Api
{
    public class Global : System.Web.HttpApplication
    {

        protected void Application_Start(object sender, EventArgs e)
        {
            GlobalConfiguration.Configure(WebApiConfig.Register);
        }

 

Step 6. Test the API

Hit F5 and browse to http://localhost:/api/test

ice_screenshot_20160508-100831

And it works. You can also consume the API with Powershell at this point:

((Invoke-WebRequest http://localhost:53601/api/test).content) | ConvertFrom-Json

It should return value1 and value2.

Done! Now let’s query a database. This will be explained in Part 2.