Browsing:

Category: node.js

Docker for impatient newbies part 1: Getting started

If you are one of the few that have not yet taken the plunge into Docker don’t feel ashamed. I haven’t either. I’ve been too busy doing other stuff lately. But yesterday I actually started using Docker because I had a real life use case. I have a Node application that I wanted to ship in a Docker container. And I wanted it quickly.

So here is what you need to package a Node application in a Docker container.

Docker for Mac and Docker for Windows

Step 1 is to install the Docker engine for Mac or Windows. Head over to https://docs.docker.com/engine/installation/mac/ and install Docker for Mac or Docker for Windows.

What happens? You just transformed your machine into a Docker engine. With this Docker engine you can host containers, from images containing your applications.
This is how you get started and start developing. Production environments are better installed in another Docker Engine, somewhere in the cloud or in a datacenter.

Docker on Linux

The Docker Engine on Linux is in its natural habitat and installing it (on Ubuntu Xenial) is as easy as:

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
apt-cache policy docker-engine
sudo apt-get install -y docker-engine

sudo usermod -aG docker $USER

First steps

First, check the installation:

docker version

1.12 is the latest version:

Let’s go ahead and create our first image for a Node Express app.

Package a Node.js Express app

Cd into a Node application you want to package, or clone mine. If you clone mine, install Node.js on your platform if you haven’t already. and check if it runs.

git clone https://github.com/jacqinthebox/node-express-starter
cd node-express-starter
npm install
gulp serve

If all is well you will be presented with a static website. The theme is from Black Tie.

First step is to create a Dockerfile.
Assuming you are still in the node-express-starter dir, run:

touch Dockerfile

This will be the folder structure from node-express-starter:

.
├── Dockerfile
├── app
├── bower.json
├── gulpfile.js
├── main.js
├── node_modules
└── package.json

This is how the Dockerfile should look like:

FROM mhart/alpine-node:6.3.0

MAINTAINER Jacqueline

# copy all the files from the Node app to /var/www in the container:
COPY  . /var/www

#set it as workdir
WORKDIR /var/www

#install dependencies from package.json
RUN npm install

# Expose port 3000 from the container to the host
EXPOSE 3000

ENTRYPOINT ["node","app.js"]

What does this mean?

  • FROM: here we need to put the base image for our image. Lots of these images yoy can find on the [Docker Hub](https://hub.docker.com/explore/). I want the image to be as small as possible. So that’s why I’m using an [Alpine](https://github.com/mhart/alpine-node) based image.
  • COPY: I copy all the files from my node project to the /var/www folder on my Docker image
  • WORKDIR: /var/www is the workdir (pwd)
  • RUN: while in /var/www, the npm install is run to install all dependencies of the Node app.
  • EXPOSE: here you can set the port for communication with the outside world. My Node app runs on port 3000
  • ENTRYPOINT: the command ‘node app.js’ will run upon starting the container (not when creating the image of course)

Now create the container image. You should still be in the node-express-starter dir. Run the following command (don’t forget the dot):

docker build -t jacqueline/impatient .

This has created a Docker image with the tag (-t) ‘jacqueline/impatient’.

Now run it

docker run -d -p 8080:3000 jacqueline/impatient

With -d the container instance is ran as a daemon and the -p is the port redirection (3000 we exposed in our image will be forwarded to port 8080)

Now head over to http://localhost :8080 and you should be presented with the Node web application:

Recap:

  • We installed the Docker Engine on our computer
  • We created a Docker image containing our Node app
  • We started an instance of this image

container

Stopping and cleaning up

If you work with containers, lots of harddrive space will be consumed. This is how you clean up stuff.

First, check which containers are currently running with ‘docker ps’:

docker ps

Obviously, the one we just ran is active:
docker
Notice the container ID. You can stop this instance with:

docker stop 45

Next, delete the container. First list all the containers:

docker ps -a

Again, notice the ID:

docker rm 45

Now that the container is deleted, we can go ahead and remove the image. First check which images are present:

docker images

There are 2. I’m only going to remove my own image, not the Alpine one.

docker rmi 78

In the screenshot above notice how I forgot to delete the container, but I could still delete the image with the -f flag.

The End. But will be continued.

Of course now that we took the Docker plunge, we now want to host our container in the cloud. At Digital Ocean, AWS or even Azure. This will be covered in part 2.


Beginning Node.js – local authentication with Passport-Local Mongoose – part 6

Disclaimer: This is a series about me, creating a web application in Node.js. The completed example is available here.

I have a deep respect for all those developers out there that write fantastic modules I can use in my projects. One little gem is passport-local-mongoose. I’ve written about it before but as you see I’m doing it again.

What is passport-local-mongoose?

You can plug-in passport-local-mongoose into your Mongoose Userschema. This decorates the User object with a set of methods. E.g.

  • authenticate
  • register
  • setPassword
  • serialize
  • deserialize

It also hashes the passwords. This saves a lot of work.

What we should work on

  • Users should be able to register
  • Users should be able to authenticate
  • Users should be able to change their password
  • Users should be able to change their e-mail address
  • There should be a ‘forgot password’ procedure
  • Users should be able to delete their accounts

This article covers only ‘register’ and ‘authenticate’. You can go ahead and clone the restaurant github repo for a full example.

To add local authentication to your app you’ll need to run:

Of course, you may add your own properties to the model:

var mongoose = require('mongoose'),
    Schema = mongoose.Schema,
    passportLocalMongoose = require('passport-local-mongoose');

var User = new Schema({
    uuid: {
        type: String,
        required: false
    },
    firstname: {
        type: String,
        required: true
    },
    active: {
        type: String,
        required: false
    }
});

var options = ({missingPasswordError: "Wrong password"});
User.plugin(passportLocalMongoose,options);

module.exports = mongoose.model('User', User)

Now let’s hook up Passport in our app.

Controller

Let’s create a user controller which contains the register, the login and the getLogin functions (to check if a user has logged in).
Create a file named controller.user.js and put it in the app folder:

var mongoose = require('mongoose');
var User = require('./model.user');

exports.register = function (req, res) {
    console.log("registering: " + req.body.firstName);
    User.register(new User({
        username: req.body.username,
        firstname: req.body.firstname
    }), req.body.password, function (err, user) {
        if (err) {
            console.log(err);
            return res.send(err);
        } else {
            res.send({
                success: true,
                user: user
            });
        }
    });
};

exports.login = function (req, res, next) {

    User.authenticate()(req.body.username, req.body.password, function (err, user, options) {
        if (err) return next(err);
        if (user === false) {
            res.send({
                message: options.message,
                success: false
            });
        } else {
            req.login(user, function (err) {
                res.send({
                    success: true,
                    user: user
                });
            });
        }
    });

};

exports.getLogin = function (req, res) {
    console.log(req.user);
    if (req.user) {

        return res.send({
            success: true,
            user: req.user
        });

    } //res.send(500, {status:500, message: 'internal error', type:'internal'}); == deprecated


    res.send({
        success: false,
        message: 'not authorized'
    });
};

What happens?
1. User.authenticate and User.register:
The User.authenticate and User.register are functions we get from passport-local-mongoose. I just took this code as an example.

2. Check if a user is logged in with ‘if(req.user)’
If a user is logged in, the req.user property is populated with the user object.
So if it exists, the user is logged in.

Routes

//mongodb
var mongoose = require('mongoose');
var User = require('./model.user');

var users = require('./controller.user');
var passport = require('passport');
var LocalStrategy = require('passport-local').Strategy;
var cookieParser = require('cookie-parser');
var session = require('express-session');


module.exports = function (app) {

    //initialize passport
    passport.use(User.createStrategy());
    // use static serialize and deserialize of model for passport session support
    passport.serializeUser(User.serializeUser());
    passport.deserializeUser(User.deserializeUser());

    //need this according to passport guide
    app.use(cookieParser());
    app.use(session({
        secret: 'the princess and the frog',
        saveUninitialized: true,
        resave: true
    }));
    app.use(passport.initialize());
    app.use(passport.session());

    //routes
    app.route('/register').post(users.register);
    app.route('/login').post(users.login);
    app.route('/login').get(users.getlogin);
   
   
};

Add this file to main.js, like so:

require('./app/routes.user')(app);

This will be your completed main.js:

var express = require('express');
var app = express();
var bodyparser = require('body-parser');
var mongoose = require('mongoose');

app.use(bodyparser.urlencoded({
  extended: true
}));

app.use(bodyparser.json());

mongoose.connect('mongodb://localhost:27017/restaurant');

require('./app/routes.food')(app);
require('./app/routes.user')(app);

app.use(express.static(__dirname + '/public'));

app.set('port', process.env.PORT || 3001);
app.listen(app.get('port'));
console.log("the server is running on http://localhost:" + app.get('port'));

Let’s try this

Install Postman or another REST API test tool.

  • Don’t forget to configure the headers: Content-Type application/json

First, let’s register a user (click POST):

Postman_passport

Second, let’s login:

Selection_012

Check the login status:

Selection_015

The end of this series

This is where this series end. I hope anyone will enjoy this and at least learns something from my struggles. I know I did!
Your feedback is more than welcome by the way.


Beginning Node.js – Testing – part 5

Testing for the newbie

To test your Node API’s you will need to install the following dependencies:

  • Mocha. Mocha is a library to run tests, AKA a test runner, or test infrastructure. Install it globally, like so: npm install -g mocha.
  • Chai. With Chai you can use “assert that.. ” or “should be.. ” or “expect that.. “. Depends on the test philosophy you are using (BDD, TDD and the likes)
  • Superagent. Superagent is a library that can test a web api, like so: superagent.get(‘http://localhost:3001/food’)

Of course there are alternatives (Jasmine, Should ..) but in my humble experience the above 3 work just fine. At first I was wondering why I needed so many libraries but I guess this all about the freedom to mix and match the modules you like the most. Which is quite nice.

Test the API

This is an example test to the ‘list’ function of all the food. Does the API return a JSON array? Well, it should, so I expect it to be.

First, Create a folder named ‘test’
Then put a file in it. (e.g. test1.js), with these contents:

var superagent = require('superagent');
var chai = require('chai');
var expect = chai.expect;

describe("Food JSON api", function() {  

  it("should return a json array", function(done) {
    superagent.get('http://localhost:3001/food')
      .end(function(err, res) {
        expect(err).to.eql(null);
        expect(res.status).to.equal(200);
        expect(res.body.success).to.equal(true);
        expect(res.type).to.equal("application/json");
        done();
      });
  });

}

Now run the test with the command: mocha.

[jacqueline@pc01 Restaurant]$ mocha

  Food JSON api
    ✓ should return a json array 


  1 passing (39ms)

Another test

With SuperAgent you can also test a POST action (be sure not to use your production db, but more about different environments later).

it("should post a new piece of food", function(done) {
    superagent.post('http://localhost:3001/food/')
      .send({
        name: "awesome food in test db from env", description: "awesome food description", price: "16,00"
      })
      .end(function(err, res) {
        expect(err).to.eql(null);
        expect(res.body._id).to.not.be.eql(null);
        id = res.body._id;
        done();
      });
  });

Run the test:

[jacqueline@pc01 Restaurant]$ mocha


  Food JSON api
    ✓ should return a json array 
    ✓ should post a new piece of food 


  2 passing (38ms)

So, this was just a wee intro to testing web API’s. There is lots more to discover about this subject!

I just put the code online at github. This is still a work in progress though.


Beginning Node.js – REST API with a Mongodb backend – part 3

This is the sequel to this article. I’m building a REST API from the ground up with Node.js and Express.js 4. I’m adding functionality as I go. I’m starting with Node.js, then I’m adding Express and now I’m adding Mongoose. In the next articles I will be adding Passport.js for authentication and start building a frontend, either with Angular of Ember.

Things have changed slightly since my former series, see here. That’s because Express.js version 3 has evolved to version 4. The main difference between version 3 and 4 is that version 3 contains middleware (JSON parsers, session support and so on). In version 4 you need to add the middleware yourself. That means installing body-parser, session and so on with NPM.

In my former article we used a JSON string with data. Now let’s use a real data backend: MongoDb. Mongoose is an ORM for MongoDb for Node.js, so let’s add the module to the project:

npm install --save mongoose

We also need to install the body-parser, which parses JSON bodies for Express.

npm install --save body-parser

Next, copy all of this to your main.js file:

var express = require('express');
var app = express();
var bodyParser = require('body-parser');
var mongoose = require('mongoose');
//mongoose uses Schema as a layer on a MongoDb document (which represents a model):
var Schema = mongoose.Schema;

app.use(bodyParser.urlencoded({
    extended: true
}));
app.use(bodyParser.json());

//connect to mongodb:
mongoose.connect('mongodb://localhost:27017/restaurant');

//create a Schema for our food:
var FoodSchema = new Schema({
    name: {
        type: String,
        index: {
            unique: true
        }
    },
    description: String,
    price: String
});


// Use the schema to register a model with MongoDb
mongoose.model('Food', FoodSchema);
var food = mongoose.model('Food');

//POST verb
app.post('/food', function (req, res) {
    food.create(req.body, function (err, food) {
        if (err) {
            res.send(401, err);
            return;
        }
        res.send(req.body);
    });
});

//GET verb
app.get('/food', function (req, res) {
    food.find(function (err, data) {
        if (err) {
            res.send(err);
        }
        res.json(data);
    });
});

//GET/id
app.get('/food/:id', function (req, res) {
    food.findOne({
        _id: req.params.id
    }, function (error, response) {
        if (error) {
            res.send(error);
        } else {
            res.send(response);
        }
    });
});

//GET by name
app.get('/foodname/:name', function (req, res) {
    food.findOne({
        name: req.params.name
    }, function (error, response) {
        if (error || !response) {
            res.send("not on the menu");
        } else {
            res.send(response);
        }
    });
});


app.set('port', process.env.PORT || 3000);
app.listen(app.get('port'));
console.log("the server is running on http://localhost:" + app.get('port'));

Now run it with the following command:

nodemon main.js

What happened?

Let’s dissect the code from the previous paragraph:

1.
We added and instantiated the prerequisites and connected to our MongodB instance

var bodyParser = require('body-parser');
var mongoose = require('mongoose');
//mongoose uses Schema as a layer on a MongoDb document (which represents a model):
var Schema = mongoose.Schema;

app.use(bodyParser.urlencoded({
    extended: true
}));
app.use(bodyParser.json());

//connect to mongodb:
mongoose.connect('mongodb://localhost:27017/restaurant');

2.
We create a Schema for our Food class:

//create a Schema for our food:
var FoodSchema = new Schema({
    name: {
        type: String,
        index: {
            unique: true
        }
    },
    description: String,
    price: String
});

3.
Then we used the schema to register a model with MongoDb
mongoose.model('Food', FoodSchema);
var food = mongoose.model('Food');

4.
And then we are ready to define our CRUD methods, e.g.

//POST verb
app.post('/food', function (req, res) {
    food.create(req.body, function (err, food) {
        if (err) {
            res.send(401, err);
            return;
        }
        res.send(req.body);
    });
});

//GET verb
app.get('/food', function (req, res) {
    food.find(function (err, data) {
        if (err) {
            res.send(err);
        }
        res.json(data);
    });
});

Using the API with Curl

You can add a menu item with the following command:

curl -H "Content-Type: application/json" -d '{ "name": "papadums", "description" : "Thin Bread", "price" : "2.00" }' http://localhost:3000/food

Issue a get:

curl http://localhost:3000/food

Parse Excel and convert it to JSON with Node.js

This is a great, simple and lovely module for Node.js to parse Excel files.

Install the module:

npm install --save excel

Use it like this:

var xls = require('excel');

xls('Sheet.xlsx', function(err, data) {
  if(err) throw err;
    // data is an array of arrays
});

As it says, the data it returns is an array of arrays. We want it to be JSON, so that we can do whatever we want with it.

This is a function that converts an array of arrays to JSON:

function convertToJSON(array) {
  var first = array[0].join()
  var headers = first.split(',');
  
  var jsonData = [];
  for ( var i = 1, length = array.length; i < length; i++ )
  {
   
    var myRow = array[i].join();
    var row = myRow.split(',');
    
    var data = {};
    for ( var x = 0; x < row.length; x++ )
    {
      data[headers[x]] = row[x];
    }
    jsonData.push(data);

  }
  return jsonData;
};

Then:

xlsx('tasks.xlsx', function(err,data) {
    if(err) throw err;
    //console.log(jsonDataArray(data));
    console.log(JSON.stringify(convertToJSON(data)));
    //console.log(data);
});

It’s almost embarrassing how Node enables me to use other people’s modules to create apps in such a simple way.