Run multiple machines in a single Vagrant file

Last Updated on by

Post summary: How to run multiple machines on Vagrant described in a single Vagrantfile.

Code bellow can be found in GitHub sample-dropwizard-rest-stub repository in Vagrantfile file. This post is part of Vagrant series. All of other Vagrant related posts as well as more theoretical information what is Vagrant and why to use it can be found in What is Vagrant and why to use it post.

Vagrantfile

As described in Vagrant introduction post all configurations are done in a single text file called Vagrantfile. Bellow is a Vagrant file which can be used to initialise two machines. One is same as described in Run Dropwizard Java application on Vagrant post, the other is the one described in Run Docker container on Vagrant post.

Vagrant.configure('2') do |config|

  config.vm.hostname = 'dropwizard'
  config.vm.box = 'opscode-centos-7.2'
  config.vm.box_url = 'http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_centos-7.2_chef-provisionerless.box'

  config.vm.synced_folder './', '/vagrant'

  config.vm.define 'jar' do |jar|
    jar.vm.network :forwarded_port, guest: 9000, host: 9100
    jar.vm.network :forwarded_port, guest: 9001, host: 9101

    jar.vm.provider :virtualbox do |vb|
      vb.name = 'dropwizard-rest-stub-jar'
    end

    jar.vm.provision :shell do |shell|
      shell.inline = <<-SHELL
        sudo service dropwizard stop
        sudo yum -y install java
        sudo mkdir -p /var/dropwizard-rest-stub
        sudo mkdir -p /var/dropwizard-rest-stub/logs
        sudo cp /vagrant/target/sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar /var/dropwizard-rest-stub/dropwizard-rest-stub.jar
        sudo cp /vagrant/config-vagrant.yml /var/dropwizard-rest-stub/config.yml
        sudo cp /vagrant/linux_service_file /etc/init.d/dropwizard
        # Replace CR+LF with LF because of Windows
        sudo sed -i -e 's/\r//g' /etc/init.d/dropwizard
        sudo service dropwizard start
      SHELL
    end
  end

  config.vm.define 'docker' do |docker|
    docker.vm.network :forwarded_port, guest: 9000, host: 9000
    docker.vm.network :forwarded_port, guest: 9001, host: 9001

    docker.vm.provider :virtualbox do |vb|
      vb.name = 'dropwizard-rest-stub-docker'
      vb.customize ['modifyvm', :id, '--memory', '768', '--cpus', '2']
    end
  
    docker.vm.provision :shell do |shell|
      shell.inline = <<-SHELL
        sudo yum -y install epel-release
        sudo yum -y install python-pip
        sudo pip install --upgrade pip
        sudo pip install six==1.4
        sudo pip install docker-py
      SHELL
    end
  
    docker.vm.provision :docker do |docker|
      docker.build_image '/vagrant/.', args: '-t dropwizard-rest-stub'
      docker.run 'dropwizard-rest-stub', args: '-it -p 9000:9000 -p 9001:9001 -e ENV_VARIABLE_VERSION=1.1.1'
    end
  end
  
end

Vagrantfile explanation

File starts with a Vagrant.configure(‘2’) do |config| which states that version 2 of Vagrant API will be used and defines constant with name config to be used bellow. Guest operating system hostname is set with config.vm.hostname. If you use vagrant-hostsupdater plugin it will add it to your hosts file and you can access it from browser in case you are developing web applications. With config.vm.box you define which would be the guest operating system. Vagrant maintains config.vm.box = “hashicorp/precise64” which is Ubuntu 12.04 (32 and 64-bit), they also recommend to use Bento’s boxes, but I found issues with Vagrant’s as well as Bento’s boxes so I’ve decided to use one I know is working. I specify where it is located with config.vm.box_url. It is It is CentOS 7.2. With config.vm.synced_folder command you specify that Vagrantfile location folder is shared as /vagrant/ in guest operating system. This makes it easy to transfer files between guest and host operating systems. Now comes the part where two different machines are defined. First one is defined with config.vm.define ‘jar’ do |jar|, which declares variable jar to be used later in configurations. All other configurations are well described in Run Dropwizard Java application on Vagrant post. Specific part here is port mapping. In order to avoid port collision port 9000 from guest is mapped to port 9100 to host with jar.vm.network :forwarded_port, guest: 9000, host: 9100 line. This is because second machine uses port 9000 from the host. Second machine is defined in config.vm.define ‘docker’ do |docker|, which declares variable docker to be used in further configurations. All other configurations are described in Run Docker container on Vagrant post.

Running Vagrant

Command to start Vagrant machine is: vagrant up. Then in order to invoke provisioning section with actual deployment you have to call: vagrant provision. All can be done in one step: vagrant up –provision. To shutdown the machine use vagrant halt. To delete machine: vagrant destroy.

Conclusion

It is very easy to create Vagrantfile that builds and runs several machines with different applications. It possible to make those machine communicate with each other, hence simulation real environment. Once created file can be reused by all team members. It is executed over and over again making provisioning extremely easy.

Read more...

Run Docker container on Vagrant

Last Updated on by

Post summary: How to run Docker container on Vagrant.

Code bellow can be found in GitHub sample-dropwizard-rest-stub repository in Vagrantfile-docker file. Since Vagrant requires to have only one Vagrantfile if you want to run this example you have to rename Vagrantfile-docker to Vagrantfile then run Vagrant commands described in the end of this post. This post is part of Vagrant series. All of other Vagrant related posts as well as more theoretical information what is Vagrant and why to use it can be found in What is Vagrant and why to use it post.

Vagrantfile

As described in Vagrant introduction post all configurations are done in a single text file called Vagrantfile. Bellow is a Vagrant file which can be used to deploy and start Docker container on Vagrant. Example here uses Dockerised application that is described in Run Dropwizard application in Docker with templated configuration using environment variables post.

Vagrant.configure('2') do |config|

  config.vm.hostname = 'dropwizard'
  config.vm.box = 'opscode-centos-7.2'
  config.vm.box_url = 'http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_centos-7.2_chef-provisionerless.box'

  config.vm.synced_folder './', '/vagrant'

  config.vm.network :forwarded_port, guest: 9000, host: 9000
  config.vm.network :forwarded_port, guest: 9001, host: 9001

  config.vm.provider :virtualbox do |vb|
    vb.name = 'dropwizard-rest-stub-docker'
    vb.customize ['modifyvm', :id, '--memory', '768', '--cpus', '2']
  end

  config.vm.provision :shell do |shell|
    shell.inline = <<-SHELL
      sudo yum -y install epel-release
      sudo yum -y install python-pip
      sudo pip install --upgrade pip
      sudo pip install six==1.4
      sudo pip install docker-py
    SHELL
  end

  config.vm.provision :docker do |docker|
    docker.build_image '/vagrant/.', args: '-t dropwizard-rest-stub'
    docker.run 'dropwizard-rest-stub', args: '-it -p 9000:9000 -p 9001:9001 -e ENV_VARIABLE_VERSION=1.1.1'
  end

end

Vagrantfile explanation

File starts with a Vagrant.configure(‘2’) do |config| which states that version 2 of Vagrant API will be used and defines constant with name config to be used bellow. Guest operating system hostname is set with config.vm.hostname. If you use vagrant-hostsupdater plugin it will add it to your hosts file and you can access it from browser in case you are developing web applications. With config.vm.box you define which would be the guest operating system. Vagrant maintains config.vm.box = “hashicorp/precise64” which is Ubuntu 12.04 (32 and 64-bit), they also recommend to use Bento’s boxes. I have found issues with Vagrant’s as well as Bento’s boxes so I’ve decided to use one I know is working. I specify where it is located with config.vm.box_url. It is CentOS 7.2. With config.vm.synced_folder command you specify that Vagrantfile location folder is shared as /vagrant/ in guest operating system. This makes it easy to transfer files between guest and host operating systems. This mount is done by default, but it is good to explicitly state it for better readability. With config.vm.network :forwarded_port port from guest OS is forwarded to your hosting OS. Without exposing any port you will not have access to guest OS, only port open by default is 22 for SSH. With config.vm.provider :virtualbox do |vb| you access VirtualBox provider for more configurations, vb.name = ‘dropwizard-rest-stub-docker’ sets the name that you see in Oracle VirtualBox Manager. With vb.customize [‘modifyvm’, :id, ‘–memory’, ‘768’, ‘–cpus’, ‘2’] you modify default hardware settings for the machine, RAM is set to 768MB and 2 CPUs are configured. Finally the provisioning part takes place which is done by shell commands inside config.vm.provision :shell do |shell| block. This block installs Python as well as docker-py. It is CentOS specific as it uses YUM which is CentOS package manager. Next provisioning part is to run docker provisioner that builds docker image and then runs it by mapping ports and setting environment variable. For more details how to build and run Docker containers read Run Dropwizard application in Docker with templated configuration using environment variables post.

Running Vagrant

Command to start Vagrant machine is: vagrant up. Then in order to invoke provisioning section with actual deployment you have to call: vagrant provision. All can be done in one step: vagrant up –provision. To shutdown the machine use vagrant halt. To delete machine: vagrant destroy.

Conclusion

It is very easy to create Vagrantfile that builds and runs Docker container. Once created file can be reused by all team members. It is executed over and over again making provisioning extremely easy.

Read more...

Run Dropwizard Java application on Vagrant

Last Updated on by

Post summary: How to run Dropwizard or any other Java application on Vagrant.

Code bellow can be found in GitHub sample-dropwizard-rest-stub repository in Vagrantfile-jar file. Since Vagrant requires to have only one Vagrantfile if you want to run this example you have to rename Vagrantfile-jar to Vagrantfile then run Vagrant commands described in the end of this post. This post is part of Vagrant series. All of other Vagrant related posts as well as more theoretical information what is Vagrant and why to use it can be found in What is Vagrant and why to use it post.

Vagrantfile

As described in Vagrant introduction post all configurations are done in a single text file called Vagrantfile. Bellow is a Vagrant file which can be used to deploy and start as service Dropwizard Java application described in Build a RESTful stub server with Dropwizard post.

Vagrant.configure('2') do |config|

  config.vm.hostname = 'dropwizard'
  config.vm.box = 'opscode-centos-7.2'
  config.vm.box_url = 'http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_centos-7.2_chef-provisionerless.box'

  config.vm.synced_folder './', '/vagrant'

  config.vm.network :forwarded_port, guest: 9000, host: 9000
  config.vm.network :forwarded_port, guest: 9001, host: 9001

  config.vm.provider :virtualbox do |vb|
    vb.name = 'dropwizard-rest-stub-jar'
  end

  config.vm.provision :shell do |shell|
    shell.inline = <<-SHELL
      sudo service dropwizard stop
      sudo yum -y install java
      sudo mkdir -p /var/dropwizard-rest-stub
      sudo mkdir -p /var/dropwizard-rest-stub/logs
      sudo cp /vagrant/target/sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar /var/dropwizard-rest-stub/dropwizard-rest-stub.jar
      sudo cp /vagrant/config-vagrant.yml /var/dropwizard-rest-stub/config.yml
      sudo cp /vagrant/linux_service_file /etc/init.d/dropwizard
      # Replace CR+LF with LF because of Windows
      sudo sed -i -e 's/\r//g' /etc/init.d/dropwizard
      sudo service dropwizard start
    SHELL
  end

end

Vagrantfile explanation

File starts with a Vagrant.configure(‘2’) do |config| which states that version 2 of Vagrant API will be used and defines constant with name config to be used bellow. Guest operating system hostname is set with config.vm.hostname. If you use vagrant-hostsupdater plugin it will add it to your hosts file and you can access it from browser in case you are developing web applications. With config.vm.box you define which would be the guest operating system. Vagrant maintains config.vm.box = “hashicorp/precise64” which is Ubuntu 12.04 (32 and 64-bit), they also recommend to use Bento’s boxes. I have found issues with Vagrant’s as well as Bento’s boxes so I’ve decided to use one I know is working. I specify where it is located with config.vm.box_url. It is CentOS 7.2. With config.vm.synced_folder command you specify that Vagrantfile location folder is shared as /vagrant/ in guest operating system. This makes it easy to transfer files between guest and host operating systems. This mount is done by default, but it is good to explicitly state it for better readability. With config.vm.network :forwarded_port port from guest OS is forwarded to your hosting OS. Without exposing any port you will not have access to guest OS, only port open by default is 22 for SSH. With config.vm.provider :virtualbox do |vb| you access VirtualBox provider for more configurations, vb.name = ‘dropwizard-rest-stub-jar’ sets the name that you see in Oracle VirtualBox Manager. Finally the deployment part takes place which is done by shell commands inside config.vm.provision :shell do |shell| block. Service dropwizard is stopped, if not existing an error is shown, but it does not interrupt provisioning process. Command yum -y install java is CentOS specific and it installs Java by YUM which is CentOS package manager. For other Linux distributions you have to use command with their package manager. Folders are created, then JAR and YML files are copied to machine. Notice that files are copied from /vagrant/ folder, this is actually the shared folder to your host OS. Installing Java application as service is done by copying linux_service_file to /etc/init.d/dropwizard. This creates service with name dropwizard. See more how to install Linux service in Install Java application as a Linux service post. Since I’m on Windows its line endings (CR+LF) are different that on Linux (LF) and service is not working, giving env: /etc/init.d/dropwizard: No such file or directory error. This is why CF+LF should be replaced with LF with sudo sed -i -e ‘s/\r//g’ /etc/init.d/dropwizard command. Finally script starts the dropwizard service. The more nicer way to do this is all installation steps to be extracted as separate batch file and in Vagrantfile just to call that file. I’ve put it in Vagrantfile just to have it on one place.

Running Vagrant

Command to start Vagrant machine is: vagrant up. Then in order to invoke provisioning section with actual deployment you have to call: vagrant provision. All can be done in one step: vagrant up –provision. To shutdown the machine use vagrant halt. To delete machine: vagrant destroy.

Conclusion

It is very easy to create Vagrantfile that install Java application. Once created file can be reused by all team members. It is executed over and over again making provisioning extremely easy.

Read more...

Install Java application as a Linux service

Last Updated on by

Post summary: Code snippet how to start Java application as a Linux service.

Code bellow can be found in GitHub sample-dropwizard-rest-stub repository in linux_service_file file. This post is related to Build a RESTful stub server with Dropwizard post. REST server build there is being setup to run as Linux service with code shown bellow.

Service snippet

This snippet can be used for other applications to be run as Linux service, not only Java.

#!/bin/bash

BASE_DIR=/var/dropwizard-rest-stub
START_COMMAND="java -jar $BASE_DIR/dropwizard-rest-stub.jar server $BASE_DIR/config.yml"
PID_FILE=$BASE_DIR/dropwizard-rest-stub.pid
LOG_DIR=$BASE_DIR/logs

start() {
  PID=`$START_COMMAND > $LOG_DIR/init.log 2>$LOG_DIR/init.error.log & echo $!`
}

case "$1" in
start)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
            start
        else
            echo "Already running [$PID]"
            exit 0
        fi
    else
        start
    fi

    if [ -z $PID ]; then
        echo "Failed starting"
        exit 1
    else
        echo $PID > $PID_FILE
        echo "Started [$PID]"
        exit 0
    fi
;;
status)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
            echo "Not running (process dead but PID file exists)"
            exit 1
        else
            echo "Running [$PID]"
            exit 0
        fi
    else
        echo "Not running"
        exit 0
    fi
;;
stop)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
            echo "Not running (process dead but PID file exists)"
            rm -f $PID_FILE
            exit 1
        else
            PID=`cat $PID_FILE`
            kill -term $PID
            echo "Stopped [$PID]"
            rm -f $PID_FILE
            exit 0
        fi
    else
        echo "Not running (PID not found)"
        exit 0
    fi
;;
restart)
    $0 stop
    $0 start
;;
*)
    echo "Usage: $0 {status|start|stop|restart}"
    exit 0
esac

Install as a Linux service

In order to make it a Linux service following file has to be copied into /etc/init.d/ Linux folder with name that you want your service to be. If you want your service to be named service_name then you put same name as filename: /etc/init.d/service_name.

Nota bene: If you are creating the service and copying the file from Windows machine it has different new line endings (CR + LF) than Linux (LF). Also by default Git amends line endings on pull and push depending on the OS. If you receive message: env: /etc/init.d/service_name: No such file or directory then you have to replace CR+LF to LF only. This can be done with following command: sed -i -e ‘s/\r//g’ /etc/init.d/service_name.

Manage service

Assume you have named your file dropwizard then you manage your service with that name. Service has 4 commands: status, start, stop and restart. You start the service with service dropwizard start command. If you input something different that 4 options given above service will output its usage pattern.

Conclusion

In current post I have provided sample bash script that is used to install Java or any other application as a Linux service and then start, stop or restart it.

Read more...

Coloured log files in Linux

Last Updated on by

Post summary: How to colour your log files for better perception under Linux.

I’m far away from being a Linux guru and honestly I like it that way. In order to be effective as QA you need to have minimal knowledge how to do certain things under Linux. This post is devoted on working with logs.

Chaining commands

Linux offers a possibility to combine several commands by chaining them. In current post I will just one of them, the PIPE (I) operator. By using it output of one command is used as input for other.

I would strongly recommend to read following post if you are interested in chaining Linux commands: 10 Useful Chaining Operators in Linux with Practical Examples.

Useful commands

Commands bellow are one I use on daily basis when working with logs. I will show basic usage, if you need more detail on certain command then you can type: man <command> e.g. man cat in Linux console and it will display you more information.

grep

It is used to search in text files or print lines matching some pattern. Usage is: grep text filename.log. If text contains spaces it should be wrapped around single quote (‘) or double quote (“). If text contains single quote, then you have to wrap it around with double quote and vice versa.

cat

Print file content on standard out put. Usage is: cat filename.log. You can concatenate several files: cat file1.log file2.log. Drawback using this command is when you have large files. It combines very well with grep to search output of the file: cat filename.log | grep text.

zcat

Print content of zipped file. Usage: zcat filename.gz. Combines with grep: zcat filename.gz | grep text.

tail

Prints last 10 lines from a file. Usage: tail filename.log. Most valuable tail usage is with -f option: tail -f filename.log. This monitor file in real time and outputs all new text appended to the file. You can also monitor several files: tail -f file1.log file2.log.

less

Used for paging through a file. It shows one page and with arrow key up and down you can scroll into the file. Usage: less filename.txt. In order to exit just type q. Valuable with this command is that you can type a search term /text and then with n go to next appearance and with N go to previous.

Colours

Commands above are nice, but using colours aid for much better perception of information in the files. In order to use colours perl -pe command will be used as chained command to colour the output of commands described above. Syntax is: perl -pe ‘s/^.*INFO.*$/\e[0;36;40m$&\e[0m/g’. It is quite complex expression and I will try to explain it in details.

Match text to be highlighted

^.*INFO.*$ is a regular expression that matches a text to be higlighted. Character ^ means from beginning of the string or line, character $ means to end of string or line. Group .* matches any character. So this regular expression means inspect every string or line and match those that contain INFO.

Text effects

\e[0;36;40m is the colouring part of the expression. 0 is value for ANSI escape code. Possible values for escape code are show in table bellow. Note that not all of them are supported by all OS.

Code Effect
0 Reset / Normal
1 Bold or increased intensity
2 Faint (decreased intensity)
3 Italic: on
4 Underline: Single
5 Blink: Slow
6 Blink: Rapid
7 Image: Negative
8 Conceal
9 Crossed-out

More codes can be found in ANSI escape code wiki.

Text colour

36 from \e[0;36;40m is colour code of text. Colour depends and is different based on escape code. Possible combinations of escape and colour codes are:

Code Colour Code Colour
0;30 Black 1;30 Dark Grey
0;31 Red 1;31 Light Red
0;32 Green 1;32 Lime
0;33 Dark Yellow 1;33 Yellow
0;34 Blue 1;34 Light Blue
0;35 Purple 1;35 Magenta
0;36 Dark Cyan 1;36 Cyan
0;37 Light Grey 1;37 White

Background colour

40m from \e[0;36;40m is colour code of background. Background colours are:

Code Colour
40m Black
41m Red
42m Green
43m Yellow
44m Blue
45m Purple
46m Cyan
47m Light Grey

Sample colour scheme for logs

One possible colour scheme I like is: cat application.log | perl -pe ‘s/^.*FATAL.*$/\e[1;37;41m$&\e[0m/g; s/^.*ERROR.*$/\e[1;31;40m$&\e[0m/g; s/^.*WARN.*$/\e[0;33;40m$&\e[0m/g; s/^.*INFO.*$/\e[0;36;40m$&\e[0m/g; s/^.*DEBUG.*$/\e[0;37;40m$&\e[0m/g’ which will produce following output:

linux-logs-colour

Conclusion

Having coloured logs makes it much easier to investigate logs. Linux provides tooling for better visualisation so it is good to take advantage of those.

Read more...