Run multiple machines in a single Vagrant file

Last Updated on by

Post summary: How to run multiple machines on Vagrant described in a single Vagrantfile.

Code bellow can be found in GitHub sample-dropwizard-rest-stub repository in Vagrantfile file. This post is part of Vagrant series. All of other Vagrant related posts as well as more theoretical information what is Vagrant and why to use it can be found in What is Vagrant and why to use it post.

Vagrantfile

As described in Vagrant introduction post all configurations are done in a single text file called Vagrantfile. Bellow is a Vagrant file which can be used to initialise two machines. One is same as described in Run Dropwizard Java application on Vagrant post, the other is the one described in Run Docker container on Vagrant post.

Vagrant.configure('2') do |config|

  config.vm.hostname = 'dropwizard'
  config.vm.box = 'opscode-centos-7.2'
  config.vm.box_url = 'http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_centos-7.2_chef-provisionerless.box'

  config.vm.synced_folder './', '/vagrant'

  config.vm.define 'jar' do |jar|
    jar.vm.network :forwarded_port, guest: 9000, host: 9100
    jar.vm.network :forwarded_port, guest: 9001, host: 9101

    jar.vm.provider :virtualbox do |vb|
      vb.name = 'dropwizard-rest-stub-jar'
    end

    jar.vm.provision :shell do |shell|
      shell.inline = <<-SHELL
        sudo service dropwizard stop
        sudo yum -y install java
        sudo mkdir -p /var/dropwizard-rest-stub
        sudo mkdir -p /var/dropwizard-rest-stub/logs
        sudo cp /vagrant/target/sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar /var/dropwizard-rest-stub/dropwizard-rest-stub.jar
        sudo cp /vagrant/config-vagrant.yml /var/dropwizard-rest-stub/config.yml
        sudo cp /vagrant/linux_service_file /etc/init.d/dropwizard
        # Replace CR+LF with LF because of Windows
        sudo sed -i -e 's/\r//g' /etc/init.d/dropwizard
        sudo service dropwizard start
      SHELL
    end
  end

  config.vm.define 'docker' do |docker|
    docker.vm.network :forwarded_port, guest: 9000, host: 9000
    docker.vm.network :forwarded_port, guest: 9001, host: 9001

    docker.vm.provider :virtualbox do |vb|
      vb.name = 'dropwizard-rest-stub-docker'
      vb.customize ['modifyvm', :id, '--memory', '768', '--cpus', '2']
    end
  
    docker.vm.provision :shell do |shell|
      shell.inline = <<-SHELL
        sudo yum -y install epel-release
        sudo yum -y install python-pip
        sudo pip install --upgrade pip
        sudo pip install six==1.4
        sudo pip install docker-py
      SHELL
    end
  
    docker.vm.provision :docker do |docker|
      docker.build_image '/vagrant/.', args: '-t dropwizard-rest-stub'
      docker.run 'dropwizard-rest-stub', args: '-it -p 9000:9000 -p 9001:9001 -e ENV_VARIABLE_VERSION=1.1.1'
    end
  end
  
end

Vagrantfile explanation

File starts with a Vagrant.configure(‘2’) do |config| which states that version 2 of Vagrant API will be used and defines constant with name config to be used bellow. Guest operating system hostname is set with config.vm.hostname. If you use vagrant-hostsupdater plugin it will add it to your hosts file and you can access it from browser in case you are developing web applications. With config.vm.box you define which would be the guest operating system. Vagrant maintains config.vm.box = “hashicorp/precise64” which is Ubuntu 12.04 (32 and 64-bit), they also recommend to use Bento’s boxes, but I found issues with Vagrant’s as well as Bento’s boxes so I’ve decided to use one I know is working. I specify where it is located with config.vm.box_url. It is It is CentOS 7.2. With config.vm.synced_folder command you specify that Vagrantfile location folder is shared as /vagrant/ in guest operating system. This makes it easy to transfer files between guest and host operating systems. Now comes the part where two different machines are defined. First one is defined with config.vm.define ‘jar’ do |jar|, which declares variable jar to be used later in configurations. All other configurations are well described in Run Dropwizard Java application on Vagrant post. Specific part here is port mapping. In order to avoid port collision port 9000 from guest is mapped to port 9100 to host with jar.vm.network :forwarded_port, guest: 9000, host: 9100 line. This is because second machine uses port 9000 from the host. Second machine is defined in config.vm.define ‘docker’ do |docker|, which declares variable docker to be used in further configurations. All other configurations are described in Run Docker container on Vagrant post.

Running Vagrant

Command to start Vagrant machine is: vagrant up. Then in order to invoke provisioning section with actual deployment you have to call: vagrant provision. All can be done in one step: vagrant up –provision. To shutdown the machine use vagrant halt. To delete machine: vagrant destroy.

Conclusion

It is very easy to create Vagrantfile that builds and runs several machines with different applications. It possible to make those machine communicate with each other, hence simulation real environment. Once created file can be reused by all team members. It is executed over and over again making provisioning extremely easy.

Read more...

Run Dropwizard Java application on Vagrant

Last Updated on by

Post summary: How to run Dropwizard or any other Java application on Vagrant.

Code bellow can be found in GitHub sample-dropwizard-rest-stub repository in Vagrantfile-jar file. Since Vagrant requires to have only one Vagrantfile if you want to run this example you have to rename Vagrantfile-jar to Vagrantfile then run Vagrant commands described in the end of this post. This post is part of Vagrant series. All of other Vagrant related posts as well as more theoretical information what is Vagrant and why to use it can be found in What is Vagrant and why to use it post.

Vagrantfile

As described in Vagrant introduction post all configurations are done in a single text file called Vagrantfile. Bellow is a Vagrant file which can be used to deploy and start as service Dropwizard Java application described in Build a RESTful stub server with Dropwizard post.

Vagrant.configure('2') do |config|

  config.vm.hostname = 'dropwizard'
  config.vm.box = 'opscode-centos-7.2'
  config.vm.box_url = 'http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_centos-7.2_chef-provisionerless.box'

  config.vm.synced_folder './', '/vagrant'

  config.vm.network :forwarded_port, guest: 9000, host: 9000
  config.vm.network :forwarded_port, guest: 9001, host: 9001

  config.vm.provider :virtualbox do |vb|
    vb.name = 'dropwizard-rest-stub-jar'
  end

  config.vm.provision :shell do |shell|
    shell.inline = <<-SHELL
      sudo service dropwizard stop
      sudo yum -y install java
      sudo mkdir -p /var/dropwizard-rest-stub
      sudo mkdir -p /var/dropwizard-rest-stub/logs
      sudo cp /vagrant/target/sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar /var/dropwizard-rest-stub/dropwizard-rest-stub.jar
      sudo cp /vagrant/config-vagrant.yml /var/dropwizard-rest-stub/config.yml
      sudo cp /vagrant/linux_service_file /etc/init.d/dropwizard
      # Replace CR+LF with LF because of Windows
      sudo sed -i -e 's/\r//g' /etc/init.d/dropwizard
      sudo service dropwizard start
    SHELL
  end

end

Vagrantfile explanation

File starts with a Vagrant.configure(‘2’) do |config| which states that version 2 of Vagrant API will be used and defines constant with name config to be used bellow. Guest operating system hostname is set with config.vm.hostname. If you use vagrant-hostsupdater plugin it will add it to your hosts file and you can access it from browser in case you are developing web applications. With config.vm.box you define which would be the guest operating system. Vagrant maintains config.vm.box = “hashicorp/precise64” which is Ubuntu 12.04 (32 and 64-bit), they also recommend to use Bento’s boxes. I have found issues with Vagrant’s as well as Bento’s boxes so I’ve decided to use one I know is working. I specify where it is located with config.vm.box_url. It is CentOS 7.2. With config.vm.synced_folder command you specify that Vagrantfile location folder is shared as /vagrant/ in guest operating system. This makes it easy to transfer files between guest and host operating systems. This mount is done by default, but it is good to explicitly state it for better readability. With config.vm.network :forwarded_port port from guest OS is forwarded to your hosting OS. Without exposing any port you will not have access to guest OS, only port open by default is 22 for SSH. With config.vm.provider :virtualbox do |vb| you access VirtualBox provider for more configurations, vb.name = ‘dropwizard-rest-stub-jar’ sets the name that you see in Oracle VirtualBox Manager. Finally the deployment part takes place which is done by shell commands inside config.vm.provision :shell do |shell| block. Service dropwizard is stopped, if not existing an error is shown, but it does not interrupt provisioning process. Command yum -y install java is CentOS specific and it installs Java by YUM which is CentOS package manager. For other Linux distributions you have to use command with their package manager. Folders are created, then JAR and YML files are copied to machine. Notice that files are copied from /vagrant/ folder, this is actually the shared folder to your host OS. Installing Java application as service is done by copying linux_service_file to /etc/init.d/dropwizard. This creates service with name dropwizard. See more how to install Linux service in Install Java application as a Linux service post. Since I’m on Windows its line endings (CR+LF) are different that on Linux (LF) and service is not working, giving env: /etc/init.d/dropwizard: No such file or directory error. This is why CF+LF should be replaced with LF with sudo sed -i -e ‘s/\r//g’ /etc/init.d/dropwizard command. Finally script starts the dropwizard service. The more nicer way to do this is all installation steps to be extracted as separate batch file and in Vagrantfile just to call that file. I’ve put it in Vagrantfile just to have it on one place.

Running Vagrant

Command to start Vagrant machine is: vagrant up. Then in order to invoke provisioning section with actual deployment you have to call: vagrant provision. All can be done in one step: vagrant up –provision. To shutdown the machine use vagrant halt. To delete machine: vagrant destroy.

Conclusion

It is very easy to create Vagrantfile that install Java application. Once created file can be reused by all team members. It is executed over and over again making provisioning extremely easy.

Read more...

Build a Dropwizard project with Gradle

Last Updated on by

Post summary: Code examples how to create Dropwizard project with Gradle.

Code sample here can be found as buildable and runable project in GitHub sample-dropwizard-rest-stub repository in separate git branch called gradle.

Project structure

All classes have been thoroughly described in Build a RESTful stub server with Dropwizard post. In this post I will describe how to make project build-able with Gradle. In order to make it more understandable I will compare with Maven’s pom.xml file elements by XPath.

Gradle

Gradle is an open source build automation system that builds upon the concepts of Apache Ant and Apache Maven and introduces a Groovy-based domain-specific language (DSL) instead of the XML form used by Apache Maven for declaring the project configuration. Gradle is much more powerful and more complex than Maven. There is significant tendency for Java projects moving towards Gradle so I’ve decided to make this post.

Gradle artefacts

In order to make your project work with Gradle you need several files. List bellow is how files are placed in project’s root folder:

  • gradle/wrapper/gradle-wrapper.jar – Gradle Wrapper allows you to make builds without installing Gradle on your machine. This is very convenient and makes Gradle usage easy. This JAR is managing the Gradle Wrapper automatic download and installation on first build.
  • gradle\wrapper\gradle-wrapper.properties – configuration which Gradle Wrapper version to be downloaded and installed on first build.
  • build.gradle – the most import file. This is where you configure your project.
  • gradlew – this Gradle Wrapper executable for Linux.
  • gradlew.bat – this is Gradle Wrapper executable for Windows.
  • settings.gradle – Project settings. Mainly used in case of multi-module projects.

setting.gradle file

This file is mainly used in case of multi-module project. In it we currently define project name: rootProject.name = ‘sample-dropwizard-rest-stub’. This is same value as in /project/name form pom.xml file.

Constructing build.gradle file

This is main file where you configure your project. You need to define version (/project/version in pom.xml), group (/project/groupId in pom.xml) and optionally description. Since this is Java project you need to apply plugin: ‘java’. Also you need need to specify Java version, 1.8 in this case by sourceCompatibility and targetCompatibility values. Next is to set repositories. You can use mavenCentral or add a custom one by following code, which is not shown in example bellow: maven { url ‘https://plugins.gradle.org/m2/’ }. You need to define dependencies (/project/dependencies/dependency in pom.xml file) to tell Gradle what libraries this project needs. In current example it is compile dependency to io.dropwizard:dropwizard-core:0.8.0 and testCompile dependency to junit:junit:4.12. This is enough to have fully functional Dropwizard project with code examples given in Build a RESTful stub server with Dropwizard post.

version '1.0-SNAPSHOT'
group 'com.automationrhapsody.reststub'
description 'Sample Dropwizard REST Stub'

apply plugin: 'java'

sourceCompatibility = 1.8
targetCompatibility = 1.8

repositories {
	mavenCentral()
}

dependencies {
	compile 'io.dropwizard:dropwizard-core:0.8.0'

	testCompile 'junit:junit:4.12'
}

The beauty of Dropwizard is the ability to pack everything into a single JAR file and then run that file. In Maven this was done by maven-shade-plugin in Gradle the best way to do it is Shadow JAR plugin. You need to define it via plugins closure. Now lets configure shadowJar. You can specify archiveName or exclude some artefacts from packed JAR. Optionally you can enhance you MANIFEST.MF file by adding more details to manifest closure. Nice thing for Gradle is that you can use Groovy as well as pure Java code. Constructing Build-Time requires import some Java DateTime classes and using them to make human readable time. Next piece that you need to add to your build.gradle file is:

import java.time.ZoneId
import java.time.ZonedDateTime
import java.time.format.DateTimeFormatter

plugins {
	id 'com.github.johnrengelman.shadow' version '1.2.4'
}

mainClassName = 'com.automationrhapsody.reststub.RestStubApp'

shadowJar {
	mergeServiceFiles()
	exclude 'META-INF/*.DSA', 'META-INF/*.RSA', 'META-INF/*.SF'
	manifest {
		attributes 'Implementation-Title': rootProject.name
		attributes 'Implementation-Version': rootProject.version
		attributes 'Implementation-Vendor-Id': rootProject.group
		attributes 'Build-Time': ZonedDateTime.now(ZoneId.of("UTC"))
				.format(DateTimeFormatter.ISO_ZONED_DATE_TIME)
		attributes 'Built-By': InetAddress.localHost.hostName
		attributes 'Created-By': 'Gradle ' + gradle.gradleVersion
		attributes 'Main-Class': mainClassName
	}
	archiveName 'sample-dropwizard-rest-stub.jar'
}

Once you build your JAR file with command: gradlew shadowJar you can run it with java -jar build/sample-dropwizard-rest-stub.jar server config.yml command. Gradle has another option to run your project for testing purposes. It is done by first apply plugin: ‘application’. You need to specify which is mainClassName to be run and configure run args. In order to run your project from Gradle with gradlew run command you just add:

apply plugin: 'application'

mainClassName = 'com.automationrhapsody.reststub.RestStubApp'

run {
	args = ['server', 'config.yml']
}

build.gradle file

Full build.gradle file content is shown bellow:

import java.time.ZoneId
import java.time.ZonedDateTime
import java.time.format.DateTimeFormatter

plugins {
	id 'com.github.johnrengelman.shadow' version '1.2.4'
}

version '1.0-SNAPSHOT'
group 'com.automationrhapsody.reststub'
description 'Sample Dropwizard REST Stub'

apply plugin: 'java'
apply plugin: 'application'

sourceCompatibility = 1.8
targetCompatibility = 1.8

repositories {
	mavenCentral()
}

dependencies {
	compile 'io.dropwizard:dropwizard-core:0.8.0'

	testCompile 'junit:junit:4.12'
}

mainClassName = 'com.automationrhapsody.reststub.RestStubApp'

run {
	args = ['server', 'config.yml']
}

shadowJar {
	mergeServiceFiles()
	exclude 'META-INF/*.DSA', 'META-INF/*.RSA', 'META-INF/*.SF'
	manifest {
		attributes 'Implementation-Title': rootProject.name
		attributes 'Implementation-Version': rootProject.version
		attributes 'Implementation-Vendor-Id': rootProject.group
		attributes 'Build-Time': ZonedDateTime.now(ZoneId.of("UTC"))
				.format(DateTimeFormatter.ISO_ZONED_DATE_TIME)
		attributes 'Built-By': InetAddress.localHost.hostName
		attributes 'Created-By': 'Gradle ' + gradle.gradleVersion
		attributes 'Main-Class': mainClassName
	}
	archiveName 'sample-dropwizard-rest-stub.jar'
}

Conclusion

This post is extension to Build a RESTful stub server with Dropwizard post, in which I have described how to build REST service with Dropwizard and Maven. In current post I have shown how to do the same with Gradle.

Read more...

Run Dropwizard application in Docker with templated configuration using environment variables

Last Updated on by

Post summary: How to run Dropwizard application inside a Docker container with configuration template file which later get replaced by environment variables.

In Build a RESTful stub server with Dropwizard post I have described how to build a Dropwizard application and run it. In current post I will show how this application can be inserted into Docker container and run with different configurations, based on different environment variables. Code sample here can be found as buildable and runable project in GitHub sample-dropwizard-rest-stub repository.

Dropwizard templated config

Dropwizard supports replacing variables from config file with environment variables if such are defined. First step is to make the config file with special notation.

version: ${ENV_VARIABLE_VERSION:- 0.0.2}

Dropwizard substitution is based on Apache’s StrSubstitutor library. ${ENV_VARIABLE_VERSION:- 0.0.2} means that Dropwizard will search for environment variable with name ENV_VARIABLE_VERSION and replace its value with given variable. If no variable is configured then it will replace with default value of 0.0.2. If default is not needed then just ${ENV_VARIABLE_VERSION} can be used.

Next step is to make Dropwizard substitute config file with environment variables on its startup before file is being read. This is done with following code:

@Override
public void initialize(Bootstrap<RestStubConfig> bootstrap) {
	bootstrap.setConfigurationSourceProvider(new SubstitutingSourceProvider(
			bootstrap.getConfigurationSourceProvider(), 
			new EnvironmentVariableSubstitutor(false)));
}

EnvironmentVariableSubstitutor is used with false in order to suppress throwing of UndefinedEnvironmentVariableException in case environment variable is not defined. There is approach to pack config.yml file into JAR file and use it from there, then bootstrap.getConfigurationSourceProvider() should be changed with path -> Thread.currentThread().getContextClassLoader().getResourceAsStream(path).

Docker

Docker is platform for building software inside containers which then can be deployed on different environments. A container is like a virtual machine with significant difference that it does not build full operation system. In this way host’s resources are optimised, container consumes as much memory as needed by application. Virtual machine itself consumes memory to run the operation system as well. Containers have resource (CPU, RAM, HDD) isolation so that application sees the container as separate operating system.

Dockerfile

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession. In given example Dockerfile looks like this:

FROM openjdk:8u121-jre-alpine
MAINTAINER Automation Rhapsody https://automationrhapsody.com/

WORKDIR /var/dropwizard-rest-stub

ADD target/sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar /var/dropwizard-rest-stub/dropwizard-rest-stub.jar
ADD config-docker.yml /var/dropwizard-rest-stub/config.yml

EXPOSE 9000 9001

ENTRYPOINT ["java", "-jar", "dropwizard-rest-stub.jar", "server", "config.yml"]

Dockerfile starts with FROM openjdk:8u121-jre-alpine which instructs docker to use this already prepared image as a base image for container. At https://hub.docker.com/ there are numerous of images maintained by different organisations or individuals that provide different frameworks and tools. They can save you a lot of work allowing you to skip configuration and installation of general things. In given case openjdk:8u121-jre-alpine has OpenJDK JRE 1.8.121 already installed with minimalist Linux libraries. MAINTAINER documents who is the author of this file. WORKDIR sets working directory of the container. Then ADD copies JAR file and config.yml to container. ENTRYPOINT configures container to be run as executable. This instruction translates to java -jar dropwizard-rest-stub.jar server config.yml command. More info can be found in Dockerfile reference link.

Build Docker container

Docker container can be build with following command:

docker build -t dropwizard-rest-stub .

dropwizard-rest-stub is the name of container, it is later used to run container with this name. Note that dot in the end is mandatory.

Run Docker container

docker run -it -p 9000:9000 -p 9001:9001 -e ENV_VARIABLE_VERSION=1.1.1 dropwizard-rest-stub

Name of the container is dropwizard-rest-stub and is put in the end. With -p 9000:9000 port 9000 form guest machine is exposed to host machine, otherwise container will not be accessible. With -e ENV_VARIABLE_VERSION=1.1.1 environment variable is being set. If not passed it will be substituted with 0.0.2 in config.yml file as described above. In case of many environment variables it is more comfortable to put them in a file with content KEY=VALUE. Then use this file with –env-file= instead of specifying each individual variable.

Conclusion

Dropwizard has very easy mechanism for making configuration dependant on environment variables. This makes Dropwizard applications extremely suitable to be build into Docker containers and run on different environments just by changing environment variables being passed to container.

Read more...

Implement secure API authentication over HTTP with Dropwizard

Last Updated on by

Post summary: Reference implementation on suggested in How to implement secure REST API authentication over HTTP post authentication mechanism.

API authentication mechanism

Suggested authentication mechanism consists of following steps:

  • Secret key that is known only by API consumer and API provider is needed along with API key.
  • Secret key is used to one way hash a token which is send to server along with API key in the API call.
  • Token consists of: API key + Secret key + Current time in seconds, which then gets hashed with SHA-256 algorithm preferably.
  • Server recreates all the tokens locally for every second for some time in the future, preferably not too long – 30~120 seconds.
  • Server recreates all the tokens for 30~120 seconds in the past, to take into account the time needed for request to reach the server.
  • Server compares each of the tokens with received one.
  • If there is match consumer is authenticated and response is returned.

Dropwizard implementation

Dropwizard stub introduced in Build a RESTful stub server with Dropwizard post will be used to create authentication. Full example can be found in GitHib sample-dropwizard-rest-stub repository. Implementation consists of following steps:

  • Implement javax.ws.rs.container.ContainerRequestFilter interface. Implementation will inspect every request and verify authentication.
  • Create custom annotation
  • Annotate RequestFilter and Dropwizard resource (API service) on which authentication should be applied.
  • Register RequestFilter implementation class into Dropwizard Jersey environment.

Create custom annotation

Starting with the easiest step. Creating custom annotation is pretty easy. It could be applied to a class (ElementType.TYPE) or to a method (ElementType.METHOD). It should live as long as program runs (RetentionPolicy.RUNTIME). In order to make it possible annotated request filter to be applied on specific resource only @NameBinding annotation is a must in Jersey. If not specified request filter will apply on all resources. Needed annotation is:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.ws.rs.NameBinding;

@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(value = RetentionPolicy.RUNTIME)
@NameBinding
public @interface Authenticator {
}

ContainerRequestFilter implementation

Container request filter is applied on incoming requests. If used with @NameBinding annotation it is applied only where needed, if not it is applied globally. Mandatory is to override filter() method:

import com.automationrhapsody.reststub.persistence.AuthDB;

import java.io.IOException;
import java.util.List;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;

@Authenticator
public class AuthenticateFilter implements ContainerRequestFilter {

	private static final String PARAM_API_KEY = "apiKey";
	private static final String PARAM_TOKEN = "token";
	private static final long SECONDS_IN_MILLISECOND = 1000L;
	private static final int TTL_SECONDS = 60;

	@Override
	public void filter(ContainerRequestContext context) throws IOException {
		final String apiKey = extractParam(context, PARAM_API_KEY);
		if (StringUtils.isEmpty(apiKey)) {
			context.abortWith(responseMissingParameter(PARAM_API_KEY));
		}

		final String token = extractParam(context, PARAM_TOKEN);
		if (StringUtils.isEmpty(token)) {
			context.abortWith(responseMissingParameter(PARAM_TOKEN));
		}

		if (!authenticate(apiKey, token)) {
			context.abortWith(responseUnauthorized());
		}
	}
}

As seen above two GET parameters are mandatory in the request: “apiKey” and “token”. Those are first extracted and verified. If some of them is not existing BAD_REQUEST (HTTP Status code 400) Response is returned with error message. Methods that extract params and build error response are:

private String extractParam(ContainerRequestContext context, String param) {
	final UriInfo uriInfo = context.getUriInfo();
	final List user = uriInfo.getQueryParameters().get(param);
	return CollectionUtils.isEmpty(user) ? null : String.valueOf(user.get(0));
}

private Response responseMissingParameter(String name) {
	return Response.status(Response.Status.BAD_REQUEST)
		.type(MediaType.TEXT_PLAIN_TYPE)
		.entity("Parameter '" + name + "' is required.")
		.build();
}

If both are present then code tried to authenticate the call by rebuilding all the hashes for 60 seconds in the past because request cannot arrive instantly it takes some time. If network is slower this time can be increased. It also rebuilds all hashes for 60 seconds in the future, this is token time to live. Server has access to Secret key for any given API key. In example above they are stored in fake DB provider and obtained by AuthDB.getSecretKey(apiKey):

private boolean authenticate(String apiKey, String token) {
	final String secretKey = AuthDB.getSecretKey(apiKey);

	// No need to calculate digest in case of wrong apiKey
	if (StringUtils.isEmpty(secretKey)) {
		return false;
	}

	final long nowSec = System.currentTimeMillis() / SECONDS_IN_MILLISECOND;
	long startTime = nowSec - TTL_SECONDS;
	long endTime = nowSec + TTL_SECONDS;
	for (; startTime < endTime; startTime++) {
		final String toHash = apiKey + secretKey + startTime;
		final String sha1 = DigestUtils.sha256Hex(toHash);
		if (sha1.equals(token)) {
			return true;
		}
	}

	return false;
}

As seen above server uses SHA-256 cryptographic algorithm. It is the best solution in terms of speed and security. In MD5, SHA-1, SHA-256 and SHA-512 speed performance post a comparison between MD5, SHA-1, SHA-256 and SHA-512 is made. If authentication cannot be verified then UNAUTHORIZED (HTTP Status code 401) Response response is returned:

private Response responseUnauthorized() {
	return Response.status(Response.Status.UNAUTHORIZED)
		.type(MediaType.TEXT_PLAIN_TYPE)
		.entity("Unauthorized")
		.build();
}

This is the hardest part. Now this filter has to be registered with Jersey and applied to needed resources (services). See more on ContainerRequestFilter interface and @NameBinding annotation in Jersey filters and interceptors page.

Apply authentication filter on a resource

Indicating that given resource should be checked for authentication is done with custom @Authenticator annotation created previously. If needed just for specific API call it can be applied also on a method level:

import com.automationrhapsody.reststub.data.Book;
import com.automationrhapsody.reststub.filters.Authenticator;
import com.automationrhapsody.reststub.persistence.BookDB;
import com.codahale.metrics.annotation.Timed;

import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Authenticator
@Path("/secure/books")
public class BooksSecureService {

	@GET
	@Timed
	@Produces(MediaType.APPLICATION_JSON)
	public List<Book> getBooks() {
		return BookDB.getAll();
	}
}

Register in Dropwizard Jersey

Last step is to register the request filter and resource with Dropwizard’s Jersey:

@Override
public void run(RestStubConfig config, Environment env) {

	env.jersey().register(BooksSecureService.class);
	env.jersey().register(AuthenticateFilter.class);

}

Conclusion

Very easy to implement in Dropwizard and relatively secure way to provide API authentication over HTTP protocol. For mission critical application definitely more strict consideration and review of this authentication mechanism is needed.

Read more...

JPS alternative for Dropwizard – Servlet with Apache Velocity template engine

Last Updated on by

Post summary: How to create a web application with Dropwizard with Servlet and Apache Velocity template engine since JSP is not supported.

JSP

JSP (Java Server Pages) is used to create web applications. It easy to separate presentation from business logic. JSP defines a page template with static content and tags inside which are evaluated in business logic and replaced in template.

Dropwizard

Dropwizard is Java framework for building RESTful web server in very short time. It has incorporated proven libraries like Jetty, Jersey, Jackson and many more to reliably do the job in shortest possible time. They have very good getting started tutorial how to make a project from scratch. Dropwizard doesn’t have support for JPS. This is because its purpose is to be REST server, not a web application server.

Dropwizard servlet support

JSP support is not in scope of Dropwizard as it is designed for microservices not for Web application server see more in Any JSP support going on? thread. Dropwizard has servlet support though. Internally JSP is compiled to servlet so it is alternative to use some template engine within a servlet.

Dropwizard views

Dropwizard provides so called Views which actually uses FreeMarker or Mustache template engines. See more about views in Dropwizard Views page. This is Dropwizard’s build in JSP alternative. If you want to go that way then rest of this post is not really helpful to you.

Apache Velocity

Apache Velocity is a template engine. HTML code is defined as a template with dynamic values substituted with special tags, similar as JSP.

Define Velocity template

Template contains HTML code and tags starting with $ sign, in example bellow this is $productId. Template is put in project’s “resource” folder.

This is 'Product $productId name' details page.

Initialise Velocity engine

This is done in Servlet’s init() method. It is important to give as a property where Velocity should look for its resources. It is at class loader.

private Template details;

public void init() throws ServletException {
	Properties props = new Properties();
	props.setProperty("resource.loader", "class");
	props.setProperty("class.resource.loader.class",
		"org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader");
	VelocityEngine engine = new VelocityEngine(props);
	engine.init();
	details = engine.getTemplate("velocity.details.html");
}

Render template

In order to render the template $variable needs to be substituted with valued values in a VelocityContext map. Note that if given value is null then $variable is being output directly, so this case should be handled correctly.

public void doGet(HttpServletRequest request,
		HttpServletResponse response) throws ServletException, IOException {
	String productId = request.getParameter("id");
	if (StringUtils.isEmpty(productId)) {
		productId = "";
	}
	VelocityContext context = new VelocityContext();
	context.put("productId", productId);
	StringWriter writer = new StringWriter();
	details.merge(context, writer);
	String result = writer.toString();

	// Output
	response.setContentType("text/html");
	PrintWriter out = response.getWriter();
	out.println(result);
}

Full code

Full example can be found in GitHib sample-dropwizard-rest-stub repository ProductsServlet class.

Conclusion

This is pretty easy way to create web application of Dropwizard with keeping the presentation code separate from business logic.

Read more...

REST performance problems with Dropwizard and Jersey JAXB provider

Last Updated on by

Post summary: Dropwizard’s performance highly degrades when using REST with XML caused by Jersey’s Abstract JAXB provider. Solution is to inject your own JAXB provider.

Dropwizard is Java based framework for building RESTful web server in very short time. I have created short tutorial how to do so in Build a RESTful stub server with Dropwizard post.

Short overview

Current application is a Dropwizard based serving as a hub between several systems. Running on Java 7, it receives REST with XML and sends XML over REST to other services. JAXB is a framework for converting XML document to Java objects and vice versa. In order to do so JAXB needs to instantiate a context for each and every Java object. Context creation is an expensive operation.

Problem

Jersey’s Abstract JAXB provider has weak references to JAXB contexts by using WeakHashMap. This causes contexts map to be garbage collected very often and new contexts to be added again to that map. Both garbage collection and context creation are expensive operations causing 100% CPU load and very poor performance.

Solution

Solution is to create your own JAXB context provider which keeps context forever. One approach is HashMap with context created on the fly on first access of specific Java object:

import javax.ws.rs.ext.ContextResolver;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import java.util.HashMap;
import java.util.Map;

public class CustomJAXBContextProvider implements ContextResolver<JAXBContext> {
	private static final Map<Class, JAXBContext> JAXB_CONTEXT
			= new HashMap<Class, JAXBContext>();

	public JAXBContext getContext(Class<?> type) {
		try {
			JAXBContext context = JAXB_CONTEXT.get(type);
			if (context == null) {
				context = JAXBContext.newInstance(type);
				JAXB_CONTEXT.put(type, context);
			}
			return context;
		} catch (JAXBException e) {
			// Do something
			return null;
		}
	}
}

Other approach is one big context created for all the Java objects from specific packages separated with colon:

import javax.ws.rs.ext.ContextResolver;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;

public class CustomJAXBContextProvider implements ContextResolver<JAXBContext> {
	private static JAXBContext jaxbContext;

	public JAXBContext getContext(Class<?> type) {
		try {
			if (jaxbContext == null) {
				jaxbContext = JAXBContext
						.newInstance("com.acme.foo:com.acme.bar");
			}
			return jaxbContext;
		} catch (JAXBException e) {
			// Do something
			return null;
		}
	}
}

Both approaches have pros and cons. First approach has fast start up time, but first request will be slow. Second approach will have fast first request, but slow server start up time. Once JAXB context is created in Dropwizard Application class a Jersey client should be created with this context and used for REST requests:

Client client = new JerseyClientBuilder(environment)
		.using(configuration.getJerseyClientConfiguration())
		.withProvider(CustomJAXBContextProvider.class).build(getName());

Conclusion

There is no practical need to garbage collect JAXB context so it should stay as log as application lives. This is why custom JAXB provider is a good solution even there are not actual performance issues.

Read more...

Build a RESTful stub server with Dropwizard

Last Updated on by

Post summary: How to make RESTful server that can be used for stub during testing.

It might happen that you are testing a REST client against server that is not under your control. It might happen that server is not in your network, server is not very stable, has sensitive data, has changing and unstable data, etc. In such cases it might be hard to do proper automation testing. Solution to such situation is a server stub that responds to REST request in a predictable manner. This is tutorial how to do it.

Dropwizard

Dropwizard is Java framework for building RESTful web server in very short time. It has incorporated proven libraries like Jetty, Jersey, Jackson and many more to reliably do the job in shortest possible time. They have very good getting started tutorial how to make a project from scratch. I’ve used it to create project on my own. Steps are described bellow.

How to do it

  1. Create Maven project
  2. Add Dropwizard dependency
  3. Build with Maven
  4. Add configuration file
  5. Add configuration class
  6. Add data classes
  7. Add service classes
  8. Add health check
  9. Add Dropwizard application
  10. Build everything into a single JAR file
  11. Run it
  12. Test and enjoy

Create Maven project

Maven is central build repository for JARs. It makes it very easy to manage dependencies between libraries. Before getting started with Maven it should be installed. Once you do this path to Maven bin folder should be added in your Path environment variable (Windows). Once you do this open command prompt an type “mvn –version” to test if everything is configured correctly. If OK then make project with command bellow. Important in command is “groupId” this is Java package and “artifactId” this is project name:

mvn -B archetype:generate
	-DarchetypeGroupId=org.apache.maven.archetypes
	-DgroupId=com.automationrhapsody.reststub
	-DartifactId=sample-dropwizard-rest-stub

Project can be created directly from IntelliJ, but I would recommend to create it with maven to get acknowledged to it.

Build with Gradle

How to build same project with Gradle instead of Maven can be found in Build a Dropwizard project with Gradle post.

Add Dropwizard dependency

Run your favorite IDE and import already created Maven project. In this tutorial I’ll use IntelliJ. From project structure open pom.xml file. If project was created with Maven there should be <dependencies> section with junit in it. You can remove junit and add following XML instead.

<dependency>
	<groupId>io.dropwizard</groupId>
	<artifactId>dropwizard-core</artifactId>
	<version>0.8.0</version>
</dependency>

Build with Maven

Since you have created project with Maven you have it configured and know how to use it. Navigate to projects folder and run “mvn package” command. When run first time it takes a while since all dependencies are being loaded to local Maven repository.

Once build is done go to IntelliJ and refresh Maven JARs. Right click on project -> Maven (in the bottom) -> Reimport.

Add configuration file

Configurations in Dropwizard are managed with YAML. In short key value pairs are separated with colon. Child elements are indented with two spaces from their parent. Repeating items are shown with dash in front. Configuration file is with *.yml extension. Add “config.yml” file in IntelliJ project. Bellow is use configuration we are about to use in this tutorial. “version” is our custom property to illustrate working with configurations. “server” is standard Dropwizard property. With those part we set application listen port to 9000 and administration port to 9001. With “-type” is shown repetitive sequence. In current situation it is http, but there may be several protocols provided. “port” is its child key/value pair.

version: 0.0.1

# Change default server ports
server:
  applicationConnectors:
  - type: http
    port: 9000
  adminConnectors:
  - type: http
    port: 9001

Add configuration class

Once we have configuration file we need a class that will handle it. As I said “version” is our custom configuration property. In order to handle it our class should extend Configuration. Define field with getter and setter. Annotate getter and setter with @JsonProperty and you are ready to go. If more properties are needed more fields with getters and setters should be defined in class.

package com.automationrhapsody.reststub;

import com.fasterxml.jackson.annotation.JsonProperty;
import io.dropwizard.Configuration;
import org.hibernate.validator.constraints.NotEmpty;

public class RestStubConfig extends Configuration {
	@NotEmpty
	private String version;

	@JsonProperty
	public String getVersion() {
		return version;
	}

	@JsonProperty
	public void setVersion(String version) {
		this.version = version;
	}
}

Create data classes

Term in Dropwizard for those POJOs is “Representation Class” but in general ther are objects to exchange data. In our example we have Person class which has very basic attributes. It has only getters in order to be immutable. Getters are annotated with @JsonProperty  which allows Jackson to serialize and deserialize from JSON. Note that there is empty constructor which is needed for Jackson’s deserialization.

package com.automationrhapsody.reststub.data;

import com.fasterxml.jackson.annotation.JsonProperty;

public class Person {
	private int id;
	private String firstName;
	private String lastName;
	private String email;

	public Person() {
		// Needed by Jackson deserialization
	}

	public Person(int id, String firstName, String lastName, String email) {
		this.id = id;
		this.firstName = firstName;
		this.lastName = lastName;
		this.email = email;
	}

	@JsonProperty
	public int getId() {
		return id;
	}

	@JsonProperty
	public String getFirstName() {
		return firstName;
	}

	@JsonProperty
	public String getLastName() {
		return lastName;
	}

	@JsonProperty
	public String getEmail() {
		return email;
	}
}

If data to be exchanged gets too much data classes will become enormous. One solution to reduce their size is to use Lombok. See how it is done in Get rid of Getters and Setters post.

Create service

Term in Dropwizard is “Resource Class” but this actually is the RESTful service with its endpoints. @Path provides where the endpoint is. In current example I have “/person” for whole class and different paths for different operations. The result is that paths are concatenated. @GET and @POST indicate type of the request. @Timed is put for analytics purposes. @Produces and @Consumes provide type of data that is being exchanged. @PathParam indicates that “id” is part of the URL.

package com.automationrhapsody.reststub.resources;

import com.automationrhapsody.reststub.data.Person;
import com.automationrhapsody.reststub.persistence.PersonDB;
import com.codahale.metrics.annotation.Timed;

import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import java.util.List;

@Path("/person")
public class PersonService {

	public PersonService() {
	}

	@GET
	@Timed
	@Path("/get/{id}")
	@Produces(MediaType.APPLICATION_JSON)
	public Person getPerson(@PathParam("id") int id) {
		return PersonDB.getById(id);
	}

	@GET
	@Timed
	@Path("/remove")
	@Produces(MediaType.TEXT_PLAIN)
	public String removePerson() {
		PersonDB.remove();
		return "Last person remove. Total count: " + PersonDB.getCount();
	}

	@GET
	@Timed
	@Path("/all")
	@Produces(MediaType.APPLICATION_JSON)
	public List<Person> getPersons() {
		return PersonDB.getAll();
	}

	@POST
	@Timed
	@Path("/save")
	@Produces(MediaType.TEXT_PLAIN)
	@Consumes({MediaType.APPLICATION_JSON})
	public String addPerson(Person person) {
		return PersonDB.save(person);
	}
}

Service operations

Example above is about RESTful service dealing with person data. There are 4 operations exposed on following URLs:

  • /person/get/{id} – by provided person unique “id” it returns JSON with person data
  • /person/remove – removes one person on random basis
  • /person/all – returns JSON with all persons data
  • /person/save – receives JSON with person data and saves it to persons if “id” is unique, if not updating person by its id.

Business logic

It is little overrated to call it business logic but this is how we manage persons. If this was a production application you might have lots of business logic and some DB (SQL or no-SQL). Since this is just a test stub it is enough to have some data structure where to keep persons. In our case HashMap is selected. There are static methods manipulating data.

package com.automationrhapsody.reststub.persistence;

import com.automationrhapsody.reststub.data.Person;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class PersonDB {
	private static Map<Integer, Person> persons = new HashMap<Integer, Person>();

	static {
		persons.put(1, new Person(1, "FN1", "LN1", "email1@email.com"));
		persons.put(2, new Person(2, "FN2", "LN2", "email2@email.com"));
		persons.put(3, new Person(3, "FN3", "LN3", "email3@email.com"));
		persons.put(4, new Person(4, "FN4", "LN4", "email4@email.com"));
	}

	public static Person getById(int id) {
		return persons.get(id);
	}

	public static List<Person> getAll() {
		List<Person> result = new ArrayList<Person>();
		for (Integer key : persons.keySet()) {
			result.add(persons.get(key));
		}
		return result;
	}

	public static int getCount() {
		return persons.size();
	}

	public static void remove() {
		if (!persons.keySet().isEmpty()) {
			persons.remove(persons.keySet().toArray()[0]);
		}
	}

	public static String save(Person person) {
		String result = "";
		if (persons.get(person.getId()) != null) {
			result = "Updated Person with id=" + person.getId();
		} else {
			result = "Added Person with id=" + person.getId();
		}
		persons.put(person.getId(), person);
		return result;
	}
}

Create health check

Health check is smoke test that can be called from admin panel to give you information about the status of the system. In production systems you might do things like checking DB connection, checking file system or network, checking important functionality. In example here just to illustrate the functionality my health check is count of persons in memory. If it goes to 0 then something is wrong and system is not healthy. Also to illustrate how properties are used “version” is passed from configuration file to health check via its constructor.

package com.automationrhapsody.reststub;

import com.automationrhapsody.reststub.persistence.PersonDB;
import com.codahale.metrics.health.HealthCheck;

public class RestStubCheck extends HealthCheck {
	private final String version;

	public RestStubCheck(String version) {
		this.version = version;
	}

	@Override
	protected Result check() throws Exception {
		if (PersonDB.getCount() == 0) {
			return Result.unhealthy("No persons in DB! Version: " +
					this.version);
		}
		return Result.healthy("OK with version: " + this.version +
				". Persons count: " + PersonDB.getCount());
	}
}

Create application

This is the final piece. Once we have all (data, service, health check) application is binding piece that brings them together. This is execution entry point. In “main” method new application is created and its run() method is called. This is it. In order to actually work service and health check should be registered. This is done in the run method. You create instance of both service and health check. Configuration is passed in health check’s constructor.

package com.automationrhapsody.reststub;

import com.automationrhapsody.reststub.resources.BookService;
import com.automationrhapsody.reststub.resources.PersonService;
import io.dropwizard.Application;
import io.dropwizard.setup.Environment;

public class RestStubApp extends Application<RestStubConfig> {

	public static void main(String[] args) throws Exception {
		new RestStubApp().run(args);
	}

	@Override
	public void run(RestStubConfig config, Environment env) {
		final PersonService personService = new PersonService();
		env.jersey().register(personService);

		env.healthChecks().register("template", 
			new RestStubCheck(config.getVersion()));
	}
}

Build a single JAR

This was it now all have to be packed into a JAR. The strategy is to build everything into one JAR and just run it. It could not be more simple. Open pom.xml file. Add <build><plugins> … </plugins></build> in the end. Add XML bellow into this snippet. Only <mainClass> is customisable and should be changed according to your project structure.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-shade-plugin</artifactId>
	<version>1.6</version>
	<configuration>
		<createDependencyReducedPom>true</createDependencyReducedPom>
		<filters>
			<filter>
				<artifact>*:*</artifact>
				<excludes>
					<exclude>META-INF/*.SF</exclude>
					<exclude>META-INF/*.DSA</exclude>
					<exclude>META-INF/*.RSA</exclude>
				</excludes>
			</filter>
		</filters>
	</configuration>
	<executions>
		<execution>
			<phase>package</phase>
			<goals>
				<goal>shade</goal>
			</goals>
			<configuration>
				<transformers>
					<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
					<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
						<mainClass>com.automationrhapsody.reststub.RestStubApp</mainClass>
					</transformer>
				</transformers>
			</configuration>
		</execution>
	</executions>
</plugin>

Build and run

Once this is done use “mvn package” to make the JAR. Navigate to target folder in your project and run the JAR. Two arguments are needed in order to run the JAR. First is “server” which instructs Dropwizard to run as server. Second is path to *.yml configuration file.

java -jar sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar server ../config.yml

If everything is fine you should see something like code bellow which will mean server is ready.

GET     /person/all (...)
GET     /person/get/{id} (...)
GET     /person/remove (...)
POST    /person/save (...)

Test and enjoy

Once all this hard work has been done it is time to enjoy our RESTful server. List with all persons can be found on this URL: http://localhost:9000/person/all. You can get person by id: http://localhost:9000/person/get/1.

Health checks are found in admin panel: http://localhost:9001. Try removing all persons by invoking several times this URL: http://localhost:9000/person/remove

And the hardest part is to save a person. I’m using Postman Chrome plugin but you can use any REST client you want. You have to put POST data against http://localhost:9000/person/remove URL.

{"id":10,"firstName":"FN10","lastName":"LN10","email":"email10@email.com"}

And most important DO not forget to put “Content-Type: application/json” in request header. If you do not put you will get “Error 415 Unsupported Media Type” error.

PostmanRequest

Sample application can be found in GitHub sample-dropwizard-rest-stub repository. Postman requests can be downloaded from Dropwizard Postman requests link and directly imported into Postman.

Another way to test the stub is by building a client as described in Create simple REST API client using Jersey post.

Run with Docker

In Run Dropwizard application in Docker with templated configuration using environment variables post I have described how to make Dropwizard application configuration be changed with environment variables which makes it very easy to build and run in inside Docker container.

Conclusion

It could not be more easier. If you really need to stub a RESTful server this is the proper way to do it.

Read more...