Mock/Stub REST API with WireMock for better unit testing

Last Updated on by

Post summary: Examples how to use WireMock to stub (mock also is possible as a term) REST API in order make better unit testing.

Code shown in examples bellow is available in GitHub java-samples/wiremock repository.

WireMock

WireMock is a simulator for HTTP-based APIs. Some might consider it a service virtualisation tool or a mock server. It enables you to stay productive when an API you depend on doesn’t exist or isn’t complete. It supports testing of edge cases and failure modes that the real API won’t reliably produce. And because it’s fast it can reduce your build time from hours down to minutes.

When to use it

One case where WireMock is very helpful is when building a REST API client. Create simple REST API client using Jersey post describes a way to achieve this with Jersey. In most of the cases REST API might not be forced to fail with certain errors, so WireMock is excellent addition to standard functional tests to verify that client is working correctly in corner cases. Also it is mandatory for unit testing because it eliminates dependencies to external services. Mock server is extremely fast and under complete control. Other case where WireMock helps is if you need to create API tests, but API is not ready yet or not working. WireMock can be used to stub the service in order to make testing framework and structure. Once real server is ready tests will just be elaborated and details cleared up.

How to use it

WireMock is used in JUnit as a rule. More info on JUnit rules can be found in Use JUnit rules to debug failed API tests post. There is WireMockClassRule and WireMockRule. The most appropriate is the class rule, there is no need to create mock server for each and every test, also additional logic is needed for port collision avoidance. In case you use other unit testing framework there is WireMockServer which can be started before tests and stopped afterwards. Code given bellow is used to REST API client from Create simple REST API client using Jersey post. First JUnit class rule is created.

public class JerseyPersonRestClientTest {

private static final int WIREMOCK_PORT = 9999;

@ClassRule
public static final WireMockClassRule WIREMOCK_RULE
= new WireMockClassRule(WIREMOCK_PORT);

private JerseyPersonRestClient clientUnderTest;

@Before
public void setUp() throws Exception {
clientUnderTest
= new JerseyPersonRestClient("http://localhost:" + WIREMOCK_PORT);
}
}

Port should be free, otherwise there is com.github.tomakehurst.wiremock.common.FatalStartupException: java.lang.RuntimeException: java.net.BindException: Address already in use: bind exception thrown.

Usage is very simple. There are several methods which are important. Method stubFor() is initialising the stub. Method get() notifies that stub is called with HTTP GET request. Method urlMatching() uses regular expression to match which API path is invoked, then willReturn() returns aResponse() withBody(). There are several with() methods which gives variety of options for testing. Complete test is bellow:

@Test
public void testGet_WithBody_PersonJson() {
String personString = "{\"firstName\":\"FN1\",\"lastName\":\"LN1\"}";
stubFor(get(urlMatching(".*/person/get/.*"))
.willReturn(aResponse().withBody(personString)));

Person actual = clientUnderTest.get(1);

assertEquals("FN1", actual.getFirstName());
assertEquals("LN1", actual.getLastName());
}

This is very straight forward case, where client should work, but when you start to elaborate on with() scenarios you can sometimes catch a issue with code being tested. See test bellow is working correctly in case where API returns HTTP response code 500 – Internal Server Error. Client might need to add some verification on response codes as well:

@Test
public void testGet_WithStatus() {
String personString = "{\"firstName\":\"FN1\",\"lastName\":\"LN1\"}";
stubFor(get(GET_URL)
.willReturn(aResponse()
.withStatus(500)
.withBody(personString)));

Person actual = clientUnderTest.get(1);

assertEquals("FN1", actual.getFirstName());
assertEquals("LN1", actual.getLastName());
}

Difference between stub and mock

In Mock JUnit tests with Mockito example post I’ve shown how yo use Mockito to mock given classes and control their behaviour in order to control and eliminate dependencies. Mockito is not suitable for this particular case because if you mock JerseyPersonRestClient‘s get() method it will just return an object, there is no testing whatsoever. Stubbing with WireMock on other hand tests all code for invoking the service, getting a response (controlled by you) and deserialising this response from network stream to Java object. It is much more adequate and close to reality testing.

Conclusion

WireMock is very powerful framework for API stubbing in order to make your test better and it is a must for unit testing some REST API client.

Read more...

Create simple REST API client using Jersey

Last Updated on by

Post summary: Code examples how to create REST API client using Jersey.

In current port I will give code examples how to build REST API client using Jersey. Code shown in examples bellow is available in GitHub java-samples/wiremock repository.

REST API client

REST API client is needed when you want to consume given REST API, either for production usage or for testing this API. In the latter case client does not need to be very sophisticated since it is used just for testing the API with Java code. In current post I will show how to create REST API client for Persons functionality of Dropwizard Rest Stub described in Build a RESTful stub server with Dropwizard post.

Jersey 2 and Jackson

Jersey is a framework which allows easier building of RESTful services. It is one of the most used such framework nowadays. Jackson is JSON parser for Java. It is also one of the most used ones. First step is to import libraries you are going to use:

<dependency>
	<groupId>org.glassfish.jersey.core</groupId>
	<artifactId>jersey-client</artifactId>
	<version>2.25.1</version>
</dependency>
<dependency>
	<groupId>org.glassfish.jersey.media</groupId>
	<artifactId>jersey-media-json-jackson</artifactId>
	<version>2.25.1</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-databind</artifactId>
	<version>2.8.7</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.core</groupId>
	<artifactId>jackson-annotations</artifactId>
	<version>2.8.7</version>
</dependency>

Not mandatory but it is good practice to create an interface for this client and then do implementations. The idea is you may have different implementations and switch between them.

import com.automationrhapsody.wiremock.model.Person;

import java.util.List;

public interface PersonRestClient {

	List<Person> getAll();

	Person get(int id);

	String save(Person person);

	String remove();
}

Now you can start with implementation. In given example constructor take the host which also includes port and scheme. Then it creates ClientConfig object with specific properties. Full list is shown in ClientProperties javadoc. In the example I set up timeouts only. Next is to create WebTarget object to query different API endpoints. It could not be simpler than that:

import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.WebTarget;

import org.glassfish.jersey.client.ClientConfig;
import org.glassfish.jersey.client.ClientProperties;
import org.glassfish.jersey.filter.LoggingFilter;

public class JerseyPersonRestClient implements PersonRestClient {

	private final WebTarget webTarget;

	public JerseyPersonRestClient(String host) {
		ClientConfig clientConfig = new ClientConfig()
				.property(ClientProperties.READ_TIMEOUT, 30000)
				.property(ClientProperties.CONNECT_TIMEOUT, 5000);

		webTarget = ClientBuilder
				.newClient(clientConfig)
				.register(new LoggingFilter())
				.target(host);
	}
}

Once WebTarget is instantiated it will be used to query all the endpoints. I will show implementation of one GET and one POST endpoints consumption:

@Override
public List<Person> getAll() {
	Person[] persons = webTarget
		.path(ENDPOINT_GET_ALL)
		.request()
		.get()
		.readEntity(Person[].class);
	return Arrays.stream(persons).collect(Collectors.toList());
}

@Override
public String save(Person person) {
	if (person == null) {
		throw new RuntimeException("Person entity should not be null");
	}
	return webTarget
		.path(ENDPOINT_SAVE)
		.request()
		.post(Entity.json(person))
		.readEntity(String.class);
}

Full code can be seen in GitHub repo: JerseyPersonRestClient full code.

jersey-media-json-jackson

This is the bonding betheww Jersey and Jackson. It should be used otherwise Jersey’s readEntity(Class var1) method throws:

Exception in thread “main” org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyReader not found for media type=application/json, type=class …

or

Exception in thread “main” org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyWriter not found for media type=application/json, type=class …

Client builder

In code there is class called PersonRestClientBuilder. In current case it does not do many things, but in reality it might turn out that a lot of configurations or input is provided to build a REST API client instance. This is where such builder becomes very useful:

public class PersonRestClientBuilder {

	private String host;

	public PersonRestClientBuilder setHost(String host) {
		this.host = host;
		return this;
	}

	public PersonRestClient build() {
		return new JerseyPersonRestClient(host);
	}
}

Unit testing

It is common and best practice that each piece of code is covered by unit tests. In Mock JUnit tests with Mockito example post I’ve described how Mockito can be used. Problem in current example is if we use Mockito we have to mock readEntity() method to return some response objects. This is way too much mocking and will not do adequate testing, actually it does not testing at all. We want to test that out REST API client successfully communicates over the wire. In order to do proper testing we need to use library called WireMock. In subsequent post I will add more details how to use it.

Conclusion

REST API consuming or testing requires building a client. Jersey is perfect candidate to be used as underlying framework. WireMock can be used for unit testing the REST API client you have created.

Read more...

Introduction to Postman with examples

Last Updated on by

Post summary: This post is demonstrating different Postman features with examples.

All examples shown in this post are available at Postman Examples link and can be imported in Postman. Environments are also used in attached examples and are available at Admin environment and User environment. In order to run all the examples you need to download and run Dropwizard stub described in Build a RESTful stub server with Dropwizard post and available in sample-dropwizard-rest-stub GitHub repo, otherwise you can just see the Postman code and screenshots.

Postman

Postman is a Chrome add-on and Mac application which is used to fire requests to an API. It is very lightweight and fast. Requests can be organised in groups, also tests can be created with verifications for certain conditions on the response. With its features it is very good and convenient API tool. It is possible to make different kinds of HTTP requests – GET, POST, PUT, PATCH and DELETE. It is possible to add headers in the requests. In current post I will write about more interesting features it has: Variables, Pre-Request Script, Environments and Tests.

postman-main

Variables

There are two types of variables – global and environment. Global variables are for all requests, environment variables are defined per specific environment which can selected from a drop-down or no environment can be selected. Environments will be discussed in details later in current port. Global variables are editable by small eye-shaped icon in the top right corner. Once defined variables can be used in request with format surrounded by curly brackets: {{VARIABLE_NAME}}.

postman-globals

Pre-Request Script

Postman allows users to do some JavaScript coding with which to manipulate the data being sent with request. One such example is when testing and API with security as explained in How to implement secure REST API authentication over HTTP post – SHA256 hash (build from apiKey + secretKey + timestamp in seconds) is sent as request parameter with the request. Calculating SHA256 hash is done with following pre-request script and then stored as global variable token.

var timeInSeconds = parseInt((new Date()).getTime() / 1000);
var sigString = postman.getGlobalVariable("apiKey") + 
	postman.getGlobalVariable("secretKey") + timeInSeconds
var token = CryptoJS.SHA256(sigString);
postman.setGlobalVariable("token", token);

Here CryptoJS library is used to create SHA256 hash. All available libraries in Postman are described in Postman Testing Sandbox page. Global variable {{token}} is then send as token request parameter.

postman-pre-request-script

Environments

Code shown above is working fine with just one set of credentials because they are stored as global variables. If you need to switch between different credentials this is where environments come in play. By switching environment and with no change in the request you can send different parameters to API. Environments are managed from Settings icon in the top right corner which opens menu with “Manage Environments” link.

postman-environments

Postman supports so called shared environments, which means whole team can use one and the same credentials managed centrally. It requires sign in and some plan though, but might be good investment in the long run.

In order to use environments pre-request script has to be changed to:

var timeInSeconds = parseInt((new Date()).getTime() / 1000);
var sigString = environment.apiKey + environment.secretKey + timeInSeconds
var token = CryptoJS.SHA256(sigString);
postman.setEnvironmentVariable("token", token);

Both apiKey and secretKey are read from environment. Environment can be changed from top right corner.

Nota bene: There is specific behaviour (I would not call it bug as it makes sense) in Postman. If select “No Environment” and fire request above Postman will automatically create environment with name “No Environment”. This is because it actually needs an environment to store variable into. This might be very confusing first time.

Post-Request Script

There is no such term defined in Postman. The idea is that in many cases you will need to do something with the response and extract variable from it in order to use it at later stage. This can be done in “Tests” tab. Example given bellow is to take all persons with API call and then to process response and at random select one id which is stored as global variable and then used in next request. You can put whatever JavaScript code you like in order to fulfil your logic.

var jsonData = JSON.parse(responseBody)
var size = jsonData.length
var index = parseInt(Math.random() * size)
postman.setGlobalVariable("userId", index);

postman-post-request

Then in subsequent request you can use GET call to URL: http://localhost:9000/person/get/{{userId}}

Tests

After response is received Postman has functionality to make verifications on it. This is done in “Tests” tab. Bellow is example on different verifications. Most interesting part is in case of JSON response it can be parsed to an array and then elements accessed by index and value jsonData[0].id or even iterated as shown bellow. Format is: tests[“TEST_NAME”] = BOOLEAN_CONDITION.

tests["Status code is 200"] = responseCode.code === 200;

tests["Response time is less than 200ms"] = responseTime < 200;

var expected = "email1@email.na"
tests["Body cointains string: " + expected] = responseBody.has(expected);

var jsonData = JSON.parse(responseBody);
var expectedCount = 4
tests["Response count is: " + expectedCount] = jsonData.length === expectedCount;

for(var i=1; i<=expectedCount; i++) {
	tests["Verify id is: " + i] = jsonData[i-1].id === i;
}

postman-test-response

postman-test-results

Nota bene: if you use responseTime verification you have to know that it measures just the TTFB (time to first bite) it does not measure time needed to transfer the data. If you have API with big responses or network is slow you may fire the request, wait a lot and then Postman shows very small response time which might be confusing.

Run from command line

In order to run Postman tests in command line as part of some CI process there is separate tool called Newman. It requires NodeJS to be installed and runs on NodeJS environment. It is very well described in How to write powerful automated API tests with Postman, Newman and Jenkins.

Code reuse between requests

It is very convenient some piece of code to be re-used between request to prevent copy/paste it. Postman does not support yet code re-use between requests. Good thing is that there is workaround for this. It is possible to do it by defining a helper function with verifications which is saved as global variable in first request from your test scenario:

postman.setGlobalVariable("loadHelpers", function loadHelpers() {
	let helpers = {};

	helpers.verifyCount = function verifyCount(expectedCount) {
		var jsonData = JSON.parse(responseBody);
		tests["Response count is: " + expectedCount] 
			= jsonData.length === expectedCount;
	}

	// ...additional helpers

	return helpers;
} + '; loadHelpers();');

Then from other requests helpers are taken from global variables and verification functions can be used:

var helpers = eval(globals.loadHelpers);
helpers.verifyCount(4);

See more in Reusing pre-request scripts across requests in a collection issue thread.

Conclusion

Postman is very nice tool to use when develop your API or manual test it. I would definitely recommend Postman, I use it on daily basis for probing the API. For serious API functional tests automation I would say Postman is not ready yet and you’d better go for another approach. Good thing is that there is big community around it which is growing and new features are added.

Read more...

JSON format to register service with Eureka

Last Updated on by

Post summary: What JSON data is needed to register service node with Eureka server.

Eureka is a REST based service that is primarily used in the AWS cloud for locating services for the purpose of load balancing and failover of middle-tier servers.

Jersey 1 vs Jersey 2

As of now Eureka client works only with Jersey 1. There is PR to create Jersey 2 client, but by the time this post is created it is still not developed. Jersey 1 and Jersey 2 are mutually exclusive. If your product works with Jersey 2 then in order to register with Eureka you have to write your own client.

Registering with Eureka

Eureka documentation is giving example how to register a server or service node with Eureka server. Given example is an XML one and there is no JSON example. The XML example cannot be straight-forward converted for JSON because JSON does not support attributes as XML does.

Solution

Use following JSON in order to register with Eureka server with custom REST client:

{
	"instance": {
		"hostName": "WKS-SOF-L011",
		"app": "com.automationrhapsody.eureka.app",
		"vipAddress": "com.automationrhapsody.eureka.app",
		"secureVipAddress": "com.automationrhapsody.eureka.app"
		"ipAddr": "10.0.0.10",
		"status": "STARTING",
		"port": {"$": "8080", "@enabled": "true"},
		"securePort": {"$": "8443", "@enabled": "true"},
		"healthCheckUrl": "http://WKS-SOF-L011:8080/healthcheck",
		"statusPageUrl": "http://WKS-SOF-L011:8080/status",
		"homePageUrl": "http://WKS-SOF-L011:8080",
		"dataCenterInfo": {
			"@class": "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo", 
			"name": "MyOwn"
		},
	}
}

Implementation details

Important part which is not clear enough in standard documentation is: “securePort”: {“$”: “8443”, “@enabled”: “true”}. Note that “securePort”: “8443” would also work, but this will just set the port number without enabling it. By default secure port is disabled unless register call enables it.

I would recommend to use one and the same for “vipAddress” and “secureVipAddress”. When searching for HTTP endpoint Eureka server uses “vipAddress” and if you switch your client to search for HTTPS port Eureka server will now search for “secureVipAddress”. If both are different this may lead to confusion why no endpoint is returned although there is one with HTTPS enabled.

Implementation is needed for DataCenterInfo interface. One such implementation is:

private static final class DefaultDataCenterInfo implements DataCenterInfo {
	private final Name name;

	private DefaultDataCenterInfo(Name name) {
		this.name = name;
	}

	@Override
	public Name getName() {
		return name;
	}

	public static DataCenterInfo myOwn() {
		return new DefaultDataCenterInfo(Name.MyOwn);
	}
}

As seen in JSON example application status is STARTING. This is good practice to keep STARTING state until application is fully started. This will prevent current not yet ready node to be returned for usage by Eureka server. Once application is fully started then with subsequent call you can change the status to UP. This is done with REST PUT call to /eureka/v2/apps/appID/instanceID/status?value=UP. InstanceID is basically the hostname. AppID is the one registered with “app” in JSON. Also good idea is to have “app” same as “vipAddress” to minimise confusion.

Conclusion

Eureka is a really nice tool. It has a default client which can be used out of the box. Problem is that this client uses Jersey 1. If you need Jersey 2 client by the time of this post you have to make it on your own. This post gives basic direction how to do this. Since official documentation is lacking details how to resister a node with JSON this port gives more clarity. The most important part is: “securePort”: {“$”: “8443”, “@enabled”: “true”}.

Read more...

Unmarshal/Convert JSON data to JAXBElement object

Last Updated on by

Post summary: How to marshal and unmarshal JAXBElement to JSON with Jackson.

This post gives solution for following usecase

Usecase

XML document -> POJO containing JAXBElement -> JSON -> POJO containing JAXBElement.

For some reason there is a POJO which has some JAXBElement. This usually happens when mixing SOAP and REST services with XML and JSON. This POJO is easily converted to JSON data. Then from this JSON data a POJO containing JAXBElement has to be unmarshalled.

Problem

By default Jackson’s ObjectMapper is unable to unmarshall JSON data into a JAXBElement object. An exception is thrown:

No suitable constructor found for type [simple type, class javax.xml.bind.JAXBElement]: can not instantiate from JSON object (missing default constructor or creator, or perhaps need to add/enable type information?)

Solution

Although somewhere it is recommended to use com.fasterxml.jackson.module.jaxb.JaxbAnnotationModule it might not work. The solution is to create custom MixIn and register it with ObjectMapper. MixIn class is:

import javax.xml.bind.JAXBElement;
import javax.xml.namespace.QName;

@JsonIgnoreProperties(value = {"globalScope", "typeSubstituted", "nil"})
public abstract class JAXBElementMixIn<T> {

	@JsonCreator
	public JAXBElementMixIn(@JsonProperty("name") QName name,
			@JsonProperty("declaredType") Class<T> declaredType,
			@JsonProperty("scope") Class scope,
			@JsonProperty("value") T value) {
	}
}

ObjectMapper is instantiated with following code:

import com.fasterxml.jackson.databind.ObjectMapper;

ObjectMapper objectMapper = new ObjectMapper();
objectMapper.addMixIn(JAXBElement.class, JAXBElementMixIn.class);

Conclusion

Jackson’s ObjectMapper does not support JSON to JAXBElement conversion by default. This is solved by creating a custom MixIn as described in current post and register it with ObjectMapper.

Read more...

Implement secure API authentication over HTTP with Dropwizard

Last Updated on by

Post summary: Reference implementation on suggested in How to implement secure REST API authentication over HTTP post authentication mechanism.

API authentication mechanism

Suggested authentication mechanism consists of following steps:

  • Secret key that is known only by API consumer and API provider is needed along with API key.
  • Secret key is used to one way hash a token which is send to server along with API key in the API call.
  • Token consists of: API key + Secret key + Current time in seconds, which then gets hashed with SHA-256 algorithm preferably.
  • Server recreates all the tokens locally for every second for some time in the future, preferably not too long – 30~120 seconds.
  • Server recreates all the tokens for 30~120 seconds in the past, to take into account the time needed for request to reach the server.
  • Server compares each of the tokens with received one.
  • If there is match consumer is authenticated and response is returned.

Dropwizard implementation

Dropwizard stub introduced in Build a RESTful stub server with Dropwizard post will be used to create authentication. Full example can be found in GitHib sample-dropwizard-rest-stub repository. Implementation consists of following steps:

  • Implement javax.ws.rs.container.ContainerRequestFilter interface. Implementation will inspect every request and verify authentication.
  • Create custom annotation
  • Annotate RequestFilter and Dropwizard resource (API service) on which authentication should be applied.
  • Register RequestFilter implementation class into Dropwizard Jersey environment.

Create custom annotation

Starting with the easiest step. Creating custom annotation is pretty easy. It could be applied to a class (ElementType.TYPE) or to a method (ElementType.METHOD). It should live as long as program runs (RetentionPolicy.RUNTIME). In order to make it possible annotated request filter to be applied on specific resource only @NameBinding annotation is a must in Jersey. If not specified request filter will apply on all resources. Needed annotation is:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.ws.rs.NameBinding;

@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(value = RetentionPolicy.RUNTIME)
@NameBinding
public @interface Authenticator {
}

ContainerRequestFilter implementation

Container request filter is applied on incoming requests. If used with @NameBinding annotation it is applied only where needed, if not it is applied globally. Mandatory is to override filter() method:

import com.automationrhapsody.reststub.persistence.AuthDB;

import java.io.IOException;
import java.util.List;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;

@Authenticator
public class AuthenticateFilter implements ContainerRequestFilter {

	private static final String PARAM_API_KEY = "apiKey";
	private static final String PARAM_TOKEN = "token";
	private static final long SECONDS_IN_MILLISECOND = 1000L;
	private static final int TTL_SECONDS = 60;

	@Override
	public void filter(ContainerRequestContext context) throws IOException {
		final String apiKey = extractParam(context, PARAM_API_KEY);
		if (StringUtils.isEmpty(apiKey)) {
			context.abortWith(responseMissingParameter(PARAM_API_KEY));
		}

		final String token = extractParam(context, PARAM_TOKEN);
		if (StringUtils.isEmpty(token)) {
			context.abortWith(responseMissingParameter(PARAM_TOKEN));
		}

		if (!authenticate(apiKey, token)) {
			context.abortWith(responseUnauthorized());
		}
	}
}

As seen above two GET parameters are mandatory in the request: “apiKey” and “token”. Those are first extracted and verified. If some of them is not existing BAD_REQUEST (HTTP Status code 400) Response is returned with error message. Methods that extract params and build error response are:

private String extractParam(ContainerRequestContext context, String param) {
	final UriInfo uriInfo = context.getUriInfo();
	final List user = uriInfo.getQueryParameters().get(param);
	return CollectionUtils.isEmpty(user) ? null : String.valueOf(user.get(0));
}

private Response responseMissingParameter(String name) {
	return Response.status(Response.Status.BAD_REQUEST)
		.type(MediaType.TEXT_PLAIN_TYPE)
		.entity("Parameter '" + name + "' is required.")
		.build();
}

If both are present then code tried to authenticate the call by rebuilding all the hashes for 60 seconds in the past because request cannot arrive instantly it takes some time. If network is slower this time can be increased. It also rebuilds all hashes for 60 seconds in the future, this is token time to live. Server has access to Secret key for any given API key. In example above they are stored in fake DB provider and obtained by AuthDB.getSecretKey(apiKey):

private boolean authenticate(String apiKey, String token) {
	final String secretKey = AuthDB.getSecretKey(apiKey);

	// No need to calculate digest in case of wrong apiKey
	if (StringUtils.isEmpty(secretKey)) {
		return false;
	}

	final long nowSec = System.currentTimeMillis() / SECONDS_IN_MILLISECOND;
	long startTime = nowSec - TTL_SECONDS;
	long endTime = nowSec + TTL_SECONDS;
	for (; startTime < endTime; startTime++) {
		final String toHash = apiKey + secretKey + startTime;
		final String sha1 = DigestUtils.sha256Hex(toHash);
		if (sha1.equals(token)) {
			return true;
		}
	}

	return false;
}

As seen above server uses SHA-256 cryptographic algorithm. It is the best solution in terms of speed and security. In MD5, SHA-1, SHA-256 and SHA-512 speed performance post a comparison between MD5, SHA-1, SHA-256 and SHA-512 is made. If authentication cannot be verified then UNAUTHORIZED (HTTP Status code 401) Response response is returned:

private Response responseUnauthorized() {
	return Response.status(Response.Status.UNAUTHORIZED)
		.type(MediaType.TEXT_PLAIN_TYPE)
		.entity("Unauthorized")
		.build();
}

This is the hardest part. Now this filter has to be registered with Jersey and applied to needed resources (services). See more on ContainerRequestFilter interface and @NameBinding annotation in Jersey filters and interceptors page.

Apply authentication filter on a resource

Indicating that given resource should be checked for authentication is done with custom @Authenticator annotation created previously. If needed just for specific API call it can be applied also on a method level:

import com.automationrhapsody.reststub.data.Book;
import com.automationrhapsody.reststub.filters.Authenticator;
import com.automationrhapsody.reststub.persistence.BookDB;
import com.codahale.metrics.annotation.Timed;

import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Authenticator
@Path("/secure/books")
public class BooksSecureService {

	@GET
	@Timed
	@Produces(MediaType.APPLICATION_JSON)
	public List<Book> getBooks() {
		return BookDB.getAll();
	}
}

Register in Dropwizard Jersey

Last step is to register the request filter and resource with Dropwizard’s Jersey:

@Override
public void run(RestStubConfig config, Environment env) {

	env.jersey().register(BooksSecureService.class);
	env.jersey().register(AuthenticateFilter.class);

}

Conclusion

Very easy to implement in Dropwizard and relatively secure way to provide API authentication over HTTP protocol. For mission critical application definitely more strict consideration and review of this authentication mechanism is needed.

Read more...

How to implement secure REST API authentication over HTTP

Last Updated on by

Post summary: How to implement secure API authentication even over HTTP.

Important: this post is not a complete and expert guide on API security. It is mainly done to test Postman Pre-request hook that is described in Introduction to Postman with examples post. It does not go into all the details about API security, SSL certificates, encrypting the data, etc. It gives basic information how you can protect your API’s consumers against their network traffic being sniffed and credentials, apiKeys, session keys, etc stolen.

Authentication vs. Authorisation

Authentication is defined as “Who you are”. It deals with usernames and password. Authorisation is defined as “What you can do”. It deals with permissions. Before dealing with permissions application must know the user, so Authorisation comes after Authentication.

Basic Authentication

As it is stated it is very basic. The idea is to sent Base64 encoded username and password in the header of the request in following format:

Authorization: Basic dXNlcjpwYXNz

Server decodes the username and password and use them to authenticate and authorise the user. Problem with Basic authentication is it must be used only over HTTPS, since network traffic is encrypted. Over HTTP request can be easily sniffed. Base64 is reversible and there are numerous tools on the web where you can put dXNlcjpwYXNz and they will return user:pass as plain text.

Nota bene: Never use this one without HTTPS.

OAuth and OAuth 2.0

OAuth is authorisation protocol. It is intended mainly for web, but can be used in API authorisation. The idea is that authentication and authorisation is done by third party like Microsoft, Google, Facebook, Twitter, etc. This is easy for API as it does not have to deal with user data. Customer logins to third party and access token is being issued. Token has some validity which is not too long, but not too short, usually 1 or 2 days. The API can obtain user details from third party by this token. User authenticates itself to the API with this access token by sending it in the request header:

Authorization: Bearer 66408bd9-2bc0-40c3-9823-e9bec390532a

Problem with OAuth is it also must be used over HTTPS. Over HTTP traffic can be sniffed and token can be stolen. Although token has some expiry time, it is long enough for a hacker to use API from your behalf.

Nota bene: Never use this one without HTTPS.

API keys

API keys has become the standard when consuming an API. API key is some random hash which uniquely identifies the consumer. API keys have numerous benefits over username/password mechanism. Again in case of HTTP network traffic can be sniffed and API key stolen.

HTTPS

Reading post so far turned out there is not a single API authentication protocol that is secure if not used over HTTPS. In current a solution is proposed. It is commonly used in public APIs, it is possible to exist as a standard I’m just not aware of its name, which provides secure API authentication even over HTTP.

Implement API security over HTTP

In short in order to have security over HTTP following steps should be done:

  • Secret key that is known only by API consumer and API provider is needed along with API key.
  • Secret key is used to one way hash a token which is send to server along with API key in the API call.
  • Token consists of: API key + Secret key + Current time in seconds, which then gets hashed with SHA-256 algorithm preferably.
  • Server recreates all the tokens locally for every second for some time in the future, preferably not too long – 30~120 seconds.
  • Server recreates all the tokens for 30~120 seconds in the past, to take into account the time needed for request to reach the server.
  • Server compares each of the tokens with received one.
  • If there is match consumer is authenticated and response is returned.

Cryptographic hash algorithms

Most used hash algorithms nowadays are: MD5, SHA-1, SHA-256, SHA-512. MD5 and SHA-1 are to week and are not recommended. SHA-512 takes more time to compute the hashes. SHA-256 is the most appropriate solution in terms of security and speed. In MD5, SHA-1, SHA-256 and SHA-512 speed performance post all 4 algorithms have been tested and compared with Apache’s Commons Codec implementation.

Hash with Salt

Time stamp into hashed token is used for so called salt, an random data that is used to differentiate the hashed data against dictionary attacks. If just API key + Secret key are hashed, then hash will always be one and the same. Intruder will take the hash and just use it. Using time stamp makes the hash always different. Another function of time stamp is to set expiration time on the token, so even if stolen not to be used for a long period of time.

Conclusion

There are already established standards to secure an API, but all of them are effective only over HTTPS. In current post is given a proposal for secure API authentication which is very simple and relatively safe even over HTTP. Cons of this method is server has to recalculate hash many times, which in massive load would require some caching. In Implement secure API authentication over HTTP with Dropwizard post there is reference implementation of proposed solution with Dropwizard.

Read more...

REST performance problems with Dropwizard and Jersey JAXB provider

Last Updated on by

Post summary: Dropwizard’s performance highly degrades when using REST with XML caused by Jersey’s Abstract JAXB provider. Solution is to inject your own JAXB provider.

Dropwizard is Java based framework for building RESTful web server in very short time. I have created short tutorial how to do so in Build a RESTful stub server with Dropwizard post.

Short overview

Current application is a Dropwizard based serving as a hub between several systems. Running on Java 7, it receives REST with XML and sends XML over REST to other services. JAXB is a framework for converting XML document to Java objects and vice versa. In order to do so JAXB needs to instantiate a context for each and every Java object. Context creation is an expensive operation.

Problem

Jersey’s Abstract JAXB provider has weak references to JAXB contexts by using WeakHashMap. This causes contexts map to be garbage collected very often and new contexts to be added again to that map. Both garbage collection and context creation are expensive operations causing 100% CPU load and very poor performance.

Solution

Solution is to create your own JAXB context provider which keeps context forever. One approach is HashMap with context created on the fly on first access of specific Java object:

import javax.ws.rs.ext.ContextResolver;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import java.util.HashMap;
import java.util.Map;

public class CustomJAXBContextProvider implements ContextResolver<JAXBContext> {
	private static final Map<Class, JAXBContext> JAXB_CONTEXT
			= new HashMap<Class, JAXBContext>();

	public JAXBContext getContext(Class<?> type) {
		try {
			JAXBContext context = JAXB_CONTEXT.get(type);
			if (context == null) {
				context = JAXBContext.newInstance(type);
				JAXB_CONTEXT.put(type, context);
			}
			return context;
		} catch (JAXBException e) {
			// Do something
			return null;
		}
	}
}

Other approach is one big context created for all the Java objects from specific packages separated with colon:

import javax.ws.rs.ext.ContextResolver;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;

public class CustomJAXBContextProvider implements ContextResolver<JAXBContext> {
	private static JAXBContext jaxbContext;

	public JAXBContext getContext(Class<?> type) {
		try {
			if (jaxbContext == null) {
				jaxbContext = JAXBContext
						.newInstance("com.acme.foo:com.acme.bar");
			}
			return jaxbContext;
		} catch (JAXBException e) {
			// Do something
			return null;
		}
	}
}

Both approaches have pros and cons. First approach has fast start up time, but first request will be slow. Second approach will have fast first request, but slow server start up time. Once JAXB context is created in Dropwizard Application class a Jersey client should be created with this context and used for REST requests:

Client client = new JerseyClientBuilder(environment)
		.using(configuration.getJerseyClientConfiguration())
		.withProvider(CustomJAXBContextProvider.class).build(getName());

Conclusion

There is no practical need to garbage collect JAXB context so it should stay as log as application lives. This is why custom JAXB provider is a good solution even there are not actual performance issues.

Read more...

Build a RESTful stub server with Dropwizard

Last Updated on by

Post summary: How to make RESTful server that can be used for stub during testing.

It might happen that you are testing a REST client against server that is not under your control. It might happen that server is not in your network, server is not very stable, has sensitive data, has changing and unstable data, etc. In such cases it might be hard to do proper automation testing. Solution to such situation is a server stub that responds to REST request in a predictable manner. This is tutorial how to do it.

Dropwizard

Dropwizard is Java framework for building RESTful web server in very short time. It has incorporated proven libraries like Jetty, Jersey, Jackson and many more to reliably do the job in shortest possible time. They have very good getting started tutorial how to make a project from scratch. I’ve used it to create project on my own. Steps are described bellow.

How to do it

  1. Create Maven project
  2. Add Dropwizard dependency
  3. Build with Maven
  4. Add configuration file
  5. Add configuration class
  6. Add data classes
  7. Add service classes
  8. Add health check
  9. Add Dropwizard application
  10. Build everything into a single JAR file
  11. Run it
  12. Test and enjoy

Create Maven project

Maven is central build repository for JARs. It makes it very easy to manage dependencies between libraries. Before getting started with Maven it should be installed. Once you do this path to Maven bin folder should be added in your Path environment variable (Windows). Once you do this open command prompt an type “mvn –version” to test if everything is configured correctly. If OK then make project with command bellow. Important in command is “groupId” this is Java package and “artifactId” this is project name:

mvn -B archetype:generate
	-DarchetypeGroupId=org.apache.maven.archetypes
	-DgroupId=com.automationrhapsody.reststub
	-DartifactId=sample-dropwizard-rest-stub

Project can be created directly from IntelliJ, but I would recommend to create it with maven to get acknowledged to it.

Build with Gradle

How to build same project with Gradle instead of Maven can be found in Build a Dropwizard project with Gradle post.

Add Dropwizard dependency

Run your favorite IDE and import already created Maven project. In this tutorial I’ll use IntelliJ. From project structure open pom.xml file. If project was created with Maven there should be <dependencies> section with junit in it. You can remove junit and add following XML instead.

<dependency>
	<groupId>io.dropwizard</groupId>
	<artifactId>dropwizard-core</artifactId>
	<version>0.8.0</version>
</dependency>

Build with Maven

Since you have created project with Maven you have it configured and know how to use it. Navigate to projects folder and run “mvn package” command. When run first time it takes a while since all dependencies are being loaded to local Maven repository.

Once build is done go to IntelliJ and refresh Maven JARs. Right click on project -> Maven (in the bottom) -> Reimport.

Add configuration file

Configurations in Dropwizard are managed with YAML. In short key value pairs are separated with colon. Child elements are indented with two spaces from their parent. Repeating items are shown with dash in front. Configuration file is with *.yml extension. Add “config.yml” file in IntelliJ project. Bellow is use configuration we are about to use in this tutorial. “version” is our custom property to illustrate working with configurations. “server” is standard Dropwizard property. With those part we set application listen port to 9000 and administration port to 9001. With “-type” is shown repetitive sequence. In current situation it is http, but there may be several protocols provided. “port” is its child key/value pair.

version: 0.0.1

# Change default server ports
server:
  applicationConnectors:
  - type: http
    port: 9000
  adminConnectors:
  - type: http
    port: 9001

Add configuration class

Once we have configuration file we need a class that will handle it. As I said “version” is our custom configuration property. In order to handle it our class should extend Configuration. Define field with getter and setter. Annotate getter and setter with @JsonProperty and you are ready to go. If more properties are needed more fields with getters and setters should be defined in class.

package com.automationrhapsody.reststub;

import com.fasterxml.jackson.annotation.JsonProperty;
import io.dropwizard.Configuration;
import org.hibernate.validator.constraints.NotEmpty;

public class RestStubConfig extends Configuration {
	@NotEmpty
	private String version;

	@JsonProperty
	public String getVersion() {
		return version;
	}

	@JsonProperty
	public void setVersion(String version) {
		this.version = version;
	}
}

Create data classes

Term in Dropwizard for those POJOs is “Representation Class” but in general ther are objects to exchange data. In our example we have Person class which has very basic attributes. It has only getters in order to be immutable. Getters are annotated with @JsonProperty  which allows Jackson to serialize and deserialize from JSON. Note that there is empty constructor which is needed for Jackson’s deserialization.

package com.automationrhapsody.reststub.data;

import com.fasterxml.jackson.annotation.JsonProperty;

public class Person {
	private int id;
	private String firstName;
	private String lastName;
	private String email;

	public Person() {
		// Needed by Jackson deserialization
	}

	public Person(int id, String firstName, String lastName, String email) {
		this.id = id;
		this.firstName = firstName;
		this.lastName = lastName;
		this.email = email;
	}

	@JsonProperty
	public int getId() {
		return id;
	}

	@JsonProperty
	public String getFirstName() {
		return firstName;
	}

	@JsonProperty
	public String getLastName() {
		return lastName;
	}

	@JsonProperty
	public String getEmail() {
		return email;
	}
}

If data to be exchanged gets too much data classes will become enormous. One solution to reduce their size is to use Lombok. See how it is done in Get rid of Getters and Setters post.

Create service

Term in Dropwizard is “Resource Class” but this actually is the RESTful service with its endpoints. @Path provides where the endpoint is. In current example I have “/person” for whole class and different paths for different operations. The result is that paths are concatenated. @GET and @POST indicate type of the request. @Timed is put for analytics purposes. @Produces and @Consumes provide type of data that is being exchanged. @PathParam indicates that “id” is part of the URL.

package com.automationrhapsody.reststub.resources;

import com.automationrhapsody.reststub.data.Person;
import com.automationrhapsody.reststub.persistence.PersonDB;
import com.codahale.metrics.annotation.Timed;

import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import java.util.List;

@Path("/person")
public class PersonService {

	public PersonService() {
	}

	@GET
	@Timed
	@Path("/get/{id}")
	@Produces(MediaType.APPLICATION_JSON)
	public Person getPerson(@PathParam("id") int id) {
		return PersonDB.getById(id);
	}

	@GET
	@Timed
	@Path("/remove")
	@Produces(MediaType.TEXT_PLAIN)
	public String removePerson() {
		PersonDB.remove();
		return "Last person remove. Total count: " + PersonDB.getCount();
	}

	@GET
	@Timed
	@Path("/all")
	@Produces(MediaType.APPLICATION_JSON)
	public List<Person> getPersons() {
		return PersonDB.getAll();
	}

	@POST
	@Timed
	@Path("/save")
	@Produces(MediaType.TEXT_PLAIN)
	@Consumes({MediaType.APPLICATION_JSON})
	public String addPerson(Person person) {
		return PersonDB.save(person);
	}
}

Service operations

Example above is about RESTful service dealing with person data. There are 4 operations exposed on following URLs:

  • /person/get/{id} – by provided person unique “id” it returns JSON with person data
  • /person/remove – removes one person on random basis
  • /person/all – returns JSON with all persons data
  • /person/save – receives JSON with person data and saves it to persons if “id” is unique, if not updating person by its id.

Business logic

It is little overrated to call it business logic but this is how we manage persons. If this was a production application you might have lots of business logic and some DB (SQL or no-SQL). Since this is just a test stub it is enough to have some data structure where to keep persons. In our case HashMap is selected. There are static methods manipulating data.

package com.automationrhapsody.reststub.persistence;

import com.automationrhapsody.reststub.data.Person;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class PersonDB {
	private static Map<Integer, Person> persons = new HashMap<Integer, Person>();

	static {
		persons.put(1, new Person(1, "FN1", "LN1", "email1@email.com"));
		persons.put(2, new Person(2, "FN2", "LN2", "email2@email.com"));
		persons.put(3, new Person(3, "FN3", "LN3", "email3@email.com"));
		persons.put(4, new Person(4, "FN4", "LN4", "email4@email.com"));
	}

	public static Person getById(int id) {
		return persons.get(id);
	}

	public static List<Person> getAll() {
		List<Person> result = new ArrayList<Person>();
		for (Integer key : persons.keySet()) {
			result.add(persons.get(key));
		}
		return result;
	}

	public static int getCount() {
		return persons.size();
	}

	public static void remove() {
		if (!persons.keySet().isEmpty()) {
			persons.remove(persons.keySet().toArray()[0]);
		}
	}

	public static String save(Person person) {
		String result = "";
		if (persons.get(person.getId()) != null) {
			result = "Updated Person with id=" + person.getId();
		} else {
			result = "Added Person with id=" + person.getId();
		}
		persons.put(person.getId(), person);
		return result;
	}
}

Create health check

Health check is smoke test that can be called from admin panel to give you information about the status of the system. In production systems you might do things like checking DB connection, checking file system or network, checking important functionality. In example here just to illustrate the functionality my health check is count of persons in memory. If it goes to 0 then something is wrong and system is not healthy. Also to illustrate how properties are used “version” is passed from configuration file to health check via its constructor.

package com.automationrhapsody.reststub;

import com.automationrhapsody.reststub.persistence.PersonDB;
import com.codahale.metrics.health.HealthCheck;

public class RestStubCheck extends HealthCheck {
	private final String version;

	public RestStubCheck(String version) {
		this.version = version;
	}

	@Override
	protected Result check() throws Exception {
		if (PersonDB.getCount() == 0) {
			return Result.unhealthy("No persons in DB! Version: " +
					this.version);
		}
		return Result.healthy("OK with version: " + this.version +
				". Persons count: " + PersonDB.getCount());
	}
}

Create application

This is the final piece. Once we have all (data, service, health check) application is binding piece that brings them together. This is execution entry point. In “main” method new application is created and its run() method is called. This is it. In order to actually work service and health check should be registered. This is done in the run method. You create instance of both service and health check. Configuration is passed in health check’s constructor.

package com.automationrhapsody.reststub;

import com.automationrhapsody.reststub.resources.BookService;
import com.automationrhapsody.reststub.resources.PersonService;
import io.dropwizard.Application;
import io.dropwizard.setup.Environment;

public class RestStubApp extends Application<RestStubConfig> {

	public static void main(String[] args) throws Exception {
		new RestStubApp().run(args);
	}

	@Override
	public void run(RestStubConfig config, Environment env) {
		final PersonService personService = new PersonService();
		env.jersey().register(personService);

		env.healthChecks().register("template", 
			new RestStubCheck(config.getVersion()));
	}
}

Build a single JAR

This was it now all have to be packed into a JAR. The strategy is to build everything into one JAR and just run it. It could not be more simple. Open pom.xml file. Add <build><plugins> … </plugins></build> in the end. Add XML bellow into this snippet. Only <mainClass> is customisable and should be changed according to your project structure.

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-shade-plugin</artifactId>
	<version>1.6</version>
	<configuration>
		<createDependencyReducedPom>true</createDependencyReducedPom>
		<filters>
			<filter>
				<artifact>*:*</artifact>
				<excludes>
					<exclude>META-INF/*.SF</exclude>
					<exclude>META-INF/*.DSA</exclude>
					<exclude>META-INF/*.RSA</exclude>
				</excludes>
			</filter>
		</filters>
	</configuration>
	<executions>
		<execution>
			<phase>package</phase>
			<goals>
				<goal>shade</goal>
			</goals>
			<configuration>
				<transformers>
					<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
					<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
						<mainClass>com.automationrhapsody.reststub.RestStubApp</mainClass>
					</transformer>
				</transformers>
			</configuration>
		</execution>
	</executions>
</plugin>

Build and run

Once this is done use “mvn package” to make the JAR. Navigate to target folder in your project and run the JAR. Two arguments are needed in order to run the JAR. First is “server” which instructs Dropwizard to run as server. Second is path to *.yml configuration file.

java -jar sample-dropwizard-rest-stub-1.0-SNAPSHOT.jar server ../config.yml

If everything is fine you should see something like code bellow which will mean server is ready.

GET     /person/all (...)
GET     /person/get/{id} (...)
GET     /person/remove (...)
POST    /person/save (...)

Test and enjoy

Once all this hard work has been done it is time to enjoy our RESTful server. List with all persons can be found on this URL: http://localhost:9000/person/all. You can get person by id: http://localhost:9000/person/get/1.

Health checks are found in admin panel: http://localhost:9001. Try removing all persons by invoking several times this URL: http://localhost:9000/person/remove

And the hardest part is to save a person. I’m using Postman Chrome plugin but you can use any REST client you want. You have to put POST data against http://localhost:9000/person/remove URL.

{"id":10,"firstName":"FN10","lastName":"LN10","email":"email10@email.com"}

And most important DO not forget to put “Content-Type: application/json” in request header. If you do not put you will get “Error 415 Unsupported Media Type” error.

PostmanRequest

Sample application can be found in GitHub sample-dropwizard-rest-stub repository. Postman requests can be downloaded from Dropwizard Postman requests link and directly imported into Postman.

Another way to test the stub is by building a client as described in Create simple REST API client using Jersey post.

Run with Docker

In Run Dropwizard application in Docker with templated configuration using environment variables post I have described how to make Dropwizard application configuration be changed with environment variables which makes it very easy to build and run in inside Docker container.

Conclusion

It could not be more easier. If you really need to stub a RESTful server this is the proper way to do it.

Read more...