Monthly Archives: July 2016

MD5, SHA-1, SHA-256 and SHA-512 speed performance

Last Updated on by

Post summary: Speed performance comparison of MD5, SHA-1, SHA-256 and SHA-512 cryptographic hash functions in Java.

For Implement secure API authentication over HTTP with Dropwizard post an one way hash function was needed. Several factors are important when choosing hash algorithm: security, speed and purpose of use.

Security

MD5 and SHA-1 are compromised. Those shall not be used unless their speed is several times slower than SHA-256 or SHA-512. Other that remain are SHA-256 and SHA-512. They are from SHA-2 family and are much more secure. SHA-256 is computed with 32 bit words, SHA-512 with 64 bit words.

Hash implementations

For generating cryptographic hashes in Java there is Apache Commons Codec library which is very convenient.

Speed performance

In order to test the speed sample code is used:

import java.util.UUID;

import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.lang.time.StopWatch;

public class Test {

	private static final int TIMES = 1_000_000;
	private static final String UUID_STRING = UUID.randomUUID().toString();

	public static void main(String[] args) {
		System.out.println(generateStringToHash());
		System.out.println("MD5: " + md5());
		System.out.println("SHA-1: " + sha1());
		System.out.println("SHA-256: " + sha256());
		System.out.println("SHA-512: " + sha512());
	}

	public static long md5() {
		StopWatch watch = new StopWatch();
		watch.start();
		for (int i = 0; i < TIMES; i++) {
			DigestUtils.md5Hex(generateStringToHash());
		}
		watch.stop();
		System.out.println(DigestUtils.md5Hex(generateStringToHash()));
		return watch.getTime();
	}

	public static long sha1() {
		...
		System.out.println(DigestUtils.sha1Hex(generateStringToHash()));
		return watch.getTime();
	}

	public static long sha256() {
		...
		System.out.println(DigestUtils.sha256Hex(generateStringToHash()));
		return watch.getTime();
	}

	public static long sha512() {
		...
		System.out.println(DigestUtils.sha512Hex(generateStringToHash()));
		return watch.getTime();
	}

	public static String generateStringToHash() {
		return UUID.randomUUID().toString() + System.currentTimeMillis();
	}
}

Several measurements were done. Two groups – one with smaller length string to hash and one with longer. Each group had following variations of generateStringToHash() method:

  • cached UUID – no extra time should be consumed
  • cached UUID + current system time – in this case time is consumed to get system time
  • new UUID + current system time – in this case time is consumed for generating the UUID and to get system time

Raw results

Five measurements were made for each case and average value calculated. Time is in milliseconds per 1 000 000 calculations. System is 64 bits Windows 10 with 1 core Intel i7 2.60GHz and 16GB RAM.

  • generateStringToHash() with: return UUID_STRING;

Data to encode is ~36 characters in length (f5cdcda7-d873-455f-9902-dc9c7894bee0). UUID is cached and time stamp is not taken. No additional time is wasted.

Hash #1 (ms) #2 (ms) #3 (ms) #4 (ms) #5 (ms) Average per 1M (ms)
MD5 649 623 621 624 620 627.4
SHA-1 608 588 630 600 594 604
SHA-256 746 724 741 720 758 737.8
SHA-512 1073 1055 1050 1052 1052 1056.4
  • generateStringToHash() with: return UUID_STRING + System.currentTimeMillis();

Data to encode is ~49 characters in length (aa096640-21d6-4f44-9c49-4115d3fa69381468217419114). UUID is cached.

Hash #1 (ms) #2 (ms) #3 (ms) #4 (ms) #5 (ms) Average per 1M (ms)
MD5 751 789 745 806 737 765.6
SHA-1 768 765 694 763 751 748.2
SHA-256 842 876 848 839 850 851
SHA-512 1161 1152 1164 1154 1163 1158.8
  • generateStringToHash() with: return UUID.randomUUID().toString() + System.currentTimeMillis();

Data to encode is ~49 characters in length (1af4a3e1-1d92-40e7-8a74-7bb7394211e01468216765464). New UUID is generated on each calculation so time for its generation is included in total time.

Hash #1 (ms) #2 (ms) #3 (ms) #4 (ms) #5 (ms) Average per 1M (ms)
MD5 1505  1471 1518 1463 1487 1488.8
SHA-1 1333 1309 1323 1326 1334 1325
SHA-256 1505 1496 1507 1498 1516 1504.4
SHA-512 1834 1827 1833 1836 1857 1837.4
  • generateStringToHash() with: return UUID_STRING + UUID_STRING;

Data to encode is ~72 characters in length (57149cb6-991c-4ffd-9c98-d823ee8a61f757149cb6-991c-4ffd-9c98-d823ee8a61f7). UUID is cached and time stamp is not taken. No additional time is wasted.

Hash #1 (ms) #2 (ms) #3 (ms) #4 (ms) #5 (ms) Average per 1M (ms)
MD5 856 824 876 811 828 839
SHA-1 921 896 970 904 893 916.8
SHA-256 1145 1137 1241 1141 1177 1168.2
SHA-512 1133 1131 1116 1102 1110 1118.4
  • generateStringToHash() with: return UUID_STRING + UUID_STRING + System.currentTimeMillis();

Data to encode is ~85 characters in length (759529c5-1f57-4167-b289-899c163c775e759529c5-1f57-4167-b289-899c163c775e1468218673060). UUID is cached.

Hash #1 (ms) #2 (ms) #3 (ms) #4 (ms) #5 (ms) Average per 1M (ms)
MD5 1029 1035 1034 1012 1037 1029.4
SHA-1 1008 1016 1027 1007 990 1009.6
SHA-256 1254 1249 1290 1259 1248 1260
SHA-512 1228 1221 1232 1230 1226 1227.4
  • generateStringToHash() with: final String randomUuid = UUID.randomUUID().toString();
    return randomUuid + randomUuid + System.currentTimeMillis();

Data to encode is ~85 characters in length (2734b31f-16db-4eba-afd5-121d0670ffa72734b31f-16db-4eba-afd5-121d0670ffa71468217683040). New UUID is generated on each calculation so time for its generation is included in total time.

Hash #1 (ms) #2 (ms) #3 (ms) #4 (ms) #5 (ms) Average per 1M (ms)
MD5 1753 1757 1739 1751 1691 1738.2
SHA-1 1634 1634 1627 1634 1633 1632.4
SHA-256 1962 1956 1988 1988 1924 1963.6
SHA-512 1909 1946 1936 1929 1895 1923

Aggregated results

Results from all iterations are aggregated and compared in table bellow. There are 6 main cases. They are listed bellow and referenced in the table:

  • Case 1 – 36 characters length string, UUID is cached
  • Case 2 – 49 characters length string, UUID is cached and system time stamp is calculated each iteration
  • Case 3 – 49 characters length string, new UUID is generated on each iteration and system time stamp is calculated each iteration
  • Case 4 – 72 characters length string, UUID is cached
  • Case 5 – 85 characters length string, UUID is cached and system time stamp is calculated each iteration
  • Case 6 – 85 characters length string, new UUID is generated on each iteration and system time stamp is calculated each iteration

All times bellow are per 1 000 000 calculations:

Hash Case 1 (ms) Case 2 (ms) Case 3 (ms) Case 4 (ms) Case 5 (ms) Case 6 (ms)
MD5 627.4 765.6 1488.8 839 1029.4 1738.2
SHA-1 604 748.2 1325 916.8 1009.6 1632.4
SHA-256 737.8 851 1504.4 1168.2 1260 1963.6
SHA-512 1056.4 1158.8 1837.4 1118.4 1227.4 1923

Compare results

Some conclusions of the results based on two cases with short string (36 and 49 chars) and longer string (72 and 85 chars).

  • SHA-256 is faster with 31% than SHA-512 only when hashing small strings. When string are longer SHA-512 is faster with 2.9%.
  • Time to get system time stamp is ~121.6 ms per 1M iterations.
  • Time to generate UUID is ~670.4 ms per 1M iterations.
  • SHA-1 is fastest hashing function with ~587.9 ms per 1M operations for short strings and 881.7 ms per 1M for longer strings.
  • MD5 is 7.6% slower than SHA-1 for short strings and 1.3% for longer strings.
  • SHA-256 is 15.5% slower than SHA-1 for short strings and 23.4% for longer strings.
  • SHA-512 is 51.7% slower that SHA-1 for short strings and 20% for longer.

Hash sizes

Important data to consider is hash size that is produced by each function:

  • MD5 produces 32 chars hash – 5f3a47d4c0f703c5d83265c3669f95e6
  • SHA-1 produces 40 chars hash – 2c5a70165585bd4409aedeea289628fa6074e17e
  • SHA-256 produces 64 chars hash – b6ba4d0a53ddc447b25cb32b154c47f33770d479869be794ccc94dffa1698cd0
  • SHA-512 produces 128 chars hash – 54cdb8ee95fa7264b7eca84766ecccde7fd9e3e00c8b8bf518e9fcff52ad061ad28cae49ec3a09144ee8f342666462743718b5a73215bee373ed6f3120d30351

Purpose of use

In specific case this research was made for hashed string will be passed as API request. It is constructed from API Key + Secret Key + current time in seconds. So if API Key is something like 15-20 chars, Secret Key is 10-15 chars and time is 10 chars, total length of string to hash is 35-45 chars. Since it is being passed as request param it is better to be as short as possible.

Select hash function

Based on all data so far SHA-256 is selected. It is from secure SHA-2 family. It is much faster than SHA-512 with shorter stings and it produces 64 chars hash.

Conclusion

Current post gives comparison of MD5, SHA-1, SHA-256 and SHA-512 cryptographic hash functions. Important is that comparison is very dependant on specific implementation (Apache Commons Codec), specific purpose of use (generate secure token to be sent with API call). It is good MD5 and SHA-1 to be avoided as they are compromised and not secure. If their speed for given context is several times faster than secure SHA-2 ones and security is not that much important they can be chosen though. When choosing cryptographic hash function everything is up to a context of usage and benchmark tests for this context are needed.

Read more...

Implement secure API authentication over HTTP with Dropwizard

Last Updated on by

Post summary: Reference implementation on suggested in How to implement secure REST API authentication over HTTP post authentication mechanism.

API authentication mechanism

Suggested authentication mechanism consists of following steps:

  • Secret key that is known only by API consumer and API provider is needed along with API key.
  • Secret key is used to one way hash a token which is send to server along with API key in the API call.
  • Token consists of: API key + Secret key + Current time in seconds, which then gets hashed with SHA-256 algorithm preferably.
  • Server recreates all the tokens locally for every second for some time in the future, preferably not too long – 30~120 seconds.
  • Server recreates all the tokens for 30~120 seconds in the past, to take into account the time needed for request to reach the server.
  • Server compares each of the tokens with received one.
  • If there is match consumer is authenticated and response is returned.

Dropwizard implementation

Dropwizard stub introduced in Build a RESTful stub server with Dropwizard post will be used to create authentication. Full example can be found in GitHib sample-dropwizard-rest-stub repository. Implementation consists of following steps:

  • Implement javax.ws.rs.container.ContainerRequestFilter interface. Implementation will inspect every request and verify authentication.
  • Create custom annotation
  • Annotate RequestFilter and Dropwizard resource (API service) on which authentication should be applied.
  • Register RequestFilter implementation class into Dropwizard Jersey environment.

Create custom annotation

Starting with the easiest step. Creating custom annotation is pretty easy. It could be applied to a class (ElementType.TYPE) or to a method (ElementType.METHOD). It should live as long as program runs (RetentionPolicy.RUNTIME). In order to make it possible annotated request filter to be applied on specific resource only @NameBinding annotation is a must in Jersey. If not specified request filter will apply on all resources. Needed annotation is:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.ws.rs.NameBinding;

@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(value = RetentionPolicy.RUNTIME)
@NameBinding
public @interface Authenticator {
}

ContainerRequestFilter implementation

Container request filter is applied on incoming requests. If used with @NameBinding annotation it is applied only where needed, if not it is applied globally. Mandatory is to override filter() method:

import com.automationrhapsody.reststub.persistence.AuthDB;

import java.io.IOException;
import java.util.List;

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;

@Authenticator
public class AuthenticateFilter implements ContainerRequestFilter {

	private static final String PARAM_API_KEY = "apiKey";
	private static final String PARAM_TOKEN = "token";
	private static final long SECONDS_IN_MILLISECOND = 1000L;
	private static final int TTL_SECONDS = 60;

	@Override
	public void filter(ContainerRequestContext context) throws IOException {
		final String apiKey = extractParam(context, PARAM_API_KEY);
		if (StringUtils.isEmpty(apiKey)) {
			context.abortWith(responseMissingParameter(PARAM_API_KEY));
		}

		final String token = extractParam(context, PARAM_TOKEN);
		if (StringUtils.isEmpty(token)) {
			context.abortWith(responseMissingParameter(PARAM_TOKEN));
		}

		if (!authenticate(apiKey, token)) {
			context.abortWith(responseUnauthorized());
		}
	}
}

As seen above two GET parameters are mandatory in the request: “apiKey” and “token”. Those are first extracted and verified. If some of them is not existing BAD_REQUEST (HTTP Status code 400) Response is returned with error message. Methods that extract params and build error response are:

private String extractParam(ContainerRequestContext context, String param) {
	final UriInfo uriInfo = context.getUriInfo();
	final List user = uriInfo.getQueryParameters().get(param);
	return CollectionUtils.isEmpty(user) ? null : String.valueOf(user.get(0));
}

private Response responseMissingParameter(String name) {
	return Response.status(Response.Status.BAD_REQUEST)
		.type(MediaType.TEXT_PLAIN_TYPE)
		.entity("Parameter '" + name + "' is required.")
		.build();
}

If both are present then code tried to authenticate the call by rebuilding all the hashes for 60 seconds in the past because request cannot arrive instantly it takes some time. If network is slower this time can be increased. It also rebuilds all hashes for 60 seconds in the future, this is token time to live. Server has access to Secret key for any given API key. In example above they are stored in fake DB provider and obtained by AuthDB.getSecretKey(apiKey):

private boolean authenticate(String apiKey, String token) {
	final String secretKey = AuthDB.getSecretKey(apiKey);

	// No need to calculate digest in case of wrong apiKey
	if (StringUtils.isEmpty(secretKey)) {
		return false;
	}

	final long nowSec = System.currentTimeMillis() / SECONDS_IN_MILLISECOND;
	long startTime = nowSec - TTL_SECONDS;
	long endTime = nowSec + TTL_SECONDS;
	for (; startTime < endTime; startTime++) {
		final String toHash = apiKey + secretKey + startTime;
		final String sha1 = DigestUtils.sha256Hex(toHash);
		if (sha1.equals(token)) {
			return true;
		}
	}

	return false;
}

As seen above server uses SHA-256 cryptographic algorithm. It is the best solution in terms of speed and security. In MD5, SHA-1, SHA-256 and SHA-512 speed performance post a comparison between MD5, SHA-1, SHA-256 and SHA-512 is made. If authentication cannot be verified then UNAUTHORIZED (HTTP Status code 401) Response response is returned:

private Response responseUnauthorized() {
	return Response.status(Response.Status.UNAUTHORIZED)
		.type(MediaType.TEXT_PLAIN_TYPE)
		.entity("Unauthorized")
		.build();
}

This is the hardest part. Now this filter has to be registered with Jersey and applied to needed resources (services). See more on ContainerRequestFilter interface and @NameBinding annotation in Jersey filters and interceptors page.

Apply authentication filter on a resource

Indicating that given resource should be checked for authentication is done with custom @Authenticator annotation created previously. If needed just for specific API call it can be applied also on a method level:

import com.automationrhapsody.reststub.data.Book;
import com.automationrhapsody.reststub.filters.Authenticator;
import com.automationrhapsody.reststub.persistence.BookDB;
import com.codahale.metrics.annotation.Timed;

import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Authenticator
@Path("/secure/books")
public class BooksSecureService {

	@GET
	@Timed
	@Produces(MediaType.APPLICATION_JSON)
	public List<Book> getBooks() {
		return BookDB.getAll();
	}
}

Register in Dropwizard Jersey

Last step is to register the request filter and resource with Dropwizard’s Jersey:

@Override
public void run(RestStubConfig config, Environment env) {

	env.jersey().register(BooksSecureService.class);
	env.jersey().register(AuthenticateFilter.class);

}

Conclusion

Very easy to implement in Dropwizard and relatively secure way to provide API authentication over HTTP protocol. For mission critical application definitely more strict consideration and review of this authentication mechanism is needed.

Read more...

How to implement secure REST API authentication over HTTP

Last Updated on by

Post summary: How to implement secure API authentication even over HTTP.

Important: this post is not a complete and expert guide on API security. It is mainly done to test Postman Pre-request hook that is described in Introduction to Postman with examples post. It does not go into all the details about API security, SSL certificates, encrypting the data, etc. It gives basic information how you can protect your API’s consumers against their network traffic being sniffed and credentials, apiKeys, session keys, etc stolen.

Authentication vs. Authorisation

Authentication is defined as “Who you are”. It deals with usernames and password. Authorisation is defined as “What you can do”. It deals with permissions. Before dealing with permissions application must know the user, so Authorisation comes after Authentication.

Basic Authentication

As it is stated it is very basic. The idea is to sent Base64 encoded username and password in the header of the request in following format:

Authorization: Basic dXNlcjpwYXNz

Server decodes the username and password and use them to authenticate and authorise the user. Problem with Basic authentication is it must be used only over HTTPS, since network traffic is encrypted. Over HTTP request can be easily sniffed. Base64 is reversible and there are numerous tools on the web where you can put dXNlcjpwYXNz and they will return user:pass as plain text.

Nota bene: Never use this one without HTTPS.

OAuth and OAuth 2.0

OAuth is authorisation protocol. It is intended mainly for web, but can be used in API authorisation. The idea is that authentication and authorisation is done by third party like Microsoft, Google, Facebook, Twitter, etc. This is easy for API as it does not have to deal with user data. Customer logins to third party and access token is being issued. Token has some validity which is not too long, but not too short, usually 1 or 2 days. The API can obtain user details from third party by this token. User authenticates itself to the API with this access token by sending it in the request header:

Authorization: Bearer 66408bd9-2bc0-40c3-9823-e9bec390532a

Problem with OAuth is it also must be used over HTTPS. Over HTTP traffic can be sniffed and token can be stolen. Although token has some expiry time, it is long enough for a hacker to use API from your behalf.

Nota bene: Never use this one without HTTPS.

API keys

API keys has become the standard when consuming an API. API key is some random hash which uniquely identifies the consumer. API keys have numerous benefits over username/password mechanism. Again in case of HTTP network traffic can be sniffed and API key stolen.

HTTPS

Reading post so far turned out there is not a single API authentication protocol that is secure if not used over HTTPS. In current a solution is proposed. It is commonly used in public APIs, it is possible to exist as a standard I’m just not aware of its name, which provides secure API authentication even over HTTP.

Implement API security over HTTP

In short in order to have security over HTTP following steps should be done:

  • Secret key that is known only by API consumer and API provider is needed along with API key.
  • Secret key is used to one way hash a token which is send to server along with API key in the API call.
  • Token consists of: API key + Secret key + Current time in seconds, which then gets hashed with SHA-256 algorithm preferably.
  • Server recreates all the tokens locally for every second for some time in the future, preferably not too long – 30~120 seconds.
  • Server recreates all the tokens for 30~120 seconds in the past, to take into account the time needed for request to reach the server.
  • Server compares each of the tokens with received one.
  • If there is match consumer is authenticated and response is returned.

Cryptographic hash algorithms

Most used hash algorithms nowadays are: MD5, SHA-1, SHA-256, SHA-512. MD5 and SHA-1 are to week and are not recommended. SHA-512 takes more time to compute the hashes. SHA-256 is the most appropriate solution in terms of security and speed. In MD5, SHA-1, SHA-256 and SHA-512 speed performance post all 4 algorithms have been tested and compared with Apache’s Commons Codec implementation.

Hash with Salt

Time stamp into hashed token is used for so called salt, an random data that is used to differentiate the hashed data against dictionary attacks. If just API key + Secret key are hashed, then hash will always be one and the same. Intruder will take the hash and just use it. Using time stamp makes the hash always different. Another function of time stamp is to set expiration time on the token, so even if stolen not to be used for a long period of time.

Conclusion

There are already established standards to secure an API, but all of them are effective only over HTTPS. In current post is given a proposal for secure API authentication which is very simple and relatively safe even over HTTP. Cons of this method is server has to recalculate hash many times, which in massive load would require some caching. In Implement secure API authentication over HTTP with Dropwizard post there is reference implementation of proposed solution with Dropwizard.

Read more...