Monthly Archives: December 2019

Serialize and deserialize enum values to custom string in C# with Json.NET

Last Updated on by

Post summary: How to serialize and deserialize C# enum values to customs strings.

The code used for this blog post is located in dotnet.core.templates GitHub repository.

Use case description

Enums provide an efficient way to define a set of named integral constants that may be assigned to a variable. They are strongly typed values and are preferred choice over the string constants. In the given example there is an API that provides functionality to save or get movies with title and genre. Although not the best example, as the genre can have lots of values, but it still can be presented as an enum. In the same setup, there is a user interface that consumes the get API. In the UI it is more convenient to directly display the genre as text, so control over naming convention happens in the backend and there is no mapping in the frontend, and localization is ignored in the current example.

Serialize and deserialize

Serialization and deserialization to a custom string can be done with two steps. The first is to add an attribute to all enum values which gives the preferred string mapping.

using System.Runtime.Serialization;

public enum MovieGenre
{
	[EnumMember(Value = "Action Movie")]
	Action,
	[EnumMember(Value = "Drama Movie")]
	Drama
}

Then Json.NET is instructed to serialize/deserialize the enum value as a string with [JsonConverter(typeof(StringEnumConverter))] attribute.

using Newtonsoft.Json;
using Newtonsoft.Json.Converters;

public class Movie
{
	public string Title { get; set; }

	[JsonConverter(typeof(StringEnumConverter))]
	public MovieGenre Genre { get; set; }
}

This results in the following JSON:

{
	"Title": "Die Hard",
	"Genre": "Action Movie"
}

In the example above, if [EnumMember(Value = “Action Movie“)] is not provided in the enum declaration, then the string representation of the enum value is taken:

{
	"Title": "Die Hard",
	"Genre": "Action"
}

Conclusion

Although the current example is not ideal from a use-case point of view, it shows technically how Newtonsoft Json.NET can be instructed to serialize/deserialize enum values to custom strings or just string representation of the enum value.

Read more...

Dockerize React application with a Docker multi-staged build

Last Updated on by

Post summary: How to build React application inside a Docker container, with a multi-staged build and then run it with NGINX or Caddy.

In the current post, I am not going to compare NGINX vs. Caddy. I will show how to build a React application and package it into a Docker container with both of them. Examples code is located in cypress-testing-framework GitHub repository.

NGINX

NGINX is open-source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers.

Caddy

Caddy is an open-source, HTTP/2-enabled web server written in Go. It uses the Go standard library for its HTTP functionality. One of Caddy’s most notable features is enabling HTTPS by default.

Building

Docker multi-staged building is going to be used in the current post. I have slightly touched the topic in the Optimize the size of Docker images post. The main idea is to optimize the Docker images, so they become smaller. In the current post, I will show two flavors of builds. One is with the standard NPM package manager and is described in Build and run with NGINX section.

The other is with Yarn package manager and is described in Build and run with Caddy section. Current examples are configured to use Yarn. I personally prefer Yarn as for local development it has very effective caching and also it has a reliable dependency locking mechanism.

Build and run with NGINX

Following Dockerfile is describing the building of the React application with NPM package manager and packaging it into NGINX image.

# ========= BUILD =========
FROM node:8.16.0-alpine as builder

WORKDIR /app

COPY package.json .
COPY package-lock.json .
RUN npm install --production

COPY . .

RUN npm run build

# ========= RUN =========
FROM nginx:1.17

COPY conf/nginx.conf /etc/nginx/nginx.conf
COPY --from=builder /app/build /usr/share/nginx/html

The keyword as builder is used to put the name to the image. Both package.json and package-lock.json are copied to the already configured work directory /app. Installation of the packages is done with npm install –production, where the –production switch is used to skip the devDependencies. In the current example, Cypress takes a lot of time to install, and it is not needed for a production build. Afterward, all project files are copied to the image. The files configured in .dockerignore are skipped. All source code files are intentionally copied to the image only after the NPM packages installation. Packages installation takes time, and they need to be installed only if the package.json file has been changed. In case of code changes only, Docker cache is used for the packages layer, this speeds up the build. The build is initiated with npm run build and takes quite a time. Now there the build artifacts are ready. Next stage is to copy the artifacts to nginx:1.17 image into /usr/share/nginx/html folder from builder image’s /app/build folder. Also, NGINX configuration file is copied.

worker_processes auto;
worker_rlimit_nofile 8192;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/mime.types;
  sendfile on;
  tcp_nopush on;

  gzip on;
  gzip_static on;
  gzip_types
    text/plain
    text/css
    text/javascript
    application/json
    application/x-javascript
    application/xml+rss;
  gzip_proxied any;
  gzip_vary on;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;

  server {
    listen 3000;
    server_name localhost;
    root /usr/share/nginx/html;
    auth_basic off;

    location / {
      try_files $uri $uri/ /index.html;
    }

    # 404 if a file is requested (so the main app isn't served)
    location ~ ^.+\..+$ {
      try_files $uri =404;
    }
  }
}

I will not go into NGINX configuration details, the configuration can be checked in details in NGINX documentation. Important in the configuration above is that gzip compression is enabled and NGINX listens to port 3000. Then with try_files unknown routes are redirected to index.html, so React can bootstrap the routes.

Build and run with Caddy

Following Dockerfile is describing the building of the React application with Yarn package manager and packaging it into Caddy image.

# ========= BUILD =========
FROM node:8.16.0-alpine as builder

WORKDIR /app

RUN npm install yarn -g

COPY package.json .
COPY yarn.lock .
RUN yarn install --production=true

COPY . .

RUN yarn build

# ========= RUN =========
FROM abiosoft/caddy:1.0.3

COPY conf/Caddyfile /etc/Caddyfile
COPY --from=builder /app/build /usr/share/caddy/html

Absolutely the same logic applies here as above. Yarn is installed as an additional Linux package, then package.json and yarn.lock files are copied. It is very important to copy the yarn.lock, otherwise every run lates dependencies will be fetched, and there might be inconsistent behavior. Only production dependencies are installed with yarn install –production=true. After the application is built with yarn build it is being copied to abiosoft/caddy:1.0.3 image in /usr/share/caddy/html folder from builder image. Caddyfile is copied as well to configure Caddy.

0.0.0.0:3000 {
	gzip
	log / stdout "{method} {path} {status}"
	root /usr/share/caddy/html
	rewrite {
		regexp .*
		to {path} /
	}
}

Caddy is configured to listen to port 3000, gzip compression is enabled and there is rewrite rule which redirects unknown paths to the main path, so React can bootstrap the router.

Conclusion

In the current post, I have shown how to build React application inside a Docker image with both NPM and Yarn and then pack the build artifacts to NGINX or Caddy Docker image, which later can be run as a container. This process optimizes the Docker image size and also it does not put extra requirements to the build machine to have Node JS installed, as Node JS is inside the builder image.

Related Posts

Read more...

Create .NET Core health checks with custom response payload

Last Updated on by

Post summary: How to extend custom .NET Core health checks so the response JSON provides more information.

The code used for this blog post is located in dotnet.core.templates GitHub repository.

Heath checks in .NET Core

Health checks in .NET Core is a middleware that provides a possibility to report an application’s health. This allows monitoring of the application and taking corrective actions in case of issues. For e.g., if an application reports to be unhealthy, then the load balancer can exclude it from the infrastructure and appropriate alarms to be raised. More in about health checks can be read in Health checks in ASP.NET Core page.

Adding a health check

In order to add a health check in a .NET Core application, then reference to Microsoft.AspNetCore.Diagnostics.HealthChecks package has to be added. Health checks themselves are classes implementing IHealthCheck interface.

public class VersionHealthCheck : IHealthCheck
{
	private readonly AppConfig _config;

	public VersionHealthCheck(IOptions<AppConfig> options)
	{
		_config = options.Value;
	}

	public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
				CancellationToken cancellationToken = new CancellationToken())
	{
		return Task.FromResult(string.IsNullOrEmpty(_config.Version)
			? HealthCheckResult.Unhealthy()
			: HealthCheckResult.Healthy());
	}
}

Health checks have to be registered in Startup.cs file:

public void ConfigureServices(IServiceCollection services)
{
	services.AddHealthChecks()
		.AddCheck<VersionHealthCheck>("Version Health Check");
}

As a last step health checks endpoint has to be configured in Startup.cs file:

public void Configure(IApplicationBuilder app)
{
	app.UseEndpoints(endpoints =>
	{
		endpoints.MapHealthChecks("/health");
	});
}

Now health checks report is available at <HOSTNAME>/health URL. If everything is good, the response is 200 OK with content Healthy. In case of issues, the response is 503 Service Unavailable with content Unhealthy.

Extend the health checks response

As stated above health checks are mainly intended for machine usage. I have had cases in practice, in which just looking into the health check allows faster problem solving rather than looking into the logs. For this reason, investing in more explanatory health checks is worth it. Below is a code snippet on how to have more information into the health check response payload. A new static class HealthCheckExtensions with MapCustomHealthChecks method can be added.

public static class HealthCheckExtensions
{
	public static IEndpointConventionBuilder MapCustomHealthChecks(
		this IEndpointRouteBuilder endpoints, string serviceName)
	{
		return endpoints.MapHealthChecks("/health", new HealthCheckOptions
		{
			ResponseWriter = async (context, report) =>
			{
				var result = JsonConvert.SerializeObject(
					new HealthResult
					{
						Name = serviceName,
						Status = report.Status.ToString(),
						Duration = report.TotalDuration,
						Info = report.Entries.Select(e => new HealthInfo
						{
							Key = e.Key,
							Description = e.Value.Description,
							Duration = e.Value.Duration,
							Status = Enum.GetName(typeof(HealthStatus),
													e.Value.Status),
							Error = e.Value.Exception?.Message
						}).ToList()
					}, Formatting.None,
					new JsonSerializerSettings
					{
						NullValueHandling = NullValueHandling.Ignore
					});
				context.Response.ContentType = MediaTypeNames.Application.Json;
				await context.Response.WriteAsync(result);
			}
		});
	}
}

All the formatting in the code depends on two additional data classes HealthInfo and HealthResult.

public class HealthInfo
{
	public string Key { get; set; }
	public string Description { get; set; }
	public TimeSpan Duration { get; set; }
	public string Status { get; set; }
	public string Error { get; set; }
}
public class HealthResult
{
	public string Name { get; set; }
	public string Status { get; set; }
	public TimeSpan Duration { get; set; }
	public ICollection<HealthInfo> Info { get; set; }
}

Registering the endpoint happens with the same code as in the default case, with the difference that the MapCustomHealthChecks extension method is used:

public void Configure(IApplicationBuilder app)
{
	app.UseEndpoints(endpoints =>
	{
		endpoints.MapCustomHealthChecks("Service Name");
	});
}

Now it is possible to have some more elaborate health checks, which can capture exception for e.g. and return it as well.

public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
		CancellationToken cancellationToken = new CancellationToken())
{
	try
	{
		var message = $"Version is healthy: {_config.Version}";
		return Task.FromResult(HealthCheckResult.Healthy(message));
	}
	catch (Exception ex)
	{
		var message = "There is an error with version health check";
		return Task.FromResult(HealthCheckResult.Unhealthy(message, ex));
	}
}

In the case of 503 Service Unavailable, health check gives more details, which in some cases can be enough to resolve the issue without having to dig into the logs.

{
	"Name": "Service Name",
	"Status": "Unhealthy",
	"Duration": "00:00:00.0159186",
	"Info": [
		{
			"Key": "Version Health Check",
			"Description": "There is an error with version health check",
			"Duration": "00:00:00.0010564",
			"Status": "Unhealthy",
			"Error": "Exception's message text"
		}
	]
}

Conclusion

.NET Core health checks are a convenient way for automatic service monitoring and taking corrective actions. With a small effort, they can be enhanced so they can be made useful for people trying to identify what the issues with the services are.

Related Posts

Read more...

Optimize the size of Docker images

Last Updated on by

Post summary: How to optimize the size of the Docker images, by using intermediate build image and final runtime image.

The code used for this blog post is located in dotnet.core.templates GitHub repository. The code examples below are for .NET Core 3.0, but principles applied in this article are valid for any programming language, so it is worth reading.

Docker layers and images

Docker image is an executable version of a given application that runs on top of an operating system’s kernel. Docker image is the result of the execution of a Dockerfile. Usually, Dockerfile starts from some base image, for e.g. an operating system. Then commands are built on top of this base image and the result is a new image. This new image can be used as a base image somewhere else. Each and every command in Dockerfile results in a layer. This layering system is used for better reusability, as several images can reuse a given layer. The more layers are added to the image the bigger it gets in size.

All docker images can be listed with docker images command. Size is also present as an output of the command. Then for a given image, it is possible to list all the layers with docker history <IMAGE_NAME> command, which also shows the size of a given layer.

Images are kept in a Docker repository, either public or private. The bigger the image, the more time it takes to upload, to download and the more space it consumes in the repository. It is a good practice to optimize the images in terms of size.

Optimize the size

Usually when building software much more resources are needed, such as SDK, or compiler, or additional libraries, than if the software is run. One strategy for optimization is to build the software on a special build machine and then pack it to a Docker image. In this approach, the build machine should have the needed build software. This puts some demand on the build machine and also makes the image creating process dependant on certain software packages being installed. A more convenient option is to build the application as part of the Docker image creating and then packet into a separate container. See Dockerfile below.

FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/core/sdk:3.0-buster AS build
WORKDIR /src
COPY . .
RUN dotnet restore
RUN dotnet publish -c Release -o /pub

FROM base AS final
WORKDIR /app
COPY --from=build /pub .
ENTRYPOINT ["dotnet", "PROJECT_NAME.dll"]

In short, image sdk:3.0-buster is used to publish the application as it has .NET Core SDK on it, and then application code is copied into aspnet:3.0-buster-slim which has only the .NET Core runtime and is low in size.

No matter how the software is built, the most optimal image in terms of size and capabilities has to be selected to pack the code into. For e.g. Google provides “Distroless”, images that do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution. This makes images smaller and much more secure. I tried to build the application I am experimenting with into Distroless image and it gets 136MB in size, where if I pack it into .NET 3.0 runtime image it gets 209MB. Unfortunately, there is no Distroless image for .NET Core 3.0, so my experiment image fails to run, and I have to use aspnet:3.0-buster-slim in order to run my sample application.

.NET Core different images

.NET Core has different images, which are very well explained into .NET Core SDK images page. They are:

  • buster – Debian 10
  • alpine – Alpine
  • bionic – Ubuntu 18.04
  • disco – Ubuntu 19.04
  • stretch – Debian 9

.NET Core 3.0 error in stretch images

This section is not directly contributing to the main point of the topic, but it might be helpful to someone. When I experimented, I initially started with stretch base images. And I got the following errors:

  • System.MissingMethodException: Method not found: ‘Void Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream..ctor(System.IO.Stream, Int32)’
  • System.TypeLoadException: Could not load type ‘Microsoft.AspNetCore.WebUtilities

These errors were not present when switching to buster base images.

Conclusion

In the current post, I describe how to construct Docker files so the build is done in Docker, eliminating the need of having specific software in order to pack the images. No matter how the software is built it is very important to pack it into the smallest possible image in order to save bandwidth and storage space during image usage. Google provides Distroless images that seem very lightweight and also secure as they do not contain package managers, shells or any other programs. Examples in this post are in .NET Core 3.0, but principles can be applied to different programming languages and technologies.

Related Posts

Read more...

Create project for .NET Core custom template

Last Updated on by

Post summary: How to create a custom .NET Core template, install it and create projects from it.

The code used for this blog post is located in dotnet.core.templates GitHub repository.

.NET Core

In short .NET Core is a cross-platform development platform supporting Windows, macOS, and Linux, and can be used in device, cloud, and embedded/IoT scenarios. It is maintained by Microsoft and the .NET community on GitHub. More can be read on .NET Core Guide.

Why .NET Core templates

When you create a new .NET Core project with dotnet new command it is possible to select from a list of predefined templates. This is a very handy feature, but out of the box templates are not really convenient, additional changes are required afterward to make the project fit for purpose. In a situation where many projects with similar structures are created, such as many micro-services, a custom template is very helpful. Users can define a custom template and easily create new projects out of it. In the current post, I will describe how to create an elaborate custom template.

Create template

What has to be done is just to create a project and customize it according to the needs. Once code is ready, a file named template.json located in .template.config folder should be added. The file should conform to http://json.schemastore.org/template JSON schema. See the file below, it is more or less self-explanatory. An important field is identity, which is basically the unique template name, it is not visible to users though. What is visible are name and shortName. ShortName is actually used when creating a project from this template: dotnet new shortName. Guids used in the template are defined in guids section in template.json file and they are replaced with a fresh set of guids on each project creation. More details on each and every possible option can be found in “Runnable-Project”-Templates page.

{
	"$schema": "http://json.schemastore.org/template",
	"author": "Lyudmil Latinov",
	"classifications": [
		"Common",
		"Code"
	],
	"identity": "dotnet.core.micro.service",
	"name": ".NET Core 3.0 micro-service",
	"shortName": "microservice",
	"tags": {
		"language": "C#",
		"type": "item"
	},
	"guids": [
		"9A19103F-16F7-4668-BE54-9A1E7A4F7556"
	]
}

This is it, the template is now ready to be installed and used.

Install template and create project

The template is installed with dotnet new -i <PATH_TO_FOLDER>. Once the template is installed dotnet new command lists it along with all predefined templates.

Creating a project is similar as it is created from standard templates: dotnet new <SHORT_NAME>, dotnet new microservice in this example.

Uninstalling of a template is done with dotnet new -u <PATH_TO_FOLDER> command. There is another command which uninstalls all custom template from the system: dotnet new –debug:reinit.

Replace project name

A custom template is a very handy thing and I would like to go one step further in making this template more flexible. When a new project is created it is good to have a proper name, which is reflected in the file names, folders’ names and into the namespace. For this reason, a custom replace task can be added to the template definition. A placeholder PROJECT_NAME is added to namespaces, Dockerfile, folder names, solution file name, and .csproj files names. This is done by adding and external required parameter in symbols section of template.json file. On project creation, this parameter is provided with value, which gets replaced in files content and file names.

{
	"$schema": "http://json.schemastore.org/template",
	...
	"symbols": {
		"ProjectName": {
			"type": "parameter",
			"replaces": "PROJECT_NAME",
			"FileRename": "PROJECT_NAME",
			"isRequired": true
		}
	}
}

When dotnet new microservice -h is executed it outputs the possible parameters that are needed to create a project from this template:

.NET Core 3.0 micro-service (C#)
Author: Lyudmil Latinov
Options:
  -P|--ProjectName
                    string - Required

Now, the project cannot be created without providing this parameter. Command dotnet new microservice returns error Mandatory parameter –ProjectName missing for template .NET Core 3.0 micro-service. The proper command now is dotnet new microservice –ProjectName SampleMicroservice.

Conditional files and features

So far template is much nicer and looks more real. It can be also further enhanced. For e.g. there are two major types of projects needed. One option is to have two separate templates, which means maintaining two templates. Another option, in case of insignificant differences, is to have conditional functionality added based on a parameter during template creation. A boolean parameter is added into symbols section of template.json file. Another thing to be done is to exclude files depending on this parameter. This is done in the sources section of template.json file.

{
	"$schema": "http://json.schemastore.org/template",
	...
	"symbols": {
		...
		"AddHealthChecks": {
			"type": "parameter",
			"datatype": "bool",
			"defaultValue": "false"
		}
	},
	"sources": [
		{
			"modifiers": [
				{
					"condition": "(!AddHealthChecks)",
					"exclude": [
						"src/**/HealthChecks/**",
						"test/**/Client/HealthCheckClient.cs",
						"test/**/Tests/HealthCheckTest.cs"
					]
				}
			]
		}
	]
}

Customize parameters switches

If you add several of those parameters, then .NET gives them an automatic name, which may not be very meaningful. So the names of those can be customized. In the example here, which is located in dotnet.core.templates GitHub repository, the default parameter names are (dotnet new microservice -h):

.NET Core 3.0 micro-service (C#)
Author: Lyudmil Latinov
Options:
  -P|--ProjectName
                         string - Required

  -A|--AddHealthChecks
                         bool - Optional
                         Default: false / (*) true

  -Ad|--AddSqsPublisher
                         bool - Optional
                         Default: false / (*) true

  -p:A|--AddSqsConsumer
                         bool - Optional
                         Default: false / (*) true


* Indicates the value used if the switch is provided without a value.

Short names like -p:A, -Ad does not seem too convenient. Those can be easily customized by adding dotnetcli.host.json file into .template.config folder:

{
	"$schema": "http://json.schemastore.org/dotnetcli.host",
	"symbolInfo": {
		"ProjectName": {
			"longName": "ProjectName",
			"shortName": "pn"
		},
		"AddHealthChecks": {
			"longName": "AddHealthChecks",
			"shortName": "ah"
		},
		"AddSqsPublisher": {
			"longName": "AddSqsPublisher",
			"shortName": "ap"
		},
		"AddSqsConsumer": {
			"longName": "AddSqsConsumer",
			"shortName": "ac"
		}
	}
}

Now options are much different:

.NET Core 3.0 micro-service (C#)
Author: Lyudmil Latinov
Options:
  -pn|--ProjectName
                         string - Required

  -ah|--AddHealthChecks
                         bool - Optional
                         Default: false / (*) true

  -ap|--AddSqsPublisher
                         bool - Optional
                         Default: false / (*) true

  -ac|--AddSqsConsumer
                         bool - Optional
                         Default: false / (*) true


* Indicates the value used if the switch is provided without a value.

Conclusion

.NET Core provides an easy and extensible way to make project templates, which are stored and maintained under version control and can be used for creating new projects. Those templates can be project-specific, product-specific, company-specific.

Related Posts

Read more...

Restore deleted Git stash

Last Updated on by

Post summary: How to restore deleted Git stash.

Git

Git is a version control system, which is conceptually different than others. It is a mini file system, which has all the information locally. Git support fully local work, no internet connection is needed once the project is checked out. All changes are done locally and saved to the local database. Once there is internet connection changes can be synced to the server and available for others as well. See more in What is Git? page.

Git stash

Git offers so-called stashing. Current work can be temporarily saved, without being committed. When current work is stashed, the repository is reverted back to the original state. One possible use case is when new conflicting changes from the upstream are coming. Current work has to be saved, remote changes applied and then, the current work to be completed.

Many changes can be stashed. The problem with having several stashes is that there is no easy way to merge the stashes. So it is recommended not to have more than one stash at a time.

Common stash operations

The most important actions that can be done on a stash are:

  • git stash – saves current work to stash
  • git stash list – shows all stashed changes
  • git stash apply – apply the latest stash
  • git stash clear – removes all stashes

More details can be found in git-stash page.

Restore deleted Git stash

It happened several times that I am working on something important with many changes, but then I need to switch to another thing. I do not want to commit the work, as it is messy, so I have to stash it. Then I accidentally deleted the stash. The worst thing that happened was two weeks of work stash to get deleted, quite upsetting.

Luckily Git is a really sophisticated version control system and it saves intermediates states, so stash is not really lost. It can be restored. I have a favorite article on the topic, Recover a dropped Git stash. There are some command line suggestions in it, but what I love is:

gitk --all $(git fsck --no-reflogs | awk '/dangling commit/ {print $3}')

In the screenshot, it is very clearly shown the stash and the file changes into it. Then those changes can be manually applied.

Conclusion

Git is a very sophisticated version control system, more like a mini file system. It allows you to stash changes that are not ready to be committed yet. Stashes can be accidentally deleted. The good thing is there is a mechanism to restore deleted stashes.

Read more...