AWS examples in C# – run the solution

Last Updated on by

Post summary: Explanation of how to install and use the solution in AWS examples in C# blog post series.

This post is part of AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository. In the current post, I give information on how to install and run the project.

Disclaimer

The solution has commands to deploy to the cloud as well as to clean resources. Note not all resources are cleaned, read more in the Cleanup section. In order to be run in AWS valid account is needed. I am not going to describe how to create an account. If an account is present, then there is also knowledge and awareness of how to use it.

Important: current examples generate costs on AWS account. Use cautiously at your own risk!

Restrictions

The project was tested to be working on Linux and Windows. For Windows, it is working only with Git Bash. The project requires a valid AWS account.

Required installations

In order to fully run and enjoy the project following needs to be installed:

Configurations

AWS CLI has to be configured in order to run properly. It happens with aws configure. If there is no AWS account, this is not an issue, put some values for access and secret key and put a correct region, like us-east-1.

Import Postman collection, in order to be able to try the examples. Postman collection is in aws.examples.csharp.postman_collection.json file in the code. This is an optional step, below there are cURL examples also.

Run the project in AWS

Running on AWS requires the setting of environment variables:

export AwsAccessKey=KIA57FV4.....
export AwsSecretKey=mSgsxOWVh...
export AwsRegion=us-east-1

Then the solution is deployed to AWS with ./solution-deploy.sh script. Note that the output of the command gives the API Gateway URL and API key, as well as the SqsWriter and SqsReader endpoints. See image below:

Run the project partly in AWS

The most expensive part of the current setup is the running of the docker containers in EC2 (Elastic Compute Cloud). I have prepared a script called ./solution-deploy-hybrid.sh, which runs the containers locally and all other things are in AWS. Still, environment variables from the previous section are mandatory to be set. I believe this is the optimal variant if you want to test and run the code in a “production-like” environment.

Run the project in LocalStack

LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. It spins up a testing environment on local machines that provide the same functionality and APIs as the real AWS cloud environment. I have experimented with LocalStack, there is a localstack branch in GitHub. The solution can be run with solution-deploy-localstack.sh command. I cannot really estimate if this is a good alternative, because I am running the free tier in AWS and the most expensive part is ECS, which I skip by running the containers locally, instead of AWS. See the previous section on how to run a hybrid.

Usage

There is a Postman collection which allows easy firing of the requests. Another option is to use cURL, examples of all requests with their Postman names are available below.

SqsWriter

SqsWriter is a .NET Core 3.0 application, that is dockerized and run in AWS ECS (Elastic Container Service). It exposes an API that can be used to publish Actor or Movie objects. There is also a health check request. After AWS deployment proper endpoint is needed. The endpoint can be found as an output of deployment scripts. See the image above.

PublishActor

curl --location --request POST 'http://localhost:5100/api/publish/actor' \
--header 'Content-Type: application/json' \
--data-raw '{
	"FirstName": "Bruce",
	"LastName": "Willis"
}'

PublishMovie

curl --location --request POST 'http://localhost:5100/api/publish/movie' \
--header 'Content-Type: application/json' \
--data-raw '{
	"Title": "Die Hard",
	"Genre": "Action Movie"
}'

When Actor or Movie is published, it goes to the SQS queue, SqsReader picks it up from there and processes it. What is visible in the logs is that both LogEntryMessageProcessor and ActorMessageProcessor are invoked. See the screenshot:

SqsWriterHealthCheck

curl --location --request GET 'http://localhost:5100/health'

SqsReader

SqsReader is a .NET Core 3.0 application, that is dockerized and run in AWS ECS. It has a consumer that listens to the SQS queue and processes the messages by writing them into appropriate AQS DynamoDB tables. It also exposes API to stop or start processing, as long as reprocess the dead-letter queue or simply get the queue status. After AWS deployment proper endpoint is needed. The endpoint can be found as an output of deployment scripts. See the image above.

ConsumerStart

curl --location --request POST 'http://localhost:5200/api/consumer/start' \
--header 'Content-Type: application/json' \
--data-raw ''

ConsumerStop

curl --location --request POST 'http://localhost:5200/api/consumer/stop' \
--header 'Content-Type: application/json' \
--data-raw ''

ConsumerStatus

curl --location --request GET 'http://localhost:5200/api/consumer/status'

ConsumerReprocess
If this one is invoked with no messages in the dead-letter queue then it takes 20 seconds to finish, because it actually waits for long polling timeout.

curl --location --request POST 'http://localhost:5200/api/consumer/reprocess' \
--header 'Content-Type: application/json' \
--data-raw ''

SqsReaderHealthCheck

curl --location --request GET 'http://localhost:5200/health'

ActorsServerlessLambda

This lambda is managed by the Serverless framework. It is exposed as REST API via AWS API Gateway. It also has a custom authorizer as well as API Key attached. Those are described in a further post.

ServerlessActors

In the case of AWS, the API Key and URL are needed, those can be obtained from deployment command logs. See the screenshot above. Put correct values to CHANGE_ME and REGION placeholders. Request is:

curl --location --request POST 'https://CHANGE_ME.execute-api.REGION.amazonaws.com/dev/actors/search' \
--header 'Content-Type: application/json' \
--header 'x-api-key: CHANGE_ME' \
--header 'Authorization: Bearer validToken' \
--data-raw '{
    "FirstName": "Bruce",
    "LastName": "Willis"
}'

MoviesServerlessLambda

ServerlessMovies

Put correct values to CHANGE_ME and REGION placeholders. Request is:

curl --location --request GET 'https://CHANGE_ME.execute-api.REGION.amazonaws.com/dev/movies/title/Die Hard'

Cleanup

Nota bene: This is a very important step, as leaving the solution running in AWS will accumulate costs.

In order to stop and clean up all AWS resources run ./solution-delete.sh script.

Nota bene: There a resource that is not automatically deleted by the scripts. This is a Route53 resource created by AWS Cloud Map. It has to be deleted with the following commands. Note that the id in the delete command comes from the result of list-namespaces command.

aws servicediscovery list-namespaces
aws servicediscovery delete-namespace --id ns-kneie4niu6pwwela

Verify cleanup

In order to be sure there are no leftovers from the examples, following AWS services has to be checked:

  • SQS
  • DynamoDB
  • IAM -> Roles
  • EC2 -> Security Groups
  • ECS -> Clusters
  • ECS -> Task Definitions
  • ECR -> Repositories
  • Lambda -> Functions
  • Lambda -> Applications
  • CloudFormation -> Stacks
  • S3
  • CloudWatch -> Log Groups
  • Route 53
  • AWS Cloud Map

On top of it, Billing should be regularly monitored to ensure no costs are applied.

Conclusion

This post describes how to run and they the solution described in AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series

Related Posts

Read more...

Git clone with predefined user email and user name

Last Updated on by

Post summary: Small bash script to clone a Git repository and set user.email and user.name.

Usecase

There are cases when committing with a different user to a different Git repository is needed. Git offers a very easy command to change user.email and user.name, as long as you remember to do so.

git config user.name "Firstname Lastname"
git config user.email "Firstname.LastnameDoe@somemailhost.com"

I always forget to do it, so I made up a small script that I use to clone a repository and it does it for me.

Git also offers a command to globally change user.name and user.email and this is valid for each and every repository that is cloned. If the use case is to work with one name and email only, then maybe this is the best option.

git config --global user.name "Firstname Lastname"
git config --global user.email "Firstname.LastnameDoe@somemailhost.com"

Script

#!/bin/bash

if [ -z "$1" ]
then
  echo "Please provide the Git repo as argument"
  exit 1
fi

if [ -z "$2" ]
then
  echo "Please provide the user.name repo as argument"
  exit 1
fi

if [ -z "$3" ]
then
  echo "Please provide the user.email repo as argument"
  exit 1
fi

IFS='/' read -r -a urlParts <<< "$1"
urlPartsLast=${urlParts[${#urlParts[@]}-1]}

IFS="." read -r -a repoParts <<< "$urlPartsLast"
repoPartsLast=${repoParts[${#repoParts[@]}-1]}
if [ "$repoPartsLast" == "git" ]
then
  unset 'repoParts[${#repoParts[@]}-1]'
fi
repoName=$(printf ".%s" "${repoParts[@]}")
repoName=${repoName:1}

git clone "$1"

cd $repoName
git config user.name "$2"
git config user.email "$3"

The script file should be made executable with chmod +x git-clone.sh and then the script can be invoked with the following command:

./git-clone.sh https://github.com/llatinov/aws.examples.csharp.git "Firstname Lastname" Firstname.LastnameDoe@somemailhost.com

Script insights

The script checks for empty arguments and returns error in case of such. Note that user.name and user.email can be hardcoded into the script itself, this makes it easier to invoke. Then the script splits by slash (/) the Git URL into different parts. It takes the last part, which is supposed to be the repository name. The last part is additionally split by dot (.) and the git suffix is ignored. Script clones the repository and navigates to the folder where it sets the user.name and user.email.

Conclusion

This script is helping not to forget to clone a Git repository with correct user.name and user.email.

Read more...

Restore deleted Git stash

Last Updated on by

Post summary: How to restore deleted Git stash.

Git

Git is a version control system, which is conceptually different than others. It is a mini file system, which has all the information locally. Git support fully local work, no internet connection is needed once the project is checked out. All changes are done locally and saved to the local database. Once there is internet connection changes can be synced to the server and available for others as well. See more in What is Git? page.

Git stash

Git offers so-called stashing. Current work can be temporarily saved, without being committed. When current work is stashed, the repository is reverted back to the original state. One possible use case is when new conflicting changes from the upstream are coming. Current work has to be saved, remote changes applied and then, the current work to be completed.

Many changes can be stashed. The problem with having several stashes is that there is no easy way to merge the stashes. So it is recommended not to have more than one stash at a time.

Common stash operations

The most important actions that can be done on a stash are:

  • git stash – saves current work to stash
  • git stash list – shows all stashed changes
  • git stash apply – apply the latest stash
  • git stash clear – removes all stashes

More details can be found in git-stash page.

Restore deleted Git stash

It happened several times that I am working on something important with many changes, but then I need to switch to another thing. I do not want to commit the work, as it is messy, so I have to stash it. Then I accidentally deleted the stash. The worst thing that happened was two weeks of work stash to get deleted, quite upsetting.

Luckily Git is a really sophisticated version control system and it saves intermediates states, so stash is not really lost. It can be restored. I have a favorite article on the topic, Recover a dropped Git stash. There are some command line suggestions in it, but what I love is:

gitk --all $(git fsck --no-reflogs | awk '/dangling commit/ {print $3}')

In the screenshot, it is very clearly shown the stash and the file changes into it. Then those changes can be manually applied.

Conclusion

Git is a very sophisticated version control system, more like a mini file system. It allows you to stash changes that are not ready to be committed yet. Stashes can be accidentally deleted. The good thing is there is a mechanism to restore deleted stashes.

Read more...