AWS examples in C# – AWS CLI commands

Last Updated on by

Post summary: Important AWS CLI commands used in AWS examples in C#.

This post is part of AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository.

Introduction

In AWS examples in C# – run the solution post I have described how to install/uninstall current examples. In the current post, I am going to show in detail individual commands used. The configuration parameters in the command below will be given with capital letters and starting with a dollar sign, e.g. $CONFIGURATION_PARAMETER. Each AWS command has its code representation in the SDK for the desired programming language.

AWS Command Line Interface

The AWS Command Line Interface (CLI) is a unified tool to manage AWS services. Control of multiple AWS services from the command line and automate them through scripts can be done with just one tool to download and configure. The full list of services that can be controlled is listed in the AWS Command Line Interface reference page. Each service has a subpage with a list of all available commands. All commands return JSON as a response. In a subsequent post, I will describe how to manage the JSON in the command line. All operations in the current post are done after AWS credentials are set as environment variables:

export AWS_ACCESS_KEY_ID=KIA57FV4.....
export AWS_SECRET_ACCESS_KEY=mSgsxOWVh...
export AWS_DEFAULT_REGION=us-east-1

SQS operations

The full list can be found in aws sqs CLI reference page. More information about SQS can be found in AWS examples in C# – create a service working with SQS post.

Create

Initially, all queues are listed with list-queues, in order to check if the queue already exists.

aws sqs list-queues

The queue is created with create-queue command, the result of the command returns the queue URL.

aws sqs create-queue --queue-name $QUEUE_NAME

After queues are created, the re-drive policy has to be set up. The ARN of the dead-letter queue can be obtained with get-queue-attributes command by providing the queue URL.

aws sqs get-queue-attributes \
	--queue-url $DEAD_LETTER_QUEUE_URL \
	--attribute-names QueueArn

The re-drive policy is set with set-queue-attributes command.

aws sqs set-queue-attributes \
	--queue-url $QUEUE_URL \
	--attributes "{\"RedrivePolicy\":\"{\\\"maxReceiveCount\\\":\\\"3\\\",\\\"deadLetterTargetArn\\\":\\\"$DEAD_LETTER_QUEUE_ARN\\\"}\",\"ReceiveMessageWaitTimeSeconds\":\"$LONG_POLLING_TIMEOUT\"}"

Delete

In order to delete the queue, its URL is needed. The URL is obtained with get-queue-url command.

aws sqs get-queue-url --queue-name $QUEUE_NAME

Deletion happens with delete-queue command.

aws sqs delete-queue --queue-url $QUEUE_URL

DynamoDB operations

The full list can be found in aws dynamodb CLI reference page. More information about DynamoDB can be found in AWS examples in C# – create a service working with DynamoDB post.

Create

The table data is obtained with describe-table command.

aws dynamodb describe-table --table-name $TABLE_NAME

If the table does not exist, it is created with create-table command. The table command has all the data needed. See more about table attributes in AWS examples in C# – create a service working with DynamoDB post.

aws dynamodb create-table \
	--table-name $TABLE_NAME \
	--attribute-definitions 'AttributeName=FirstName,AttributeType=S' 'AttributeName=LastName,AttributeType=S' \
	--key-schema 'AttributeName=FirstName,KeyType=HASH' 'AttributeName=LastName,KeyType=RANGE' \
	--provisioned-throughput 'ReadCapacityUnits=5,WriteCapacityUnits=5' \
	--stream-specification 'StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES'

Delete

The table is deleted by name with delete-table command.

aws dynamodb delete-table --table-name $TABLE_NAME

IAM roles operations

The full list can be found in aws iam CLI reference page.

Create

Roles are listed with list-roles command to check if the role exists.

aws iam list-roles

The role is created with create-role command.

aws iam create-role \
	--role-name $ROLE_NAME \
	--assume-role-policy-document file://assume-role-policy-document.json

This is the only case in the current examples where an additional JSON document is needed alongside a command. It is not possible to pass this JSON inline as it is with aws sqs set-queue-attributes command. This JSON allows certain services to be accessed by this role.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Service": [
					"lambda.amazonaws.com",
					"ec2.amazonaws.com",
					"ecs.amazonaws.com",
					"ecs-tasks.amazonaws.com",
					"batch.amazonaws.com"
				]
			},
			"Action": "sts:AssumeRole"
		}
	]
}

List policies to get the policy ARN with list-policies command. Basically, to make things easier, AdministratorAccess existing policy is used with its ARN.

aws iam list-policies

Attach the policy to the role with attach-role-policy command.

aws iam attach-role-policy \
	--role-name $ROLE_NAME \
	--policy-arn $POLICY_ARN

Delete

List policies with list-policies command to get the ARN, then detach the policy from the role.

aws iam detach-role-policy \
	--role-name $ROLE_NAME \
	--policy-arn $POLICY_ARN

After the policy is detached, role is deleted with delete-role command.

aws iam delete-role --role-name $ROLE_NAME

AWS Lambda operations

The full list can be found in aws lambda CLI reference page.

Create

List functions with list-functions command to check if the function exists.

aws lambda list-functions

Creating a function is done with create-function command and takes many arguments. Most of the parameters are self-explanatory. Timeout is important, the lambda function execution is suspended after the timeout passes, in current examples, it is 30 seconds, I found that cold start could take up to 15 seconds some times. The lambda configurations are described in AWS examples in C# – create basic Lambda function post.

aws lambda create-function \
	--function-name $FUNCTION_NAME \
	--runtime dotnetcore2.1 \
	--role $ROLE_ARN \
	--handler $HANDLER_STRING_WITH_NAMESPACE_CLASS_METHOD \
	--environment "Variables={AWS_SQS_QUEUE_NAME=$QUEUE_NAME, AWS_SQS_IS_FIFO=$IS_QUEUE_FIFO}" \
	--timeout $FUNCTION_TIMEOUT \
	--zip-file fileb://$PATH_TO_ZIP_FILE)

Once the function is created, it can be linked to an event source, such as DynamoDB. This happens by DynamoDB stream ARN. Once a record is inserted, updated or deleted in DynamoDB, the lambda function is called with this event.

aws lambda create-event-source-mapping \
	--function-name $FUNCTION_NAME \
	--event-source-arn $DYNAMODB_STREAM_ARN \
	--starting-position LATEST)

In case of function already exists, but its code has to be updated, this is done with update-function-code command.

aws lambda update-function-code \
	--function-name $FUNCTION_NAME \
	--zip-file fileb://$PATH_TO_ZIP_FILE)

Along with the code, function configuration can be updated as well with update-function-configuration command.

aws lambda update-function-configuration \
	--function-name $FUNCTION_NAME  \
	--role $ROLE_ARN\
	--handler $HANDLER_STRING_WITH_NAMESPACE_CLASS_METHOD  \
	--environment "Variables={AWS_SQS_QUEUE_NAME=$QUEUE_NAME, AWS_SQS_IS_FIFO=$IS_QUEUE_FIFO}" \
	--timeout $FUNCTION_TIMEOUT

Delete

In order to delete, then the event source UUID has to be obtained, this is done with list-event-source-mappings command.

aws lambda list-event-source-mappings --function-name $FUNCTION_NAME

Then event source mapping is deleted with delete-event-source-mapping command.

aws lambda delete-event-source-mapping --uuid $EVNET_SOURCE_UUID

And finally, the function itself is deleted with delete-function command.

aws lambda delete-function --function-name $FUNCTION_NAME

ECS (Elastic Container Service) operations

The full list can be found in aws ecs CLI reference page.

Create

Before doing anything with ECR, docker login command should be created with get-login, so docker is authenticated with AWS ECR. With eval function, the docker login command is directly executed.

eval $(aws ecr get-login --no-include-email)

Clusters are first listed, in order to evaluate if the application is already deployed.

aws ecs list-clusters

Cluster is created with create-cluster command. A cluster consists of services.

aws ecs create-cluster --cluster-name $CLUSTER_NAME

Existing task definitions are listed, to evaluate whether they are published or not. Task definitions are Docker configurations.

aws ecs describe-task-definition --task-definition $TASK_DEFINITION_NAME

Task definition is created with register-task-definition command.

aws ecs register-task-definition \
	--family $TASK_DEFINITION_NAME \
	--execution-role-arn $ROLE_ARN\
	--network-mode awsvpc \
	--container-definitions $CONTAINER_DEFINITIONS \
	--requires-compatibilities "FARGATE" \
	--cpu "256" \
	--memory "512"

$CONTAINER_DEFINITIONS is a Docker configuration which defines the task definition:

name=$TASK_DEFINITION_NAME,\
image=$IMAGE_TAG,\
environment=[\
	{name=AwsQueueIsFifo,value=$_IS_QUEUE_FIFO},\
	{name=AwsRegion,value=$REGION},\
	{name=AwsQueueName,value=$QUEUE_NAME},\
	{name=AwsAccessKey,value=$AWS_ACCESS_KEY},\
	{name=AwsSecretKey,value=$AWS_SECRET_KEY},\
	{name=AwsQueueAutomaticallyCreate,value=$AWS_QUEUE_AUTO_CREATE},\
	{name=AwsQueueLongPollTimeSeconds,value=$AWS_POLL_TIME_SECONDS}\
],\
logConfiguration={\
	logDriver=awslogs,\
	options={\
		awslogs-group=ecs/$SERVICE_NAME,\
		awslogs-region=$REGION,\
		awslogs-stream-prefix=ecs\
	}\
}

Before creating a service, existing ones are listed with describe-services command. Service has one or more running instances of a task definition. This is how service can scale.

aws ecs describe-services \
	--cluster $CLUSTER_NAME\
	--services $SERVICE_NAME

Creating a service is done with create-service command. $TASK_REVISION is the result of the register-task-definition command. $SUBNET_ID is returned by aws ec2 describe-subnets command.

aws ecs create-service --cluster $CLUSTER_NAME \
	--service-name $SERVICE_NAME \
	--task-definition "$TASK_DEFINITION_NAME:$TASK_REVISION" \
	--desired-count 1 \
	--launch-type "FARGATE" \
	--network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SECURITY_GROUP_ID],assignPublicIp=ENABLED}")

Updating of the service is done with a very update-service similar command.

aws ecs update-service --cluster $CLUSTER_NAME \
	--service $SERVICE_NAME \
	--task-definition "$TASK_DEFINITION_NAME:$TASK_REVISION" \
	--desired-count 1 \
	--network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SECURITY_GROUP_ID],assignPublicIp=ENABLED}")

Delete

In order to delete task definitions, they should be first listed with list-task-definitions command, so the task definition version is available.

aws ecs list-task-definitions

Removing of the task definition is done with deregister-task-definition command. Note that the command does what it says, deregister, it does not delete. The task definition is kept in history in status INACTIVE.

aws ecs deregister-task-definition --task-definition "$TASK_DEFINITION_VERSION"

Deleting the service is done with the delete-service command, the –force parameter also stops the running tasks.

aws ecs delete-service \
	--cluster $CLUSTER_NAME \
	--service $SERVICE_NAME \
	--force

In the end, the whole cluster is deleted with delete-cluster command.

aws ecs delete-cluster --cluster $CLUSTER_NAME

ECR (Elastic Container Registry) operations

The full list can be found in aws ecr CLI reference page.

Delete

The repository is created by Docker when the image is pushed to it. Repository and images inside are deleted with delete-repository command.

aws ecr delete-repository \
	--repository-name $REPOSITORY_NAME \
	--force

EC2 (Elastic Compute Cloud) operations

The full list can be found in aws ec2 CLI reference page.

Create

EC2 is responsible for security groups, which expose the service to the world by applying firewall rules. Before creating the group, it is first searched for presence with describe-security-groups command.

aws ec2 describe-security-groups

The security group is created with create-security-group command.

aws ec2 create-security-group \
	--description $SECIRITY_GROUP_DESCRIPTION\
	--group-name $SECIRITY_GROUP_NAME

Inbound rules are defined with authorize-security-group-ingress command, where ip_permission is a bash function generation the JSON for better reuse.

aws ec2 authorize-security-group-ingress \
	--group-id $SECURITY_GROUP_ID \
	--ip-permissions "[$(ip_permission $SERVICE_PORT)]"

Function generation firewall rule JSON, $1 is an argument given to the function.

function ip_permission() {
	echo "{\"IpProtocol\": \"tcp\", \"FromPort\": $1, \"ToPort\": $1, \"IpRanges\": [{\"CidrIp\": \"0.0.0.0/0\", \"Description\": \"Port $1\"}]}"
}

Subnets are listed with describe-subnets command. Each subnet has 3 availability zones.

aws ec2 describe-subnets

Finally, in order to report the IP of the deployed service, describe-network-interfaces command is used.

aws ec2 describe-network-interfaces --filters "Name=network-interface-id,Values=$networkInterfaceId"

Delete

A security group is deleted by name with delete-security-group command.

aws ec2 delete-security-group --group-name $SECURITY_GROUP

CloudWatch operations

The full list can be found in aws logs CLI reference page.

Delete

CloudWatch logs are created by default from the services. Deleting the logs is done with delete-log-group command. Note that I am using Git Bash on Windows and MSYS_NO_PATHCONV=1 is mandatory because the log group name starts with /.

MSYS_NO_PATHCONV=1 aws logs delete-log-group --log-group-name ecs/$SERVICE_NAME

Conclusion

AWS command-line interface provides tooling to handle all needed operations of the AWS services. It is the preferred way to manage services over the Web user interface.

Related Posts

Read more...

AWS examples in C# – introduction to Serverless framework

Last Updated on by

Post summary: Introduction to Serverless framework and .NET code example of a lambda function with API Gateway.

This post is part of AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository.

When speaking about Serverless there are two concepts and terms that need to be clarified.

Serverless architecture

Serverless architecture is an application architectural concept of the cloud, enables shifting more of your operational responsibilities to the cloud. In the current examples, AWS is used, but this is a valid concept for Azure and Google cloud. Serverless allows building and running applications and services without thinking about servers.

Serverless framework

Serverless framework is a toolset that makes deployment of serverless applications to different cloud providers extremely easy and streamlined. It supports the following cloud providers: AWS, Google Cloud, Azure, OpenWhisk, and Kubeless and following programming languages: nodeJS, Go, Python, Swift, Java, PHP, and Ruby.

AWS Lambda

AWS Lambda allows easy ramp-up of service without all the hassle to manage servers and environments. The ready code is uploaded to Lambda and automatically run. More detailed information and design considerations about AWS Lambda can be found in AWS examples in C# – working with Lambda functions post.

CloudFormation

AWS CloudFormation provides infrastructure as a code (IoC) capabilities. It defines a common language to model and provision AWS application resources. AWS resources and applications are described in YAML or JSON files, which are then provisioned by CloudFormation. This gives a single source of truth. The Serverless framework uses applications when creating the underlying AWS JSON CloudFormation templates. What Serverless framework is doing is to translate its custom YAML format to a JSON CloudFormation templates, which can be found in .serverless folder of DynamoDbServerless after the deployment to AWS is done.

Create a Serverless application

Before creating the application, Serverless needs to be installed. Installation is possible as a standalone binary or as a Node.JS package. I prefer the latter because it is much simpler.

npm install -g serverless

Creating an empty application is done with the following command:

sls create --template aws-csharp --path MyService

The command outputs a nice message:

$ sls create --template aws-csharp --path MyService
Serverless: Generating boilerplate...
Serverless: Generating boilerplate in "C:\MyService"
 _______                             __
|   _   .-----.----.--.--.-----.----|  .-----.-----.-----.
|   |___|  -__|   _|  |  |  -__|   _|  |  -__|__ --|__ --|
|____   |_____|__|  \___/|_____|__| |__|_____|_____|_____|
|   |   |             The Serverless Application Framework
|       |                           serverless.com, v1.61.3
 -------'

Serverless: Successfully generated boilerplate for template: "aws-csharp"

Apart from the default lambda project created by AWS tools, see AWS examples in C# – create basic Lambda function post for more details, the Serverless project is not split to src and test. It is a good idea to manually split the project in order to add tests. The configuration of Serverless projects is done in serverless.yml file. The default one is very simple, it states the provider and runtime, which is aws (Amazon) and dotnetcore2.1. The handler shows which method is being called when this lambda is invoked. In the default example, the handler is CsharpHandlers::AwsDotnetCsharp.Handler::Hello, which means CsharpHandlers is the assembly name, configured in aws-csharp.csproj file. The namespace is AwsDotnetCsharp, the class name is Handler and the method is Hello.

service: myservice

provider:
  name: aws
  runtime: dotnetcore2.1

package:
  individually: true

functions:
  hello:
    handler: CsharpHandlers::AwsDotnetCsharp.Handler::Hello

    package:
      artifact: bin/release/netcoreapp2.1/hello.zip

After the application is created, it has to be built with build.cmd or build.sh scripts.

Before deployment, AWS credential should be set:

export AWS_ACCESS_KEY_ID=KIA57FV4.....
export AWS_SECRET_ACCESS_KEY=mSgsxOWVh...
export AWS_DEFAULT_REGION=us-east-1

Deployment happens with sls deploy –region $AWS_DEFAULT_REGION command, the result of deployment command is:

Testing of the function can be done with sls invoke -f hello –region $AWS_DEFAULT_REGION command. Result of testing is:

{
	"Message": "Go Serverless v1.0! Your function executed successfully!",
	"Request": {
		"Key1": null,
		"Key2": null,
		"Key3": null
	}
}

Finally, the stack can be deleted with sls remove –region $AWS_DEFAULT_REGION command.

Use API Gateway

The default, Hello World example is pretty simple. In current examples, I have elaborated a bit on Lambda usage with API Gateway in order to expose it as a RESTful service. Two lambda functions are defined inside functions node, for movies and actions. Each has a handler node which defines where is the code that lambda function is executing. With events/http node and API Gateway endpoint is created. Each endpoint has a path and a method. Movies are using GET method, Actors are working via POST method. Both functions’ handler methods receive APIGatewayProxyRequest object and return APIGatewayProxyResponse object. The payload is found in Body string property of APIGatewayProxyRequest object.

Actors function is querying the Actors table. Mode details on the query and how it is constructed in BuildQueryRequest method can be found in Querying using the low-level interface section in AWS examples in C# – basic DynamoDB operations post.

Movies lambda function is getting the JSON document from Movies table. More details on getting the data can be found in Get item using document interface section in AWS examples in C# – basic DynamoDB operations post.

Actors lambda function has two more security features defined. One is an API Key defined with private: true setting. The request should have x-api-key header with the value which is returned by deployment command, otherwise, an HTTP status code 403 (Forbidden) is returned by API Gateway. See Run the project in AWS section in AWS examples in C# – run the solution post how to obtain the proper value of aws-examples-csharp-api-key API key. The example given here is not really scalable, more details on how to manage properly API keys can be found in Managing secrets, API keys and more with Serverless article.

The other security feature is lambda authorizer configured with authorizer: authorizer setting. The authorizer lambda function is not really doing anything significant in the examples, it uses UserManagement class to check if the Authorization header has proper value.  In the real-life, this would be a database call to get the user authorizations, not like in the examples with a dummy token. The request should have Authorization header with value Bearer validToken, otherwise, an HTTP status code 401 (Unauthorized) is returned by API Gateway.

serverless.yml

service: DynamoDbServerless

provider:
  name: aws
  runtime: dotnetcore2.1
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - dynamodb:Query
      Resource: ${env:actorsTableArn}
    - Effect: "Allow"
      Action:
        - dynamodb:DescribeTable
        - dynamodb:GetItem
      Resource: ${env:moviesTableArn}
  apiKeys:
    - aws-examples-csharp-api-key

package:
  individually: true
  artifact: bin/release/netcoreapp2.1/DynamoDbServerless.zip

functions:
  movies:
    handler: DynamoDbServerless::DynamoDbServerless.Handlers.MoviesHandler::GetMovie
    events:
      - http:
          path: movies/title/{title}
          method: get

  actors:
    handler: DynamoDbServerless::DynamoDbServerless.Handlers.ActorsHandler::QueryActors
    events:
      - http:
          path: actors/search
          method: post
          private: true
          authorizer: authorizer

  authorizer:
    handler: DynamoDbServerless::DynamoDbServerless.Handlers.AuthorizationHandler::Authorize

ActorsFunction

public async Task<APIGatewayProxyResponse> QueryActors(APIGatewayProxyRequest request, ILambdaContext context)
{
	context.Logger.LogLine($"Query request: {_jsonConverter.SerializeObject(request)}");

	var requestBody = _jsonConverter.DeserializeObject<ActorsSearchRequest>(request.Body);
	if (string.IsNullOrEmpty(requestBody.FirstName))
	{
		return new APIGatewayProxyResponse
		{
			StatusCode = (int)HttpStatusCode.BadRequest,
			Body = "FirstName is mandatory"
		};
	}
	var queryRequest = BuildQueryRequest(requestBody.FirstName, requestBody.LastName);

	var response = await _dynamoDbReader.QueryAsync(queryRequest);
	context.Logger.LogLine($"Query result: {_jsonConverter.SerializeObject(response)}");

	var queryResults = BuildActorsResponse(response);

	return new APIGatewayProxyResponse
	{
		StatusCode = (int)HttpStatusCode.OK,
		Body = _jsonConverter.SerializeObject(queryResults)
	};
}

MoviesFunction

public async Task<APIGatewayProxyResponse> GetMovie(APIGatewayProxyRequest request, ILambdaContext context)
{
	context.Logger.LogLine($"Query request: {_jsonConverter.SerializeObject(request)}");

	var title = WebUtility.UrlDecode(request.PathParameters["title"]);
	var document = await _dynamoDbReader.GetDocumentAsync(TableName, title);
	context.Logger.LogLine($"Query response: {_jsonConverter.SerializeObject(document)}");

	if (document == null)
	{
		return new APIGatewayProxyResponse { StatusCode = (int)HttpStatusCode.NotFound };
	}

	var movie = new Movie
	{
		Title = document["Title"],
		Genre = (MovieGenre)int.Parse(document["Genre"])
	};
	return new APIGatewayProxyResponse
	{
		StatusCode = (int)HttpStatusCode.OK,
		Body = _jsonConverter.SerializeObject(movie)
	};
}

MoviesFunction

public async Task<APIGatewayCustomAuthorizerResponse> Authorize(APIGatewayCustomAuthorizerRequest request, ILambdaContext context)
{
	context.Logger.LogLine($"Query request: {_jsonConverter.SerializeObject(request)}");

	var userInfo = await _userManager.Authorize(request.AuthorizationToken?.Replace("Bearer ", string.Empty));

	return new APIGatewayCustomAuthorizerResponse
	{
		PrincipalID = userInfo.UserId,
		PolicyDocument = new APIGatewayCustomAuthorizerPolicy
		{
			Version = "2012-10-17",
			Statement = new List<APIGatewayCustomAuthorizerPolicy.IAMPolicyStatement>
			{
				new APIGatewayCustomAuthorizerPolicy.IAMPolicyStatement
				{
					Action = new HashSet<string> {"execute-api:Invoke"},
					Effect = userInfo.Effect.ToString(),
					Resource = new HashSet<string> { request.MethodArn }
				}
			}
		}
	};
}

UserManager

public interface IUserManager
{
	Task<UserInfo> Authorize(string token);
}

public class UserManager : IUserManager
{
	private const string ValidToken = "validToken";
	private const string UserId = "usedId";

	public async Task<UserInfo> Authorize(string token)
	{
		var userInfo = new UserInfo
		{
			UserId = UserId,
			Effect = token == ValidToken ? EffectType.Allow : EffectType.Deny
		};

		return userInfo;
	}
}

Conclusion

The Serverless framework is making deployment and maintenance of lambdas very easy. It also supports different cloud providers.

Related Posts

Read more...

AWS examples in C# – create basic Lambda function

Last Updated on by

Post summary: Code examples on how to create AWS Lambda function in .NET Core.

This post is part of AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository.

AWS Lambda

AWS Lambda allows easy ramp-up of service without all the hassle to manage servers and environments. The ready code is uploaded to Lambda and automatically run. More detailed information and design considerations about AWS Lambda can be found in AWS examples in C# – working with Lambda functions post.

Create Lambda

Amazon provides Amazon.Lambda.Templates NuGet package which contains a lot of templates for Lambda functions. The NuGet package can be installed with dotnet new -i Amazon.Lambda.Templates. Once templates are installed, a new empty function is created with:

dotnet new lambda.EmptyFunction --name MyFunction

Two projects are created: src and test. The source project is with netcoreapp2.1 runtime and has reference to Amazon.Lambda.Core and Amazon.Lambda.Serialization.Json NuGet packages. It has only one simple function that takes a string input and converts it to uppercase. The test project has a unit test that tests this function.

Once the function is ready, it can be deployed to AWS Lambda and tested. In order to deploy, the Amazon lambda tools should be installed with: dotnet tool install -g Amazon.Lambda.Tools. Once the tool is installed, there should be an IAM role created with name MyRole for e.g. Before deploying the lambda, environment variable with AWS access information should be set:

export AwsAccessKey=KIA57FV4.....
export AwsSecretKey=mSgsxOWVh...
export AwsRegion=us-east-1

Then lambda is deployed with:

dotnet lambda deploy-function MyFunction \
	--function-role MyRole \
	--project-location src/MyFunction \
	--region $AwsRegion \
	--aws-access-key-id $AwsAccessKey \
	--aws-secret-key $AwsSecretKey

The function can be tested with:

dotnet lambda invoke-function MyFunction \
	--payload "Just Testing the Payload" \
	--project-location src/MyFunction \
	--region $AwsRegion \
	--aws-access-key-id $AwsAccessKey \
	--aws-secret-key $AwsSecretKey

The result is shown:

Amazon Lambda Tools for .NET Core applications (3.3.1)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Payload:
"JUST TESTING THE PAYLOAD"

Log Tail:
START RequestId: 3f1844e0-7437-4219-a391-621a55dec0e9 Version: $LATEST
END RequestId: 3f1844e0-7437-4219-a391-621a55dec0e9
REPORT RequestId: 3f1844e0-7437-4219-a391-621a55dec0e9  Duration: 925.38 ms    Billed Duration: 1000 ms Memory Size: 256 MB     Max Memory Used: 62 MB  Init Duration: 196.85 ms

The duration in the example above is 925.38ms, billed duration is 1000ms. This is the first run, the cold start as described in the previous section, which took almost a second for a very simple function. Next run the duration was 0.28ms.

The lambda can also manually be deployed and tested, see how in Run a “Hello, World!” with AWS Lambda article.

Listen to DynamoDB events

In AWS examples in C# – create a service working with DynamoDB post, I have described more about DynamoDB and its streams are very well integrated with AWS Lambda. In the current examples, the lambda functions are designed to process DynamoDB stream events. DynamoDB stream ARN (Amazon Resource Name) is defined as an event source for the lambda. Then the lambda receives DynamoDBEvent object, which is defined in Amazon.Lambda.DynamoDBEvents NuGet package. There is not much business logic in the lambda function, once the event object is received, it is read, logged to AWS CloudWatch with ILambdaContext, and its data is written to another DynamoDB table and to an SQS queue. For this purpose, references to AWSSDK.DynamoDBv2 and AWSSDK.SQS NuGet packages are made. In the example code, I have created SQS and DynamoDB proxies, so functionalities are isolated and only what is needed is exposed, these are DynamoDbWriter and SqsWriter. In both proxies, the configuration is done with Region, which is read by AWS_REGION environment variable. This variable is always present in the AWS Lambda environment. In this case, there is no need for authentication with AWSCredentials class, as everything happens inside AWS. The lambda function has two constructors, one is receiving instances of both proxies, if none are passed it instantiates them. An automatic dependency injection is not used with lambda. This constructor is used by the unit tests to pass mocked objects. The parameterless constructor is needed by AWS Lambda to instantiate the function class.

MoviesFunction

public class MoviesFunction
{
	private readonly ISqsWriter _sqsWriter;
	private readonly IDynamoDbWriter _dynamoDbWriter;

	public MoviesFunction() : this(null, null) { }

	public MoviesFunction(ISqsWriter sqsWriter, IDynamoDbWriter dynamoDbWriter)
	{
		_sqsWriter = sqsWriter ?? new SqsWriter();
		_dynamoDbWriter = dynamoDbWriter ?? new DynamoDbWriter();
	}

	public async Task FunctionHandler(DynamoDBEvent dynamoEvent, ILambdaContext context)
	{
		context.Logger.LogLine($"Beginning to process {dynamoEvent.Records.Count} records...");

		foreach (var record in dynamoEvent.Records)
		{
			context.Logger.LogLine($"Event ID: {record.EventID}");
			context.Logger.LogLine($"Event Name: {record.EventName}");

			var streamRecordJson = _dynamoDbWriter.SerializeStreamRecord(record.Dynamodb);
			context.Logger.LogLine($"DynamoDB Record:{streamRecordJson}");
			context.Logger.LogLine(streamRecordJson);

			var logEntry = new LogEntry
			{
				Message = $"Movie '{record.Dynamodb.NewImage["Title"].S}' processed by lambda",
				DateTime = DateTime.Now
			};
			await _sqsWriter.WriteLogEntryAsync(logEntry);
			await _dynamoDbWriter.PutLogEntryAsync(logEntry);
		}

		context.Logger.LogLine("Stream processing complete.");
	}
}

DynamoDbWriter

public interface IDynamoDbWriter
{
	Task PutLogEntryAsync(LogEntry logEntry);
	string SerializeStreamRecord(StreamRecord streamRecord);
}

public class DynamoDbWriter : IDynamoDbWriter
{
	private static readonly string Region = Environment.GetEnvironmentVariable("AWS_REGION") ?? "us-east-1";

	private readonly IAmazonDynamoDB _dynamoDbClient;
	private readonly JsonSerializer _jsonSerializer;

	public DynamoDbWriter()
	{
		var dynamoDbConfig = new AmazonDynamoDBConfig
		{
			RegionEndpoint = RegionEndpoint.GetBySystemName(Region)
		};
		_dynamoDbClient = new AmazonDynamoDBClient(dynamoDbConfig);
		_jsonSerializer = new JsonSerializer();
	}

	public async Task PutLogEntryAsync(LogEntry logEntry)
	{
		var request = new PutItemRequest
		{
			TableName = "LogEntries",
			Item = new Dictionary<string, AttributeValue>
			{
				{"Message", new AttributeValue {S = logEntry.Message}},
				{"DateTime", new AttributeValue {S = logEntry.ToString()}}
			}
		};

		await _dynamoDbClient.PutItemAsync(request);
	}

	public string SerializeStreamRecord(StreamRecord streamRecord)
	{
		using (var writer = new StringWriter())
		{
			_jsonSerializer.Serialize(writer, streamRecord);
			return writer.ToString();
		}
	}
}

SqsWriter

public interface ISqsWriter
{
	Task WriteLogEntryAsync(LogEntry logEntry);
}

public class SqsWriter : ISqsWriter
{
	private static readonly string QueueName = Environment.GetEnvironmentVariable("AWS_SQS_QUEUE_NAME");
	private static readonly bool IsQueueFifo = bool.Parse(Environment.GetEnvironmentVariable("AWS_SQS_IS_FIFO") ?? "false");
	private static readonly string Region = Environment.GetEnvironmentVariable("AWS_REGION") ?? "us-east-1";

	private readonly IAmazonSQS _sqsClient;

	public SqsWriter()
	{
		var sqsConfig = new AmazonSQSConfig
		{
			RegionEndpoint = RegionEndpoint.GetBySystemName(Region)
		};
		_sqsClient = new AmazonSQSClient(sqsConfig);
	}

	public async Task WriteLogEntryAsync(LogEntry logEntry)
	{
		var queueUrl = await _sqsClient.GetQueueUrlAsync(QueueName);
		var sendMessageRequest = new SendMessageRequest
		{
			QueueUrl = queueUrl.QueueUrl,
			MessageBody = JsonConvert.SerializeObject(logEntry),
			MessageAttributes = SqsMessageTypeAttribute.CreateAttributes(typeof(LogEntry).Name)
		};
		if (IsQueueFifo)
		{
			sendMessageRequest.MessageGroupId = typeof(LogEntry).Name;
			sendMessageRequest.MessageDeduplicationId = Guid.NewGuid().ToString();
		}

		await _sqsClient.SendMessageAsync(sendMessageRequest);
	}
}

Conclusion

As shown in the current post, it is very easy to create lambda function, deploy and run it. The current post shows a lambda function that listens to DynamoDB events and processes those events by writing into another DynamoDB table and SQS queue.

Related Posts

Read more...

AWS examples in C# – working with Lambda functions

Last Updated on by

Post summary: Iintroduction to AWS Lambda functions.

This post is part of AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository.

AWS Lambda

AWS Lambda allows easy ramp-up of service without all the hassle to manage servers and environments. The ready code is uploaded to Lambda and automatically run. AWS Lambda automatically scales applications by running code in response to each trigger. The code runs in parallel and processes each trigger individually, scaling precisely with the size of the workload.

Main concepts

There are several terms that need to be briefly explained to get some understanding of what AWS Lambda offers.

  • Function – A code written in a programming language, for supported runtime, that does some computational work.
  • Runtime – Allow running of functions in different programming languages, supported languages are: Node.js, Python, Ruby, Java, Go, .NET.
  • Event – A JSON formatted document that contains data for a function to process, which is converted to object by the runtime and passed to the function.
  • Concurrency – The number of requests that your function is serving at any given time. If a function is invoked, meanwhile executing another task, then another instance is provisioned, increasing the function’s concurrency.
  • Trigger – A resource or configuration that invokes a Lambda function. This includes AWS services, applications, and event source mappings.
  • Event source mapping – A resource in Lambda that reads items from a stream or queue and invokes a function.

AWS Lambda applications

Lambda is the actual name of serverless functions in AWS. Along with the lambda functions, AWS supports also a concept of an application, which is a combination of Lambda functions, event sources, and other resources that work together to perform tasks. AWS CloudFormation is used to collect application’s components into a single package that can be deployed and managed as one resource. Applications make Lambda projects portable.

CloudFormation

AWS CloudFormation provides infrastructure as a code (IoC) capabilities. It defines a common language to model and provision AWS application resources. AWS resources and applications are described in YAML or JSON files, which are then provisioned by CloudFormation. This gives a single source of truth.

API Gateway

API Gateway is a fully managed service that makes it easy to create, publish, maintain, monitor, and secure APIs. APIs act as the “front door” for applications to access data, business logic, or functionality from backend services. It is very easy to create RESTful APIs and WebSocket APIs with API Gateway. It supports traffic management, CORS support, authorization, access control, throttling, monitoring, and API version management.

API Keys

API keys are the way to create usage plans, so APIs can be given to customers as a product offering with predefined request rates and quotas. A usage plan is created in AWS and it has a throttling limit, which is basically the request rate limit that is applied to each API key that you add to the usage plan. A quota is configured to the usage plan and applied to its API keys. This is the maximum number of requests with a given API key that can be submitted within a specified time interval. API keys can be provided to API Gateway in the X-API-Key header, this is what is shown in the current examples. Another way to work with API keys is with a lambda authorizer function, which returns the API key as part of the authorization response. API Keys can be created or imported from a file. Important is that API keys are not used to manage authentication and authorization.

Access control

Access control to a REST API in API Gateway can be done with several mechanisms:

  • Resource policies
  • Standard AWS IAM roles and policies
  • IAM tags can be used together with IAM policies to control access
  • Endpoint policies for interface VPC endpoints
  • Lambda authorizers
  • Amazon Cognito user pools

Lambda (custom) authorizers

In the examples given, lambda (formerly knows as custom) authorizer is used. API Gateway uses a dedicated Lambda function to do the authorization. More details on how to use authorizers can be found in AWS examples in C# – introduction to Serverless framework post.

CloudWatch

CloudWatch is a monitoring and observability service. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing a unified view of AWS resources, applications, and services. CloudWatch can be used to detect anomalous behavior, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues. By default, AWS Lambda is logging into CloudWatch. This makes it very easy to trace lambda function issues.

Design considerations

There are some specifics that have to be taken into consideration when using lambdas. One of the benefits of lambdas is to be cost-effective. Users can select what amount of RAM to set for the lambda function when it is created. This is done with –memory-size in aws lambda create-function command, see more in AWS examples in C# – deploy with AWS CLI commands post. The default value is 128MB and CPU is allocated proportionally. Sometimes defining too low memory can end up in unexpected performance issues. This should be monitored and optimized based on specific programming language and code. Lambdas are paid per 100ms execution time, so this also should be taken into consideration when tweaking the memory setting. In terms of cost-effectiveness, it is more expensive to add more RAM in order to optimize from 100ms to 50ms execution time, because 100ms on the higher amount of RAM is being paid. It has to be analyzed how much it makes sense for the end-users. Also, another consideration is that API Gateway adds additional delay in total time for the request. CloudWatch logs cost money, so awareness is needed about how much data a lambda function is logging. More pitfalls with more details using lambdas can be found in Serverless Pitfalls: Issues With Running a Startup on AWS Lambda article.

Still, the main consideration for lambda performance is so-called cold start. If the function has not been run for a while then it needs some time for the first request to go through. I’ve seen up to 4 seconds when experimenting, although I had not really measured it. Theoretically, there is an option to ping your API at a certain amount of time to keep it “warm”. In practice, for heavy loads, AWS runs parallel instances of the lambda, in order to handle the traffic, and each new instance will have a cold start.

Create lambda

Practical examples of how to create AWS Lambda functions are available in the following posts:

Conclusion

AWS Lambda is a very convenient and easy way to create running applications with minimal overhead. There are certain design considerations such as lambda cold start that has to be taken into consideration when deciding on lambda usage.

Related Posts

Read more...

AWS examples in C# – run the solution

Last Updated on by

Post summary: Explanation of how to install and use the solution in AWS examples in C# blog post series.

This post is part of AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository. In the current post, I give information on how to install and run the project.

Disclaimer

The solution has commands to deploy to the cloud as well as to clean resources. Note not all resources are cleaned, read more in the Cleanup section. In order to be run in AWS valid account is needed. I am not going to describe how to create an account. If an account is present, then there is also knowledge and awareness of how to use it.

Important: current examples generate costs on AWS account. Use cautiously at your own risk!

Restrictions

The project was tested to be working on Linux and Windows. For Windows, it is working only with Git Bash. The project requires a valid AWS account.

Required installations

In order to fully run and enjoy the project following needs to be installed:

Configurations

AWS CLI has to be configured in order to run properly. It happens with aws configure. If there is no AWS account, this is not an issue, put some values for access and secret key and put a correct region, like us-east-1.

Import Postman collection, in order to be able to try the examples. Postman collection is in aws.examples.csharp.postman_collection.json file in the code. This is an optional step, below there are cURL examples also.

Run the project in AWS

Running on AWS requires the setting of environment variables:

export AwsAccessKey=KIA57FV4.....
export AwsSecretKey=mSgsxOWVh...
export AwsRegion=us-east-1

Then the solution is deployed to AWS with ./solution-deploy.sh script. Note that the output of the command gives the API Gateway URL and API key, as well as the SqsWriter and SqsReader endpoints. See image below:

Run the project partly in AWS

The most expensive part of the current setup is the running of the docker containers in EC2 (Elastic Compute Cloud). I have prepared a script called ./solution-deploy-hybrid.sh, which runs the containers locally and all other things are in AWS. Still, environment variables from the previous section are mandatory to be set. I believe this is the optimal variant if you want to test and run the code in a “production-like” environment.

Run the project in LocalStack

LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. It spins up a testing environment on local machines that provide the same functionality and APIs as the real AWS cloud environment. I have experimented with LocalStack, there is a localstack branch in GitHub. The solution can be run with solution-deploy-localstack.sh command. I cannot really estimate if this is a good alternative, because I am running the free tier in AWS and the most expensive part is ECS, which I skip by running the containers locally, instead of AWS. See the previous section on how to run a hybrid.

Usage

There is a Postman collection which allows easy firing of the requests. Another option is to use cURL, examples of all requests with their Postman names are available below.

SqsWriter

SqsWriter is a .NET Core 3.0 application, that is dockerized and run in AWS ECS (Elastic Container Service). It exposes an API that can be used to publish Actor or Movie objects. There is also a health check request. After AWS deployment proper endpoint is needed. The endpoint can be found as an output of deployment scripts. See the image above.

PublishActor

curl --location --request POST 'http://localhost:5100/api/publish/actor' \
--header 'Content-Type: application/json' \
--data-raw '{
	"FirstName": "Bruce",
	"LastName": "Willis"
}'

PublishMovie

curl --location --request POST 'http://localhost:5100/api/publish/movie' \
--header 'Content-Type: application/json' \
--data-raw '{
	"Title": "Die Hard",
	"Genre": "Action Movie"
}'

When Actor or Movie is published, it goes to the SQS queue, SqsReader picks it up from there and processes it. What is visible in the logs is that both LogEntryMessageProcessor and ActorMessageProcessor are invoked. See the screenshot:

SqsWriterHealthCheck

curl --location --request GET 'http://localhost:5100/health'

SqsReader

SqsReader is a .NET Core 3.0 application, that is dockerized and run in AWS ECS. It has a consumer that listens to the SQS queue and processes the messages by writing them into appropriate AQS DynamoDB tables. It also exposes API to stop or start processing, as long as reprocess the dead-letter queue or simply get the queue status. After AWS deployment proper endpoint is needed. The endpoint can be found as an output of deployment scripts. See the image above.

ConsumerStart

curl --location --request POST 'http://localhost:5200/api/consumer/start' \
--header 'Content-Type: application/json' \
--data-raw ''

ConsumerStop

curl --location --request POST 'http://localhost:5200/api/consumer/stop' \
--header 'Content-Type: application/json' \
--data-raw ''

ConsumerStatus

curl --location --request GET 'http://localhost:5200/api/consumer/status'

ConsumerReprocess
If this one is invoked with no messages in the dead-letter queue then it takes 20 seconds to finish, because it actually waits for long polling timeout.

curl --location --request POST 'http://localhost:5200/api/consumer/reprocess' \
--header 'Content-Type: application/json' \
--data-raw ''

SqsReaderHealthCheck

curl --location --request GET 'http://localhost:5200/health'

ActorsServerlessLambda

This lambda is managed by the Serverless framework. It is exposed as REST API via AWS API Gateway. It also has a custom authorizer as well as API Key attached. Those are described in a further post.

ServerlessActors

In the case of AWS, the API Key and URL are needed, those can be obtained from deployment command logs. See the screenshot above. Put correct values to CHANGE_ME and REGION placeholders. Request is:

curl --location --request POST 'https://CHANGE_ME.execute-api.REGION.amazonaws.com/dev/actors/search' \
--header 'Content-Type: application/json' \
--header 'x-api-key: CHANGE_ME' \
--header 'Authorization: Bearer validToken' \
--data-raw '{
    "FirstName": "Bruce",
    "LastName": "Willis"
}'

MoviesServerlessLambda

ServerlessMovies

Put correct values to CHANGE_ME and REGION placeholders. Request is:

curl --location --request GET 'https://CHANGE_ME.execute-api.REGION.amazonaws.com/dev/movies/title/Die Hard'

Cleanup

Nota bene: This is a very important step, as leaving the solution running in AWS will accumulate costs.

In order to stop and clean up all AWS resources run ./solution-delete.sh script.

Nota bene: There a resource that is not automatically deleted by the scripts. This is a Route53 resource created by AWS Cloud Map. It has to be deleted with the following commands. Note that the id in the delete command comes from the result of list-namespaces command.

aws servicediscovery list-namespaces
aws servicediscovery delete-namespace --id ns-kneie4niu6pwwela

Verify cleanup

In order to be sure there are no leftovers from the examples, following AWS services has to be checked:

  • SQS
  • DynamoDB
  • IAM -> Roles
  • EC2 -> Security Groups
  • ECS -> Clusters
  • ECS -> Task Definitions
  • ECR -> Repositories
  • Lambda -> Functions
  • Lambda -> Applications
  • CloudFormation -> Stacks
  • S3
  • CloudWatch -> Log Groups
  • Route 53
  • AWS Cloud Map

On top of it, Billing should be regularly monitored to ensure no costs are applied.

Conclusion

This post describes how to run and they the solution described in AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS series

Related Posts

Read more...

AWS examples in C# – working with SQS, DynamoDB, Lambda, ECS

Last Updated on by

Post summary: Overview of the AWS examples in C# series.

In several blog posts, I give some practical examples of how to use AWS SQS, DynamoDB, Lambda with C# code. The code used for this series of blog posts is located in aws.examples.csharp GitHub repository.

Introduction

AWS stands for Amazon Web Services, it is a subsidiary of Amazon that provides on-demand cloud computing platforms and APIs to individuals, companies, and governments, on a metered pay-as-you-go basis. AWS is one of the big cloud service providers. The others are Microsoft Azure and Google Cloud. All three cloud service providers have functions that are semantically common but differ in practical implementation. Also, every one of them has its own flavors. I have chosen to use AWS for these examples as it is something I have used before and I am most comfortable with it.

Architectural overview

In order to get a full understanding of the architecture, I have prepared this very basic diagram. It illustrates what services are there and how they communicate.

SqsReader and SqsWriter

Both are .NET Core 3.0 microservices running in docker containers. The images are uploaded in AWS ECR (Elastic Container Registry) and containers are run in AWS ECS (Elastic Container Service). SqsReader has a REST endpoint, by which an Actor or Movie can be posted. Both are pushed as a message to AWS SQS (Simple Queue Service). SqsWriter is listening to the SQS and in case of a message, it processes it. If the message is from type Actor or Movie then SqsReader saves it to the respective AWS DynamoDB tables. If the message is LogEntry, then the message is only output into SqsReader logs.

ActorsLambdaFunction and MoviessLambdaFunction

Both are .NET Core 2.1 lambda functions run in AWS Lambda. They listen to Actors and Movies DynamoDB tables changes and in case of new entries, they write to LogEntries DynamoDB table. Also, they write SQS messages from type LogEntry, which are then read by SqsReader.

ActorsServerlessLambda and MoviesServerlessLambda

Those are again lambda functions by are fully managed by the Serverless framework. They have a lambda application defined as well as Cloud Formation templates. They expose a REST API trough AWS API Gateway, by which the Actors table can be queried or a movie can be got from the Movies table.

Post in the series

This is a long series of posts describing into detail all the pieces of the architectural diagram above. Also, every aspect of the code in the repository is explained in detail in subsequent blog posts. It was a very interesting learning opportunity for me, which I would like to share. Here are the posts in the series:

Future plans

There are several topics I would like to go into as well, but there is no code yet for them into the GitHub repository. Those are:

  • AWS examples in C# – manage with Terraform
  • AWS examples in C# – use AWS Cognito for API Gateway authorizer

Conclusion

These series of posts are intended to give some basic overview of important AWS services and how to use them in C# code.

Related Posts

Read more...