GitLab CI template for Amazon Web Services¶

This project implements a GitLab CI/CD template to deploy your application to Amazon Web Services.

Usage¶

This template can be used both as a CI/CD component or using the legacy include:project syntax.

Use as a CI/CD component¶

Add the following to your .gitlab-ci.yml:

include:
  # 1: include the component
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws@5.2.3
    # 2: set/override component inputs
    inputs:
      # ⚠ this is only an example
      base-app-name: wonderapp
      review-enabled: true
      staging-enabled: true
      prod-enabled: true

Use as a CI/CD template (legacy)¶

Add the following to your .gitlab-ci.yml:

include:
  # 1: include the template
  - project: 'to-be-continuous/aws'
    ref: '5.2.3'
    file: '/templates/gitlab-ci-aws.yml'

variables:
  # 2: set/override template variables
  # ⚠ this is only an example
  AWS_BASE_APP_NAME: wonderapp
  AWS_REVIEW_ENABLED: "true"
  AWS_STAGING_ENABLED: "true"
  AWS_PROD_ENABLED: "true"

Understand¶

This chapter introduces key notions and principle to understand how this template works.

Managed deployment environments¶

This template implements continuous delivery/continuous deployment for projects hosted on AWS.

It allows you to manage automatic deployment & cleanup of standard predefined environments. Each environment can be enabled/disabled by configuration. If you're not satisfied with predefined environments and/or their associated Git workflow, you may implement you own environments and workflow, by reusing/extending the base (hidden) jobs. This is advanced usage and will not be covered by this documentation.

The following chapters present the managed predefined environments and their associated Git workflow.

Review environments¶

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment¶

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments¶

Lastly, the template supports 2 environments associated to your production branch (master or main by default):

a staging environment (an iso-prod environment meant for testing and validation purpose),
the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Supported authentication methods¶

The AWS template supports two kinds of authentication:

basic authentication with AWS access key ID & secret access key,
or federated authentication using OpenID Connect.

Basic authentication¶

When using basic authentication (AWS access key ID & secret access key), this template does not manage AWS authentication.

That means you'll have to manage AWS authentication by yourself, according to the aws CLI configuration options (configuration file, CLI options, environment variables).

For credentials management, we strongly advise to use environment variables configuration, managed as GitLab CI secret variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, possibly AWS_ROLE_ARN).

If you have to manage different set of authentication credentials depending on managed environments, you shall either use GitLab scoped variables or our scoped variables syntax to limit/override some variables values, using $CI_ENVIRONMENT_NAME as the conditional variable.

Example: different credentials for production ( declared as project variables)

# global AWS credentials
AWS_ACCESS_KEY_ID: "<my-nonprod-access-key-id>"
AWS_SECRET_ACCESS_KEY: "<my-nonprod-secret-access-key>"

# overridden configuration for production
scoped__AWS_ACCESS_KEY_ID__if__CI_ENVIRONMENT_NAME__equals__production: "<my-prod-access-key-id>"
scoped__AWS_SECRET_ACCESS_KEY__if__CI_ENVIRONMENT_NAME__equals__production: "<my-prod-secret-access-key>"

Federated authentication using OpenID Connect¶

The AWS template supports OpenID Connect to retrieve temporary credentials.

If you wish to use this authentication mode, please activate and configure the OIDC variant.

Deployment context variables¶

In order to manage the various deployment environments, this template provides a couple of dynamic variables that you might use in your hook scripts, deployment manifests and other deployment resources:

${environment_type}: the current deployment environment type (review, integration, staging or production)
${environment_name}: a generated application name to use for the current deployment environment (ex: myapp-review-fix-bug-12 or myapp-staging) - details below

Generated environment name¶

The ${environment_name} variable is generated to designate each deployment environment with a unique and meaningful application name. By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels... It is built from:

the application base name (defaults to $CI_PROJECT_NAME but can be overridden globally and/or per deployment environment - see configuration variables)
GitLab predefined $CI_ENVIRONMENT_SLUG variable (sluggified name, truncated to 24 characters)

The ${environment_name} variable is then evaluated as:

<app base name> for the production environment
<app base name>-$CI_ENVIRONMENT_SLUG for all other deployment environments
${environment_name} can also be overriden per environment with the appropriate configuration variable

Examples (with an application's base name myapp):

`$environment_type`	Branch	`$CI_ENVIRONMENT_SLUG`	`$environment_name`
`review`	`feat/blabla`	`review-feat-bla-xmuzs6`	`myapp-review-feat-bla-xmuzs6`
`integration`	`develop`	`integration`	`myapp-integration`
`staging`	`main`	`staging`	`myapp-staging`
`production`	`main`	`production`	`myapp`

Deployment and cleanup scripts¶

The AWS template requires you to provide a shell script that fully implements your application deployment and cleanup using the aws CLI and all other tools available in the selected Docker image.

The deployment script is searched as follows:

look for a specific aws-deploy-$environment_type.sh in the $AWS_SCRIPTS_DIR directory in your project (e.g. aws-deploy-staging.sh for staging environment),
if not found: look for a default aws-deploy.sh in the $AWS_SCRIPTS_DIR directory in your project,
if not found: the deployment job will fail.

The cleanup script is searched as follows:

look for a specific aws-cleanup-$environment_type.sh in the $AWS_SCRIPTS_DIR directory in your project (e.g. aws-cleanup-staging.sh for staging environment),
if not found: look for a default aws-cleanup.sh in the $AWS_SCRIPTS_DIR directory in your project,
if not found: the cleanup job will fail.

Your deployment (and cleanup) scripts have to be able to cope with various environments, each with different application names, exposed routes, settings, ... Part of this complexity can be handled by the lookup policies described above (ex: one script per env) and also by using available environment variables:

deployment context variables provided by the template:

${environment_type}: the current environment type (review, integration, staging or production)

${environment_name}: the application name to use for the current environment (ex: myproject-review-fix-bug-12 or myproject-staging)

${hostname}: the environment hostname, extracted from the current environment url (after late variable expansion - see below)

any GitLab CI variable

any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)

Environments URL management¶

The AWS template supports two ways of providing your environments url:

a static way: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
a dynamic way: when the url cannot be known before the deployment job is executed.

The static way can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):

$AWS_ENVIRONMENT_URL to define a default url pattern for all your envs,
$AWS_REVIEW_ENVIRONMENT_URL, $AWS_INTEG_ENVIRONMENT_URL, $AWS_STAGING_ENVIRONMENT_URL and $AWS_PROD_ENVIRONMENT_URL to override the default.

Each of those variables support a late variable expansion mechanism with the %{somevar} syntax, allowing you to use any dynamically evaluated variables such as ${environment_name}.

Example:

variables:
  AWS_BASE_APP_NAME: "wonderapp"
  # global url for all environments
  AWS_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"
  # override for prod (late expansion of $AWS_BASE_APP_NAME not needed here)
  AWS_PROD_ENVIRONMENT_URL: "https://$AWS_BASE_APP_NAME.acme.domain"
  # override for review (using separate resource paths)
  AWS_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"

To implement the dynamic way, your deployment script shall simply generate a environment_url.txt file in the working directory, containing only the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

Deployment output variables¶

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

$environment_type: set to the type of environment (review, integration, staging or production),
$environment_name: the application name (see below),
$environment_url: set to the environment URL (whether determined statically or dynamically).

Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

You may also add and propagate your own custom variables, by pushing them to the aws.env file in your deployment script.

Configuration reference¶

Secrets management¶

Here are some advices about your secrets (variables marked with a ):

Manage them as project or group CI/CD variables:
- masked to prevent them from being inadvertently displayed in your job logs,
- protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
Don't forget to escape special characters (ex: $ -> $$).

Global configuration¶

The AWS template uses some global configuration used throughout all jobs and environments.

Input / Variable	Description	Default value
`cli-image` / `AWS_CLI_IMAGE`	the Docker image used to run AWS CLI commands	`registry.hub.docker.com/amazon/aws-cli:latest`
`base-app-name` / `AWS_BASE_APP_NAME`	Base application name	`$CI_PROJECT_NAME` (see GitLab doc)
`environment-url` / `AWS_ENVIRONMENT_URL`	Default environments url (only define for static environment URLs declaration) supports late variable expansion (ex: `https://%{environment_name}.aws.acme.com`)	none
`scripts-dir` / `AWS_SCRIPTS_DIR`	Directory where AWS scripts (deploy & cleanup) are located	`.` (root project dir)

Review environments configuration¶

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the AWS_REVIEW_ENABLED variable (see below).

Here are variables supported to configure review environments:

Input / Variable	Description	Default value
`review-enabled` / `AWS_REVIEW_ENABLED`	AWS project ID for `review` env	none (disabled)
`review-app-name` / `AWS_REVIEW_APP_NAME`	Application name for `review` env	`"${AWS_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`)
`review-environment-url` / `AWS_REVIEW_ENVIRONMENT_URL`	The review environments url (only define for static environment URLs declaration and if different from default)	`$AWS_ENVIRONMENT_URL`
`review-autostop-duration` / `AWS_REVIEW_AUTOSTOP_DURATION`	The amount of time before GitLab will automatically stop `review` environments	`4 hours`

Integration environment configuration¶

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the AWS_INTEG_ENABLED variable (see below).

Here are variables supported to configure the integration environment:

Input / Variable	Description	Default value
`integ-enabled` / `AWS_INTEG_ENABLED`	AWS project ID for `integration` env	none (disabled)
`integ-app-name` / `AWS_INTEG_APP_NAME`	Application name for `integration` env	`${AWS_BASE_APP_NAME}-integration`
`integ-environment-url` / `AWS_INTEG_ENVIRONMENT_URL`	The integration environment url (only define for static environment URLs declaration and if different from default)	`$AWS_ENVIRONMENT_URL`

Staging environment configuration¶

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (main or master by default).

It is disabled by default and can be enabled by setting the AWS_STAGING_ENABLED variable (see below).

Here are variables supported to configure the staging environment:

Input / Variable	Description	Default value
`staging-enabled` / `AWS_STAGING_ENABLED`	AWS project ID for `staging` env	none (disabled)
`staging-app-name` / `AWS_STAGING_APP_NAME`	Application name for `staging` env	`${AWS_BASE_APP_NAME}-staging`
`staging-environment-url` / `AWS_STAGING_ENVIRONMENT_URL`	The staging environment url (only define for static environment URLs declaration and if different from default)	`$AWS_ENVIRONMENT_URL`

Production environment configuration¶

The production environment is the final deployment environment associated with your production branch (main or master by default).

It is disabled by default and can be enabled by setting the AWS_PROD_ENABLED variable (see below).

Here are variables supported to configure the production environment:

Input / Variable	Description	Default value
`prod-enabled` / `AWS_PROD_ENABLED`	AWS project ID for `production` env	none (disabled)
`prod-app-name` / `AWS_PROD_APP_NAME`	Application name for `production` env	`$AWS_BASE_APP_NAME`
`prod-environment-url` / `AWS_PROD_ENVIRONMENT_URL`	The production environment url (only define for static environment URLs declaration and if different from default)	`$AWS_ENVIRONMENT_URL`
`prod-deploy-strategy` / `AWS_PROD_DEPLOY_STRATEGY`	Defines the deployment to production strategy. One of `manual` (i.e. one-click) or `auto`.	`manual`

Examples¶

AWS CloudFormation stack¶

This chapter gives some implementation hints to implement continuous deployment based on AWS CloudFormation.

It enables review, staging and production environments.

`.gitlab-ci.yml`¶

include:
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws@5.2.3
    inputs:
      review-enabled: true
      staging-enabled: true
      prod-enabled: true
      # authentication means defined as secret CI/CD variable

AWS scripts¶

`aws-deploy.sh`¶

This script is executed by the template to perform the application(s) deployment based on aws CLI, and uses deployment context variables provided by the template (${environment_name} is used as the CloudFormation stack name).

It implements dynamic environment URLs, by generating the environment_url.txt file, containing the dynamically generated url at the end of the deployment script.

#!/usr/bin/env bash
set -e # fail on error

echo "[aws-deploy] Deploying $environment_name..."

# disable AWS CLI pager
export AWS_PAGER=""

template_file=file://MyStack.template
# retrieve environment type ($environment_type) from template
# retrieve $AWS_KEYPAIR_NAME from project secret variables
params_opts="--parameters ParameterKey=EnvType,ParameterValue=$environment_type ParameterKey=KeyName,ParameterValue=$AWS_KEYPAIR_NAME"

if aws cloudformation describe-stacks --output text --stack-name "$environment_name" > /dev/null
then
  echo -e "Stack exists: update..."
  aws cloudformation update-stack --output text --stack-name "$environment_name" --template-body $template_file $parameters

  echo "Waiting for stack to be updated..."
  aws cloudformation wait stack-update-complete --stack-name "$environment_name"
else
  echo -e "Stack doesn't exist: create..."
  aws cloudformation create-stack --output text --stack-name "$environment_name" --template-body $template_file $parameters

  echo "Waiting for stack to be created..."
  aws cloudformation wait stack-create-complete --stack-name "$environment_name"
fi

# Retrieve outputs (use cloudformation query)
webserver_url=$(aws cloudformation describe-stacks --output text --stack-name "$environment_name" --query 'Stacks[0].Outputs[?OutputKey==`WebServerUrl`].OutputValue')

echo "Stack created/updated:"
echo " - WebServerUrl: $webserver_url"

# Finally set the dynamically generated WebServer Url
echo "$webserver_url" > environment_url.txt

`aws-cleanup.sh`¶

This script is executed by the template to perform the application(s) cleanup based on aws CLI (review env only).

#!/usr/bin/env bash
set -e # fail on error

echo "[aws-cleanup] Cleanup $environment_name..."

# disable AWS CLI pager
export AWS_PAGER=""

aws cloudformation delete-stack --stack-name "$environment_name"

AWS Serverless¶

This chapter gives some implementation hints to implement continuous deployment of a Serverless application based on Serverless Application Model.

It enables review, staging and production environments.

`.gitlab-ci.yml`¶

include:
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws@5.2.3
    inputs:
      # use an image with both aws and sam CLI
      cli-image: "pahud/aws-sam-cli:latest"
      review-enabled: true
      staging-enabled: true
      prod-enabled: true
      # authentication means defined as secret CI/CD variable

AWS scripts¶

`aws-deploy.sh`¶

This script is executed by the template to perform the application(s) deployment based on samand aws CLI, and uses deployment context variables provided by the template (${environment_name} is used as the SAM/CloudFormation stack name).

It implements dynamic environment URLs, by generating the environment_url.txt file, containing the dynamically generated url at the end of the deployment script.

#!/usr/bin/env bash
set -e # fail on error

echo "[aws-deploy] Deploy $environment_name..."

# disable AWS CLI pager
export AWS_PAGER=""

# 1: build
sam build ${TRACE+--debug}

# 2: deploy (each environment is a separate stack)
# AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY defined as secret CI/CD variable
# AWS_DEFAULT_REGION and AWS_SAM_BUCKET defined as project CI/CD variable
sam deploy ${TRACE+--debug} \
  --stack-name "$environment_name" \
  --region "$AWS_DEFAULT_REGION" \
  --s3-bucket "$AWS_SAM_BUCKET" \
  --no-fail-on-empty-changeset \
  --no-confirm-changeset \
  --tags "ci-job-url=$CI_JOB_URL environment=$environment_type"

# Retrieve outputs (use cloudformation query)
api_url=$(aws cloudformation describe-stacks --stack-name "$environment_name" --output text --query 'Stacks[0].Outputs[?OutputKey==`MyProjectApiUrl`].OutputValue')

echo "Stack created/updated:"
echo " - Api URL: $api_url"

# Finally set the dynamically generated WebServer Url
echo "$api_url" > environment_url.txt

`aws-cleanup.sh`¶

This script is executed by the template to perform the application(s) cleanup based on aws CLI (review env only).

#!/usr/bin/env bash
set -e # fail on error

echo "[aws-cleanup] Cleanup $environment_name..."

# disable AWS CLI pager
export AWS_PAGER=""

aws cloudformation delete-stack --stack-name "$environment_name"

Variants¶

The AWS template can be used in conjunction with template variants to cover specific cases.

OIDC variant¶

This variant enables OpenID Connect to retrieve temporary credentials.

If you wish to use this authentication mode, please follow carefully the GitLab guide, then configure appropriately the related variables:

AWS_OIDC_ROLE_ARN for any global/common access,
AWS_REVIEW_OIDC_ROLE_ARN and/or AWS_INTEG_OIDC_ROLE_ARN and/or AWS_STAGING_OIDC_ROLE_ARN and/or AWS_PROD_OIDC_ROLE_ARN if you wish to use a separate role with any of your environments.

Configuration¶

The variant supports the following configuration:

Input / Variable	Description	Default value
`oidc-aud` / `AWS_OIDC_AUD`	The `aud` claim for the JWT	`$CI_SERVER_URL`
`oidc-role-arn` / `AWS_OIDC_ROLE_ARN`	Default IAM Role ARN associated with GitLab to authenticate using OpenID Connect	none (disabled)
`review-oidc-role-arn` / `AWS_REVIEW_OIDC_ROLE_ARN`	IAM Role ARN associated with GitLab to authenticate using OpenID Connect on `review` env (only define to override default)	none (disabled)
`integ-oidc-role-arn` / `AWS_INTEG_OIDC_ROLE_ARN`	IAM Role ARN associated with GitLab to authenticate using OpenID Connect on `integration` env (only define to override default)	none (disabled)
`staging-oidc-role-arn` / `AWS_STAGING_OIDC_ROLE_ARN`	IAM Role ARN associated with GitLab to authenticate using OpenID Connect on `staging` env (only define to override default)	none (disabled)
`prod-oidc-role-arn` / `AWS_PROD_OIDC_ROLE_ARN`	IAM Role ARN associated with GitLab to authenticate using OpenID Connect on `production` env (only define to override default)	none (disabled)

Example¶

include:
  # main template
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws@5.2.3
  # OIDC variant
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws-oidc@5.2.3
    inputs:
      # audience claim for JWT
      oidc-aud: "https://gitlab.acme.com"
      # common OIDC role ARN for non-prod envs
      oidc-role-arn: "arn:aws:iam::111111111111:role/cicd-role"
      # specific OIDC role ARN for prod
      prod-oidc-role-arn: "arn:aws:iam::222222222222:role/cicd-role"

Vault variant¶

This variant allows delegating your secrets management to a Vault server.

Configuration¶

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

Input / Variable	Description	Default value
`TBC_VAULT_IMAGE`	The Vault Secrets Provider image to use (can be overridden)	`registry.gitlab.com/to-be-continuous/tools/vault-secrets-provider:latest`
`vault-base-url` / `VAULT_BASE_URL`	The Vault server base API url	none
`vault-oidc-aud` / `VAULT_OIDC_AUD`	The `aud` claim for the JWT	`$CI_SERVER_URL`
`VAULT_ROLE_ID`	The AppRole RoleID	must be defined
`VAULT_SECRET_ID`	The AppRole SecretID	must be defined

Usage¶

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Parameter	Description
`secret_path` (path parameter)	this is your secret location in the Vault server
`field` (query parameter)	parameter to access a single basic field from the secret JSON payload

Example¶

include:
  # main template
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws@5.2.3
  # Vault variant
  - component: $CI_SERVER_FQDN/to-be-continuous/aws/gitlab-ci-aws-vault@5.2.3
    inputs:
      # audience claim for JWT
      vault-oidc-aud: "https://vault.acme.host"
      vault-base-url: "https://vault.acme.host/v1"
      # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

variables:
  # Secrets managed by Vault
  AWS_ACCESS_KEY_ID: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/aws/prod/account?field=access_key_id"
  AWS_SECRET_ACCESS_KEY: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/aws/prod/account?field=secret_access_key"

GitLab CI template for Amazon Web Services¶

Usage¶

Use as a CI/CD component¶

Use as a CI/CD template (legacy)¶

Understand¶

Managed deployment environments¶

Review environments¶

Integration environment¶

Production environments¶

Supported authentication methods¶

Basic authentication¶

Federated authentication using OpenID Connect¶

Deployment context variables¶

Generated environment name¶

Deployment and cleanup scripts¶

Environments URL management¶

Deployment output variables¶

Configuration reference¶

Secrets management¶

Global configuration¶

Review environments configuration¶

Integration environment configuration¶

Staging environment configuration¶

Production environment configuration¶

Examples¶

AWS CloudFormation stack¶

.gitlab-ci.yml¶

AWS scripts¶

aws-deploy.sh¶

aws-cleanup.sh¶

AWS Serverless¶

.gitlab-ci.yml¶

AWS scripts¶

aws-deploy.sh¶

aws-cleanup.sh¶

Variants¶

OIDC variant¶

Configuration¶

Example¶

Vault variant¶

Configuration¶

Usage¶

Example¶

`.gitlab-ci.yml`¶

`aws-deploy.sh`¶

`aws-cleanup.sh`¶

`.gitlab-ci.yml`¶

`aws-deploy.sh`¶

`aws-cleanup.sh`¶