Shell Scripting Guidelines¶

This page gives guidelines and recommandations for shell scripting developement in to-be-continuous.

Introduction¶

In to-be-continuous, a significant part of the automation is implemented using shell scripts. These scripts do not rely on a proprietary language or exotic tooling; instead, they use the standard command language available on most Unix-like systems, which is ubiquitous in CI/CD environments and container images.

In practice, TBC scripts must be executable in a wide range of system contexts - from lightweight Alpine Linux images to more feature-rich Debian or Fedora-based images. A key aspect of our approach is therefore to remain compatible with the most common shells while maximizing script portability and robustness.

On Alpine images (which are widely used for lightweight CI jobs), the default shell is Ash (the Almquist Shell), a minimalist shell that closely follows the POSIX specification. On Debian or Fedora images, the shell is typically Bash, which provides many additional features while still supporting the POSIX shell syntax.

The purpose of this page is to clarify which subset of Bash features can be safely used in Ash, so that scripts written for to-be-continuous remain both portable and maintainable, regardless of the execution environment.

In the following sections, you will find:

a reference of Bash features supported by ash,
general recommendations for writing robust shell scripts in the TBC context,
common pitfalls to avoid (notably around shell options and external command usage).

Bash features support in Ash¶

Name	Description / Example	Ash support
file conditional expressions	`[[ -a file ]]`, `[[ -f file ]]`, `[[ -d file ]]`, `[[ -x file ]]`...	✅ supported
string conditional expressions	`[[ string ]]`, `[[ -z string ]]`, `[[ -n string ]]`, `[[ string1 == string2 ]]`, `[[ string1 != string2 ]]`...	✅ supported
arithmetic conditional expressions	`[[ val1 -eq val2 ]]`, `[[ val1 -ne val2 ]]`, `[[ val1 -lt val2 ]]`, `[[ val1 -ge val2 ]]`...	✅ supported
advanced conditional expressions	`[[ -o optname ]]` (true if the shell option optname is enabled) `[[ -v varname ]]` (true if variable varname is set) `[[ -R varname ]]` (true if variable varname is set and is a name reference)	❌ not supported
regex operator	`[[ text =~ regex ]]` (true if text matches regex)	✅ supported ⚠️ except variable `$BASH_REMATCH`
unset or null substitution expansion	`${parameter:−word}` (expands to word if `$parameter` is unset or empty; else `$parameter`)	✅ supported
unset substitution expansion	`${parameter−word}` (expands to word if `$parameter` is unset; else `$parameter`)	✅ supported
set substitution expansion	`${parameter:+word}` (expands to word if `$parameter` is set and not empty; else empty string)	✅ supported
substring expansion	`${parameter:offset}` `${parameter:offset:length}`	✅ supported
length expansion	`${#parameter}` (length of `$parameter`)	✅ supported
remove prefix expansion	`${parameter#pattern}`	✅ supported
remove long prefix expansion	`${parameter##pattern}`	✅ supported
remove suffix expansion	`${parameter%pattern}`	✅ supported
remove long suffix expansion	`${parameter%%pattern}`	✅ supported
replace first match expansion	`${parameter/pattern/string}`	✅ supported
replace all expansion	`${parameter//pattern/string}`	✅ supported
indirect variable expansion	`${!varname}` (expands to variable whose name is given by `$varname`)	❌ not supported
replace prefix expansion	`${parameter/#pattern/string}` (replace) `${parameter/#pattern}` (remove)	❌ not supported
replace suffix expansion	`${parameter/%pattern/string}` (replace) `${parameter/%pattern}` (remove)	❌ not supported
uppercase expansion	`${parameter^}` (uppercase first letter) `${parameter^^}` (uppercase whole word)	❌ not supported
lowercase expansion	`${parameter,}` (lowercase first letter) `${parameter,,}` (lowercase whole word)	❌ not supported
operator expansion	`${parameter@u}`, `${parameter@U}`, `${parameter@L}`...	❌ not supported
`declare` command	`declare -A mydict` (a dictionary) `declare -i mynumber` (an integer) ...	❌ not supported
`local` comand	`local myvar=$1` (declare a local variable within a function)	✅ supported
Arithmetic expansion	`$(( total + incr ))`, `$(( ! value ))`, `$(( bits << 2 ))`, `$(( bits \\| 2 ))`...	✅ supported
Arithmetic command	`(( index++ ))` (increment integer-typed variable `index`) `(( index+=30 ))` (increase integer-typed variable `index`) ...	❌ not supported
Arrays	One-dimensional indexed array variables. See cheatsheet	❌ not supported
Dictionaries	Associative array variables. See cheatsheet	❌ not supported
Here Documents	See cheatsheet	✅ supported
Here Strings	`tr '[:lower:]' '[:upper:]' <<< "Will be uppercased"` See cheatsheet	❌ not supported
Process Substitution	`diff <(ls "$dir1") <(ls "$dir2")` See cheatsheet	✅ supported

Other recommendations¶

Shell options¶

All to-be-continuous templates must use the following shell options:

set -e: instructs bash to immediately exit if any command has a non-zero exit status
set -o pipefail: prevents errors in a pipeline from being masked. If any command in a pipeline fails, the pipeline breaks and that return code is used as the return code of the whole pipeline.

Use `grep` with caution¶

The grep command has a behavior that might cause unexpected behaviors: returns a non-zero return code when no match is found.

As a result, due to the recommended shell options used in TBC, the following script might fail unintentionally:

titles=$(grep ^# README.md)
# 💣 the above command will fail and exit the script if no match was found

As a conclusion: don't use grep to filter items out of a list. For this purpose, prefer using awk instead.

The above script can be implemented with awk:

titles=$(awk '/^#/' README.md)
# the above command will never fail, possibly returns an empty string if no match was found

The grep command might be used if the return code is tested:

if ! grep ^# README.md
then
  echo "Your README.md file doesn't contain any title!"
  continue
fi

Pattern matching should never be used for pure string matching¶

The pattern matching implementation on Alpine messes up with filename expansion.

Here is the problem:

$ docker run --rm -it alpine

# in the container
$ [[ "test" == "te"* ]]; echo $?
0
# 👍 Ok

$ touch te.file
$ [[ "test" == "te"* ]]; echo $?
1
# 💣 due to '"te"*' that messes up with 'te.file'

this is specific to Alpine Ash, the above behavior is not reproduced in Debian or Fedora Bash.

As a conclusion: glob patterns should never be used for pure string matching in to-be-continuous template scripts. Instead, prefer using the regex operator.

The above script can be implemented with the regex operator (=~):

$ docker run --rm -it alpine

# in the container
$ [[ "test" =~ ^"te" ]]; echo $?
0
# 👍 Ok

$ touch te.file
$ [[ "test" =~ ^"te" ]]; echo $?
0
# 👍 Ok: =~ is not messed up with files