paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

July 06, 2019

Basics of Shell Scripting

Master the command line basics that every engineer needs before tackling advanced automation and DevOps tooling

Introduction

Every senior engineer has a moment where they realize the command line isn't an optional skill-it's the substrate everything else runs on. Container orchestration, CI/CD pipelines, cloud provisioning scripts, and even modern IDEs all eventually funnel down to a shell executing commands. Yet many developers who are fluent in high-level languages and frameworks never formally learn the shell; they pick up fragments through trial and error, copy-pasting commands from Stack Overflow without understanding why they work. This gap becomes expensive the moment something breaks in production and the only tool available is a terminal.

This article is a grounded, practical walkthrough of the skills that form the floor of command-line competency: understanding what a shell actually is, navigating a filesystem with intention, manipulating files safely, finding things quickly, and editing content without fear. These are not advanced topics-they are the prerequisites that make advanced topics (process substitution, job control, writing production-grade automation, debugging distributed systems from a terminal) actually learnable. Skipping this foundation is why many engineers plateau at "googling commands" rather than reasoning about them.

What Is a Shell, and Why Bash

A shell is a program that provides a command-line interface between a human (or another program) and the operating system's kernel. When you type ls and press Enter, the shell parses your input, resolves ls to an executable on disk (via the PATH environment variable), forks a child process, and executes that binary, capturing its output and returning control to you. This sounds mechanical, but understanding it changes how you debug problems-"command not found" is a PATH resolution failure, not a mystery.

Bash (Bourne Again SHell) is the most common shell on Linux systems and was, until recently, the default on macOS (which switched to zsh starting with macOS Catalina in 2019). Bash is a superset of the original Bourne shell (sh), adding features like command history, tab completion, arrays, and improved scripting constructs. Because Bash is nearly universal on servers, containers, and CI runners, it remains the most portable and valuable shell to learn deeply, even if your personal terminal uses something else like zsh or fish.

It's worth distinguishing between interactive use (typing commands and reading output in real time) and scripting (writing a sequence of commands into a .sh file to be executed non-interactively). The same language underlies both, but scripts require more discipline: error handling, exit codes, and avoiding assumptions about the environment. A script that works perfectly when you run it by hand can fail silently in a cron job or CI pipeline if it depends on an alias, an interactive prompt, or an assumed working directory. This distinction is one of the most common sources of "it works on my machine" bugs in automation.

#!/usr/bin/env bash
# A minimal, defensive script header
set -euo pipefail
IFS=$'\n\t'

echo "Shell: $SHELL"
echo "Bash version: ${BASH_VERSION}"
echo "Current PATH entries:"
echo "$PATH" | tr ':' '\n'

Core CLI Commands and Everyday Workflows

Before diving into specific categories like navigation or file handling, it helps to see how commands compose. Unix philosophy favors small, single-purpose tools connected via pipes (|), where the output of one command becomes the input of the next. This compositional model is arguably more important to internalize than memorizing any individual command, because it's what lets you solve novel problems with familiar building blocks rather than searching for a single tool that does exactly what you want.

A typical workflow might involve listing processes, filtering them, and acting on the result:

# Find and inspect a process by name, then show its resource usage
ps aux | grep '[n]ode' | awk '{print $2, $3, $4, $11}'

# Chain commands: count unique IP addresses in a log file, sorted by frequency
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

Notice the pattern: each stage does one thing (extract a column, sort, deduplicate, count, sort again, truncate) and the pipe glues them together. Once this pattern is internalized, commands like grep, sort, uniq, cut, awk, and wc stop being isolated trivia and become a toolkit you can recombine on demand. This is the 80/20 insight of CLI proficiency: a handful of composable text-processing tools, combined fluently, solve the overwhelming majority of day-to-day terminal tasks-far more than memorizing dozens of specialized commands with narrow use cases.

Navigating the Filesystem

Navigation starts with understanding that Unix-like systems present a single, unified directory tree rooted at /, regardless of how many physical disks or network mounts are involved. Every file and directory has a path, either absolute (starting from /, like /home/user/projects) or relative (starting from your current location, like ../projects or ./scripts). The commands pwd (print working directory), cd (change directory), and the special path components . (current directory) and .. (parent directory) are the primitives you'll use constantly.

A few conventions make navigation faster once internalized. cd - returns you to the previous directory, which is invaluable when bouncing between two locations. cd ~ (or just cd with no arguments) takes you home. Tab completion, available in virtually every modern shell, should be used aggressively-typing a few characters and pressing Tab is faster and less error-prone than typing full paths, and it also protects against typos that could otherwise point a destructive command at the wrong directory.

# Common navigation patterns
cd /var/log                 # absolute path
cd ../nginx                 # relative path, up one level then into nginx
cd -                        # jump back to the previous directory
pushd /tmp && ls && popd    # visit a directory and return automatically

The pushd/popd pair, often overlooked, maintains a stack of directories, which is genuinely useful in scripts that need to temporarily operate elsewhere (building an artifact in /tmp, for instance) and reliably return to the original location afterward. Environment variables like OLDPWD and PWD are maintained automatically by the shell, and understanding that they exist explains why cd - and similar shortcuts work the way they do rather than feeling like magic.

Viewing and Listing Directory Content

The ls command is deceptively simple on the surface and deep underneath. Running it bare gives a basic listing, but the flags transform it into a diagnostic tool. ls -la shows hidden files (those starting with a dot) alongside a long-format listing that includes permissions, ownership, size, and modification time-information you need constantly when debugging permission errors or figuring out what changed and when. ls -lh adds human-readable sizes (KB, MB, GB) instead of raw byte counts, and ls -lt sorts by modification time, which is often the first thing you want when investigating "what just changed on this box."

ls -lah /var/www/html          # long, human-readable, include hidden files
ls -lt /var/log | head -5      # 5 most recently modified files
ls -R /etc/nginx               # recursive listing of a directory tree

Beyond ls, the tree command (not always installed by default, but available via most package managers) gives a visual hierarchy of a directory structure, which is particularly useful when onboarding onto an unfamiliar codebase or documenting a project layout. For disk usage specifically, du -sh * within a directory shows the size of each item, and df -h shows overall filesystem capacity-two commands that are often the first line of defense when a server runs out of disk space.

Handling Files: Create, Copy, Move, and Remove

File handling commands are simple individually but dangerous in combination, which is why they deserve deliberate practice rather than casual memorization. touch creates an empty file or updates its timestamp if it already exists. mkdir -p creates a directory along with any missing parent directories in one call, which avoids the tedious error of creating nested directories one level at a time. cp copies files, and cp -r copies directories recursively; mv moves or renames files and directories (the same command does both, since a rename is just a move within the same directory).

mkdir -p ~/projects/app/src/{components,utils,tests}  # brace expansion creates all at once
touch ~/projects/app/src/index.ts
cp -r ~/projects/app ~/backups/app-$(date +%Y%m%d)     # timestamped backup copy
mv old-name.txt new-name.txt                           # rename in place

The command that deserves the most respect is rm, specifically rm -rf, which recursively and forcibly deletes without confirmation. There is no trash bin recovery on most Unix systems by default-a deleted file is gone unless you have backups or snapshots. A well-known category of production incidents involves a misplaced space or a wrong variable expansion turning rm -rf "$BUILD_DIR" into rm -rf / when $BUILD_DIR is unexpectedly empty. This is precisely why defensive scripting habits (quoting variables, validating they're non-empty before using them in destructive commands) are not pedantic-they are load-bearing safety practices.

# Defensive deletion pattern used in real automation scripts
BUILD_DIR="${1:?Usage: $0 <build-dir>}"
if [[ -d "$BUILD_DIR" && "$BUILD_DIR" != "/" ]]; then
    rm -rf -- "$BUILD_DIR"
else
    echo "Refusing to delete: '$BUILD_DIR' is empty or root" >&2
    exit 1
fi

Finding Files and Comparing Content

Locating files by name, type, size, or modification time is a daily necessity, and find is the canonical tool for it. Unlike ls, which lists what's in a directory, find traverses a directory tree and filters based on arbitrary predicates, making it suitable for scripted automation rather than just visual inspection. Combined with -exec or piped into xargs, find becomes a way to perform bulk operations safely, such as deleting all log files older than 30 days or finding every file over 100MB in a repository before a commit.

find . -name "*.log" -mtime +30 -delete                 # delete logs older than 30 days
find . -type f -size +100M                               # find files larger than 100MB
find /etc -type f -newer /etc/passwd                      # files modified after a reference file
find . -name "*.ts" -exec grep -l "TODO" {} \;            # files containing TODO comments

Comparing files is a related but distinct skill. diff shows line-by-line differences between two files or directories (diff -r for directories), which is the same underlying mechanism Git uses when showing you a changeset. cmp does a byte-by-byte comparison, useful for binary files where line-based diffing doesn't make sense. md5sum or sha256sum generate checksums, letting you verify that two files are identical even across different machines without transferring the entire file-a technique commonly used to validate downloaded artifacts against published checksums.

Viewing File Content

Reading file content efficiently depends on matching the tool to the file's size and your intent. For small files, cat dumps the entire content to standard output in one shot-fine for a 20-line config file, but a poor choice for a 2GB log file, where it will flood your terminal and provide no way to navigate. For anything beyond trivial size, less is the right tool: it loads content lazily, supports searching (/pattern), scrolling, and doesn't choke on large files the way cat effectively does when combined with a terminal's scrollback buffer.

less +F /var/log/application.log    # start in "follow" mode, like tail -f but navigable
head -n 50 data.csv                  # first 50 lines
tail -n 100 -f service.log           # last 100 lines, then follow new output live
grep -n "ERROR" service.log | less   # search with line numbers, paged

head and tail are indispensable for quickly sampling a file without loading it entirely-head for the beginning (useful for checking a CSV's header row) and tail for the end (useful for recent log entries). The -f flag on tail turns it into a live stream, which is the standard way to watch a log file update in real time during a deployment or incident. For structured data like JSON, piping through jq transforms raw text into something genuinely readable and queryable, which is worth learning early given how common JSON configuration and API responses have become in modern engineering work.

A less obvious but valuable technique is combining watch with another command to observe changing state over time, such as watch -n 2 'ls -la /var/spool/mail' to see a directory's contents refresh every two seconds. This bridges the gap between a one-off ls and writing a full monitoring script, and it's often the fastest way to visually confirm that a long-running process is producing expected output.

Editing File Content

Editing files directly from the terminal is unavoidable once you're working on remote servers, inside containers, or in environments without a GUI editor attached. The two editors worth being functionally competent in are nano, which is approachable and displays its shortcuts on screen, and vim (or its predecessor vi), which has a steeper learning curve but is installed virtually everywhere, including minimal container images and recovery environments where nothing else is available. Being stuck in a vim session with no idea how to save and exit (:wq) or discard changes (:q!) is a rite of passage, but a brief investment in the fundamentals-insert mode, normal mode, basic navigation, search, save/quit-pays for itself many times over.

For non-interactive editing, particularly within scripts, sed and awk allow you to transform file content programmatically rather than opening an editor at all. sed excels at search-and-replace and line-based transformations; awk excels at field-based processing of structured text. These tools let you make a change across dozens of files in one command, which is both a massive productivity multiplier and a potential hazard if used carelessly without testing on a copy first.

# In-place replacement with sed, with a backup created first (safer pattern)
sed -i.bak 's/old_hostname/new_hostname/g' /etc/hosts

# Using awk to reformat a CSV column before loading into another system
awk -F',' 'BEGIN{OFS=","} {$3 = toupper($3); print}' input.csv > output.csv

Shell Scripting Basics

Once the individual commands are comfortable, scripting is simply the act of sequencing them with logic: variables, conditionals, loops, and functions. A script begins with a shebang line (#!/usr/bin/env bash) telling the kernel which interpreter to use, and should be made executable with chmod +x script.sh. Variables are assigned without spaces around the equals sign (name="value") and referenced with a dollar sign ($name or, preferably, "${name}" with quotes to prevent word-splitting on values containing spaces).

Control flow in Bash mirrors other languages conceptually but has distinct syntax: if/then/fi for conditionals, for/do/done and while/do/done for loops, and functions defined with function_name() { ... }. Exit codes matter enormously in scripting-every command returns a numeric status (0 for success, non-zero for failure), and $? captures the most recent one. Scripts that ignore exit codes silently continue past failures, which is a common source of bugs in CI pipelines that report "success" despite a step having actually failed.

#!/usr/bin/env bash
set -euo pipefail

deploy_service() {
    local service_name="$1"
    local environment="$2"

    if [[ ! -f "deploy/${environment}/${service_name}.yaml" ]]; then
        echo "Error: no deploy config for ${service_name} in ${environment}" >&2
        return 1
    fi

    echo "Deploying ${service_name} to ${environment}..."
    kubectl apply -f "deploy/${environment}/${service_name}.yaml"
}

for svc in api worker scheduler; do
    deploy_service "$svc" "staging" || { echo "Deployment failed for $svc"; exit 1; }
done

The three flags in set -euo pipefail are worth memorizing deliberately: -e exits immediately on any command failure, -u treats unset variables as errors rather than silently substituting empty strings, and pipefail ensures a pipeline's exit status reflects any failed stage, not just the last one. Together they convert Bash from a forgiving, error-swallowing language into something closer to the fail-fast behavior engineers expect from statically typed languages, catching mistakes early rather than letting them cascade silently into production.

Pitfalls and Trade-offs

The most common mistake among developers new to shell scripting is treating Bash like a general-purpose programming language for complex logic. Bash is excellent for orchestration-gluing commands together, handling files, managing processes-but it is a poor choice for anything involving complex data structures, intricate string parsing, or numerical computation. When a script starts accumulating nested conditionals, associative arrays used as makeshift objects, and regex gymnastics, that's usually a signal to switch to Python or another scripting language better suited to the complexity, and call it from Bash if orchestration is still needed.

Another recurring pitfall is unquoted variable expansion, which causes word-splitting and glob expansion in ways that are rarely intended. A path containing a space, or a variable that happens to expand to multiple words, can silently break a script or-worse-cause a destructive command to operate on the wrong set of files. Tools like shellcheck catch this class of bug automatically and should be a standard part of any script-writing workflow, the same way a linter is standard for application code. Relying on manual review alone to catch quoting issues is unrealistic even for experienced engineers, because the failure mode is often invisible until the exact edge case triggers it in production.

Best Practices

Good shell scripting habits compound over time in the same way good software engineering habits do. Always start scripts with set -euo pipefail unless you have a specific reason not to, quote all variable expansions by default ("$var" rather than $var), and prefer [[ ]] over the older [ ] test syntax in Bash, since it handles string comparisons and pattern matching more predictably. Writing small, composable functions within a script-mirroring the Unix philosophy applied at the command level-makes scripts easier to test and reason about than one long sequential block.

Version-controlling scripts, even small ones, is non-negotiable in a professional context. A deployment or maintenance script that lives only on one engineer's laptop is a liability; it should be in the same repository as the code it operates on, reviewed in pull requests like any other change. Adding comments explaining why a command does something non-obvious (rather than restating what it does) makes scripts maintainable by people other than the original author, which matters enormously once a script becomes part of a team's shared tooling rather than a personal convenience.

Finally, test scripts against non-production data or in a sandboxed environment before running them against anything that matters. This sounds obvious, but the speed and low ceremony of writing a shell command creates a false sense of safety-unlike deploying application code, there's often no staging gate or code review required to run a one-off command directly against production. Building the habit of a dry-run flag (--dry-run) that prints intended actions without executing them is a small addition that prevents a large class of irreversible mistakes.

What to Build Toward Next

Once navigation, file handling, searching, viewing, and editing feel automatic rather than effortful, the natural next layer includes process management (ps, top, kill, job control with &, fg, bg), permissions and ownership (chmod, chown, understanding the rwx and numeric permission model), and environment configuration (.bashrc, .bash_profile, exporting variables, and understanding shell startup order). These sit just above the fundamentals and are prerequisites in turn for genuinely intermediate topics like writing portable POSIX-compliant scripts, understanding signal handling, or working with process substitution and named pipes.

Beyond that lies the territory most people mean by "advanced": writing idempotent infrastructure scripts, integrating shell scripts into CI/CD pipelines robustly, understanding how shells interact with containers (why ENTRYPOINT and CMD behave differently in Docker, for instance), and knowing when to stop using Bash altogether in favor of a proper language with better error handling and testing support. None of that is reachable comfortably without the fluency this article covers-trying to learn process substitution before you're comfortable with quoting and pipes is like trying to learn concurrency before you're comfortable with functions.

Key Takeaways

Conclusion

Shell proficiency is not a glamorous skill, but it is a force multiplier that touches nearly every other part of software engineering-debugging, deployment, infrastructure, and even day-to-day navigation of a codebase. The commands covered here (navigation, listing, file handling, finding, viewing, and editing) are deliberately unglamorous because they are the floor, not the ceiling. Fluency here is what makes the more interesting layers-automation, infrastructure as code, observability tooling-approachable rather than intimidating.

The path forward is straightforward: use the terminal deliberately rather than defaulting to a GUI file manager or IDE panel for tasks that are faster and more precise from the command line. Write small scripts to automate repetitive tasks, even trivial ones, since that repetition is what builds the muscle memory and mental models needed for the more complex scripting and tooling work ahead. The gap between "can use a terminal" and "is fluent at the terminal" closes faster than most engineers expect once the fundamentals are treated with the same seriousness as any other core engineering skill.

References