paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

April 02, 2022

Copying Files on Linux with Progress: rsync, pv, dd, and tar Compared

A practical guide to monitoring large file transfers on the command line without guessing whether they've stalled

Introduction

Copying a few kilobytes with cp is instantaneous and requires no feedback. Copying a 200GB virtual machine image, a directory tree with millions of small files, or a database dump across a slow network link is a different problem entirely-one where the absence of feedback becomes a real operational risk. Is the transfer still running? Has it stalled on a bad sector or a flaky network mount? Will it finish in five minutes or five hours? Standard cp and dd (in their default form) answer none of these questions, which is precisely why a cluster of tools and flags exists specifically to add progress visibility to file copying on Linux.

This matters more than it might seem for a purely cosmetic feature. In production operations-migrating data before a server decommission, restoring a backup under time pressure during an incident, or moving large build artifacts between hosts-knowing whether a transfer is progressing, stalled, or simply slow changes what action you take next. This guide walks through the practical options available on Linux: rsync's built-in progress reporting, pv as a general-purpose throughput monitor, dd's native status flag, and a few less common but situationally useful alternatives, with an eye toward when each is the right choice.

Why Progress Visibility Matters for File Transfers

A file copy with no progress indicator forces you into a binary judgment: either trust that it's working, or kill it and start over out of uncertainty, both of which carry real costs. Trusting blindly risks letting a stalled transfer run for hours before anyone notices nothing has happened-a mount point that silently dropped, a network path that stopped responding while the local process stays alive. Killing and restarting unnecessarily wastes time and, in the case of a partially complete copy, risks leaving a corrupted or truncated file in place if the interruption isn't handled cleanly.

Progress reporting resolves this ambiguity directly by surfacing throughput, elapsed time, and estimated completion, turning a black-box operation into one you can reason about. This is especially relevant in infrastructure contexts: copying disk images during a migration, synchronizing large datasets between environments, or restoring from backup during an incident where every minute of uncertainty has a cost. The tools below differ mainly in what they're built for-rsync is a synchronization tool with progress as a feature, pv is a dedicated throughput monitor that can be inserted into almost any pipeline, and dd's status=progress is a narrow but genuinely useful built-in option for its specific use case of raw block-level copying.

rsync: Synchronization with Built-in Progress Reporting

rsync is primarily a synchronization tool-it copies only the data that has changed between source and destination, using a delta-transfer algorithm that makes it dramatically more efficient than a plain copy for repeated transfers of mostly-unchanged data. Its progress reporting comes in two forms, and understanding the difference matters for anything beyond a single small file. The older --progress flag reports progress per file, which is informative for a handful of large files but produces an overwhelming wall of output when copying a directory containing thousands of small files, since each one gets its own progress line.

rsync -ah --progress source destination

The -a flag enables archive mode, which is the same concept discussed when comparing cp -a to cp -r: it preserves permissions, ownership, timestamps, and symbolic links, making it the correct default for anything resembling a backup or migration rather than a casual one-off copy. The -h flag renders sizes in human-readable units rather than raw byte counts.

For directory-level transfers, the newer --info=progress2 (or the shorthand in some versions, --info=progress=2) is the more useful option, since it reports aggregate progress across the entire transfer rather than file by file-giving you a single running total, overall throughput, and an estimated time remaining for the whole operation:

rsync -ah --info=progress2 big-directory/ back-up-directory/

This distinction-per-file versus aggregate progress-is the single most important thing to know about rsync's reporting options, because defaulting to --progress on a large directory tree produces output that's technically informative but practically unreadable, scrolling past faster than anyone could usefully read it. --info=progress2 is almost always the better choice once more than a handful of files are involved, and it's worth noting that rsync's delta-transfer efficiency is most valuable on repeated syncs of mostly-unchanged data (incremental backups, ongoing mirroring) rather than a single one-time copy, where the overhead of computing deltas provides less benefit over a straightforward copy.

pv: A General-Purpose Pipe Monitor

Where rsync has progress reporting built into its own logic, pv (Pipe Viewer) is a standalone utility designed to sit inside any pipeline and report on the data flowing through it-essentially a transparent pass-through that adds a progress bar, throughput rate, and elapsed time without altering the data itself. This makes it more flexible than rsync's progress flags, since it isn't tied to any specific copying tool and can be inserted wherever data moves through a pipe.

For a straightforward single-file copy, pv can stand in for cp directly:

pv large_file.iso > new_location/large_file.iso

This works because pv reads from its input (here, the file given as an argument) and writes to standard output, reporting progress as it goes-functionally equivalent to cat large_file.iso > new_location/large_file.iso, but with visibility into the transfer's progress. pv automatically knows the total size of a regular file it's reading directly, which lets it calculate a percentage and estimated time remaining without additional input.

The more powerful use case is wrapping pv around a different transfer mechanism that lacks its own progress reporting, such as a tar pipeline used to copy a directory tree while preserving structure:

tar cf - my_directory/ | pv -s $(du -sb my_directory/ | awk '{print $1}') | tar xf - -C destination/

This command deserves unpacking because the pattern-tar piped through pv piped into another tar-is a genuinely useful technique beyond this specific example. The first tar cf - streams the directory's contents to standard output rather than writing an archive file to disk; pv -s <size> sits in the middle, and because a pipe doesn't carry size information the way a regular file does, the -s flag is given the total size upfront (computed here via du -sb summing bytes, piped through awk to extract just the number) so pv can calculate an accurate percentage rather than just reporting raw throughput; the final tar xf - extracts the stream directly into the destination. The result is a directory copy with accurate progress reporting and no intermediate archive file written to disk, which is particularly useful for large directories where creating a full intermediate .tar file would itself consume significant time and disk space.

dd and Alternative Tools for Progress Reporting

dd occupies a different niche from rsync and pv-it operates at the block level, copying raw data between a source and destination without any awareness of filesystem structure, which makes it the standard tool for tasks like writing a disk image to a USB drive or cloning a raw partition. Historically, dd offered no progress feedback by default, leading to a long-standing workaround of sending it a SIGUSR1 signal to force a progress report on demand. Since GNU coreutils 8.24, dd supports a native status=progress option that eliminates the need for that workaround, printing periodic updates on bytes transferred and throughput directly:

dd if=source of=destination status=progress

This is worth using by default for any dd invocation beyond a trivial, fast copy, since the alternative-waiting silently for a block-level copy of an entire disk image to complete-offers no indication of whether the operation is proceeding normally or has stalled on a failing sector, which is exactly the situation where you most want visibility.

Beyond these core tools, two other utilities solve narrower versions of the same problem. progress (sometimes packaged as progress-cli or similar, depending on distribution) is a utility that inspects already-running coreutils operations (cp, mv, dd, and similar) and reports their progress by reading /proc to determine how far each has gotten-useful specifically when you've already started a plain cp without foresight and want visibility after the fact, rather than needing to restart with a different tool:

cp bigfile newfile &
progress -mp $!

Here, the cp runs in the background, and progress -mp $! attaches to that specific process ID ($! expands to the PID of the most recently backgrounded job) to monitor and report its progress without needing to have anticipated the need beforehand. This is a genuinely useful pattern for the common situation where a long copy was started without planning for feedback, since starting over with a progress-aware tool would waste the elapsed time already spent. Readers researching this area may also encounter cpg, a cp-like utility shipped with GNU coreutils in some distributions that supports a progress option (-g); as a less universally available tool than the others covered here, it's worth checking your distribution's package repository before depending on it in scripts meant to run across different environments.

Trade-offs Between Progress-Reporting Approaches

Each of these tools makes different trade-offs, and choosing the right one depends on what you're actually copying and why. rsync is the right default for anything involving directory trees, remote hosts, or situations where you might need to resume or repeat the transfer-its delta algorithm and --info=progress2 aggregate reporting are purpose-built for exactly this. But rsync carries more overhead than a plain copy for a single, one-time local file transfer, since it performs additional bookkeeping (checksums, file comparison) that a simple pipe doesn't need.

pv is the more flexible tool precisely because it doesn't care what's producing or consuming the data-it can be inserted into virtually any pipeline, which makes it valuable in situations rsync doesn't cover well, such as monitoring a tar stream, a database dump being piped to a remote host, or any other data flow that isn't natively a "file copy." The trade-off is that pv requires you to construct the pipeline yourself, and for directory copies, that means managing tar or a similar archiving step explicitly rather than getting directory-aware copying for free the way rsync -a provides it.

dd status=progress solves a narrow but important problem: it only matters for block-level copies, where rsync or cp wouldn't normally be used in the first place (writing disk images, cloning partitions). It offers no file-level awareness and isn't a substitute for rsync or pv in general file-copying scenarios-reaching for dd to copy an ordinary directory of files would be the wrong tool regardless of progress reporting, since dd has no concept of files at all when operating on raw devices. The progress utility's niche is narrower still: it's valuable specifically as a retrofit for commands already running without foresight, not as a first choice when you're planning a transfer in advance and could simply choose a progress-aware tool from the start.

Best Practices for Monitoring Large File Transfers

For most directory-level copies, particularly backups, migrations, and anything involving a remote host, default to rsync -ah --info=progress2 rather than plain cp -r, since the aggregate progress reporting and archive-mode metadata preservation address two common pain points at once. Reserve --progress (the per-file variant) for situations where you're copying a small number of large files and genuinely want to see each one's individual progress rather than an aggregate total.

For pipelines that don't naturally fit rsync's model-archiving before extraction, piping data to a remote host via ssh, or monitoring a custom data-processing pipeline-insert pv deliberately, and remember to supply -s with an accurate size estimate whenever the data source doesn't let pv determine size automatically (as with the tar piping pattern), since without it pv can only report throughput and elapsed time, not a percentage or estimated completion. For raw block-level operations, make status=progress a habit on every dd invocation rather than an afterthought reached for only after a copy has already been running silently for an uncomfortable length of time.

Key Takeaways

Conclusion

Progress visibility during file copying is not a cosmetic nicety-it's an operational necessity whenever the data being moved is large enough that a silent failure could go unnoticed for a meaningful amount of time. rsync, pv, and dd's native progress reporting each solve a slightly different version of this problem: synchronization with aggregate reporting, flexible pipeline monitoring, and block-level transfer visibility respectively, and knowing which fits a given situation is more valuable than memorizing any single command.

The underlying pattern across all of these tools is one familiar from the rest of the Unix toolkit: small, composable utilities, each doing one thing well, that can be combined into exactly the solution a specific situation calls for. Whether you're migrating a production database, cloning a disk image, or syncing a backup directory across a slow link, the right combination of these tools turns an anxious wait into an observable, reasoned process.

References