Notes on file syncing across unreliable networks

Syncing between two machines is easy when the link is good and hard when it is not. The failure mode is not usually "the sync breaks" but "the sync takes hours and you cannot tell whether it finished".

Resumability is the whole design

On a link that drops every few minutes, a transfer protocol that restarts from byte zero will never complete a large file. Everything else is secondary to being able to resume.

rsync with --partial keeps the incomplete file and picks up where it stopped:

rsync -avz --partial --append-verify src/ dst/

--append-verify is worth the extra read. Plain --append trusts the existing prefix blindly; if a previous run was interrupted mid-write, the file is silently corrupted and rsync will happily extend it. --append-verify checks the prefix checksum and falls back to a full transfer if it does not match.

Compression helps less than you think

-z compresses during transfer, which is a clear win on text and a clear loss on already-compressed data — media, archives, anything gzip would not shrink further. On a slow CPU with a fast link, compression is net negative.

Check what you are actually moving before deciding:

du -sh --apparent-size src/
file src/* | grep -c compressed

If most of the payload is already compressed, drop -z and spend the CPU elsewhere.

The checksum problem

rsync's default quick check is size plus mtime. That is fast and usually right, but it misses a file whose content changed without the timestamp changing — restored from a backup, written by a tool that preserves mtime, or touched by a clock skew.

When it matters, force a content comparison:

rsync -avzc --partial src/ dst/

The cost is reading every file on both ends. On a large tree over a slow link this is expensive, so the usual compromise is quick check for routine syncs and a periodic full-checksum pass on a schedule.

Clock skew breaks everything quietly

mtime-based comparison assumes both clocks agree. A machine whose clock drifted by minutes will either re-sync everything on every run or skip changes it should have caught. If syncs behave strangely, check time first:

timedatectl status
chronyc tracking

This is a common cause of "rsync re-copies files that did not change" and it looks like an rsync bug until you check.

Bidirectional is a different problem

One-way sync is a solved problem. Two-way sync is conflict resolution, and no tool makes it painless. The honest options are:

• One writer — designate a primary and sync outward. Conflicts cannot happen.
• Versioned storage — let both sides write, keep history, resolve manually.

Anything in between will eventually delete something you wanted, usually during the one week you were not watching.

Verify the result, not the exit code

A zero exit from rsync means it did what it was asked, not that the destination is correct. After a large transfer, do a dry run in the other direction — it should report nothing to do:

rsync -avzn --delete src/ dst/

If it lists files, something did not land. That check costs one directory walk and catches the failure mode where a transfer "succeeded" into a path that was not where you thought.