Backups

For the past couple of years I haven't run regular backups of any of my systems.
It feels vaguely dirty to say it, but it is also true - I just had my data
replicated in enough places to avoid ever really losing anything, and thankfully
I have a handful of physically separate machines, so I could just basically
ignore the problem. I also have a relatively small set of data I actually care
about retaining - mainly keys, config files, source code, and writing - so it
was and is relatively cheap to just keep a lot of full copies.

A couple of days ago I decided to migrate my passwords from Dashlane (a
proprietary thing that admittedly does a pretty decent job) to Pass, which is
a Unixy free-software thing. You can read about it at:

=> https://passwordstore.org

but the short version is it's a command-line tool that manages a
GPG-encrypted collection of per-site info, and has an interface for unwrapping
info for specific sites, rotating keys, etc etc. There are a bunch of plugins
for stuff like TOTP as well, although I haven't really tried to use that.

Anyway, digression aside, I started moving passwords into pass and therefore
out of the cloud. Of course, the longevity of that data was always my problem,
but before I avoided thinking about it, because Cloud. Now, with all of it
sitting basically in a directory on a single disk, I started more seriously
considering how to properly take care of it. Ideally, I want to sync it to
other machines so I have access to it there, but even more than that I want
to make sure I never lose it, since resetting passwords is a colossal pain in
the butt[1].

Backup tools are one of those things where your individual needs significantly
influence what the right choice is for you, so I sat down to think about what
I wanted to do:

* On-device "snapshot" type backups, as protection against mistakes[2]
* Off-device but in my house local backups, as protection against device
  death
* Outside-house backups, as protection against various house misfortunes

I also wanted:

* Encrypted backup data, in case I misplace a backup itself
* Easy restoration without any special tools
* Something simple enough that I can fix it myself if it breaks

And I thought a bit about my use case:

* I don't really take or keep photos or videos
* I don't bother backing up games[3] - those I trust Steam to give me copies
  of if I need them
* I don't have much of a local music collection
* I do have a lot of config files, keys, program source, and written text
  files (like this one) that it would grieve me to lose

I considered using something off-the-shelf - there are a lot of good options -
but ultimately I felt better having no new dependencies than one, so I ended up
just writing something from scratch. The resulting tool is called "bkup".

It has a config directory called ~/.config/bkup; in that directory, there are[4]:

	data: files to put into backups, encrypted
	notdata: globs to exclude from backups, even if they appear in data
	plain: files to put into backups, unencrypted
	rcpt: the GPG key ID to encrypt backups to
	signer: the GPG key ID to sign backups from

and there's one directory, called dests, which contains executables. When
doing a backup, all of those executables are run on the final backup image,
and they can do things like squirrel it away locally, drop it on external
media, or scp/rsync it around.

Here's how it works - actually this is just the entire source, annotated with
comments:

#!/bin/sh

CONFDIR=$HOME/.config/bkup

# Regrettably, Linux has no shasum(1) (at least on my system) and macOS has no
# sha256sum(1). Try either, and strip the filename out of the result.
checksum() {
	if type sha256sum >/dev/null 2>/dev/null; then
		sha256sum "$1" | cut -f1 -d' ';
	elif type shasum >/dev/null 2>/dev/null; then
		shasum -a 256 "$1" | cut -f1 -d' ';
	fi
}

# Produce a compressed tarball containing all the stuff listed in /data that
# doesn't match a glob listed in /notdata. This is the inner tarball. The
# inner tarball is compressed before encryption, since compressing encrypted
# data is generally useless.
tar -X $CONFDIR/notdata -C / -czf /tmp/bkup-data-$$.tar.gz -T $CONFDIR/data

# Sign and encrypt the inner tarball, producing a .gpg file.
gpg --sign --encrypt --local-user $(cat $CONFDIR/signer) \
    --recipient $(cat $CONFDIR/rcpt) /tmp/bkup-data-$$.tar.gz

# Produce the outer tarball, which contains the encrypted inner tarball,
# plus the files listed in /plain. My /plain generally looks like:
#   d/keypair.pgp
#   b/backup
#   b/restore
# i.e., the unencrypted part contains the pgp keypair needed to decrypt
# the backups (protected with a very strong passphrase) as insurance
# against me losing my keys, as well as the tools needed to generate
# and unpack the backup in the first place, mostly as documentation of
# how it was made.
tar -C / -cf /tmp/bkup-$$.tar -T $CONFDIR/plain tmp/bkup-data-$$.tar.gz.gpg

# Give the outer tarball its final filename, containing the date of the
# backup and the expected checksum of the outer tarball. Having the checksum
# in the name names it easy to check whether backup tarballs have gotten
# damaged.
day=$(date -u +%Y-%m-%d)
sha=$(checksum "/tmp/bkup-$$.tar")
mv /tmp/bkup-$$.tar /tmp/bkup-$day-$sha.tar

# Run each of the dest scripts so they can deliver the tarball to wherever
# they want to put it.
for x in $CONFDIR/dests/*; do
	"$x" /tmp/bkup-$day-$sha.tar
done

# Clean up the files we made.
rm /tmp/bkup-data-$$.tar.gz /tmp/bkup-data-$$.tar.gz.gpg
rm /tmp/bkup-$day-$sha.tar

[1]: The situation is even worse for TOTP, which I currently store in
     andOTP on my phone. I may migrate them to pass-otp(1) in the future,
     but the mobile story there is not as good.

[2]: So far in my life, "oops, I shouldn't have rmed that" has vastly
     outweighed data loss to all other causes.

[3]: There are a handful I do back up, but they are either ROMs of old
     games that are not easy to come by (small) or historical games in
     source form (small).

[4]: This way of using the file system as a key-value store, with a bunch
     of config files with one item per file, is underrated. For simple
     programs it can save significant parsing.