in which i am still too lazy to garbage-collect my mxlinux partition but have somehow scrounged up the willpower to optimize my backups
currently waiting for borg to import-tar all my crud so here's a blogpost
while i watch my simple little free-space chocobar counter
while df -h /dev/sda8 | sed 1d | tr -s ' \t'; do sleep 1; done | chocobar -Z -y-
pretty satisfying to see the number go down
ok back to topic!
my storage partition has several years' worth of alpine linux backups and all in all they're wasting around ~30GB
naming format is from lbu: $HOSTNAME.$(date -u '+%Y%m%d%H%M%S').tar.gz
but the newer ones are aes encrypted and have an extra .aes-256-cbc suffix
so we have a mix of unencrypted and encrypted tar.gz archives
after finding my way out of the borg2 docs and back to stable, i can finally init the repo
borg init apkovls -e repokey
borg has a convenient import-tar option which can handle our unencrypted tar.gz's just fine
unfortunately it's not magic enough to decrypt so guess i gotta do that first
nopenopenope
the relevant code is in unpack_apkovl():
# unpack archive on LBU_MEDIA to given dir
unpack_apkovl() {
and i must have been immediately dissuaded by the mention of LBU_MEDIA
because after re-reading the function it looks like i just had to export $LBU_BACKUPDIR and be done with it
oh nevermind the function is only used for diffing
:/
if="$1"
enc="${if##*.}"
openssl enc -d -"$enc" -in "$if" ${PASSWORD:+ -k "$PASSWORD"} > "$of"
then match all the *.aes-256-cbc to decrypt them
for f in *.aes-256-cbc; do ./apkovldec "$f" && rm "$f" & :; done
in parallel because i'm impatient (:
... some waiting later...
yippee that's all 40 jobs done
and now we have a simple collection of tar.gz's to shove into borg :D
file naming is alpine.$(date -u '+%Y%m%d%H%M%S').tar.gz so i wanna use the date part for the archive name
once borg has accepted our tar we can remove it
for f in apkovl/*.tar.gz; do echo "$f"; n="${f#*.}";n="${n%%.*}"; borg import-tar apkovls::"$n" "$f" && rm -- "$f"; done
oh. hold on lemme prettify that
for f in apkovl/*.tar.gz; do
echo "$f"
n="${f#*.}";n="${n%%.*}"
borg import-tar apkovls::"$n" "$f" && rm -- "$f"
done
corrupted archives? no problem! just shove them into a separate folder for future me to salvage i will never touch them again
borg dedupes file chunks so it's perfect for backups that don't change much
but still pretty crazy to see my 30GB go down to 8.6GB
that's a solid 1 - 8.6/30 = 71.33% size reduction!
alright, time to update my backup scripts to use borg
borg has options to include and exclude file patterns but im tired and lazy to read allat so lets just do this the simple way seriously why does a simple help text take so long to show and why is there so much of it i would be appreciative but beyond a certain point it simply belongs in the man page
one thing that immediately pops out to me is the "Feeding all file paths from externally" section
mostly because it's at the bottom of the screen right above my shell prompt
If you need more control and you want to give every single fs object path to borg (maybe implementing your own recursion or your own rules), you can use --paths-from-stdin or --paths-from-command (with the latter, borg will fail to create an archive should the command fail).
conveniently enough lbu ls lists all files to be backed up
is this really as simple as lbu ls | borg create apkovls::"$(date -u '+%Y%m%d%H%M%S')" --paths-from-stdin?
nope! we can use --paths-from-command to avoid the pipe
so, is this really as simple as borg create apkovls::"$(date -u '+%Y%m%d%H%M%S')" --paths-from-command lbu ls ?
still nope!
lbu ls lists relative to root (there's no leading slash)
so we can either
sed 's,^,/,' shim to add back the slash1 is cleaner so imma go with that
and it just worksTM :D
borg best backup brogram
made with <3 and /.gen.sh