Today, a change of pace: getting 52GB of image files synced with git-annex in a way that still provides for a nice editing/publishing workflow and keeps history for blogposts around.

Goals

  1. hands-off file synchronisation
  2. keep history of posts and pages
  3. automatic server-side jekyll rendering

Results

  1. Using the assistant through git annex webapp on linux and the dmg on MacOS allows hands off syncing of git commits of blogs posts and annexed image files
  2. git retains all the history. With a backup remote in annex image data can also be archived
  3. I likely can hook into the regular post-receive hooks on the server to launch processing, but need to be cognizant of the extra branches annex is maintaining for state.

I’ve now got a git-annex’d repo on my server, my workstation and my wife’s workstation happily keeping 52GB of images (in 27k files) in sync.

Insights

  • 27k files in a repository make git-annex assistant slow to start up (it’s rather annoying too to see only 10% of your SSD bandwidth and 12% CPU being used)
  • stopping syncing in the git-annex web ui works to stop it from autocommitting and syncing half-finished work
  • for big changes the CLI is a lot better than the assistant, as the CLI can batch and parallelise. The assistant would upload each file individually
  • I would have saved a lot of time (and bandwidth) had I done the initial ingestion on the server instead of over the network
  • my devolo ‘dLAN 500 AVplus’ loses connection due to overheating after hours of full load
  • some weekends are less productive than others
  • don’t kill the assistant with ^C on the terminal
  • People are seriously running Windows 10 in KVM with their GPU on PCI pass-through for gaming.