This is a
You can (safely) follow along by typing or copying all the commands into your terminal. You'll need a relatively recent Linux or Mac system, and maybe an USB stick. The Stick can be simulated by any folder on the computer, so you can also try it out on any machine. Commands are lines start start with a '$' sign, the '$' is not part of the command.
Try it out now:
^ There are snippets of the expected output, where it is important.
For educational purposes, we define an alias,
so it is more clear when we are using
alias magic="git annex"
We will be defining more variables in this adventure,
so make sure to not close your shell (or edit
Official instructions and downloads for every OS are here:
tar xvzf git-annex-standalone-i386.tar.gz
Drag and Drop App to
It should work via
If it doesn't work:
~/.bashrc is enough for this adventure
git-annex version: 5.20131117-gbd514dc build flags: Assistant Webapp … key/value backends: SHA256E WORM … remote types: git directory…
ssh localhost 'git annex version'
git-annex version: 5.20131117-gbd514dc …
We start with a new folder. This could be your new Dropbox.
Initialized empty Git repository …
magic init 'My Laptop'
init My Laptop ok (Recording state in git...)
We make a sub-folder with a new text file inside.
mkdir foo && echo 'simsalabim' > foo/bar.txt
magic add foo/bar.txt
add foo/bar.txt (checksum...) ok (Recording state in git...)
git commit -m 'added'
[master (root-commit) 67a2114] added 1 file changed, 1 insertion(+) create mode 120000 foo/bar.txt
To make things easier, you can always use
Should be done after every change (or in intervals w/
It does the following:
echo '1' > one.txt
Now clone this repo using any git transport (
file works with everything you can mount somehow,
including Dropbox, SFTP, SMB, NFS, etc..
git clone file://$HOME/magicfolder "$STICK/magicfolder"
git clone user@host:magicfolder "$STICK/magicfolder"
magic init 'My USB Stick'
init My USB Stick ok (Recording state in git...)
(merging origin/git-annex into git-annex...) (Recording state in git...) commit ok pull origin ok push origin Writing objects: 100% (8/8), 776 bytes … * [new branch] git-annex -> synced/git-annex ok
We seem to have our file…
(bear with me for a second)
Standard git setup.
For the Laptop:
git remote add stick "$STICK/magicfolder"
For the stick:
origin remote is not used by
magic, can be removed or not.
git remote remove origin # optional
git remote add laptop ~/magicfolder
With the configured remotes,
sync now also does
(merging synced/git-annex into git-annex...) commit ok pull stick From /Volumes/USBSTICK/magicfolder * [new branch] git-annex -> stick/git-annex * [new branch] master -> stick/master * [new branch] synced/master -> stick/synced/master ok
We have just cloned the meta data.
All the files and folder are on the stick, as symlinks!
magic takes care of the symlinks.
Metadata includes info about the remotes, and what files they have available!
ls foo/bar.txt # foo/bar.txt
magic geta file
So, how do we get the content?
To manually transfer files, use
magic get foo/bar.txt cat foo/bar.txt
magic dropa file
To delete a file just from a remote, use
This is useful for making free space on Laptops with small hard drives, etc.
magic drop foo/bar.txt
cat foo/bar.txt # > "No such file or directory"
Isn't that a fancy word to say
is this safe?
(unless you use the
This is why we want to have a record of all files and in which remotes they are.
Let's try 'dropping' it on our Laptop, too:
magic drop foo/bar.txt
drop foo/bar.txt (merging synced/git-annex into git-annex...) (unsafe) Could only verify the existence of 0 out of 1 necessary copies (Use --force to override this check, or adjust annex.numcopies.) failed git-annex: drop: 1 failed
What if I want to push a file to the stick?
copy --toa remote
magic copy foo/bar.txt --to stick
copy foo/bar.txt (to stick...) SHA256E-s11--9062535d58….txt 11 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1) sent 173 bytes received 42 bytes 430.00 bytes/sec total size is 11 speedup is 0.05 ok (Recording state in git...)
We can also just get a list of a file availability in other remotes,
without trying to drop it:
magic whereis foo/bar.txt
whereis foo/bar.txt (2 copies) ab8aad2f-f87e-440e-baef-… -- here (My Laptop) f7feb954-0250-4117-a368-… -- stick (My USB Stick) ok
OK so far? Cool.
cd ; \ sudo rm -rf \ ~/magicfolder \ "$STICK/magicfolder"
Files are locked by default!
Also, while the content is not tracked in git by
you can still commit with git, and
magic will take of it (w/
echo 'fail' > foo/bar.txt
We need to unlock it first!
magic unlock foo/bar.txt
echo 'booya' > foo/bar.txt # No error!
Now "commit and push".
magic add foo/bar.txt
git commit -m 'changed foo bar'
Note: We didn't really have to
magic sync would have done that for us,
but committing let us put in our own message.
To get the file to the stick,
sync, which fetches the meta data with
get . (everything) to actually transfer the file.
magic sync && magic get .
Doesn't this locking/unlocking seem complicated
in comparison to, say, Dropbox?
YES. That is why we use 'direct mode'.
You may now forget the slides about locking/editing.
… direct ok
All available files are normal files;
all unavailable files are symlinks
(and we can use
magic get to make them available).
echo 'pow' > foo/bar.txt
magic sync && magic get .
cat foo/bar.txt #
Note: We could also have used
copy --to stick.
Usage: If you just want to copy somewhere, for backup or transfer.
If you'll look at the directory, it will just be a bunch of object files.
Special remotes are (
GPG) encrypted by default,
so you have to explicitly turn it off (
Once setup with
magic can use them like any other remote – it just works™.
The remote will have a name and a GPG key.
BOX=dropbox # how to call the remote BOX_PATH="~/Dropbox/BACKUP/magic/"
KEY=965113EA # YOUR GPG keyid - don't take mine ;)
Make a new folder in the Dropbox:
mkdir -p "$BOX_PATH"
magic initremote $BOX type=directory \ directory="$BOX_PATH" \ keyid=$KEY
initremote dropbox (encryption setup) (hybrid cipher with gpg key 3771835A3BADB56D) ok
Alternative if you don't have a key handy (unecrypted):
magic initremote $BOX type=directory \ directory="$BOX_PATH" \ encryption=none
Like any other remote, just a password prompt for the key.
magic copy . --to $BOX
copy foo/bar.txt (gpg) You need a passphrase to unlock the secret key for user: "Max F. Albrecht <firstname.lastname@example.org>" 4096-bit RSA key, ID 965113EA, created 2013 (main key ID FOO) (to dropbox...) ok
how files are matched with they meta data.
obviously very important.
SHA256E, a file hash plus the file extension
WORM: alternative if you don't want hashing, uses just file name, date, etc. usefull if:
echo '* annex.backend=WORM' > .gitattributes
Use the direct mode for normal folders,
and the indirect mode for normal git repos + magic.
sync to sync meta data between remotes.
move to sync content between remotes.
magic assistant # or 'magic webapp' to re-open
--fast do stuff faster.
For example, rely on local data instead of updating before checking.
magic fsck: for the data conscious
magic describe: change the description
that was set with
magic init 'My Laptop'.
always try to use the faster machine for hashing…