Post categories



Recommended Wordpress Hosting from WPEngine

  • I use these guys for some of my clients; absolutely rock solid and blazingly fast - check them out!





Here are a couple of ads whose clicks and views help me pay for my server hosting











50gb of “cloud” space with Box, automatically sync’d on Ubuntu/Linux with webdav and unison

Box (used to be box.net) is an online storage service that’s been around for a while – it’s quite popular and gives you more storage space than Dropbox usually does. And, if you’ve got an Android or iPhone, getting the mobile Box app (for Box.com) unlocks 50gb of storage immediately, for free. 50GB of storage. Unfortunately, it doesn’t have a linux syncing client yet – wouldn’t it be nice if you could create your own?


Contents


Box supports webdav; webdav is an extension to HTTP that allows handling of remote filesystems – thankfully, there are a number of packages in linux available to let you mount these webdav folders locally and have them behave as local folders. However, performance on these usually is a bit slow (each read/write involves a few round-trips to the web server) so what we’ll get here is:

  • a locally mounted webdav folder that will have the “live” contents of what Box has, but quite slow
  • a local copy of the contents of the webdav folder that gets kept in sync with the online Box copy every, or automatically when you change a file – but because it’s a local copy, it’s no different to normal files

(Anywhere I’ve used vim, feel free to use your editor of choice – just replace it with something like gedit if you’d rather a gui)


Sign up for Box!

So, first up, you’ll need a Box account.

Just visit box.com to create one, or signup from your mobile (then you get 50gb space from the getgo!). Once you’ve got your account all set up and confirmed, continue!

Note: this currently doesn’t work with Box’s 2-factor authentication as it requires entering the code they send you as the password; so for now this just assumes plain password authentication. Thanks Sam S for the update!

Mount the Box folder locally using webdav

Next we need to mount the Box web folder as a webdav folder – the davfs2 package will do the job:

sudo apt-get install davfs2

Now create somewhere for the webdav folder to mount:

sudo mkdir /media/box.net

Then we’ll add an entry in fstab (the file that Ubuntu reads to work out what’s mountable)

sudo vim /etc/fstab

and put this in it at the end:

https://dav.box.com/dav /media/box.net davfs defaults,rw,user,noauto 0 0

(update 2013-12-06: updated the url to dav.box.com instead of www.box.net, thanks to a comment from here)

Now we’ll change the davfs config so that users in the users group can mount the webdav share whenever they need, and also to disable the use of file locks – they don’t seem to work properly with the Box.com webdav service

sudo vim /etc/davfs2/davfs2.conf

and put this in it (there might already be something similar to this somewhere in the file – just replace it with this):

dav_group users
use_locks 0
(Thanks to johnreid.it for the pointers about use_locks)

Next, we need to add our user to the users group – (my user’s called seb, replace seb with you obviously!)

sudo addgroup seb users

Now we need to store our Box.com username and password so that it doesn’t prompt every time we try and use it:

sudo vim /etc/davfs2/secrets

and put this at the end:

https://dav.box.com/dav <your_user> <your_password>

At this point, everything should be ready to test the webdav setup; run this to mount it:

mount /media/box.net

(it might say something strange about system option in user configuration file – I don’t really know why that is, but it doesn’t appear to break anything)
(it might also say program is not setuid root – you can fix this by running sudo chmod u+s /sbin/mount.davfs then running the mount command again)

If that worked, then you should be able to open a filemanager (like nautilus or thunar) and visit /media/box.net and view the contents of your Box account! However, you might notice that it’s really not very responsive – and sometimes freezes up the file manager… which is what the next section will fix.


Setup Unison to keep a local folder and your Box folder in sync

Unison is a 2-way synchronization tool that can do all sorts of powerful things; here though we’re just going to use it to keep 2 folders in sync with each other – I’m going to put my local Box folder in /home/seb/box.net and keep it in sync with the existing /media/box.net. So first make sure you’ve got the box.net folder:

cd ~
mkdir box.net

Now you need to install unison (if you’ve not got it already) then run unison once to create the .unison config folder:

sudo apt-get install unison
unison # if you've already used unison before to create a profile, this might run it! So be careful...

You should now have a .unison folder in your home directory. We’re going to create a new unison profile to keep our folders in sync:

vim .unison/box.prf

and put this in it (obviously again replacing seb with your username!):

root = /home/seb/box.net
root = /media/box.net

ignore = Name *~
ignore = Name .*~

auto = true

retry = 2

logfile = /tmp/unisonlog

batch = true

That profile ignores any files with filenames ending in ~ (like vim and emacs’ backup files), sets automatic and batchmode to true (so it’ll try and do everything automatically, without asking for confirmation all the time) and write out a log of what happened to /tmp/unisonlog.

So let’s try it:

unison -ui text box

This runs unison in non-gui mode (thanks for Onoman for the heads up!) with the “box” profile. This should print out a load of stuff about what it’s doing, what it’s copying, conflicts etc, then write something (hopefully!) like:

UNISON 2.32.52 finished propagating changes at 08:30:46 on 24 Feb 2012
Saving synchronizer state
Synchronization complete at 08:30:46

If it did, then the contents if your box.net account should be in the box.net folder in your home directory. Brilliant.

So, now you can edit files, move them around, change, rename, whatever files in the /home//box.net foder, run unison box, then your box account will update itself. Likewise, you can put files in your box account on your phone, upload files through the website, email files in, run unison box again and your box.net folder in your home dir should update itself. AMAZING!

It’s a bit of a pain in the bum having to manually run unison everytime we want updates though…


Create a cron job to run unison automagically

Cron lets your system run commands automatically in the background on timed intervals, without you having to do anything. So we’ll create a cron job to automatically run our unison box command every hour. You can fiddle with that number to suit, but bear in mind that if you need some files sync’d immediately, just run unison box again.

Update: as mentioned by Markus in the comments, we really want to prevent unison from running more than one copy at a time; this can be arranged by creating a wrapper script that uses flock to run unison for you, and making sure there’s only one copy by creating a lock file:

/home/seb/bin/unison_wrapper.sh:

#!/bin/bash
flock -n /var/lock/my_unison_lock unison -ui text box

flock is a tool that is in Ubuntu by default – the manpage description is flock - manage locks from shell scripts which is exactly what we want; the -n specifies the name of the lock file to monitor; if another copy of our script is run but the lock is already locked, flock returns immediately and doesn’t run another (potentially clobbering) copy of unison.

So, now that we have the wrapper script setup, edit your crontab (your user’s list of scheduled cron jobs):

crontab -e

and put this at the bottom of it:
0 * * * * /home/seb/bin/unison_wrapper.sh

Cron’s format is a bit odd – it’s:

m    h    dom    mon    dow    command

So the crontab line we put in will run /home/seb/bin/unison_wrapper.sh on the 0th minute of * hours (every hour), * days of the month, * months, * days of the week – which is exactly what we want.

And with that, you should be done! Feel free to drop me a line if you have any problems…

It is possible to get unison to run automatically every time you change a file in your local copy of the folder using dnotify or famd. Basically, you get dnotify or famd to “watch” the folder, then run unison for you everytime something changes. However, I’m not sure whether using this is a good idea as if you change a lot of files in short succession, it’ll create a lot of instances of unison running on top of each other and they might collide, creating havoc in your box! It’ll also mean that everytime a file changes, it creates a load of https requests to Box.com (which they may or may not be too happy about if it happens a lot!)I haven’t tried this myself so won’t provide details, but if you’re feeling adventurous and would like to give it a whirl, apparently the following dnotify command will do roughly what you want (again, untested!):

	dnotify -CDMr --background /home/seb/sync -e "unison box"

I didn’t have dnotify on my box, and searching for it in the apt-cache suggested fam – I didn’t have time to try configuring and setting this up, but if you do, let me know how you get on!

Posted in Android, Linux | 50 Comments »

50 Responses to “50gb of “cloud” space with Box, automatically sync’d on Ubuntu/Linux with webdav and unison”

  1. John Reid says:

    Thanks for the link to my te. I’m glad the article helped.

    I use Box but haven’t tried setting up a synchronisation. Ik probably give your article a whirl and see how it behaves.

    Nice one.

    • seb says:

      No problem – thanks for pointing me in the right direction re locks :)

      Box seems rather neat so far and mixed with webdav and syncing I reckon it could replace my dropbox setup – time will tell :)

      Let me know if you have any issues…

  2. murr4y says:

    Thanks for the article!
    Now my free 50GB box.net account will no longer be useless! :-D

    • Seb says:

      Hooray for free online storage! Let me know how you get on – seems to be working quite well here – I made a few changes earlier online at work and they appear to have synced nicely to my PC at home whilst I’ve been away :)

  3. MorayJ says:

    Hi, thanks for tipping Unison as I’m starting to use Box and looking for something like this.

    You mention the auto (crontab) idea…what happens if your box folder isn’t mounted for any reason…is there a danger that you will sync to zero?

    • Seb says:

      Glad you found it useful :)
      There is a small chance that unison would clobber all your files. However, most of the time if you have a lot of items in your box folder that a unison would otherwise delete, it simply refuses to do it – so check your logs every now and again :)

      For example, I just umount’d my /media/box and ran unison box:

      Contacting server…
      Looking for changes
      Reconciling changes
      The root of one of the replicas has been completely emptied.
      Unison may delete everything in the other replica. (Set the
      ‘confirmbigdel’ preference to false to disable this check.)

      Aborting…

      So I think it’s safe enough :)

  4. onoman says:

    A little investigation on dnotify showed it as obsoleted. As a replacement for it you could use incron’s inotify which watches for inode events and then can preform an action. The web site describing it is here:

    http://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/

    I have not tried this because a simple 60 minute cron job is enough for me.

  5. onoman says:

    The new unison has some command line options that kind of need to be used (especially with cron). I installed unison package on a fedora system (unison240-2.40.63-6.fc17.i686.rpm) and just running unison box defaults to using the GUI. So to use the command line you have to run something like:

    unison -ui text box

    With cron, unison need to be silenced and you can do this with something like:

    unison -ui text -silent box

    and this works fine in the users crontab.

    • Seb says:

      Hi Onoman, thanks for checking that – I just gave it a go on my box and you’re right; -ui text seems to sort it so thanks for the heads up – post updated :)

      And yes; I used the 1 hour cron job as that’s plenty sufficient for my needs (which are mainly just backing things up) – I use GIT for when I need proper remote version-tracked storage, and Dropbox for when I need to sync things with others…

  6. Stefano says:

    Hi! This guide is very clear, but unfortunately I got a long log mainly of errors like:

    UNISON 2.40.65 started propagating changes at 21:29:01.81 on 02 Aug 2012
    [CONFLICT] Skipping
    [ERROR] Skipping lost+found
    Error in scanning directory:
    Permission denied [opendir(/home/.../box/lost+found)]
    [BGN] Copying Comparative_Government.pdf from /home/…/prova to /home/doc/box
    Failed: Error in copying locally:………..

    • Seb says:

      Hi Stefano,
      Sorry you’re having issues – from some of the errors:
      /home/…/prova to /home/doc/box
      Have you made sure that /home/doc/box is writeable? Also did mounting of the webdav folder previously work before trying to get Unison working with it?
      Cheers

      • Stefano says:

        Hi Seb, thanks for reply. Yes box folder was working before I tried Unison. But after a reboot the folder is again not working, I open it and nothing shows up. Permission are normal I’d say, write/read for everyone and user/group is both my user.
        This problem seems to happen with all the guides I found on the net, it’s incredible :(

  7. Durand says:

    Works well, thanks! I’m still not sure if I need another cloud storage solution though…

  8. Sagar Behere says:

    I recently got a box.com account and this article describes pretty much what I wish to do.

    Wouldn’t it be better to call a script from cron that first checks if the mounted webdav folder is functioning properly, and only if it is, then it calls unison? I often have to unplug my laptop and go to meetings where I don’t have internet access. In such a meeting, if I change contents of the local folder and unison tries to sync to the webdav mounted volume, then what happens?

    More worryingly, if for some reason, the webdav volume is not mounted, then the mount directory will be empty. Will unison then empty my local folder too, to keep it in sync with the empty webdav mountpoint? That could mean losing data.

    Would be much obliged if you know the answers to these questions.
    Thanks in advance,
    Sagar

    • Seb says:

      Unison usually fails quite gracefully if one side of a sync has disappeared – it quits with something like “refusing to sync, large data loss” or something along those lines… so I think you’re relatively safe it won’t delete all your data if you go offline!

      But the idea of the script for cron that checks it’s mounted is a good idea; it could just run something like (providing you’ve only got one davfs mount) (completely untested!):

      num_mounted_davfs=$(mount | grep davfs | wc -l)
      if [[ "$num_mounted_davfs" == "1" ]] ; then
          echo “seems mounted, checking files”
          num_mounted_files=$(ls -l /media/box.net | wc -l)
          if [[ "$num_mounted_files" == "0" ]] ; then
              echo “Nothing listed on box; aborting sync”
          else
              echo doing unison
              unison box
          fi
      else
          echo not doing unison
      fi

      although the num_mounted_files check is probably a bit flaky (not sure what it would give if you weren’t connected but the share was still mounted- probably not 0 lines!)

  9. Bill Kristan says:

    Great, thanks for this – I’m finding that when I try to copy my files to box that it changes the modification dates. If I use copy -p the dates will be correct initially, but then they change to the date and time of copy. Are you getting that too, or am I doing something wrong? Seems that would screw up unison, since it would think the box copies were more recent than the local ones, and it would want to copy everything back to the local folder.

  10. Donny says:

    This is great. It works in my Ubuntu box. However, when I upload a file via web, the file can not be synced to Ubuntu with the following error:

    Error in digesting /media/box.net/:
    /media/box.net/: No such file or directory

    The file is shown while “ls” in the /media/box.net, but I cannot open the file with the same No such file or directory error.

    Do you have any idea why this happens?

    • Seb says:

      Have you checked the read permissions on the folder that the file is supposedly in? Seems a bit weird that the file exists by unison can’t read it; unless the davfs driver is malfunctioning?

  11. Sagar Behere says:

    Hmm,

    I got around to trying this today, and got a strange error with box.com

    It seems that box.com does not support creation of files/folders whose name begins with a . (dot). Unison attempts to create a temporary directory named .unison, and the directory creation fails. Therefore, synchronization fails.

    I tried to cd to the mounted box.com webdav volume and manually tried `touch .foo` and `mkdir .foo`, both of which failed. Then I went to the web interface and tried to create a new folder named .foo, whereupon I got the message,

    “The folder name you specified contains special characters (, /, “, :, , |, *, ?, .) or words that are not allowed. Modify the folder name and try again.”

    Funny that I get this error but it seems to be working for everyone else :(

    Regards,
    Sagar

    • Bill Kristan says:

      I had the same problem.

      I also have a problem with anything I copy to my box.com webdav directory (using cp -p or rsync -a) losing its modification date. Isn’t that a problem for syncing?

    • Diaz says:

      Hi,

      same problem here.
      Box.com does not like unison temporary directory .unison/… because it starts with a “.”.

      I read the manual, and there is not luck to change the name of the temp directory.

      @seb don’t you get any issues with this directory ? Is it successfully created on your box.com share ?

      Does anyone have an idea hos to fix this issue ?
      Apart from patching the source and rebuilding – I think it will end like this.

      Thank you for the how-to by the way !
      Jeremy.

    • Chris Z says:

      I have exactly the same problem when I set up Box sync on my dockstar file server. If I set use_locks = 0, however, dot file can be created, but dot directories can not. Any solutions? ideas?

      • jsp says:

        Unison puts a hardcoded prefix “.unison” before any temporary files, and this fails with directories, since box does not allow leading dots in directory names.

        To fix this, you need to change the unison source and then recompile unicode yourself (which is actually not difficult).

        In the file os.ml of the unicode source, change the line
        let tempFilePrefix = “.unison.”
        to something else, e.g.
        let tempFilePrefix = “_unsn.”

        • iomartin says:

          jsp,

          Your fix worked for me. It was indeed very easy to compile it myself (just follow the manual at their website), and now usison is working.

          Thanks!

  12. Markus says:

    You definitely should make sure that only one instance of Unison is running at a time. Even more so if you are using inotify to launch Unison automatically when changes are detected or when using a shorter interval for the cron job.

    Fortunately, this is possible with a tool like flock, which only starts Unison if it is not already running.

    You can use it like

    flock -n /var/lock/unison_box -c “unison -ui box”

  13. chopchop says:

    I cant get this to work, I’m sure I missed something. Followed the steps and all is well, the box.net folder is synched to a folder in my home folder. If I then run unison -ui text box it takes a matter of seconds for it to check for changes. If I log out and then back in and run unison -ui text box it takes hours to check for changes. What am I doing wrong. Help!

  14. Klaus says:

    Hi, I have set everything up as described, and it sorta works.

    When I issue ‘unison box’, it starts scanning for changes, but eventually stops with this error:

    The root of one of the replicas has been completely emptied.
    Unison may delete everything in the other replica. (Set the
    ‘confirmbigdel’ preference to false to disable this check.)

    That seems fair since the /home/klaus/box.com dir is empty – this is the initial run!

    I couldn’t find anything helpful in the man page. Can you help?

    /klaus

  15. iomartin says:

    jsp,

    Your fix worked for me. It was indeed very easy to compile it myself (just follow the manual at their website), and now usison is working.

    Thanks!

    • jonathan says:

      Make sure your box.net is mounted. You could also add a mount /media/box.net (or whereever youve mounted it to) to the wrapper script.

  16. […] 50gb of “cloud” space with Box, automatically sync’d on Ubuntu/Linux with webdav and unison […]

  17. Marcus says:

    So I’ve configured everything as described, but I’m having a problem: it’s not actually syncing anything. I run “unison -ui text box” and my filename is box.prf so that checks out. Anyway, it checks for differences which takes hours, then does nothing. The most I’ve gotten it to do is create the empty directory structure.

    Any ideas?

  18. James Tylee says:

    Hey, I noticed in your entry you have:
    “Now we need to store our Box.com username and password so that it doesn’t prompt every time we try and use it:

    sudo vim /etc/davfs2/secrets

    and put this at the end:

    https://www.box.net/dav

    it will still prompt you for the username and password, change the line to:

    https://dav.box.com/dav

  19. Sam S says:

    Halfway through setting this up at the moment, just thought I’d mention that to get box to mount through webdav I had to disable two factor authentication on box, as once you try to connect with username and password, box SMS’s you a code – you are then supposed to try and connect again with the code in the password field… ugh.

    I’m planning to try using a script, as mentioned above, to see if it’s mounted – which should be fun. I may even set the script to try and mount it if it finds it hasn’t been already.

    • Seb says:

      Good point! I’ll update the post to mention this, thanks :)

      • Sam S says:

        Incidentally, I’ve worked out that the “system option in user config” message is due to the fact the davfs config files have a few settings that are marked as only being usable in the system config (in /etc/davfs2/davfs2.conf) or the user config (in ~/.davfs2/davfs2.conf) – and for some reason the default user config has one of these system options in it by default.

        You can edit ~/.davfs2/davfs2.conf and comment out the line beginning ignore_home and it will work fine (the option mentioned is still set, because it’s also in the system config where it belongs)

  20. Mohammad says:

    I tried a different way that worked fine. But, I cannot seem to be able to get box to mount automatically when booting. before trying your way, does it fix that problem? did you mange to make it moutn automatically? thank you

    • Seb says:

      Yes, because this is all done in fstab, it will be mounted on boot

      • Sam S says:

        In the fstab we have the noauto switch, which I believe prevents the drive from mounting automatically

        I assumed this was deliberate – it’s tricky to mount a networked drive at boot time because we’re not sure if the network is available yet (particularly with wifi)

        So to get it to mount “at boot” I’d suggest keeping noauto in the fstab to stop it from automatically mounting, and maybe having a script that runs just after boot that checks for an active internet connection, and then mounts box only if one is found

        • Seb says:

          Oh yes, my bad – it’s been a while (nearly 2 years!) since I wrote this… I’ve since moved to owncloud so I’m probably a little out of date :)

          To get it to run just after network up, you could put a script in /etc/if-up.d/ (possibly with a timeout or somesuch) which should run after the network interface is connected (or up at least)

Leave a Reply