For the last couple of weeks I finally decided that it's time to get an online backup plan of some sort to do 'live' backups of my data and work environment. I figured this process would be fairly straight forward since online backup services have been around for a while now. To my chagrin though I discovered that while there are plenty of services out there, finding one that provides the right features at the right price was anything but trivial. I spent a bunch of time playing around with various services. Here's what I played with:

• DropBox
• SkyDrive
• Carbonite
• CrashPlan
• SugarSync

This list is not exhaustive by any means - there are many more services out there but these are the ones I looked at based on recommendations and after poking around a bit before playing. In this post I'll describe what I found. I'd be curious to hear what some of you are using and how you fared.

What I've been doing

To give some context first, before going down this path I've been been doing timed backups that do a backup every night to an attached USB drive that I swap between two physical devices every few days a third drive is backed up about once a month and that one is stored with a friend off site (we basically swap drives). That's actually been working pretty good except that on occasion the machine is off at night or the drive isn't connected and so a backup here and there fails.

Sure enough on one of those occasions when the backup didn't happen one night a few months ago, my SSD drive froze up and I lost a bunch of data from the previous days' work (I got it back when a couple of days later the SSD miraculously decided to start up again :-). Since then I've been a lot more meticulous in either manually backing up or making sure the box does wake up to run the backup in early AM hours.

What I have been doing - and still do for daily backups - is to simply do a batch file backup where I map all of my folders that I care about and push them off to a backup drive. The batch file I use, utilizes RoboCopy which ensures speedy and reliable copying of files using recursive directory copying/mirroring:

@echo off

set source=c:
set target=e:

if "%1"=="" goto continue
set source=%1

if "%2"=="" goto continue
set target=%2

:continue
echo Copying from drive %source% to %target%   %1 %2
echo

iisreset /stop
net stop mssqlserver

robocopy %source%\projects2010 %target%\backups\dev\projects2010 /MIR /R:2 /W:5 /MT:8
robocopy %source%\wwapps %target%\backups\wwapps /MIR /R:2 /W:5 /MT:8
robocopy %source%\westwind %target%\backups\westwind /MIR /R:2 /W:5 /MT:8

net start mssqlserver
iisreset

robocopy %source%\data %target%\backups\data /MIR /R:2 /W:5 /MT:8
robocopy %source%\saved %target%\backups\saved /MIR /R:2 /W:5 /MT:8
robocopy %source%\articles %target%\backups\articles /MIR /R:2 /W:5 /MT:8
robocopy %source%\utl %target%\backups\utl /MIR /R:2 /W:5 /MT:8

robocopy %source%\Users\ricks\Documents %target%\backups\documents /MIR /R:2 /W:5 /MT:8
robocopy %source%\Users\ricks\Music %target%\backups\music /MIR /R:2 /W:5 /MT:8
robocopy %source%\Users\ricks\Pictures %target%\backups\photos /MIR /R:2 /W:5 /MT:8

robocopy %source%\Users\ricks\AppData\Roaming %target%\backups\Roaming /MIR /R:2 /W:5 /MT:8
robocopy %source%\Users\ricks\AppData\LocalLow %target%\backups\LocalLow /MIR /R:2 /W:5 /MT:8

This file basically backs up all my development folders first - shutting down IIS and SQL Server so files locked by those are released. Then a bunch of personal data stuff is backed up individually. There might still be other stuff that's open (like an Outlook file) but the key stuff like data and dev related stuff is backed up for sure.

The batch file is then tied to Scheduled Task at 5am every morning with auto wake up of the computer enabled and I occasionally run this manually as well. The nice thing is that I can also do this while travelling when other means like online backups are not a good idea due to bandwidth constraints.

Clearly this is a most basic backup strategy, but it's actually served me really well over the years. I only back up things that matter so the back up size is somewhat reasonable and runs at about 75 gigs at the moment. I can do this easily when I travel as long as I carry a USB drive with me. RoboCopy only copies what's changed so the actual copied data is not excessive, so typical intra-day backups take something like 5 minutes or less to run. These days I often run another backup in mid-day when heading out to lunch. And since there's no compression or special storage involved I can access the backed up data directly anytime without using any software.

Why no automated backup?

You might ask, why no automated backup solution? To be honest I've looked at various automated backup software in the past, but I've never really found anything that was easy to set up and maintain and that worked reliably especially when on the go. It seems with most backup software that I've tried it's always a royal pain in the butt to set up the initial folders and it's even more of a hassle to actually make changes to a backup configuration. I also was never fond of real time backup software, because I'm running a lot of stuff that's constantly open and constantly locked. Backing up files that are always open is not very reliable if they ever get backed up at all - most backup software gets this wrong. Then there's typically the mess of incremental backups and trying to find individual files in them - it just never seems to work out well. In short, to me there were too many downsides and not enough upsides to make it worthwhile to switch from my simple batch file backups which while sort of crude catch everything I care about efficiently.

Backup to the Cloud? Yes please!

After my wake up call with the SSD drive, I finally got enough fire under my ass to take a look at online backup solutions and I started looking around what's out there. There are a lot of choices available today, and I'm finding that there's lots to like about various solutions I still haven't found one solution that works for everything I'd like to have. But I made a choice

There are really a few different approaches available for cloud storage of data:

• Pure file storage services like DropBox and SkyDrive
  These services basically provide you a shared folder on your machine - anything dropped into that folder is synced between devices that are connected to this account and have syncing enabled. These services seem to be geared at selective backups.
• Backup Services like Carbonite and CrashPlan
  Carbonite and CrashPlan bill themselves as full backup plans that allow you to do full machine backups (depending on the plan you chose). The idea with these services is that they provide a one stop solution to backing up everything and in the case of Carbonite to restore an entire machine.
• Hybrid Services like SugarSync
  SugarSync on the surface looks a lot like DropBox and SkyDrive, but provides a bit more integration with the Windows Shell. Folders can be dropped onto SugarSync to sync and you can then see the sync status. You can also access backed up files in a separate drive, so it's real easy to find and search for files that are backed up. Overall I found that this combination along with the easy UI and fast syncing of changed files was the best solution for what I needed. It blurs the line between backup and syncing services.

DropBox and SkyDrive

The first set of tools I used - some time ago and now re-evaluated - are DropBox and SkyDrive. Both services are fairly similar although I would say that DropBox worked a lot more intuitively and was much less resource intensive than SkyDrive. My first thought went to these services since I'd used them previously, not quite sure if they'd make a good fit. In short, they didn't, but in a pinch they can work for basic backup purposes.

These services seem like they are primarily designed to share files - between devices and also externally for collaboration or simple file sharing with others. For this purpose they work well, but they're not particularly well suited for more complete backup scenarios. For one thing neither of these services explicitly supports linking folders external to their shared folders (although there's a workaround for that which I describe below).

Either of these services will work for file sharing but for me at least as a backup (and sharing) solution they weren't sufficient.

DropBox
I've been using DropBox off and on for a long time, but mostly for sharing files. DropBox was one of the first cloud file services out there and it works efficiently and has a simple and logical UI. It just works and gets out of the way. It's easy to drop files into the shared folder and the Web UI makes it very easy to share files with others. It all works as expected in a no frills but efficient sort of way.

The big downside to DropBox is that the amount of free storage is very limited at 2.5 gigs. While you can boost that minimum storage a bit with a few referrals, it's still not enough for even the most basics of backups. I'd say 5gig is the absolute minimum I could get away with for the most basic backups of data I work on a daily basis. Pricing beyond that free tier is also on the very pricey side with cheapest plan at $100/yr or $10/month for 100gigs. An intermediate step for 50gigs would be much more welcoming.

The desktop 'interface' for DropBox is just the dropbox folder. The Web Interface for DropBox is basic, but easy to navigate and get to the files you need quickly.

SkyDrive
Although I had been using DropBox for sometime, the first thing I actually checked out was SkyDrive, mainly because I have an old free SkyDrive account that was capped at 25 gig, which is a stately amount of free storage that would suffice for my typical 'live' backup needs. New accounts opened now get 7 GB of free space.

SkyDrive pretty much provides the same feature set as DropBox does and adds a few pretty cool features on top of it. One particularly nice feature is the ability to share your actual live PC if it is online. You can set it up so that your machine's drive(s) are remotely accessible, with file access to all of your files from other computers or any device that is logged into your SkyDrive account.

SkyDrive also intrinsically knows about Office documents and lets you edit Office documents online in Office 365 documents (which BTW are pretty sweet looking for online versions!). If you have Office installed on the machine you're using you can also open Office documents on your SkyDrive directly and then save them right back to your SkyDrive when you're done.

The extra features are pretty nice, but unfortunately I found SkyDrive to be pretty slow uploading and syncing of my data, and at the same time chewing through CPU making my laptop sound like a jet was taking off. Once I had a good chunk of data uploaded it seemed like forever for updated files to actually sync up to the server. And because there's no easy way to see what files are actually synced I often found myself with data that's out of date. Even doing something simple like dropping a PDF file into my Public folder and then accessing it from my phone a few minutes later usually didn't work, because of the delay between drop and sync which greatly reduces the utility of a file sharing service. I didn't have these kind of issues with DropBox or any of the other services I tried which seem to sync manually dropped files much more quickly especially in the 'special' folders.

On two occasions I also found SkyDrive to temporarily lose files. I'd go to a folder see two files, go back then only see one. Go back see two again. One of the files had been changed on the client but the file was never deleted on the local drive, yet SkyDrive was missing the file for a while. Not very confidence inspiring.

I also find SkyDrive's Web interface and the mobile apps (on IOs anyway) annoying to use. SkyDrive by default uses a Windows Metro style tile UI that is a waste of screen real estate without providing any real value for all that space especially when backing up non-media files. Fortunately you can also switch to a details view which uses space more diligently and provides more information. However, SkyDrive still hides file extensions of some known file types. Here's the details view:

Of the two services, I found DropBox to be more responsive and in general easier to use and consume both on the desktop and on various devices. Both work well enough, but SkyDrive definitely feels rougher around the edges. I suspect in time Microsoft will get the roughness smoothed out and it will end up a good solution. For now, as a backup solution neither of these services really work well.

Using Windows Junction Points to sync folders

DropBox and SkyDrive expect you to drop files into their respective sharing folders (User\DropBox User\SkyDrive respectively) in order to share files, so officially they don't support 'folder syncing' directly.

However, you can actually keep folders in sync by using Directory Symbolic Links (NTFS Junctions) which link a physical folder to a 'virtual' location. You can use the MKLINK Windows utility like this:

mklink /D c:\users\ricks\dropbox\utl  c:\utl

This causes the c:\utl folder to be synced into a DropBox utl folder. The same trick also works with SkyDrive. MKLINK comes with Vista and later - if you have an XP box you can use Systernal's Junction instead. For SkyDrive there's also a SkyShellEx utility you can download that provides a shell extension shortcut on folders so you can immediately share to the SkyDrive folder.

If you do map folders into DropBox or SkyDrive, be careful when you decide to quit the service. Because these files are linked, deleting the folder in your SkyDrive or DropBox folder will also delete the original linked folder and vice versa. You have to use the same tools to unlink the junction first.

I made that mistake with SkyDrive originally. I decided it wasn't going to work for what I wanted so I deleted the test folder I set up for sharing and sure enough they were gone. Ironically I could restore from backup with CrashPlan which I had installed in the interim to test on the same folders

Full backups Carbonite and CrashPlan

The next set of tools I looked at were meant as tools for full backups more so than file sharing although both of these services also provide the ability to access files from various devices and so effectively provide file sharing.

Carbonite

I only spent a brief time with Carbonite. Carbonite is clearly geared at casual computer users and wants to hold your hand through the backup process. It makes some reasonable suggestions for backup of user data, but it's not the greatest choice if you want to customize your backups. Carbonite only includes certain files by default in your backups and you have to manually add 'special' files like EXEs or anything that Carbonite doesn't recognize as data. This means you have to manually tweak the default data set selected. The UI also provides no easy way to see what's in the default set or modify it short of uninstalling and reinstalling. I was pretty turned off by this clearly end user focused product, but also by the fact that I think that many people will actually get screwed by these default settings that are supposed to back up everything but in all reality are likely to miss a lot of vital stuff that's needed. Not cool.

On the plus side Carbonite is reasonably priced with unlimited backups of internal drives for $60. For that you can't back up attached drives. The next tier allows for any data, plus mirroring and simultaneous hard disk backups which is nice, but pricey at $99. Carbonite also shows files that it manages for backup with icons that indicate backup status - a vital feature that's unfortunately missing from many other services like CrashPlan and also SkyDrive and DropBox.

CrashPlan

CrashPlan is unabashedly an online backup tool and it does that job well, but it's not as polished as some of the other products. All products sync data of course and Crashplan's data synching worked quickly and smoothly without much CPU overhead in the background. You also have the ability to fine tune how much bandwidth and CPU it uses very finely which can be useful if you're on a slower connection or if you move around a lot. If you just need your files backed up and then in an emergency pick them back up then Crashplan is a great solution.

CrashPlan also keeps track of recent versions of your files and you easily see version and roll back to them. You can also simultaneously backup to a local folder/drive so you can have a local copy of the same backup data.

CrashPlan also has attractive pricing with unlimited data for $60/yr - the same price as Carbonite, but with the ability to back up all data on from a single computer including attached drives. There's also a free plan, but it's extremely limited to sharing backups with friends (you back each others machines up essentially) which seems like little more than a gimmick.

The biggest problem with CrashPlan is that it uses an old school hierarchical UI to manage files in the backup. It uses its own tree to show your computer's file system and you have to select the folders/files to be backed up. The same is true for restoring files and this is really the bigger issue. If you want to look at a backed up file you have to download it, push it into a different folder then open it separately. CrashPlan would benefit tremendously from the Windows Shell Integration that all of the other products provide, but then again, the primary feature is definitely for backup not for casual file access where you set up crash plan and forget about it until you have a problem or need files on one of your devices on the go.

Although you can easily access backed up files from various devices and phone apps, the process of sharing files is not quite as smooth as with other tools. You can only add files through a backup, but not from devices so in effect you can't push data from the device to the local machine.

CrashPlan also is not quite as good about new files added popping up in the backup right away. It basically schedules a backup after the last one completes, so the process is synchronous - one backup after the other. While you can control when and how files are backed up, even with the always option, new files aren't immediately detected and updated and there's no 'special folder' onto which you can drop files that are prioritized. This means that as a file sharing service CrashPlan is not ideal and you'll likely need to back it up with something like DropBox or SkyDrive.

This is not to fault CrashPlan - as a pure backup solution it works great and as such it pretty much does exactly what it promises to do.

SugarSync

One tool I looked at right at the beginning of my trials was SugarSync and I liked it right away, but I was initially put off by the higher pricing and for less data ($75/yr for 60 GB, $100 for 100 GB). As I looked at all the other choices and the pros and cons of both I found myself going back to SugarSync because to me at least it seems to strike the right balance between non-intrusive file backup and complete synching across devices.

There's lots to like about SugarSync. The first thing I noticed is that it seems a bit faster than most of the other services in getting the data pushed up to the server. Given my upload bandwidth is not exactly stellar here on Maui, I'm surprised at that figuring most of these services would be close to maxing out my connection, but it just seems like SugarSync was able to sync 6 GB of my initial test data in a few hours all the while seemingly not affecting the bandwidth while browsing too heavily. SugarSync also seems to be fairly light on the CPU usage. I let SugarSync back up my 6 gigs of test data twice and both times I was pleasantly surprised how quickly that went.

Also really nice is the syncing functionality of SugarSync which just seems much quicker than other services. Files dropped on the SugarSync Shared folder show up immediately even while other uploading is still going on. Even files changed while the initial upload is still going seem to show up quickly. It appears SugarSync is prioritizing changed files in backups which seems like a good idea that seems pretty obvious, but doesn't seem to happen in other products.

SugarSync's UI is more polished than most and it provides excellent Shell integration, so the shared folders show up in an S: drive on my system. The actual files in use are highlighted with icons that show the backup status of the actual files so it's easy to see what's changed and pending.

The Web and mobile interfaces make it easy to browse through files and pick individual files including multiple versions.

As a file sharing service, SugarSync is similar to DropBox and SkyDrive but has better Shell Integration than either of those products. I don't have to explicitly do folder mappings using mklink, but I can simply drop a folder onto SugarSync to have it show up in my shared folders. Once configured I can specify how the folders is shared (one way or two way).

On the downside it's not very easy to manage your mapped folders in SugarSync. In fact I ran into a major problem when I uninstalled SugarSync, then reinstalled a week later and all of a sudden had SugarSync back syncing from the cloud to my local machine in the process renaming a bunch of changed files. I also was unable to delete mapped folders while not connected, so it was difficult to get folders unlinked. The real trick is to mark a folder as not down-synched first then delete it, but that's all not very obvious. Still that issue was more of my problem as I just killed SugarSync and then reattached it telling it to sync up from the server, not realizing that the old files were still there.

To me SugarSync so far has the best combination of synching and file backup and the smoothest user experience. With low overhead, especially once the initial data upload has gone up there's very little activity on the wire and changes seem to appear very promptly on the cloud drive.

I'm not too thrilled with their pricing, but I for piece of mind I guess it's worth it. I just seems that the sweet spot for full back up should be right around $50/yr. Anything much above that just seems too much to spend…

No perfect Solutions

After all this trial and error with providers I still come away with partial solutions. Although SugarSync comes close it too is pretty rough. I posted on Twitter  throughout this trial and I heard quite a few horror stories about most services where files were missed or things took forever to sync up. It's surprising to see that this hasn't gotten a lot more solid yet.

Part of this too is that cloud backup are still hitting the limits of even reasonably fast connections. I have a 1.2mb up connection with RoadRunner here in Hawaii, but I rarely see that. Up speeds often hover around 750kb or so, which is pretty slow if you need to push up even 50GB of data. It'll take forever to sync. Now try to get that 1 or 2 GB database sent up everytime it changes and the backup will eat through a lot of bandwidth very quickly. I also noticed that when I was testing at some point my bandwidth started severly throttling - it's likely that RoadRunner decided to cripple my connection for a day or two of heavy uploads. We're all at the mercy of the bandwidth gate holders.

As I mentioned I've gone with SugarSync, but it was a close race between using it and CrashPlan + DropBox. CrashPlan works great for backup, and in combination with a file synching software it would make a good mix. This would allow for getting a full backup on CrashPlan and only putting a relatively small amount of shared files into DropBox or SkyDrive.

Even so after all of this I'm a little wiser and my data a is a little less susceptible to loss as I have my important files backed up in the Cloud - or they will be once they all arrive there :-)

I'd be curious to hear what some of you are using for online backup and synching and how you fared.

Resources

• SugarSync
• CrashPlan
• DropBox
• SkyDrive
• Carbonite