File Copy Disaster - self-inflicted at that
Oh man I scared the hell out of myself tonight. I thought I'd do a different backup tonight and do a full copy of my hard drive to an external hard disk. Usually I do a selected backup to DVD every week or so with a few quick copies of the stuff I'm working on to the external drive. I thought it would be good to do a more complete snapshot for a change.
Windows File Copy is not an option to do a file copy based back up. The problem with Windows is that it can't reliably copy files – one file that is locked and the whole copy process aborts. Here we are in 2006 and we're talking about 'exotic file systems' like WinFS and Windows can't even copy files with something as simple as a dialog that let's you skip a file it can't copy. Pathetic…
You can use command line tools like XCopy or RoboCopy but the experience is a little less than stellar. So, I dug out my trusty copy of Beyond Compare which besides being a great tool for file comparison can also work well as a file copy utility. So I mapped most of folders to a backup directory on the external drive, got the thing started and went off to watch a movie.
An hour and a half later I walk into the room and the copy is not even close to done – it's on file 23,000 of 75,000. Yeah it's a lot of files but it shouldn't take this long to copy files between two fast local hard drives over a firewire connection. So I look at what's going on, and I notice – wholly crap – all the files on my local drive are gone. Well, less than half actually but that's enough!!!!
It turns out when I started Beyond Compare and its copy operation I accidentally started it using MOVE instead of COPY. Move tries to move files to the target and it deletes the files it has copied from the source. Big difference! Luckily BC doesn't do a hard delete but deletes to the Recyle bin by default. Thankfully… or so I thought anyway.
Problem is, have you ever tried to open the recycle bin with nearly 30,000 files in it? It's not pretty… It's a good thing this machine is a dual core machine - one of the virtual CPUs was locked solid while trying to open the Recyle bin. Worse I had deleted a bunch of stuff over the last couple of days in the midst of doing some housekeeping without clearing out the files. Hence the extra 7,000 files or so. Some of these were also files that were then written again later over the last few days, so a straight restore wouldn't work.
The Recycle bin doesn't support selecting multiple files to delete or restore – your options are delete/restore one at a time or do the whole thing. Now with 30,000 files in there even selecting individual files is very slow and I had to go search through the deleted files up to the copy date of today…
Aaaaaargh. Pain-ful. The whole restore process took close to an hour as a had to sift through the previous deleted files and figure out which files really needed deleting so they wouldn't interfere with newer versions.
Ok. So after this episode I'm looking for a good file management and copy utility. Anybody have any suggestions for good tools? Am I just asking for trouble like this again by using Beyond Compare?
I'm happy to report that I got all of my files back and my system is back to the way it was. Almost, almost there was the opportunity to get rid of XP again and install Win2003 after all. Nat….
Other Posts you might also like
- Adding minimal OWIN Identity Authentication to an Existing ASP.NET MVC Application
- Resolving Paths To Server Relative Paths in .NET Code
- Map Physical Paths with an HttpContext.MapPath() Extension Method in ASP.NET
- Back to Basics: Rewriting a URL in ASP.NET Core
- Getting the ASP.NET Core Server Hosting Urls at Startup and in Requests
The Voices of Reason
# re: File Copy Disaster - self-inflicted at that
I also use Drive Snapshot (http://www.drivesnapshot.de/en/) for full disk backup to external disk. The nice thing with this program is that it can backup your drive while you work! And you can map you backup-file to a driveletter to access files without having to restore whole disk. It's possible to make boot CD for restore of harddisk.
Both programs are shareware.
# re: File Copy Disaster - self-inflicted at that
It's not realy a traditional backup solution since it copies every bit of the harddisk so I don't know what happens when there is a corruption in the snapshot. Maybe the whole thing becomes useless?
Good thing is that it is an exact compressed copy of the harddisk and its fast.
Tom provides good support and he recently made differential backups possible.
deffentily worth checking out this little gem.
Look beyond the crappy ui though :)
# re: File Copy Disaster - self-inflicted at that
# re: File Copy Disaster - self-inflicted at that
# re: File Copy Disaster - self-inflicted at that
How about a file copy utility that can deal with large amount of file copies and potential logging issues.
It'd be nice if you had something that can let you choose files or directories, then send them, capture errors and let you retry any files that didn't copy in the first pass.
# re: File Copy Disaster - self-inflicted at that
# re: File Copy Disaster - self-inflicted at that
# re: File Copy Disaster - self-inflicted at that
# re: File Copy Disaster - self-inflicted at that
# re: File Copy Disaster - self-inflicted at that
The best feature though is being able to save multiple copies of a deleted or changed file up to 25 previous versions. For those of us who have been through the pain of retrieving a file from VSS 6.0, having an easily accessible trail of changes is great. That 30 bucks has brought me 5 years peace-of-mind (priceless).
Your post touched a chord, as I just had an experience with Windows XP System Restore on a friends laptop - the wizard reassures the user that no documents will be lost! It fails to mention that they must exist in the 'My Documents' folder for this rule to apply. Ack!
Incidentally, if you have System Restore on, that may be why the file operations (delete in your case) were so slow - the operating system is being diligent and keeping a meticulous record (and copy) of every change that it's making ... which can bring even the mightiest of laptops to its 'knees'. Cheers,
# re: File Copy Disaster - self-inflicted at that
It is a nice utility.
# re: File Copy Disaster - self-inflicted at that
I have a simple batch file that does about 5 robocopies to another machine on my LAN in mirror mode and works nicely. Also in the batch file are a few commands to dump mySQL databases/etc. for the process.
Then every few weeks/months I just burn back the robocopied files to DVD-R.
The feature set of Robocopy, well at least in the later windows XP builds is pretty comprehensive.
# re: File Copy Disaster - self-inflicted at that
what about keeping beyond compare and using COPY instead of MOVE ? ;-)
... or reading the confirmation "pop-up" after clicking ... MOVE :-)))
just my two cents
Christophe
# re: File Copy Disaster - self-inflicted at that
"what about keeping beyond compare and using COPY instead of MOVE ? ;-)"
I've tried many backup systems over the years, though not every one of the above.
I've settled on scripted Beyond Compare to rewritable DVDs.
By scripting it, you can control things like clicking MOVE instead of COPY.
Since an example will put you way ahead in considering this, I am putting below
--a Beyond Compare script, then
--a batch file to run it then
--some miscellaneous support files.
You will see that I haven't made sophisticated use of environment variables, ...
With these shitty little languages like MS .bat files I just do the bare minimum to get what I want. (After using MULTICS com files even unix sh, csh, and the like seem annoying.)
Actually, I landed on this page as I begin looking for a discussion of why "xcopy ... /S /Y" within a .bat copies a file and then copies another file with the same name but different contents OVER THE FIRST DESTINATION!
Sorry for the clutter.
You can find an email for me on the web if you look hard.
-Fred
THE BC SCRIPT, 0001 bc2 script.txt:
# DISK 1 CONTENTS
# From "C:\A FileSystem Cleanup Helper\"
# not including "C:\C#"
# down through "C:\Documents and Settings"
#NOTE: For resilience, I try hard not to violate the notion that ALPHABETICAL RANGE <-> DISC.
log normal "C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 Bc2Synclog.txt"
criteria timestamp size
filter "-Thumbs.db;*.*"
load "C:\A FileSystem Cleanup Helper\" "E:\A FileSystem Cleanup Helper\"
sync mirror:lt->rt
load "C:\Data\" "E:\Data\"
sync mirror:lt->rt
load "C:\Documents and Settings\All Users\Application Data\Fidelity Investments\Fidelity Active Trader\data\" "E:\Documents and Settings\All Users\Application Data\Fidelity Investments\Fidelity Active Trader\data\"
sync mirror:lt->rt
load "C:\Documents and Settings\All Users\Start Menu\Programs\" "E:\Documents and Settings\All Users\Start Menu\Programs\"
sync mirror:lt->rt
# my Genealogy master database!!
load "C:\Documents and Settings\Fred\My Documents\The Master Genealogist v7\" "E:\Documents and Settings\Fred\My Documents\The Master Genealogist v7\"
sync mirror:lt->rt
load "C:\Documents and Settings\Fred\.moneydance\" "E:\Documents and Settings\Fred\.moneydance\"
sync mirror:lt->rt
load "C:\Documents and Settings\Fred\Application Data\FidelityData\" "E:\Documents and Settings\Fred\Application Data\FidelityData\"
sync mirror:lt->rt
# Firefox bookmark treasuretrove
load "C:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default" "E:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default"
filter "bookmarks.html"
sync mirror:lt->rt
filter "-Thumbs.db;*.*"
# Data for IMatch (photools) (only a few files here)
load "C:\Documents and Settings\Fred\Application Data\photools.com" "E:\Documents and Settings\Fred\Application Data\photools.com"
sync mirror:lt->rt
load "C:\Documents and Settings\Fred\Application Data\Thunderbird\Profiles\nj6nxcr6.default" "E:\Documents and Settings\Fred\Application Data\Thunderbird\Profiles\nj6nxcr6.default\"
sync mirror:lt->rt
load "C:\Documents and Settings\Fred\Desktop" "E:\Documents and Settings\Fred\Desktop"
sync mirror:lt->rt
load "C:\Documents and Settings\Fred\My Documents" "E:\Documents and Settings\Fred\My Documents"
collapse all
sync visible mirror:lt->rt
load "C:\Documents and Settings\Fred\Start Menu" "E:\Documents and Settings\Fred\Start Menu"
sync mirror:lt->rt
THE BATCH FILE INVOKING IT 0001 bc2 script.bat:
@echo off
mkdir "E:\A FileSystem Cleanup Helper\"
mkdir "E:\Data\"
mkdir "E:\Documents and Settings\All Users\Application Data\Fidelity Investments\Fidelity Active Trader\data\"
mkdir "E:\Documents and Settings\All Users\Start Menu\Programs\"
mkdir "E:\Documents and Settings\Fred\.moneydance"
mkdir "E:\Documents and Settings\Fred\Application Data\Google\GoogleEarth"
mkdir "E:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default"
mkdir "E:\Documents and Settings\Fred\Application Data\photools.com"
rem get only C:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default\bookmarks.html
mkdir "E:\Documents and Settings\Fred\Application Data\Thunderbird\Profiles\nj6nxcr6.default"
mkdir "E:\Documents and Settings\Fred\Application Data\FidelityData\"
mkdir "E:\Documents and Settings\Fred\Desktop"
mkdir "E:\Documents and Settings\Fred\My Documents"
rem get only the loose files
mkdir "E:\Documents and Settings\Fred\Start Menu"
rem my Genealogy master database!!
mkdir "E:\Documents and Settings\Fred\My Documents\The Master Genealogist v7"
@echo on
rem To COMPARE (for RESTORE), hit enter.
@echo off
set /p bkup= To BACKUP (IMMEDIATELY make the DVD like the hard disk), type backup, then enter
if %bkup% == backup goto DOIT
goto SKIP
:DOIT
"C:\Program Files\Beyond Compare 2\BC2.exe" @"C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 bc2 script.txt"
mkdir "E:\Documents and Settings\Fred\Application Data\Google\GoogleEarth"
copy /Y "C:\Documents and Settings\Fred\Application Data\Google\GoogleEarth\myplaces.kml" "E:\Documents and Settings\Fred\Application Data\Google\GoogleEarth\myplaces.kml"
rem This log was just completed.
copy /Y "C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 Bc2Synclog.txt" "E:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 Bc2Synclog.txt"
:SKIP
"C:\Program Files\Beyond Compare 2\BC2.exe" "C:\" "E:\"
THE SCRIPT INVOKER (one line) 0001 bc2 script.invoker.bat:
"C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 bc2 script.bat"
THE NEW DISK BATCH FILE 0001 newDisk.bat:
rem Disk must be formatted.
copy autorun.2.inf E:\autorun.inf
copy "0001 bc2 script.invoker.bat" "E:\bcSynchThisDisk.bat"
"E:\bcSynchThisDisk.bat"
# re: File Copy Disaster - self-inflicted at that