Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
Markdown Monster - The Markdown Editor for Windows

File Copy Disaster - self-inflicted at that


:P
On this page:

Oh man I scared the hell out of myself tonight. I thought I'd do a different backup tonight and do a full copy of my hard drive to an external hard disk. Usually I do a selected backup to DVD every week or so with a few quick copies of the stuff I'm working on to the external drive. I thought it would be good to do a more complete snapshot for a change.

 

Windows File Copy is not an option to do a file copy based back up. The problem with Windows is that it can't reliably copy files – one file that is locked and the whole copy process aborts. Here we are in 2006 and we're talking about 'exotic file systems'  like WinFS and Windows can't even copy files with something as simple as a dialog that let's you skip a file it can't copy. Pathetic…

 

You can use command line tools like XCopy or RoboCopy but the experience is a little less than stellar. So, I dug out my trusty copy of Beyond Compare which besides being a great tool for file comparison can also work well as a file copy utility. So I mapped most of folders to a backup directory on the external drive, got the thing started and went off to watch a movie.

 

An hour and a half later I walk into the room and the copy is not even close to done – it's on file 23,000 of 75,000. Yeah it's a lot of files but it shouldn't take this long to copy files between two fast local hard drives over a firewire connection. So I look at what's going on, and I notice – wholly crap – all the files on my local drive are gone. Well, less than half actually but that's enough!!!!

 

It turns out when I started Beyond Compare and its copy operation I accidentally started it using MOVE instead of COPY. Move tries to move files to the target and it deletes the files it has copied from the source. Big difference! Luckily BC doesn't do a hard delete but deletes to the Recyle bin by default. Thankfully… or so I thought anyway.

 

Problem is, have you ever tried to open the recycle bin with nearly 30,000 files in it? It's not pretty… It's a good thing this machine is a dual core machine - one of the virtual CPUs was locked solid while trying to open the Recyle bin. Worse I had deleted a bunch of stuff over the last couple of days in the midst of doing some housekeeping without clearing out the files. Hence the extra 7,000 files or so. Some of these were also files that were then written again later over the last few days, so a straight restore wouldn't work.

 

The Recycle bin doesn't support selecting multiple files to delete or restore – your options are delete/restore one at a time or do the whole thing. Now with 30,000 files in there even selecting individual files is very slow and I had to go search through the deleted files up to the copy date of today…

 

Aaaaaargh. Pain-ful. The whole restore process took close to an hour as a had to sift through the previous deleted files and figure out which files really needed deleting so they wouldn't interfere with newer versions.

 

Ok. So after this episode I'm looking for a good file management and copy utility. Anybody have any suggestions for good tools? Am I just asking for trouble like this again by using Beyond Compare?

 

I'm happy to report that I got all of my files back and my system is back to the way it was. Almost, almost there was the opportunity to get rid of XP again and install Win2003 after all. Nat….

 


The Voices of Reason


 

Christof Wollenhaupt
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

I use FileBack PC for backing up files to an external hard disk (http://www.fileback-pc.com)

Cato Antonsen
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

I like Second Copy (http://www.centered.com/) for filebackup to external disk or even FTP-server if you like.

I also use Drive Snapshot (http://www.drivesnapshot.de/en/) for full disk backup to external disk. The nice thing with this program is that it can backup your drive while you work! And you can map you backup-file to a driveletter to access files without having to restore whole disk. It's possible to make boot CD for restore of harddisk.

Both programs are shareware.

Tom Pester
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

Drive snapshot is great indeed.

It's not realy a traditional backup solution since it copies every bit of the harddisk so I don't know what happens when there is a corruption in the snapshot. Maybe the whole thing becomes useless?

Good thing is that it is an exact compressed copy of the harddisk and its fast.
Tom provides good support and he recently made differential backups possible.

deffentily worth checking out this little gem.
Look beyond the crappy ui though :)

nwells
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

Acronis backup utility is the best I have seen. Works like Ghost, only better.

Rick Strahl
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

I actually own TrueImage and I guess since I'm now running XP (rather than 2003 server) I can actually use it after all.

Rick Strahl
February 19, 2006

# re: File Copy Disaster - self-inflicted at that


How about a file copy utility that can deal with large amount of file copies and potential logging issues.

It'd be nice if you had something that can let you choose files or directories, then send them, capture errors and let you retry any files that didn't copy in the first pass.


Marc Brooks
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

Have you considered using FolderShare to sync things to another computer? That's what I do :)

David Brabant
February 19, 2006

# re: File Copy Disaster - self-inflicted at that

What's wrong with Windows Backup?

Rogas
February 20, 2006

# re: File Copy Disaster - self-inflicted at that

I would use Total Commander. It >has< the 'Skip' button :)

Dale Zimmer
February 20, 2006

# re: File Copy Disaster - self-inflicted at that

The backup utility I use is called XXCopy. Works the same as xcopy, but improved big time. Extremely fast, lightweight, 1000 more features. I have not found one thing it can not do for backup. You can include/exclude files from a list etc. Everything can be automated from commandline. Skip files automatically in use, can log errors. It is shareware and cheap too.

Sidney
February 20, 2006

# re: File Copy Disaster - self-inflicted at that

SecondCopy 2000 rocks. There's something sweet about having multiple redundant copies of your important data. Just set the copy process to be off-peak and away you go. It really is set-and-forget protection.

The best feature though is being able to save multiple copies of a deleted or changed file up to 25 previous versions. For those of us who have been through the pain of retrieving a file from VSS 6.0, having an easily accessible trail of changes is great. That 30 bucks has brought me 5 years peace-of-mind (priceless).

Your post touched a chord, as I just had an experience with Windows XP System Restore on a friends laptop - the wizard reassures the user that no documents will be lost! It fails to mention that they must exist in the 'My Documents' folder for this rule to apply. Ack!

Incidentally, if you have System Restore on, that may be why the file operations (delete in your case) were so slow - the operating system is being diligent and keeping a meticulous record (and copy) of every change that it's making ... which can bring even the mightiest of laptops to its 'knees'. Cheers,

Bobby
February 21, 2006

# re: File Copy Disaster - self-inflicted at that

I like to use Karen's Power Tools - Replicator http://www.karenware.com/powertools/ptreplicator.asp. It is freeware (although donations are appreciated) that will synch a directory (or multiple directories) to another location. You can set it up to base the output directory on the day of the week (and other options). I use it at home to back up important directories and it keeps a 7-day rolling backup of those directories.
It is a nice utility.

Peter Mescalchin
February 22, 2006

# re: File Copy Disaster - self-inflicted at that

Surprised you knocked Robocopy - I find it works great and have used it for many a situation.

I have a simple batch file that does about 5 robocopies to another machine on my LAN in mirror mode and works nicely. Also in the batch file are a few commands to dump mySQL databases/etc. for the process.

Then every few weeks/months I just burn back the robocopied files to DVD-R.

The feature set of Robocopy, well at least in the later windows XP builds is pretty comprehensive.

christophe
January 16, 2008

# re: File Copy Disaster - self-inflicted at that

hello !

what about keeping beyond compare and using COPY instead of MOVE ? ;-)

... or reading the confirmation "pop-up" after clicking ... MOVE :-)))

just my two cents

Christophe

Frederick N. Chase
August 04, 2009

# re: File Copy Disaster - self-inflicted at that

YES:
"what about keeping beyond compare and using COPY instead of MOVE ? ;-)"

I've tried many backup systems over the years, though not every one of the above.

I've settled on scripted Beyond Compare to rewritable DVDs.
By scripting it, you can control things like clicking MOVE instead of COPY.

Since an example will put you way ahead in considering this, I am putting below
--a Beyond Compare script, then
--a batch file to run it then
--some miscellaneous support files.

You will see that I haven't made sophisticated use of environment variables, ...

With these shitty little languages like MS .bat files I just do the bare minimum to get what I want. (After using MULTICS com files even unix sh, csh, and the like seem annoying.)

Actually, I landed on this page as I begin looking for a discussion of why "xcopy ... /S /Y" within a .bat copies a file and then copies another file with the same name but different contents OVER THE FIRST DESTINATION!

Sorry for the clutter.
You can find an email for me on the web if you look hard.
-Fred




THE BC SCRIPT, 0001 bc2 script.txt:
# DISK 1 CONTENTS
# From          "C:\A FileSystem Cleanup Helper\"
# not including "C:\C#"
# down through  "C:\Documents and Settings"


#NOTE:   For resilience, I try hard not to violate the notion that ALPHABETICAL RANGE <-> DISC.


log normal "C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 Bc2Synclog.txt"
criteria timestamp size
filter  "-Thumbs.db;*.*"

load "C:\A FileSystem Cleanup Helper\" "E:\A FileSystem Cleanup Helper\"
sync mirror:lt->rt

load "C:\Data\" "E:\Data\"
sync mirror:lt->rt





load "C:\Documents and Settings\All Users\Application Data\Fidelity Investments\Fidelity Active Trader\data\"   "E:\Documents and Settings\All Users\Application Data\Fidelity Investments\Fidelity Active Trader\data\"
sync mirror:lt->rt


load "C:\Documents and Settings\All Users\Start Menu\Programs\"   "E:\Documents and Settings\All Users\Start Menu\Programs\"
sync mirror:lt->rt


# my Genealogy master database!!
load "C:\Documents and Settings\Fred\My Documents\The Master Genealogist v7\"   "E:\Documents and Settings\Fred\My Documents\The Master Genealogist v7\"
sync mirror:lt->rt


load "C:\Documents and Settings\Fred\.moneydance\"    "E:\Documents and Settings\Fred\.moneydance\"
sync mirror:lt->rt


load "C:\Documents and Settings\Fred\Application Data\FidelityData\" "E:\Documents and Settings\Fred\Application Data\FidelityData\"
sync mirror:lt->rt


# Firefox bookmark treasuretrove
load "C:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default"  "E:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default"
filter "bookmarks.html"
sync mirror:lt->rt
filter  "-Thumbs.db;*.*"

# Data for IMatch (photools) (only a few files here)
load "C:\Documents and Settings\Fred\Application Data\photools.com"  "E:\Documents and Settings\Fred\Application Data\photools.com"
sync mirror:lt->rt


load "C:\Documents and Settings\Fred\Application Data\Thunderbird\Profiles\nj6nxcr6.default"   "E:\Documents and Settings\Fred\Application Data\Thunderbird\Profiles\nj6nxcr6.default\"
sync mirror:lt->rt

load "C:\Documents and Settings\Fred\Desktop"          "E:\Documents and Settings\Fred\Desktop"
sync mirror:lt->rt

load "C:\Documents and Settings\Fred\My Documents"      "E:\Documents and Settings\Fred\My Documents"
collapse all
sync visible mirror:lt->rt

load "C:\Documents and Settings\Fred\Start Menu"        "E:\Documents and Settings\Fred\Start Menu"
sync mirror:lt->rt



THE BATCH FILE INVOKING IT   0001 bc2 script.bat:
@echo off

mkdir "E:\A FileSystem Cleanup Helper\"

mkdir "E:\Data\"

mkdir "E:\Documents and Settings\All Users\Application Data\Fidelity Investments\Fidelity Active Trader\data\"
mkdir "E:\Documents and Settings\All Users\Start Menu\Programs\"
mkdir "E:\Documents and Settings\Fred\.moneydance"
mkdir "E:\Documents and Settings\Fred\Application Data\Google\GoogleEarth"
mkdir "E:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default"
mkdir "E:\Documents and Settings\Fred\Application Data\photools.com"
rem get only C:\Documents and Settings\Fred\Application Data\Mozilla\Firefox\Profiles\011ndapp.default\bookmarks.html
mkdir "E:\Documents and Settings\Fred\Application Data\Thunderbird\Profiles\nj6nxcr6.default"
mkdir "E:\Documents and Settings\Fred\Application Data\FidelityData\"
mkdir "E:\Documents and Settings\Fred\Desktop"
mkdir "E:\Documents and Settings\Fred\My Documents"
rem get only the loose files
mkdir "E:\Documents and Settings\Fred\Start Menu"
rem   my Genealogy master database!!
mkdir "E:\Documents and Settings\Fred\My Documents\The Master Genealogist v7"




@echo on
rem To COMPARE (for RESTORE), hit enter.  
@echo off
set /p bkup=  To BACKUP (IMMEDIATELY make the DVD like the hard disk), type backup, then enter  
if %bkup% == backup goto DOIT
goto SKIP

:DOIT

"C:\Program Files\Beyond Compare 2\BC2.exe" @"C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 bc2 script.txt"

mkdir                                                                                     "E:\Documents and Settings\Fred\Application Data\Google\GoogleEarth"
copy /Y "C:\Documents and Settings\Fred\Application Data\Google\GoogleEarth\myplaces.kml" "E:\Documents and Settings\Fred\Application Data\Google\GoogleEarth\myplaces.kml"



rem This log was just completed.
copy /Y "C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 Bc2Synclog.txt" "E:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 Bc2Synclog.txt"



:SKIP
"C:\Program Files\Beyond Compare 2\BC2.exe" "C:\"   "E:\"


THE SCRIPT INVOKER (one line)  0001 bc2 script.invoker.bat:
"C:\Data\Backups\Beyond Compare Scripts for DVD 2\0001 bc2 script.bat"

THE NEW DISK BATCH FILE  0001 newDisk.bat:

rem   Disk must be formatted.

copy  autorun.2.inf  E:\autorun.inf

copy  "0001 bc2 script.invoker.bat" "E:\bcSynchThisDisk.bat"
                    
"E:\bcSynchThisDisk.bat"

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024