Hello beautiful peoples!
A few days ago, I finally reinstalled Windows 11 24H2 on my main machine. The process went relatively smoothly, without any major issues. Before I formatted all my SSD drives to start fresh however, I have decided to repurpose one of my older drives – a 3.5″ WD Green 2TB (spinning rust) – for BackUp.
There are plenty of ways I could transfer my three SSDs onto this sluggish USB 2.0 drive, but over the years, I’ve grown fond of rsync. Don’t get me wrong – it has its quirks – but it solves more problems than it causes. Since I don’t need a full disk image and only want to copy all accessible files, the decision is an easy one. Rsync is a solid choice – flexible, efficient, and reliable. Even on a slow USB connection, it’ll get the job done with minimal fuss.
Since I wanted to access the files from Windows 11 later, using the EXT4 file system wasn’t an option. That left me with either FAT or NTFS – so NTFS it is. I used diskpart to clean the drive and convert it to MBR, then followed up with Disk Management to add a partition and format it as BackUp.
Next I have rebooted to my USB installed Debian Testing (any LIVE distribution will do) and mounted my SSDs accordingly as /media/andrzejl/SySSDtem/, /media/andrzejl/SSDocuments/ and /media/andrzejl/GameSSD. Next I have mounted my 2TB BackUp drive as /media/andrzejl/BackUp/ and I have created 3 folders C, D and E on it.
At first I went with a very straight-forward approach.
Copy everything
– from /media/andrzejl/SySSDtem/ to /media/andrzejl/BackUp/C/
– be vvvery vvverbose,
– show me the progress,
– ignore all errors and keep going,
– if you get interrupted mid-file make sure you can continue without starting over,
– write directly to the destination,
– if its not on the original drive but for some reason is on the BackUp drive – get rid of it,
– limit the transfers to 5 MB/s,
– don’t compress.
rsync -avvv --progress --ignore-errors --partial --inplace --delete --bwlimit=5000 --no-compress /media/andrzejl/SySSDtem/ /media/andrzejl/BackUp/C/I wrote matching commands for the other two SSDs and ran them in separate Konsole windows as root. Everything was flying. Since the transfer was going to take a good few hours – moving about 1.5 to 1.6 terabytes over sluggish USB 2.0 speeds – I disabled sleep and screen locking and then took my wife on a long walk.
When I came back to it about 2 hours later I’ve noticed an issue…
Rsync has pooped the bed with:
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(849) [generator=3.4.1] [generator] _exit_cleanup(code=10, file=io.c, line=849): about to call exit(10)
Killed
That “Broken pipe (32)” error suggests that rsync was interrupted while writing to the USB drive. Possibly due to USB disconnecting or becoming temporarily unresponsive OR buffering issues with large file transfers OR maybe power issues… Who knows…
I re-ran the command and it copied some more files and threw the same error in my face…
Not ideal. First of all, it was supposed to ignore all errors 😒 and second… I’m certainly not going to sit here babysitting the process, just to manually re-run the command every time it fails. No, this needs to loop until it’s done. Loop… Hmmm…
And that’s how the final iteration of my backup command was born:
until rsync -av --progress --ignore-errors --partial --inplace --delete --no-compress /media/andrzejl/SySSDtem/ /media/andrzejl/BackUp/C/; do
echo "Rsync failed. Retrying..."
sleep 5
doneI ditched the vvvery vvverbose output – no point in excessive details when I am away. I also removed the –bwlimit=5000 restriction. Initially, I had used it to avoid overwhelming the USB drive, but since it got overwhelmed anyway and I’m looping the whole process… sod it – we’re going full speed ahead, Captain!
After a while I came back and noticed that the command was humming along nicely. I was also lucky to catch this moment of a command doing exactly what was expected of it…
Lovely… Few hours later I came back and re-run all 3 of the commands just to be sure to be sure (if you know – you know) and few minutes later I was unmounting all the drives knowing that my BackUp operation succeeded.
I hope this helps!
Catch you on the flip side,
AndrzejL
