[SOLVED] Help Automating Backup of Drive

I need help with this problem. OK, here goes. I think I have a bad internal keyboard. I'm going to want to replace it. I need a little hardware help to do the work, I'm a software guy, not a hardware guy. I typically save a partition table, and then use partimage for backup, when backing up in this manor. I "think" I have enough space. Basically, I want to make sure software is saved so when it gets replaced, I lose nothing. Even the OS could possibly be replaced, but some of the files, CANNOT be replaced.

But partimage failed to work, probably because I see I'm using ext4, not ext3. This could be fixed, but for time, I just want to save it anyway. The OS for backup will be System Rescue CD running off of a ventoy USB stick.

I typically use dd when partimage fails like that. I do a simple:

dd if=<savedrive> of=<outputfile>

However, in this case, that was unreliable. After about 2 days, it still failed without even being completed for the command. I'm thinking that if I split each image, into my standard split size (the size of my smallest CD), 637MB, like partimage does, this would succeed. I can immediately compress the files, each one, right with bzip, right after I make the image. The extension, should probably do what dd does, and end in a number. To restore it, I think I could uncompress, then join the images, with a copy, then restore the file to the hard drive. I've only now begun to think of restoration, which IS important.

But giving each command to the OS, to do the work, is tedious, and error prone. I CANNOT afford one error in this process. Can you help me write a script for this task? Then, I can include it with my standard scripts, in which case my process for backing up, would be:

1. boot the disk
2. install standard scripts
3. run the script for backup
4. reboot into real os, and good to go

This is simplifying it, but I should be able to do this. I might need this process in the future too, which is why I thought script would be best. I may not have the latest version of the backup OS.

I cannot afford to make mistakes, and need no sudo in the script, as a requirement, if sudo is needed, I will have to sudo it by the whole script.

Code so far:

#! /bin/bash


if [ $# == 3 ]; then
 countvar=$3
 splitnum=0
 echo "Saving drive if=$1, of=$2.$splitnum <blocksize=$countvar>..."
 
 dd if=$1 of=`echo $2.$splitnum` count=$countvar

 for i in `seq 0..`
 do
 done

 echo
else
 echo "Usage: $0 <input> <output> <splitsize>"
 echo
 echo "input     - The input file."
 echo "output    - The output file."
 echo "splitsize - The size in megabytes, to split the file into for ouput."
 echo
fi

I'm open to suggestions if this is not right process. I DON'T want to use clonezilla, because it takes too much to setup and get working. I can backup online in Linux Mint, or offline in that OS. I want to do the quickest too, as well as safest. Please try to help quick, I need the replacement, but don't make an error or I'm in trouble. This could be the wrong idea.

Boot from some external device (dvd, usb) and backup the disk/partition without mounting it (or mounting it read/only mode).
Meta: maybe Programming forum is not optimal for this question.

I wondered if it fits here or not. Gray area,i thought. As far as the backup, thats what im trying to do. Should i just copy the files? It wouldnt be as easy, but im wondering if efficiant.

Why are you using a tool that states on its homepage it is no longer developed, and trying to backup a filesystem that was never supported ???. Madness.

As suggested on that page use a supported tool - I like fsarchiver for this sort of thing. Seems you might need the some extra protection like the checksumming it offers.

I'm using partimage because I always have, and the last I checked until now, there were no issues. If I can learn the new tool, I'd be happy to use it too, but I'm probably not going to right away convert every image. I am trying to backup a filesystem that had issues, because I didn't know it would. It's always worked for me, even windows, so I had no idea it would fail now.

Yes, extra protection would be great! I'll check into fsarchiver as I delete my files giving me trouble.

I'll tell you how I would do this, although it's pretty different.

1) Use dd only to back up a very small portion of the drive near the start (basically only to back up the partition table and initial bootloader).

2) record the output of "fdisk -l" and "lsblk -f" just in case I need to recreate a partition's file system from scratch. Example output here:

root@rapunzel:~# fdisk -l
Disk /dev/sda: 27.96 GiB, 30016659456 bytes, 58626288 sectors
Disk model: TOSHIBA THNSNB03
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x62f17335

Device     Boot Start      End  Sectors  Size Id Type
/dev/sda1  *     2048 58595327 58593280 27.9G 83 Linux
root@rapunzel:~# lsblk -f
NAME   FSTYPE FSVER LABEL    UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                              
└─sda1 ext4   1.0   deb9os64 567e8081-c884-4242-9c60-8920eebc9feb    1.2G    90% /
sr0

3) Use rsync to backup the partition's contents. It IS SAFE to do this while the partition itself is mounted. Just keep on doing rsync until it's not updating very many files each pass.

Example rsync:

rsync -vaxAX --delete --progress /* /mnt/USBdrive/OSbackup/

This example will replicate the contents of the current / partition to a folder "OSbackup" on a mounted "USBdrive". The various flags ensure replication of file permissions and such, as well as providing progress feedback.

In case I need to restore this, the steps are:

1) Use dd to restore the start of the drive.

2) Reformat the ext4 partition with GParted or something. Edit the UUID and Label if necessary.

3) Mount the ext4 partition and use rsync to copy the contents back.

Example rsync:

rsync -vaxAX --delete --progress /mnt/USBdrive/OSbackup/* /mnt/NewExt4Partition

If you get really precise with the "dd" command, you can make it so that you ONLY copy/overwrite the partition table and bootloader. This means you won't need to reformat the ext4 partition, and that can simplify things.

You can use gzip and split to divide the image in chunks (2G in this example) like:

 dd if=/dev/sdx conv=sync,noerror bs=64K | gzip -c | split -a3 -b2G - /path/to/backup.img.gz

and restore

 cat /path/to/backup.img.gz* | gunzip -c | dd of=/dev/sdx

Remember that dd copies the entire disk/filesystem not just the used parts. You have to create the backup from a live system due to fact the filesystem(s) can not be active.

I had a NAS corruption problem to deal with, before I had space to deal with this. I think part of my issue, is I don't have enough space, all on one drive. I need to be able to split some of this between drives. I will NOT neglect to study this last command, as I think that is what I was origionally looking for, if I could understand it. But now, I think, that just backing up my files, could be best, because of space issues, and really, my OS could be replaced, whereas the files mostly CANNOT. But backing them up to my NAS, or external drive, is good enough for me for now. I was just trying to do it the easiest way. Perhaps I will use ext3 if I ever rebuild it again.

Anyway, restarting my NAS, seems to have fixed my whole corruption problem, then I was able to delete the big files taking up space, in 2 seconds. And I didn't lose any other work now. Now I can try again to back up my files, now that I have space on the NAS.

Hey, I've been where you are now, and my experience is - don't fear ext4. It will help more than it hurts, especially in eliminating those annoying bouts of long file system check on boot.

There are tools to do everything you need with ext4.

All of that with the recommended programs and stuff, pretty much changed overnight as far as I could understand. That is why I had no idea that partimage is outdated now. It's by chance that I had ext4, and I thought that it would be supported. I heard now, it's the new "standard". I always when I knew, have been an ext* guy rather than reiserfs or anything like that. I have used reiserfs, but that was before I knew what my preferences were. Also, unless there's reasons to do so, on some server, I tend to stick to either of those two also, unless I'm on a server, or my distro recommends something else for some distro specific stuff.

I downloaded a new ISO, and will be migrating to the newer tool. But I'm glad I got the old version saved, as there's times it will be better going forward, for a transitional period. That's why my NAS is so important. I don't typically try to copy my whole NAS or anything for someone else, but I prefer programs which don't care if I distribute, and I then distribute "network" wide, which is the household, but so far, that's just me anyway. I don't always care if everything is 100% "free" by definition, at this point, just that it is a free as in beer, and allows that permission, so that I can keep a copy of the download, for reinstalling beyond where they support it. Less of that is now needed with Linux though. More pulling from the repository now. The overnight thing is how I can have no idea about that tech change, but it's a good thing, from my study.

I've almost got the files on my NAS now, just one or so (folders) to go still. In the command above, is the final dash neccessary?

dd if=/dev/sdx conv=sync,noerror bs=64K | gzip -c | split -a3 -b2G <<-> Needed??> /path/to/backup.img.gz

Or was that dash in error? I think I understand the command other than that now. If I had enough space, that's what I would be using. I found I have a space issue, which is why I finally just started copying the basic files.

Once I know that command, when I can, it will be added to my command reference I'm creating. I'm doing what Distrotube says he does, going forward.

Have you read the man page for split? It is not an error. It isn't absolutely required but more of a convention for stdin.

With no FILE, or when FILE is -, read standard input.

I read the man page, and missed that.

OK. I now understand the command given completely.

Just 4 hours, and all data will be copied off...