[SOLVED] Trying to shrink Mac OS X installation image using Linux utilities

DaneM · 03-24-2016, 04:49 PM

Edit: This problem is now solved. Refer to post #3 for the solution.

(Mint 17.3 AMD64 Cinnamon)

I keep a collection of images on my Linux workstation to expedite computer repair. Recently, I've been working on more Macs, and discovered that making installation media for that OS is a pain in the rear (takes hours, requires an iStore login, etc.). Naturally, my first thought was to use dd to clone a working USB flash drive installer into an image on my workstation's hard drive, so I can make more installer drives in a jiffy, whenever needed. That worked nicely, with one catch: the actual installation image is only 8GB or so, but because I used a 32GB flash drive to make it, the disk-dumped image on the hard drive is 32GB.

I figured out how to shrink the filesystem using the method described, below, but what I'm left with is a 32GB image that contains a couple of smaller partitions amounting to ~8GB. My goal is to shrink this down to a file of no more than 8GB; but when I try to do so, I get error messages about the GPT partition table, and am unable to read the resulting file as a flash drive image without getting some alarming errors.

I don't currently have a Mac on-hand to experiment with. (Working on that...)

The original image is ElCapInstaller.iso. I've copied this to ElCapInstaller2.iso to protect the known-working 32GB version. hfsplus, hfsutils, hfsprogs are already installed.

Code:

# modprobe loop      # insert the loopback kernel module, just to be sure
# losetup -f          # this returns the name of an unassigned loopback device
/dev/loop0
# losetup /dev/loop0 ElCapInstaller2.iso         # assign ElCapInstaller2.iso to /dev/loop0
# partprobe /dev/loop0       # check /dev/loop0 for partitions, make Linux aware of them
# blkid       # find out the filesystem types and labels of all attached partitions
[...]
/dev/loop0p1: LABEL="EFI" UUID="67E3-17ED" TYPE="vfat" 
/dev/loop0p2: UUID="fe8b1f84-4011-3229-8d2f-d7e6cf2c4b3d" LABEL="Install OS X El Capitan" TYPE="hfsplus"
[...]
# mount /dev/loop0p2 /mnt/tmp   # mount the biggest partition
mount: warning: /mnt/tmp seems to be mounted read-only.
# df -h        # see how much space is being used.  NOTE THAT I'VE ALREADY SHRUNK THIS PARTITION
Filesystem      Size  Used Avail Use% Mounted on
[...]
/dev/loop0p2    6.4G  6.0G  474M  93% /mnt/tmp
# umount /mnt/tmp        # unmount the partition
# parted /dev/loop0      # shrink the partition...but remember that I've already done so; this output is just for reference
(parted) print                                                            
Model: Loopback device (loop)
Disk /dev/loop0: 31.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name                  Flags
 1      20.5kB  210MB   210MB   fat32        EFI System Partition  boot
 2      210MB   7046MB  6836MB  hfs+         ElCapInstaller

(parted) resize                                                           
WARNING: you are attempting to use parted to operate on (resize) a file system.
parted's file system manipulation code is not as robust as what you'll find in
dedicated, file-system-specific packages like e2fsprogs.  We recommend
you use parted only to manipulate partition tables, whenever possible.
Support for performing most operations on most types of file systems
will be removed in an upcoming release.
Partition number? 2
Start?  [210MB]?
End?  [7046MB]? 7G         # being conservative.  can't be sure if df uses 1M=1000K=1000B...
[at this point, it does the operation, then returns to the parted prompt]
(parted) quit
# partprobe /dev/loop0      # update partition table in OS
# ls -sh ElCapInstaller2.iso 
29G ElCapInstaller2.iso

So, now I have the partitions shrunken nicely. Unfortunately, the image file, itself, is still huge! (Apparently, ls uses 1G=1024M=1024K.) I've tried two methods of shrinking the file:

Code:

# losetup -d /dev/loop0     # unassign loopback device, since I had previously assigned it to the image file that I'm about to change
$ dd if=<imagefile> of=<copyofimagefile> bs=1M count=<howbigitshouldbe>     # shrink image

...and...

Code:

# losetup -d /dev/loop0     # unassign loopback device, since I had previously assigned it to the image file that I'm about to change
$ truncate -s <howbigitshouldbe> <imagefile>     # shrink image

In each case, I'm conservative with the space requirements, allowing 8G, instead of 6 or 7G, to make sure I don't cut anything important off. The new file is the right size, when complete, but both methods produce the same result when trying to read the image:

Code:

# losetup /dev/loop0 ElCapInstaller2.iso # assign the shrunken image to loopback device
# partprobe /dev/loop0 # update partition info in OS
Error: Invalid argument during seek for read on /dev/loop0 # Uh-oh...
# parted /dev/loop0 print # print partition info of image; this is where it "hits the fan"
Error: Invalid argument during seek for read on /dev/loop0
Retry/Ignore/Cancel? Ignore
Error: The backup GPT table is corrupt, but the primary appears OK, so that will
be used.
OK/Cancel? OK
Backtrace has 8 calls on stack:
8: /lib/x86_64-linux-gnu/libparted.so.0(ped_assert+0x31) [0x7fdd48a9a4b1]
7: /lib/x86_64-linux-gnu/libparted.so.0(+0x3f686) [0x7fdd48aca686]
6: /lib/x86_64-linux-gnu/libparted.so.0(ped_disk_new+0x49) [0x7fdd48a9ff99]
5: parted() [0x406dff]
4: parted(non_interactive_mode+0x8f) [0x40dc5f]
3: parted(main+0x146d) [0x4064ed]
2: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fdd48278ec5]
1: parted() [0x406617]

You found a bug in GNU Parted! Here's what you have to do:

Don't panic! The bug has most likely not affected any of your data.
Help us to fix this bug by doing the following:

Check whether the bug has already been fixed by checking
the last version of GNU Parted that you can find at:

http://ftp.gnu.org/gnu/parted/

Please check this version prior to bug reporting.

If this has not been fixed yet or if you don't know how to check,
please visit the GNU Parted website:

http://www.gnu.org/libparted/labels/gpt.c:994 in function _parse_header() failed.

# fdisk -l /dev/loop0 # let's try this a different way...
WARNING: GPT (GUID Partition Table) detected on '/dev/loop0'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/loop0: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/loop0p1 1 60620799 30310399+ ee GPT

# gparted /libparted/labels/gpt.c:994 in function _parse_header() failed. # clicking No, because, for some reason, that's the only button...and now gparted has crashed/closed itself.

# gdisk -l /dev/loop0 # trying yet another way
GPT fdisk (gdisk) version 0.8.8

Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Disk /dev/loop0: 16777216 sectors, 8.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): EB724872-3B02-4554-A985-59C63E85CF17
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 60620766
Partitions will be aligned on 8-sector boundaries
Total free space is 46859054 sectors (22.3 GiB)

Number Start (sector) End (sector) Size Code Name
1 40 409639 200.0 MiB EF00 EFI System Partition
2 409640 13761718 6.4 GiB AF00 ElCapInstaller

It's worth noting that I can still mount /mnt/loop0p2, and read the files...but I'm not comfortable using this as installation media on a customer's computer, unless it's truly "right". I have very little experience with gdisk, so I'm quite hesitant to mess with it, absent advice from a pro. I haven't had a chance to test whether this boots, but even if it did, I wouldn't want to use it on someone else's computer, so long as it's spitting out all these warnings. Obviously, I'm doing something wrong, but I can't figure out what. Please advise!

Thanks. :-)

Ztcoracat · 03-27-2016, 05:27 PM

I'm not good with Mac pc's sorry but I have a few ideas.

Quote:

That worked nicely, with one catch: the actual installation image is only 8GB or so, but because I used a 32GB flash drive to make it, the disk-dumped image on the hard drive is 32GB.

Maybe get your hands on a 8 GB flash drive or maybe even a 16 GB flash drive.

Quote:

The new file is the right size, when complete, but both methods produce the same result when trying to read the image:

I don't get why it's insisting that the image is 32GB and there's certainly a lot of WARNINGS and I'm not sure what some of them mean. Sorry. Maybe our Guru yancek can help you.
He's been a great help to me with partitioning and things of the such.

I know you said that your working on getting another Mac to experiment on. I would try it with that first and see if you have success. Good luck.

http://www.rodsbooks.com/gdisk/walkthrough.html
http://ubuntuforums.org/showthread.php?t=1956173&page=2

DaneM · 03-30-2016, 02:34 AM

Update: I've verified that this method works. Marking as "Solved". :-)

As before, "#" is a root prompt, whereas "$" is a non-root prompt. You can technically use root for all of these commands, but it's naughty, from a "best practices" standpoint.

First, rip the image from the USB installer, if you haven't already done so. Plug in the USB installer and then to the following:

Code:

# blkid                                    # Note which "/dev/sdXY" entry is the correct USB flash drive.  X=letter Y=partition number.  Ignore Y.
# dd if=/dev/sdX of=ElCapInstaller.img     # Replace "X" with the appropriate letter for your USB device, and exclude any numbers after that.
$ cp ElCapInstaller.img ElCapInstaller2.img     # Do all the work on the copy, so you don't have to re-copy from a slow USB device if you mess up.  Delete the original, later, if desired.

Here's where the real work to shrink the image begins.

Code:

# modprobe loop      # insert the loopback kernel module, just to be sure
# losetup -f         # this returns the name of an unassigned loopback device
/dev/loop0
# losetup /dev/loop0 ElCapInstaller2.iso         # assign ElCapInstaller2.iso to /dev/loop0
# partprobe /dev/loop0       # check /dev/loop0 for partitions, make Linux aware of them
# blkid              # find out the filesystem types and labels of all attached partitions
[...]
/dev/loop0p1: LABEL="EFI" UUID="67E3-17ED" TYPE="vfat" 
/dev/loop0p2: UUID="fe8b1f84-4011-3229-8d2f-d7e6cf2c4b3d" LABEL="Install OS X El Capitan" TYPE="hfsplus"
[...]
# mount /dev/loop0p2 /mnt/tmp         # mount the biggest partition
mount: warning: /mnt/tmp seems to be mounted read-only.
# df -h        # see how much space is being used.  NOTE THAT I'VE ALREADY SHRUNK THIS PARTITION!  If you're doing this from scratch, you will see a larger number, here.
Filesystem      Size  Used Avail Use% Mounted on
[...]
/dev/loop0p2    6.4G  6.0G  474M  93% /mnt/tmp
# umount /mnt/tmp        # unmount the partition
# parted /dev/loop0      # shrink the partition...but remember that I've already done so; this output is just for reference

(parted) print                                                            
Model: Loopback device (loop)
Disk /dev/loop0: 31.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name                  Flags
 1      20.5kB  210MB   210MB   fat32        EFI System Partition  boot
 2      210MB   7046MB  6836MB  hfs+         ElCapInstaller

(parted) resize                                                           
WARNING: you are attempting to use parted to operate on (resize) a file system.
parted's file system manipulation code is not as robust as what you'll find in
dedicated, file-system-specific packages like e2fsprogs.  We recommend
you use parted only to manipulate partition tables, whenever possible.
Support for performing most operations on most types of file systems
will be removed in an upcoming release.
Partition number? 2
Start?  [210MB]?
End?  [7046MB]? 7G         # being conservative.  can't be sure if df uses 1M=1000K=1000B...
[at this point, it does the operation, then returns to the parted prompt]
(parted) quit

# partprobe /dev/loop0      # update partition table in OS
# ls -sh ElCapInstaller2.iso 
29G ElCapInstaller2.iso

Now, shrink the image file. Two methods shown; YMMV. I'm assuming that you will use Method 2, since it doesn't involve making another copy.

Method 1

Code:

# losetup -d /dev/loop0     # unassign loopback device, since I had previously assigned it to the image file that I'm about to change
$ dd if=<imagefile> of=<copyofimagefile> bs=1M count=<howbigitshouldbe>     # shrink image

Method 2

Code:

# losetup -d /dev/loop0     # unassign loopback device, since I had previously assigned it to the image file that I'm about to change
$ truncate -s <howbigitshouldbe> <imagefile>     # shrink image

Here's what seems to have been the missing piece: repairing the GPT table with gdisk. The filenames, below, assume that you used Method 2. If you used Method 1, replace "ElCapInstaller2.iso" with the name of the new file that you created with dd.

Code:

# losetup /dev/loop0 ElCapInstaller2.iso # re-assign the newly-shrunk image to the loop device
# partprobe /dev/loop0 # scan it for new partition information
# gdisk /dev/loop0 # run gdisk on it

GPT fdisk (gdisk) version 0.8.8

Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Command (? for help): v # verify the disk

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Problem: The secondary header's self-pointer indicates that it doesn't reside
at the end of the disk. If you've added a disk to a RAID array, use the 'e'
option on the experts' menu to adjust the secondary header's and partition
table's locations.

Problem: Disk is too small to hold all the data!
(Disk size is 16777216 sectors, needs to be 60620800 sectors.)
The 'e' option on the experts' menu may fix this problem.

Problem: GPT claims the disk is larger than it is! (Claimed last usable
sector is 60620766, but backup header is at
60620799 and disk size is 16777216 sectors.
The 'e' option on the experts' menu will probably fix this problem

Identified 4 problems!

Command (? for help): r # enter "expert" repair mode

[help menu is displayed]

Recovery/transformation command (? for help): e # rebuild partition table
Warning! This will probably do weird things if you've converted an MBR to
GPT form and haven't yet saved the GPT! Proceed? (Y/N): y

Recovery/transformation command (? for help): v # verify the disk, again, for good measure

Problem: The secondary header's self-pointer indicates that it doesn't reside
at the end of the disk. If you've added a disk to a RAID array, use the 'e'
option on the experts' menu to adjust the secondary header's and partition
table's locations.

Problem: Disk is too small to hold all the data!
(Disk size is 16777216 sectors, needs to be 60620800 sectors.)
The 'e' option on the experts' menu may fix this problem.

Problem: GPT claims the disk is larger than it is! (Claimed last usable
sector is 60620766, but backup header is at
60620799 and disk size is 16777216 sectors.
The 'e' option on the experts' menu will probably fix this problem

Identified 3 problems!

Recovery/transformation command (? for help): w # write the changes to disk

Caution! Secondary header was placed beyond the disk's limits! Moving the
header, but other problems may occur!

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/loop0.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

# sudo partprobe /dev/loop0

You can now run parted/gparted on /dev/loop0 to look the partitioning; you can mount /dev/loop0p2 wherever and look at the files, and so on. When you're satisfied that everything is working, unassign /dev/loop0.

Code:

# losetup -d /dev/loop0

To use this image to make a new USB installer, just dd it onto the USB flash drive:

Code:

# dd if=<imagefile> of=</dev/sdX> conv=notrunc,noerror,sync     # dump the image onto the USB drive
# sync                                                          # make sure the readout finishes
# partprobe /dev/sdX                                            # optional: read the partition table; if this produces warnings about the disk not being fully used, that's OK--it's what we want!

You can mount and check the partitions for readability, if you want. Just be sure to unmount ("safely remove") all the drive's partitions it before disconnecting the drive. Note that you might get further warning messages, at some point, about the partition table not extending all the way to the end of the drive. This is OK, for what we're doing, and such messages should be ignored.

Happy imaging!

DaneM · 03-30-2016, 02:35 AM

Thanks for trying to help, Ztcoracat. :-)

Germany_chris · 03-30-2016, 03:02 AM

Any OSX Image that was downloaded before the middle of last month will need to be redownloaded and redone..

Next time you get a Mac in just create a thumb drive installer it take maybe 10 minutes I carry El Cap, Yose, and Mavs in my computer bag.

http://forums.macrumors.com/threads/...pdate.1959487/

DaneM · 03-30-2016, 03:44 AM

Thanks for the heads up, Germany_chris! Looks like my El Capitan installer will work, but my Yosemite installer might not.

Yeah, it's not too difficult to make a new flash drive installer, but I'm hoping to build up enough of a collection that I'll have all the images I could possibly need on my shop computer's hard drive, so I can quickly dd them over to a flash drive, when needed, and then use those drives for other things, as appropriate. I already have just about every Windows version, all the Linuxes I use, Memtest (EFI-compatible), etc...but am pretty shy on Mac OS images. If the above method works, it will allow me (and anyone else) to make a Mac-friendly image collection of minimal size, irrespective of the medium that was used to create the installers. Here's hoping...

Ztcoracat · 03-30-2016, 08:26 PM

Quote:

Originally Posted by DaneM

Thanks for trying to help, Ztcoracat. :-)

You're Welcome.

Let us know how it goes when you get another Mac on your hands.

DaneM · 03-30-2016, 10:24 PM

I just verified the method in post #3 on an Intel Mac (Core 2 Duo). It works! Thank-you to all who put their 2 cents in! I've updated the posts, accordingly, and marked this thread as "Solved".

Ztcoracat · 03-30-2016, 10:35 PM

Quote:

Originally Posted by DaneM

I just verified the method in post #3 on an Intel Mac (Core 2 Duo). It works! Thank-you to all who put their 2 cents in! I've updated the posts, accordingly, and marked this thread as "Solved".

Congrads!

That's great news DaneM. All the time and effort you put into it were well worth it.

You're Welcome.......Enjoy:-

DaneM · 11-23-2016, 03:37 PM

As of version 3.0, the "resize" command was removed from Parted. Use "resizepart", instead.

Additionally, newer Mac OS versions (such as Sierra) are impossible to mount without disabling journaling on their filesystems from within OS X. You may not be able to mount the resulting image's partitions in Linux, even if it works properly. You can still use the shrunken image to create a working USB installer, though.

Ztcoracat · 11-23-2016, 05:03 PM

Quote:

Originally Posted by DaneM

As of version 3.0, the "resize" command was removed from Parted. Use "resizepart", instead.

Additionally, newer Mac OS versions (such as Sierra) are impossible to mount without disabling journaling on their filesystems from within OS X. You may not be able to mount the resulting image's partitions in Linux, even if it works properly. You can still use the shrunken image to create a working USB installer, though.

Thanks for the update.

Happy Thanksgiving.

DaneM · 11-24-2016, 03:33 PM

You, too. :-)