The Myth of "Once you go Slack, you never go back"

a4z · 01-30-2015, 10:58 AM

Quote:

Originally Posted by Richard Cranium

He implied that you'd examine the resulting VM after running the script, to see what (if anything) got trashed.

does not help if a install script places some stuff not into $DESTDR/usr/share ...but into /usr/share.
you would need to control the whole file system if there is anything there that should not be, makes no sense, just use fakeroot instead of root for the build and problems like this do not exist.
but sure, if a solid secure solution like this is to simple someone will always find a reason why this is not needed.

Richard Cranium · 01-30-2015, 02:46 PM

Quote:

Originally Posted by a4z

does not help if a install script places some stuff not into $DESTDR/usr/share ...but into /usr/share.
you would need to control the whole file system if there is anything there that should not be, makes no sense, just use fakeroot instead of root for the build and problems like this do not exist.
but sure, if a solid secure solution like this is to simple someone will always find a reason why this is not needed.

Does fakeroot cause the build to fail for each individual file that was handled incorrectly? (So to find N errors, you have to re-run the build at least N times.) Does it catch the case where the directory in question is already writable by members of the users group?

If you want to be sure, you'd build your package in a clean environment. You'd take a snapshot of the system prior to creating the package, another after you've created it and then compare the two systems. You could then install and check again (but you should be able to examine the package to figure out what, if anything, was missed).

A system audit tool takes no more than 777 lines of python code (which includes generating all sorts of reports telling you what's the same on both systems, what's only on one, and what's on both but is different somehow).

Richard Cranium · 01-30-2015, 03:14 PM

Quote:

Originally Posted by ruario

Yep, you can use snapshots to roll back to before install and compare but it can be tricky to know where to look. One way is to use find to generate a log of all files (in common install directories /bin, /etc, /lib, /opt, /sbin, /usr, etc.) with time stamps (or md5sums if you are really paranoid). Then do the packaging, repeat the find command and diff the results.

Easier to write an application to do it.

Rename the attached file to have a .py extension. "python platformFileAudit.py --help" provides help.

It detects differences in file mode, length, owner, group, symlink target, and sha256 of contents. You can tag files as "special", which means that the program saves a copy of the file contents into the snapshot (which is only useful sense for files with text content). You can tag files/directories as "ignored", which means the scanner won't look at them (files) or their contents (directories). You can tag files (only) as "specific", which means they will be scanned no matter what. Finally, you can put all that into a configuration file that you can reference so you don't have to remember all that stuff after you figured it out the first time.

An example configuration file's contents would look like...

Code:

[standard]
Scan = /etc
       /var
       /lib
       /lib64
       /bin
       /sbin
       /usr
       /boot
Ignore = /var/tmp
         /var/run
         /var/state
         /var/log
	 /var/spool
	 /var/cache
         /lib/udev/devices
	 /usr/local
         /var/lib/libvirt
         /var/lib/sbopkg
         /var/slapt-get
	 /boot/initrd-tree
	 /boot/initrd-tree.old
Special = /etc/fstab
          /etc/passwd
          /etc/group
Specific =

Of course, you have to run it with root permissions so you have access to all the files/directories. It's python, so if you tell it to scan iso images in your filesystem, be prepared to wait.

(When you read the code/comments, a python "pickle" is a serialized data object.)

55020 · 01-30-2015, 03:45 PM

As an alternative approach to R.Cranium's python thingie, or aide/slacktrack/checkinstall: when 3.18 first came out I experimented with using overlayfs to catch any rogue changes in the 'upper' filesystem (and even better, as a lightning quick way of uninstalling a whole dependency stack of packages). But it turned out to be difficult to unmount an 'upper' root filesystem, let alone examine it to find out what had changed. Maybe that can be made to work, by paying close attention to OpenWRT. I'm reluctant to explore btrfs snapshots, because, well, that would impose btrfs on the end user...

Edit: Two thoughts:
(1) I should have said thanks to Matteo, because aide is new to me.
(2) Maybe mount --bind all the standard Slackware root directories into an empty working directory, then mount an empty upper overlayfs on top of it, then chroot into that to do your building... that would be so cool if it works...?

Edit again: No, I tried it, and mounts on the lower filesystem are not visible on the merged filesystem. Should have expected that really

Edit Three: Well duh. Why not just mount --bind / ... and it works! :O

Edit Four: Here is the Safe Building As Root HOWTO, v0.1 (Requires 3.18 Kernel)

Code:

mkdir /tmp/root /tmp/changes /tmp/workdir /tmp/chroot
mount --bind / /tmp/root
mount -t overlay overlay -olowerdir=/tmp/root,upperdir=/tmp/changes,workdir=/tmp/workdir /tmp/chroot
mount --bind /proc /tmp/chroot/proc  (and any of the other virtual crap that you fancy)
chroot /tmp/chroot

Now run the SlackBuild that you don't trust, AS ROOT!!!

any changes will be visible in /tmp/changes, in realtime, and will *not* happen in the real /. And you get to keep the built package too, in /tmp/changes/tmp

a4z · 01-31-2015, 01:47 AM

Quote:

Originally Posted by Richard Cranium

Does fakeroot cause the build to fail for each individual file that was handled incorrectly? (So to find N errors, you have to re-run the build at least N times.) Does it catch the case where the directory in question is already writable by members of the users group?

If you want to be sure, you'd build your package in a clean environment. You'd take a snapshot of the system prior to creating the package, another after you've created it and then compare the two systems. You could then install and check again (but you should be able to examine the package to figure out what, if anything, was missed).

A system audit tool takes no more than 777 lines of python code (which includes generating all sorts of reports telling you what's the same on both systems, what's only on one, and what's on both but is different somehow).

fakeroot avoids putting files on the wrong location and ending up with incomplete packages, exact the problem that was mentioned earlier in that thread.
it gives you an error which file can not be written, you should use it already during writing the script,
you catch these kind of problems on the most early time possible what is exactly what you want when you develop anything. (ok, MS and Apple can ship and wait that problem are detected by users, but we should not take them as example)
as additions insurance you should use it always ween building a package.
it does not hinder you to use additional tools like lintpkg or other tools
not using a tool like this, lets say its just not very professional and gives a clear message that you have not understood the problem, the tool, or do just not care about what makes you differently not to a better developer.

a4z · 01-31-2015, 02:07 AM

Quote:

Originally Posted by 55020

..
Now run the SlackBuild that you don't trust, AS ROOT!!!

any changes will be visible in /tmp/changes, in realtime, and will *not* happen in the real /. And you get to keep the built package too, in /tmp/changes/tmp

this is a nice work around :-)
does not report the problem of a make DESTDIR, so if someone runs your build where your workaround worked around such a problem he/she will have the problem and you know nothing about the problem, just as now, and you will end with an incomplete package,
but I am sure you can put a workaround on to of the workaround to work around that problem, if MS can do it and work like that, we can also do it

but the bind mount overlay is nice and interesting.

and as said, trust is mostly not a problem, errors are, you do not catch them, you hide them, ignore them and leaf it to others to run into them.

other question,
or can you simply remount or clean up?
otherwise there might be the problem that when you do more packages you have to clean up in between

ponce · 01-31-2015, 03:42 AM

thanks a lot to Dave (55020) for his method, that looks real fun to play with.

following his advice I installed a kernel 3.18.5 on my vm and created two small script, overlay

Code:

mkdir -p /tmp/{root,changes,workdir,chroot/proc,chroot/dev,chroot/sys}
mount --bind / /tmp/root
mount -t overlay overlay -olowerdir=/tmp/root,upperdir=/tmp/changes,workdir=/tmp/workdir /tmp/chroot
mount -t proc proc /tmp/chroot/proc
mount --bind /sys /tmp/chroot/sys
mount --bind /dev /tmp/chroot/dev
chroot /tmp/chroot

and overlay_off

Code:

umount /tmp/chroot/dev /tmp/chroot/sys /tmp/chroot/proc /tmp/chroot /tmp/root
rm -fR /tmp/root /tmp/changes /tmp/workdir /tmp/chroot

then, to apply them to a real-life scenario, I tried using them when building zarfy.

- first I launched

Code:

overlay

and found myself in the chroot;

- for my own commodity I sourced some useful stuff

Code:

. /etc/profile

- then I proceded to build zarfy in the standard way: I already got the SlackBuild /root/zarfy, so I downloaded the sources;

- at the end of the build process I looked from outside the chroot (in another shell) at what has changed in the overlayed filesystem

Code:

# ls -la /tmp/changes/    
total 52
drwxr-xr-x  5 root root  4096 Jan 31 09:26 ./
drwxrwxrwt 13 root root 32768 Jan 31 09:26 ../
drwx--x---  3 root root  4096 Jan 31 09:25 root/
drwxrwxrwt  3 root root  4096 Jan 31 09:28 tmp/
drwxr-xr-x  4 root root  4096 Jan 30 19:43 usr/

- as I built stuff in /root, it was logical that I had changes in /root (where the sources were) and in /tmp (where /tmp/SBo and the final packages are), but there shouldn't be anything is /tmp/changes/usr if the make install have respected DESTDIR: so I inspected it

Code:

# ls -laR /tmp/changes/usr
/tmp/changes/usr:
total 16
drwxr-xr-x 4 root root 4096 Jan 30 19:43 ./
drwxr-xr-x 5 root root 4096 Jan 31 09:26 ../
drwxr-xr-x 3 root root 4096 Jan 30 18:13 man/
drwxr-xr-x 3 root root 4096 Jan 31 09:28 share/

/tmp/changes/usr/man:
total 12
drwxr-xr-x 3 root root 4096 Jan 30 18:13 ./
drwxr-xr-x 4 root root 4096 Jan 30 19:43 ../
drwxr-xr-x 2 root root 4096 Jan 31 09:28 man1/

/tmp/changes/usr/man/man1:
total 12
drwxr-xr-x 2 root root 4096 Jan 31 09:28 ./
drwxr-xr-x 3 root root 4096 Jan 30 18:13 ../
-rw-r--r-- 1 root root  972 Jan 31 09:28 zarfy.1.gz

/tmp/changes/usr/share:
total 12
drwxr-xr-x 3 root root 4096 Jan 31 09:28 ./
drwxr-xr-x 4 root root 4096 Jan 30 19:43 ../
drwxr-xr-x 2 root root 4096 Jan 31 09:28 zarfy/

/tmp/changes/usr/share/zarfy:
total 116
drwxr-xr-x 2 root root  4096 Jan 31 09:28 ./
drwxr-xr-x 3 root root  4096 Jan 31 09:28 ../
-rw-r--r-- 1 root root  1920 Jan 31 09:28 dvi.png
-rw-r--r-- 1 root root  7719 Jan 31 09:28 lcd.png
-rw-r--r-- 1 root root  1289 Jan 31 09:28 monitor.png
-rw-r--r-- 1 root root  1177 Jan 31 09:28 monitor_d.png
-rw-r--r-- 1 root root  1322 Jan 31 09:28 monitor_s.png
-rw-r--r-- 1 root root  7606 Jan 31 09:28 tv.png
-rw-r--r-- 1 root root  1897 Jan 31 09:28 vga.png
-rw-r--r-- 1 root root 66223 Jan 31 09:28 zarfy.glade
-rw-r--r-- 1 root root   573 Jan 31 09:28 zarfy.png

- assessed that zarfy has to be fixed I exited the chroot shell: from that

Code:

exit

- I cleaned the stuff with

Code:

overlay_off

and I was ready to start testing again.

a4z · 01-31-2015, 04:36 AM

good example, thanks,
so if you control the result you can detect problems,

but it seems to be a lot of effort, just for a because I can as root,
I prefer fakeroot and get for zarfy

Code:

make[2]: Entering directory `/tmp/SBo/zarfy-0.1.0'
make[2]: Nothing to be done for `install-exec-am'.
/bin/sh /tmp/SBo/zarfy-0.1.0/install-sh -d /usr/share/zarfy; \
  for fname in data/*; do \
    if test -f $fname; then \
      /usr/bin/ginstall -c -m 644 $fname /usr/share/zarfy; \
    fi \
  done;
mkdir: cannot create directory '/usr/share/zarfy': Permission denied
mkdir: cannot create directory '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied
make[2]: *** [install-data-local] Error 1
make[2]: Leaving directory `/tmp/SBo/zarfy-0.1.0'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/tmp/SBo/zarfy-0.1.0'
make: *** [install-recursive] Error 1

this seems to follow more the kiss way, doesn't it?

however, the overlay fs stuff is very nice, I am happy that I have learned something new

55020 · 01-31-2015, 05:56 AM

Three observations:

(1) I think the directory /tmp/root and mount --bind are pointless, you can just use "-olowerdir=/,upperdir=...", sorry about the derp

so (also with the virtual filesystems in the LFS style if you prefer) your overlay script could be e.g.

Code:

mkdir -p /tmp/{changes,workdir,chroot/proc,chroot/dev/pts,chroot/sys}
mount -t overlay overlay -olowerdir=/,upperdir=/tmp/changes,workdir=/tmp/workdir /tmp/chroot
mount -t devpts devpts /tmp/chroot/dev/pts
mount -t tmpfs shm /tmp/chroot/dev/shm
mount -t proc proc /tmp/chroot/proc
mount -t sysfs sysfs /tmp/chroot/sys
chroot /tmp/chroot

(2) With chroot environments you can run multiple builds at the same time, each with its own chroot. Say goodbye to "you can only run one instance of sbopkg" misery!

(3) fakeroot doesn't always tell you what failed. It just says e.g. "/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Permission denied". And then you have to guess what the problem was (probably DESTDIR, but not always -- see audio/gtkwave), and then try again (repeat...) *BUT* clearly fakeroot is a useful test, and the best way of catching errors is with several different tests. So it's all good. Edit: you can do both -- you can enter the chroot environment as an ordinary user, and run in there with fakeroot.

ivandi · 01-31-2015, 07:52 AM

FWIW here is my script to create a fake root build environment:

Code:

#!/bin/bash

BUILDDIR=/var/lib/build
DIRS="bin lib etc lib64 sbin usr"

mkdir -p $BUILDDIR
mkdir -p $BUILDDIR/{dev,sys,proc,run,root,tmp,var/tmp}

mount /dev $BUILDDIR/dev -o bind
mount -t devpts devpts $BUILDDIR/dev/pts -o gid=5,mode=620
mount -t proc proc $BUILDDIR/proc
mount -t sysfs sysfs $BUILDDIR/sys
mount -t tmpfs tmpfs $BUILDDIR/run

for d in $DIRS ; do
    mkdir -p $BUILDDIR/$d
    mount /$d $BUILDDIR/$d -o bind
    mount /$d $BUILDDIR/$d -o remount,ro,bind
done

chroot $BUILDDIR /usr/bin/env -i HOME=/root TERM=$TERM /bin/bash --login

for d in $DIRS ; do
    umount $BUILDDIR/$d
done

umount $BUILDDIR/run
umount $BUILDDIR/sys
umount $BUILDDIR/proc
umount $BUILDDIR/dev/pts
umount $BUILDDIR/dev

zarfy.SlackBuild happily fails:

Code:

mkdir: cannot create directory '/usr/share/zarfy': Read-only file system
mkdir: cannot create directory '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
/usr/bin/ginstall: cannot create regular file '/usr/share/zarfy': Read-only file system
make[2]: *** [install-data-local] Error 1
make[2]: Leaving directory `/tmp/SBo/zarfy-0.1.0'
make[1]: *** [install-am] Error 2
make[1]: Leaving directory `/tmp/SBo/zarfy-0.1.0'
make: *** [install-recursive] Error 1

Cheers

ivandi · 01-31-2015, 09:13 AM

Quote:

Originally Posted by 55020

see audio/gtkwave

Code:

make[4]: Entering directory `/tmp/SBo/gtkwave-3.3.51/share/applications'
/usr/bin/update-desktop-database
The databases in [/usr/local/share/applications, /usr/share/applications] could not be updated.
make[4]: *** [install-data-hook] Error 1
make[4]: Leaving directory `/tmp/SBo/gtkwave-3.3.51/share/applications'
make[3]: *** [install-data-am] Error 2
make[3]: Leaving directory `/tmp/SBo/gtkwave-3.3.51/share/applications'
make[2]: *** [install-am] Error 2
make[2]: Leaving directory `/tmp/SBo/gtkwave-3.3.51/share/applications'
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory `/tmp/SBo/gtkwave-3.3.51/share'
make: *** [install-recursive] Error 1

Cheers

a4z · 01-31-2015, 09:56 AM

Quote:

Originally Posted by ivandi

FWIW here is my script to create a fake root build environment:

very interesting, thanks,
how exactly does it work? are only the directories in $DIRS write protceted and a build script can only create /tmp/whatever /opt/whatever ...
and a chroot question, the -i HOME=/root option, there is no /root in $BUILDDIR, will it be created, is it the existing one?

ivandi · 01-31-2015, 10:05 AM

Quote:

Originally Posted by a4z

how exactly does it work? are only the directories in $DIRS write protceted and a build script can only create /tmp/whatever /opt/whatever ...

Exactly. We cannot directly bind mount readonly a directory, but we can remount it readonly once it is mounted.

Quote:

Originally Posted by a4z

and a chroot question, the -i HOME=/root option, there is no /root in $BUILDDIR, will it be created, is it the existing one?

Code:

mkdir -p $BUILDDIR/{dev,sys,proc,run,root,tmp,var/tmp}

Cheers

Richard Cranium · 01-31-2015, 02:03 PM

While running my scan tool, I noticed that /etc/profile.d/gtk+.sh and /etc/profile.d/gtk+.csh have an unusual userid (3356) instead of root (Slackware64 14.1).

Anyways...

After running the zarfy slackbuild, another scan, and generating a report, the relevant data was...

Code:

********************************************************************************
* Only on system B
*   A: CleanScan.pickle
*   B: zarfy.pickle
********************************************************************************
User Group Mode                 Type  Size Contents? Name                           Target Hash                                                            
root root  0100644 (-rw-r--r--) File   972 False     /usr/man/man1/zarfy.1.gz       None   214d640dddc5caabbb532f9ac39af2946895e9a46bfc9c0a4de0082d747adfdb
root root  0040755 (drwxr-xr-x) Dir      0 False     /usr/share/zarfy               None   None                                                            
root root  0100644 (-rw-r--r--) File  1920 False     /usr/share/zarfy/dvi.png       None   b3b6d878426f979572850f75f3ab712ab01676dedb3db65626712c60e3ada164
root root  0100644 (-rw-r--r--) File  7719 False     /usr/share/zarfy/lcd.png       None   9fa11cc601bab2b2246d872e50357626c545f90200fb23ff089b9fd078bd9287
root root  0100644 (-rw-r--r--) File  1289 False     /usr/share/zarfy/monitor.png   None   a09ff2ebb156ae3c343bca190d3db47bed4827f7b790c7462b2fd1bc3175610d
root root  0100644 (-rw-r--r--) File  1177 False     /usr/share/zarfy/monitor_d.png None   16aaba4a2a4132650c16c36c28de36e0bc85395482881bb4a0fd6fb71e2581b6
root root  0100644 (-rw-r--r--) File  1322 False     /usr/share/zarfy/monitor_s.png None   6fa0f7fcf7704343f59894d93bae3728eec4c0250bf35334b19b4ef37ca91dc7
root root  0100644 (-rw-r--r--) File  7606 False     /usr/share/zarfy/tv.png        None   1d015d27c6c8ac610d57df2071862912e0364fcd44100642f50f6c748f6108dc
root root  0100644 (-rw-r--r--) File  1897 False     /usr/share/zarfy/vga.png       None   f033db53dddfaef90bb3e2d41964c46a462416868199b75572c7f3372b5aaaf3
root root  0100644 (-rw-r--r--) File 66223 False     /usr/share/zarfy/zarfy.glade   None   f7c92f86e84fbf8c69aae635223d32aaf383d8bfbe190a562cfcfb022f415164
root root  0100644 (-rw-r--r--) File   573 False     /usr/share/zarfy/zarfy.png     None   cf0b340f2b6985991da7c03e949974685c4884785a928305c9f5336bca49084a

All things considered, I'd recommend using 55020's method instead if your kernel supports it.

gauchao · 02-17-2015, 09:51 AM

I have finished installing VirtualBox under my Slack64 14.1 machine. Now I can run.... Slackware 64 CURRENT! Slacking forever...