v8.4 LFS systemd - chpt 6.9 Glibc: Error during "make" No module named '_posixsubprocess'

GeekBoy · 06-03-2019, 11:38 AM

So as mentioned in title

Quote:

File "/tools/lib/python3.7/subprocess.py", line 152, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
Traceback (most recent call last):
File "../scripts/gen-as-const.py", line 28, in <module>
import glibcextract
File "/sources/glibc-2.29/scripts/glibcextract.py", line 22, in <module>
import subprocess
File "/tools/lib/python3.7/subprocess.py", line 152, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
make[2]: *** [../Makerules:271: /sources/glibc-2.29/build/tcb-offsets.h] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [../Makerules:271: /sources/glibc-2.29/build/ucontext_i.h] Error 1
Traceback (most recent call last):
File "../scripts/gen-as-const.py", line 28, in <module>
import glibcextract
File "/sources/glibc-2.29/scripts/glibcextract.py", line 22, in <module>
import subprocess
File "/tools/lib/python3.7/subprocess.py", line 152, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
Traceback (most recent call last):
File "../scripts/gen-as-const.py", line 28, in <module>
import glibcextract
File "/sources/glibc-2.29/scripts/glibcextract.py", line 22, in <module>
make[2]: *** [../Makerules:271: /sources/glibc-2.29/build/link-defines.h] Error 1
import subprocess
File "/tools/lib/python3.7/subprocess.py", line 152, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
make[2]: *** [../Makerules:271: /sources/glibc-2.29/build/tlsdesc.h] Error 1
make[2]: Leaving directory '/sources/glibc-2.29/csu'
make[1]: *** [Makefile:258: csu/subdir_lib] Error 2
make[1]: Leaving directory '/sources/glibc-2.29'
make: *** [Makefile:9: all] Error 2

To me it seems it is looking for a Python module names _posixsubprocess. Is that correct?

Shouldn't that of installed during section 5.30?

Thanks!

business_kid · 06-03-2019, 01:30 PM

That's a big page, compiling glibc. I have versions in /usr/lib64/python2.x/site-packages, and /usr/lib64/python3.x/lib-dynload/

Now You will have /usr/lib, not /usr/lib64 and probably only 1 python version in /tools. Check you have it in
/tools/usr/lib
My next suspect would be the clever stuff they did so you can see the libs in /tools because you will hardly compile without the stuff in /tools. That's what it's there for. See how you get on.

GeekBoy · 06-03-2019, 04:39 PM

There is no directory tools/usr/

I do see tools/lib/ and tools/lib64

In tools/lib there is /python3.7 directory with nothing with '_posixsubprocess' or 'posixsubprocess'

In tools/lib64 is also a /python3.7 which looks identical to tools/lib/python3.7 with no such posixsubprocess or _posixsubprocess.

Update:

I rebuilt both Python and Glibc, but this time with one difference. Instead of the usual "make -j4," this time I just did it with make. It was successful.

Any ideas on why they would be the case?

Luridis · 06-04-2019, 12:05 AM

Tar up your /sources and save it in the build host. Wipe the disk, rebuild /tools (Chapter 5) the module paths hard coded into your python build are incorrect. Never try to rebuild a missing or corrupted chapter 5 package (./configure --prefix=/tools) during chapter 6 because the search paths are different.

CH5: PATH=/tools/bin:/bin:/usr/bin (In lfs user .bash_profile)
CH6: PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin (In chapter 6 chroot command.)

This is without bringing the linker changes into it, which complicates it even further. If you've been doing this a long time you can get away with it by building some simple tools like gawk with static linking. But, Python... That's a can of worms. Start over.

PS: Just saw your second message. Do you mean you rebuilt the chapter 5 libc (the one for /tools) during chapter 6? If that even worked I am surprised. It may cause you bizarre problems later...

business_kid · 06-04-2019, 03:48 AM

There's a big difference between 'tools/usr' which searches whatever directory you happen top be in, and '/tools/usr' which searches /.

'make' uses 1 make process; make -j4 uses 4, and is therefore faster.

I have misgivings also about rebuilding python (back in Chapter 5), but if glibc compiled, I'd probably keep going. If you didn't exit the chroot and set up for chapter 5 again to build python you're in a mess. I presume that whatever you did, chapter 6's glibc worked in the chroot? I'd take the advice in the previous post then.

GeekBoy · 06-04-2019, 11:46 AM

I did rebuild Python under chroot, but under the advice of another who mentioned to exclude the sed command because the path changes.

So it did compile. I did the "make check." While it is mentioned some of the tests will probably fail, what is not mentioned is what should happen if those tests fail, or you get a passing test.

Quote:

UNSUPPORTED: elf/tst-audit10
UNSUPPORTED: elf/tst-avx512
XPASS: elf/tst-protected1a
XPASS: elf/tst-protected1b
FAIL: inet/tst-idna_name_classify
FAIL: io/tst-copy_file_range
FAIL: io/tst-copy_file_range-compat
UNSUPPORTED: math/test-double-libmvec-alias-avx512
UNSUPPORTED: math/test-double-libmvec-alias-avx512-main
UNSUPPORTED: math/test-double-libmvec-sincos-avx512
UNSUPPORTED: math/test-float-libmvec-alias-avx512
UNSUPPORTED: math/test-float-libmvec-alias-avx512-main
UNSUPPORTED: math/test-float-libmvec-sincosf-avx512
UNSUPPORTED: misc/tst-pkey
FAIL: misc/tst-ttyname
UNSUPPORTED: nptl/test-cond-printers
UNSUPPORTED: nptl/test-condattr-printers
UNSUPPORTED: nptl/test-mutex-printers
UNSUPPORTED: nptl/test-mutexattr-printers
UNSUPPORTED: nptl/test-rwlock-printers
UNSUPPORTED: nptl/test-rwlockattr-printers
FAIL: nptl/tst-create-detached
FAIL: nss/tst-nss-files-hosts-getent
UNSUPPORTED: nss/tst-nss-test3
UNSUPPORTED: resolv/tst-resolv-ai_idn
UNSUPPORTED: resolv/tst-resolv-ai_idn-latin1
Summary of test results:
6 FAIL
5958 PASS
18 UNSUPPORTED
17 XFAIL
2 XPASS
make[1]: *** [Makefile:405: tests] Error 1
make[1]: Leaving directory '/sources/glibc-2.29'
make: *** [Makefile:9: check] Error 2

The book does not mentioned what to expect at the end. Should it fail like this just because there is an expected failed test, and this is normal, or do I have an issue because of the error reported at the end?

Thanks again!

Luridis · 06-04-2019, 10:55 PM

The book does mention what to expect and most of those are covered by the section immediately following the shown "make check" command in the Chapter 6 Glibc build instructions. The exceptions are these two:

Code:

FAIL: io/tst-copy_file_range
FAIL: io/tst-copy_file_range-compat

Which are likely due to timeouts during the tests.

As to Python: If you built Python in chapter 6 again before building (g)libc, you will need to build it again at the normal step 6.51. This is necessary because if you built it before installing libc, it will be linked to /tools/lib/{glibc.so,libstdc++.so} instead of /lib/{glibc.so,libstdc++.so}. Also note that libstdc++.so is part of the GCC package, so even if you were to rebuild Python right after glibc, you'd still be linking to things in /tools. Finally, since SSL was not installed when you built Python, the _ssl module wouldn't be built either. Lots of things use Python's _ssl module so, it is needed. So... be sure to build Python again at the normal step.

GeekBoy · 06-05-2019, 01:38 PM

I am using a flash drive as the partition location.
So they could be the cause of timeout.
But still does not really answer about what I asked. The book does not mention at all what the end result should be at all. While it does mention about some of the tests failing, and how critical the test is, it still does not address if you should be getting a fail as don't worry about it, or if fail, you have an issue.

Thanks for the response.

business_kid · 06-06-2019, 06:07 AM

Quote:

Originally Posted by GeekBoy

I am using a flash drive as the partition location.

Personally, I think that's crazy. Usb drives are best used as data disk, write and read slowly, and wear very fast. Even if you haven't a spare partition, you can use this dodge: choose a spot with 5G free, and do the following

Code:

dd if=/dev/zero of=/somewhere/LFS bs= 1G count=5
mkfs.ext4 /somewhere/LFS
mount -o loop /somewhere/LFS /lfs

Luridis · 06-06-2019, 11:03 AM

Quote:

Originally Posted by GeekBoy

But still does not really answer about what I asked. The book does not mention at all what the end result should be at all. While it does mention about some of the tests failing, and how critical the test is, it still does not address if you should be getting a fail as don't worry about it, or if fail, you have an issue.

They do address it as best they can. Different tests will fail on different machines. There are many things related to the hardware and/or build host that can cause a test to fail: kernel configuration, shell environment, chroot settings, installed software, lack of microcode updates, host cpu available features, and even BIOS bugs can cause tests to fail. The problem you're having is that you expect certainty, the exact output you can expect from the test suites but, that is impossible in the open source ecosystem.

Generally, if the majority of the tests pass the system will be fine. There really is only reason for concern if hundreds of tests fail. Test suites on these core libraries look for things that most users will not encounter through the normal use of the machine. A lot of the tests look for things the developers call "corner cases". These are mostly obscure problems that only show up in very specific circumstances that will not occur for most uses of the library or tool. I would not be concerned about the failures you encountered... You can move on to the next step.

GeekBoy · 06-07-2019, 07:00 PM

Quote:

Originally Posted by business_kid

Personally, I think that's crazy. Usb drives are best used as data disk, write and read slowly, and wear very fast. Even if you haven't a spare partition, you can use this dodge: choose a spot with 5G free, and do the following

Code:

dd if=/dev/zero of=/somewhere/LFS bs= 1G count=5
mkfs.ext4 /somewhere/LFS
mount -o loop /somewhere/LFS /lfs

I have some spare SATA3 SSD drives laying around I could use with a USB 3.0 adapter instead.

GeekBoy · 06-07-2019, 07:03 PM

Quote:

Originally Posted by Luridis

They do address it as best they can. Different tests will fail on different machines. There are many things related to the hardware and/or build host that can cause a test to fail: kernel configuration, shell environment, chroot settings, installed software, lack of microcode updates, host cpu available features, and even BIOS bugs can cause tests to fail. The problem you're having is that you expect certainty, the exact output you can expect from the test suites but, that is impossible in the open source ecosystem.

Generally, if the majority of the tests pass the system will be fine. There really is only reason for concern if hundreds of tests fail. Test suites on these core libraries look for things that most users will not encounter through the normal use of the machine. A lot of the tests look for things the developers call "corner cases". These are mostly obscure problems that only show up in very specific circumstances that will not occur for most uses of the library or tool. I would not be concerned about the failures you encountered... You can move on to the next step.

Yes, I understand that, but saying it is critical to run the test but then having the "make" fail is the part which could be explained a bit better. Maybe it could be mentioned a little better why it is so critical and possibly explain when you have a problem when make check fails

Luridis · 06-15-2019, 10:03 PM

I'm confused now. You showed the results of "make check" and then mention afterward that the "make" step failed and didn't post the output of that.

GeekBoy · 06-15-2019, 10:37 PM

Quote:

Originally Posted by Luridis

I'm confused now. You showed the results of "make check" and then mention afterward that the "make" step failed and didn't post the output of that.

It's the same thing.

Anyway it is moot now, and I have started over now.
Thanks anyway for taking the time for responding.

Luridis · 06-15-2019, 11:15 PM

LOL, a fail on make is different. The "make" step builds the package, make check builds and runs the tests. If the build fails, that needs to be fixed before running tests.