A problem about gcc 5.5.0 after upgrade. It does not compile TensorFlow.

sungjin · 02-09-2018, 12:56 AM

Hi. Recently, we got a security update for Spectre. As a side-effect, we also got an updated gcc (5.3.0 -> 5.5.0). So far so good. Thank you very much.

Let me briefly introduce the situation. I'm a graduate student in CS dept. I use TensorFlow, a machine learning library developed by google, to do my experiments at my lab. Of course it runs on a slackware machine.

I usually compile the TensorFlow source code for optimization. However, it can not be compiled with updated gcc (5.5.0), like this;

Code:

INFO: From Compiling tensorflow/contrib/resampler/kernels/resampler_ops_gpu.cu.cc:
/usr/lib64/gcc/x86_64-slackware-linux/5.5.0/include/avx512fintrin.h(9220): error: argument of type "const void *" is incompatible with parameter of type "const float *"

/usr/lib64/gcc/x86_64-slackware-linux/5.5.0/include/avx512fintrin.h(9231): error: argument of type "const void *" is incompatible with parameter of type "const float *"

/usr/lib64/gcc/x86_64-slackware-linux/5.5.0/include/avx512fintrin.h(9244): error: argument of type "const void *" is incompatible with parameter of type "const double *"

/usr/lib64/gcc/x86_64-slackware-linux/5.5.0/include/avx512fintrin.h(9255): error: argument of type "const void *" is incompatible with parameter of type "const double *"

So I googled a little bit, and found the following issue:

https://github.com/tensorflow/tensorflow/issues/10220

and,

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=76731

which says that,

Quote:

All of the scatter/gather intrinsics in avx512intrin.h use int/float/double pointers, which is incorrect.

So I think that maybe we need a patch for gcc, if I tracked down the matter correctly. I'll glad to hear about this from community. Thank you.

Best regards,
sungjin.

Appendix.

How to reproduce.

0. (optionally) compile kernel to enable numa support, and install NVIDIA driver, CUDA, cuDNN.

1. download TensorFlow source code.

git clone https://github.com/tensorflow/tensorflow
cd tensorflow
git checkout r1.5

2. configure TensorFlow with GPU support.

./configure

3. compile it (you need bazel).

ponce · 02-09-2018, 01:08 AM

IMHO, the more reasonable explanation of the issue is this

https://github.com/tensorflow/tensor...ment-316887593

it's the GPU/CUDA code that doesn't support the new compiler so, if you want to build that, your only option is downgrade the compiler, until Nvidia releases a new CUDA sdk.

sungjin · 02-09-2018, 01:27 AM

Thank you for your kind remark.

Well, in fact, I'm not good at compilers. I'll investigate things as you recommended.

Thank you for the advice.

Best regards,
sungjin.

sungjin · 02-09-2018, 01:49 AM

I saw a remark about the gcc, from https://github.com/tensorflow/tensorflow/issues/10220, as mentioned already, that is;

Quote:

I think the problem here is that gcc-5.5 shipped with avx512*intrin.h headers that switched to using void* and const void* (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=76731) but without switching the builtins to do the same. This is why 5.4 works but 5.5 breaks.

So I think there are mismatches in the declaration, namely in the header files, related to avx instructions.

And what's suggests as a workaround is this;

Code:

for f in avx512fintrin.h avx512pfintrin.h avx512vlintrin.h; do
   curl -H "User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" -o $f "https://gcc.gnu.org/viewcvs/gcc/branches/gcc-5-branch/gcc/config/i386/${f}?view=co&revision=245536&content-type=text%2Fplain&pathrev=245536"
done && mv avx512*intrin.h  /usr/lib/gcc/x86_64-linux-gnu/5/include/

but I think this is very dangerous.

I agree that there's a confusion between gcc and CUDA, but yet I do not know which made this mistake. I'll study more.

Thank you for your help.

Best regards,
sungjin.

marrowsuck · 02-09-2018, 09:31 AM

I have a seperate gcc-5.4 installed under /opt to compile tensorflow with GCC_HOST_COMPILER_PATH=/opt/gcc54/bin/gcc-5.4.0
See the attached SlackBuild if you want to compile a package.

sungjin · 02-09-2018, 07:33 PM

Thank you for your remark.

In fact, I had a similar idea, but I'm considering to install gcc in my home directory.

I'll try when I go to my lab.

Thank you for the advice.

sungjin · 02-17-2018, 10:14 PM

Sorry for the late report.

I was able to compile TensorFlow with gcc 5.4 as suggested.

A subtle difference is that I installed gcc under my home directory, not opt, because in my opinion this local gcc version (5.4) will not be used frequently.

Thank you for your help.