[SOLVED] Eek ! NVidia 410.xx and Kernel 4.4.169+ and get_user_pages

kjhambrick · 01-13-2019, 02:36 PM

All --

I built 4.4.170 on my Slackware 14.2 box without issue.

I was running 4.4.167 before I booted 4.4.170

I tried running NVIDIA-Linux-x86_64-410.93.run and it failed to compile because the definition of get_user_pages( ) has changed between Linux 4.4.167 and 4.4.170.

I googled the error without any luck.

This is the definition of get_user_pages( ) in Linux 4.4.167:

Code:

# less /usr/src/linux-4.4.167.kjh/include/linux/mm.h

<<snip>> ( line 1200 )
long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
                    unsigned long start, unsigned long nr_pages,
                    int write, int force, struct page **pages,
                    struct vm_area_struct **vmas);

And this is the definition of get_user_pages( ) in Linux 4.4.170:

Code:

# less /usr/src/linux-4.4.170.kjh/include/linux/mm.h
<<snip>> ( line 1200 )
long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
                    unsigned long start, unsigned long nr_pages,
                    unsigned int gup_flags, struct page **pages,
                    struct vm_area_struct **vmas);

These are the lines from /var/log/nvidia-installer.log

Code:

   In file included from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-linux.h:21:0,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:15:
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c: In function 'os_lock_user_pages'
:
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:120:48: warning: passing argument
 6 of 'get_user_pages' makes pointer from integer without a cast [-Wint-conversion]
                                page_count, write, force, user_pages, NULL);
                                                   ^
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-mm.h:44:70: note: in definition of ma
cro 'NV_GET_USER_PAGES'
            get_user_pages(current, current->mm, start, nr_pages, write, force, pages, vmas)
                                                                         ^
   In file included from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-pgprot.h:17:0,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-linux.h:20,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:15:
   /usr/src/linux-4.4.170.kjh/include/linux/mm.h:1200:6: note: expected 'struct page **' but argument is of type 'NvBool {aka unsigned char}'
    long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
         ^
   In file included from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-linux.h:21:0,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:15:
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:120:55: warning: passing argument 7 of 'get_user_pages' from incompatible pointer type [-Wincompatible-pointer-types]
                                page_count, write, force, user_pages, NULL);
                                                          ^
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-mm.h:44:77: note: in definition of macro 'NV_GET_USER_PAGES'
            get_user_pages(current, current->mm, start, nr_pages, write, force, pages, vmas)
                                                                                ^
   In file included from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-pgprot.h:17:0,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-linux.h:20,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:15:
   /usr/src/linux-4.4.170.kjh/include/linux/mm.h:1200:6: note: expected 'struct vm_area_struct **' but argument is of type 'struct page **'
    long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
         ^
   In file included from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-linux.h:21:0,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:15:
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-mm.h:44:9: error: too many arguments to function 'get_user_pages'
            get_user_pages(current, current->mm, start, nr_pages, write, force, pages, vmas)
            ^
            ^
   /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:119:11: note: in expansion of macro 'NV_GET_USER_PAGES'
        ret = NV_GET_USER_PAGES((unsigned long)address,
              ^
   In file included from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-pgprot.h:17:0,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/common/inc/nv-linux.h:20,
                    from /tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.c:15:
   /usr/src/linux-4.4.170.kjh/include/linux/mm.h:1200:6: note: declared here
    long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
         ^
   /usr/src/linux-4.4.170.kjh/scripts/Makefile.build:277: recipe for target '/tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.o' failed
   make[3]: *** [/tmp/selfgz3548/NVIDIA-Linux-x86_64-410.93/kernel/nvidia/os-mlock.o] Error 1

I tried against 4.4.169 and got the same error.

I didn't build 4.4.168 so I reverted to Linux 4.4.167 and NVidia 410.93 and the NVidia Blob built without issues.

Note: I and still using Pat's kernel-headers-4.4.157-x86-1

This is a Q I asked a while back ... opinions on upgrading the Slackware kernel-headers with kernel headers for the running kernel and didn't get an answer.

Should I upgrade the Kernel Headers ?

If so, does anyone have a Kernel Headers SlackBuild ?

Ideas ?

Thanks !

-- kjh

abga · 01-15-2019, 02:09 AM

The kernel headers that are released in the kernel-headers-x.x.x-whatever package are the headers used for building the toolchain (glibc & co) and reside in /usr/include/
Now if you get a kernel-headers- package as an update in Slackware, that means you can use it and upgrade them and you won't break anything, it's been checked "upstairs" and maybe glibc&stuff already recompiled to match them.
Overwriting them on your own with the headers from your running kernel is not advised, unless you want to reinstall / recompile parts of your Linux system.
Just for reference, this Gentoo article is the shortest/simplest I could find ATM:
https://wiki.gentoo.org/wiki/Linux-headers

bassmadrigal · 01-15-2019, 11:44 AM

Have a look at my post here where I linked to several posts of Pat's about kernel headers.

Simply put, he always upgrades the kernel headers and never runs into any issues with it. Back in the 2.x days, there were possibilities of breakage, but it seems to be highly unlikely now. I only upgrade headers when I use a pre-built package that has them available (like from Pat or dusk), otherwise I just run my own kernels and leave the existing headers in place.

As for a kernel headers SlackBuild, if there isn't one in the source tree, you can use an excerpt from 55020's dusk.SlackBuild to easily create your own SlackBuild for it (it will need minor tweaking to set things for your environment).

Code:

#------------------------------------------------------------------------------
PRGNAM=kernel-headers
#------------------------------------------------------------------------------

PKG=$TMP/package-$PRGNAM
rm -rf $PKG
mkdir -p $PKG
cd $CWD/linux-stable

$NICE make headers_install INSTALL_HDR_PATH=$PKG/usr
find $PKG/usr -type f -name '.*' -delete

mkdir -p $PKG/install
cat $CWD/$SLKARCH/$PRGNAM-slack-desc > $PKG/install/slack-desc

cd $PKG
$NICE $CWD/makepkg -l y -c y $OUTPUT/$PRGNAM-$SLKVER-$SLKARCH-$SLKBUILD$SLKTAG.${PKGTYPE:-txz}

kjhambrick · 01-15-2019, 03:23 PM

Thanks for the replies all'Y'all ...

I am not ignoring you

I need to do some research before I stick my foot in my mouth (

or before I poke my eye out with a sharp retort

)

Here is where I am headed ...

I KNOW that the Kernel Devs have changed the argument list for get_user_pages( ) after 4.4.157 and before 4.4.169

Rhetorical Q: What good does it do to update /usr/include/mm.h ( the Shared Memory library API header ) if get_user_pages( ) is a member of some C-Library that is still based on the old argument list.

On my plate is to find out where get_user_pages() and the other functions in mm.h, etc are made available to the Linker.

If they're linked via glibc then I am in a pickle ... I had better stop playing with Kernels ...

Or am I barking up the wrong tree ???

I can't find mm.h anywhere under /usr/include/ on my Slackware 14.2 System with the Official 4.4.157 Kernel Headers ...

Anyhow, I am up to 'there' with Projects at work but I sure am itching to find out.

-- kjh

abga · 01-16-2019, 12:03 AM

@kjhambrick

I just answered your question(s):

Quote:

Note: I and still using Pat's kernel-headers-4.4.157-x86-1

This is a Q I asked a while back ... opinions on upgrading the Slackware kernel-headers with kernel headers for the running kernel and didn't get an answer.

Should I upgrade the Kernel Headers ?

If so, does anyone have a Kernel Headers SlackBuild ?

And even if bassmadrigal & Patrick (the links bassmadrigal provided) state that you won't break anything if you upgrade the kernel headers, I'd still advise to not do that, because theoretically there's a possibility to mess things up. Again, I take bassmadrigal's & Patrick's statements and experience as true&granted, as I trust them and also because I never attempted such a thing myself, knowing that I might break things.
Here is the official kernel doc I couldn't find and used that Gentoo wiki instead:
https://www.kernel.org/doc/Documenta...rs_install.txt

There's a confusion about the term "kernel headers" because some distributions (Debian? - Raspbian for sure) strip the kernel source tree of source files and only leave the headers, then pack the resulting tree in a package called kernel headers for distribution purposes. This is useful for compiling kernel modules, where usually you don't need the kernel source files but only the headers. The resulting kernel headers package is smaller (around 100MB or less) and easier to download compared to the whole kernel source tree. Slackware does not provide such a package / solution.

You can do whatever you like in your /usr/src/linux-x.x.x/include/ folders, take a big spoon and mix headers & co up, the worst case scenarios would be a module/kernel build failure or a boot failure if the module/kernel was successfully built with mixed headers but broken.
Since you only spotted the difference in mm.h , I'd take a minimally invasive approach and only change that header file first, inspect it first to make sure it's not heavily modified in the newer release - include section - some new/different headers.

Gordie · 01-16-2019, 12:27 AM

There is a SlackBuild for kernel headers in the directory tree of current. Maybe it can be of use?

https://mirrors.slackware.com/slackw...ers.SlackBuild

abga · 01-16-2019, 12:52 AM

Quote:

Originally Posted by Gordie

There is a SlackBuild for kernel headers in the directory tree of current. Maybe it can be of use?

https://mirrors.slackware.com/slackw...ers.SlackBuild

Those are the kernel headers generated by the kernel make headers_install target and residing in /usr/include, the ones I advised not to mess up with:

Code:

  make headers_install ARCH=${HEADERS_ARCH} INSTALL_HDR_PATH=$PKG/usr
  cd $PKG/usr/include

Again:
https://www.kernel.org/doc/Documenta...rs_install.txt

Ilgar · 01-17-2019, 11:18 AM

I doubt that changing the headers would solve the issue. The driver code makes calls using an old API, so it should not link against the running (new) kernel's binaries. Even if you could force it to, I would expect segfault at first call.

I don't know how practical it is to patch the Nvidia code. If you can do that, a quick Google search reveals how old code can be ported to the new format (you can also follow it through the patches applied to gup.c).

kjhambrick · 01-19-2019, 06:57 AM

Thanks All'Y'All for the replies.

P.P.P.S. I should also add that this is Slackware 14.2 + Multilib which may have something to do with my issue ( with all the recent deprecation of 32-bit computing ( long stretch )) ???

I installed 55020's 4.4.171 Kernel Stuff:

Code:

   kernel-generic-4.4.171-x86_64-1_dusk.txz
   kernel-modules-4.4.171-x86_64-1_dusk.txz
   kernel-source-4.4.171-noarch-1_dusk.txz

And I also ran `upgradepkg kernel-headers-4.4.171-x86_64-1_dusk.txz` to replace Pat's Official kernel-headers-4.4.157 with matching headers for the DUSK 4.4.171 Kernel.

I was hoping this would fix my NVIDIA .compile error, so I built an initrd, cold-booted to runlevel 3 and ...

No joy ... same errors building the NVIDIA-Linux-x86_64-410.93.run blob against the DUSK 4.4.171 as I saw in my own 4.4.170 and 4.4.171 Kernels.

I reverted to kernel-headers-4.4.157 and booted 4.4.167.kjh and reran NVIDIA-Linux-x86_64-410.93.run -- that worked.

I am leaning toward Ilgar's suggestion -- to try patching nvidia/NVIDIA-Linux-x86_64-410.93/kernel/conftest.sh so that the correct #defines are set.

If I am reading /usr/src/include/linux/mm.h and nvidia/NVIDIA-Linux-x86_64-410.93/kernel/conftest.sh correctly, I need:

Code:

   #define NV_GET_USER_PAGES_HAS_WRITE_AND_FORCE_ARGS
   #define NV_GET_USER_PAGES_HAS_TASK_STRUCT

What really confuses me is when I look at mm.h in 4.4.157 ( Pat's Kernel ), 4.4.167 ( the last one where the NVIDIA Blob works ) and 4.4.171, get_user_pages( ) is the same in all three ...

This is the prototype of get_user_pages() from /usr/src/include/linux/mm.h that leads me to believe that the #defines above are what I need:

Code:

long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
                    unsigned long start, unsigned long nr_pages,
                    int write, int force, struct page **pages,
                    struct vm_area_struct **vmas);

Step 1 will be a patch to simply echo the #defines to stderr so I can compare 4.4.167 to 4.4.171 to verify that those are the proper #defines by looking at /var/log/nvidia-installer.log ...

Anyhow, still looking but I am also still overwhelmed with stuff at work which has to go first ...

Thanks again.

-- kjh

P.S. Thanks ( abga, bassmadrigal, Gordie ) for the pointers to the kernel-headers.SlackBuild(s) ...

I went with a modified version of 55020's SlackBuild, adapted to my kernel-compiler wrapper script.

I'll stop asking that Q now

P.P.S. I also compared 55020 kernel .config to 'my' .config for 4.4.171.kjh -- they're identical except for the Kernel Version Line.

kjhambrick · 01-19-2019, 07:36 AM

hmmm ...

This looks exactly like my issue: https://www.proxytip.com/be/browse.p...d%3D675310&b=4

Even the breakage beginning with 4.4.168.

There is no solution but misery does love company

Will investigate the Patch approach.

If that works and NVIDIA actually runs (

) I'll submit a patch to NVidia.

-- kjh

Ilgar · 01-19-2019, 07:38 AM

Quote:

Originally Posted by kjhambrick

I am leaning toward Ilgar's suggestion -- to try patching nvidia/NVIDIA-Linux-x86_64-410.93/kernel/conftest.sh so that the correct #defines are set.

I think this is not a matter of #defines. Every get_user_pages() call in the Nvidia code should be ported to the new API. If they are few, you may have a chance to do it quckly. Otherwise, waiting for an updated driver might be the faster way

.

kjhambrick · 01-19-2019, 07:49 AM

Thanks Ilgar.

That's what I've been waiting for -- this is an NVidia bug and I was hoping for a fix by now

Anyhow, an update:

get_user_pages() did change between 4.4.167 and 4.4.168 but I missed it while eyeballing it.

Below is output from an sdiff wrapper I use to look at code changes

-- kjh

Code:

#
# Diff Search = '  [\|]  |  < *$|  >  |  > *$'   # ( exclude leading and trailing ' )
# Wide Lines  = ' [|]  | < *$| > | > *$'
#
# Working Dir = /usr/src
#
# ./linux-4.4.167.kjh/include/linux/mm.h                                               # ./linux-4.4.168.kjh/include/linux/mm.h
#                                                                                      # 
#ifndef _LINUX_MM_H                                                                    #ifndef _LINUX_MM_H
#define _LINUX_MM_H                                                                    #define _LINUX_MM_H

<<snip>>

extern int access_process_vm(struct task_struct *tsk, unsigned long addr, void *buf    extern int access_process_vm(struct task_struct *tsk, unsigned long addr, void *buf
extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,                  extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,
                void *buf, int len, int write);                                     |                  void *buf, int len, unsigned int gup_flags);

long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,                   long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
                      unsigned long start, unsigned long nr_pages,                                           unsigned long start, unsigned long nr_pages,
                      unsigned int foll_flags, struct page **pages,                                          unsigned int foll_flags, struct page **pages,
                      struct vm_area_struct **vmas, int *nonblocking);                                       struct vm_area_struct **vmas, int *nonblocking);
long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,                     long get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
                    unsigned long start, unsigned long nr_pages,                                           unsigned long start, unsigned long nr_pages,
                    int write, int force, struct page **pages,                      |                      unsigned int gup_flags, struct page **pages,
                    struct vm_area_struct **vmas);                                                         struct vm_area_struct **vmas);
long get_user_pages_locked(struct task_struct *tsk, struct mm_struct *mm,              long get_user_pages_locked(struct task_struct *tsk, struct mm_struct *mm,
                    unsigned long start, unsigned long nr_pages,                                           unsigned long start, unsigned long nr_pages,
                    int write, int force, struct page **pages,                      |                      unsigned int gup_flags, struct page **pages, int *locked);
                    int *locked);                                                   <
long __get_user_pages_unlocked(struct task_struct *tsk, struct mm_struct *mm,          long __get_user_pages_unlocked(struct task_struct *tsk, struct mm_struct *mm,
                               unsigned long start, unsigned long nr_pages,                                           unsigned long start, unsigned long nr_pages,
                               int write, int force, struct page **pages,           |                                 struct page **pages, unsigned int gup_flags);
                               unsigned int gup_flags);                             <
long get_user_pages_unlocked(struct task_struct *tsk, struct mm_struct *mm,            long get_user_pages_unlocked(struct task_struct *tsk, struct mm_struct *mm,
                    unsigned long start, unsigned long nr_pages,                                           unsigned long start, unsigned long nr_pages,
                    int write, int force, struct page **pages);                     |                      struct page **pages, unsigned int gup_flags);
int get_user_pages_fast(unsigned long start, int nr_pages, int write,                  int get_user_pages_fast(unsigned long start, int nr_pages, int write,
                        struct page **pages);                                                                  struct page **pages);

/* Container for pinned pfns / pages */                                                /* Container for pinned pfns / pages */
struct frame_vector {                                                                  struct frame_vector {
        unsigned int nr_allocated;      /* Number of frames we have space for */               unsigned int nr_allocated;      /* Number of frames we have space for */
        unsigned int nr_frames; /* Number of frames stored in ptrs array */                    unsigned int nr_frames; /* Number of frames stored in ptrs array */
        bool got_ref;           /* Did we pin pages by getting page ref? */                    bool got_ref;           /* Did we pin pages by getting page ref? */
        bool is_pfns;           /* Does array contain pages or pfns? */                        bool is_pfns;           /* Does array contain pages or pfns? */
        void *ptrs[0];          /* Array of pinned pfns / pages. Use                           void *ptrs[0];          /* Array of pinned pfns / pages. Use
                                 * pfns_vector_pages() or pfns_vector_pfns()                                            * pfns_vector_pages() or pfns_vector_pfns()
                                 * for access */                                                                        * for access */
};                                                                                     };

kjhambrick · 01-19-2019, 05:55 PM

All --

I posted a bug report to NVidia DevTalk

Waiting to see what the NVidia Devs say.

Thanks !

-- kjh

Update: NVidia has assigned Internal Bug #2488457 to this issue ... will follow up as things progress.

kjhambrick · 01-31-2019, 04:04 AM

All --

NVidia Dev lmiddlebrook let me know that Linux, Solaris, and FreeBSD driver 418.30 [beta release] [long-lived branch release] was available and that it addressed the get_user_pages() issue.

Sure enough, I was able to compile NVidia 418.30 Beta against the latest 4.4.172 Kernel.

And then startx and KDE 4 ran just fine on my Slackware 64 14.2 + Multilib Laptop.

However, I failed to notice that vmware workstation 12 failed to compile when I booted runlevel 3 so I've got no Win7 which I need for work.

But that's another topic for another day

I'll follow up with another Thread as I am able to fix the vmware issue.

Bottom Line: NVidia came thru for me.

-- kjh

marnold · 01-31-2019, 04:06 PM

Thanks for this info. I only wish I would have seen this before I installed the new 14.2 kernel this morning. The good news is that I learned how to do a few things on the command line that I didn't know how to do before.