If you really want to, you can natively compile on your target board or cross-compile the kernel on a host.
A long time ago I was cross-compiling the kernel and building the rootfs with scripts.
But nowadays I make my life simple by using
Buildroot to build a toolchain, the Linux kernel, Busybox and other applications and wrapped up into a rootfs, and boot programs.
Buildroot does add another layer of makefiles, so it can get a little confusing, but it can eliminate a lot of little problems like what you're encountering.
There are other build tools (such as Yocto), but they typically require more effort to use.
E.G. See
this for building Kodi on a Wandboard i.MX6Q using Buildroot
Regards