lsmod is command to list loaded modules and I referred man page of lsmod that shows it refers to /proc/modules file. In terms of my os platform, Ubuntu, almost available modules are in /lib/module directory. I think OS kernel naturally loads modules in there. So, What does kernel refer to load modules at boot time?

Is there any files to be refereed for loading or mechanism can't know easily for end user?

Looks like this is relevant for Ubuntu.

Modules needed during boot come from the initrd that the boot loader loads along with the kernel. You will find it in the /boot directory along with the kernel image(s). The initrd is a compressed cpio archive. You can decompress it and look inside if you like.

The initial ramdisk is needed for early boot: Disk controller, filesystems, everything the kernel needs to access the root filesystem. Other modules are in /lib/modules/<version>. The udev service is running to query hardware, trigger events for the kernel to load the appropriate modules, and watch for hotplugging events.

Old school wise, the /etc/modules file can contain a list of modules to load at boot time (really old school if it's /etc/conf.modules). But it's not generally populated these days. There are /etc/modprobe.d/*.conf files that can setup some things, and blacklist modules that it doesn't want loaded. Many modules load when you try to use them, like restoring mixer settings or by udev / systemd. Also note that some modules will not show up in lsmod if they were compiled into the kernel. And some modules will have entries in the ps output, but the name of the process doesn't always match the module name that uses or lauches that process.