Don't worry about temperatures that are within spec.

Instead, try to minimize running processes. That will give better responsiveness and battery life. It is also something that the user can control.
Ed

But they are decidedly not using the same software. There are many differences even with the same hardware. Do not expect everything to match 1 for 1. You will see differences in performance (linux is often better) and temp is one of the factors reflecting how busy the cpu is.

Bear in mind that from a cpu POV, itis better to have a steadier temperature than a cycling on. R & D uses temperature cycling to artificially age products.

So, at no load, your pc gets cooler on windows than the same pc(?) does on linux. But at no load, the pc can go off, surely? Try measuring the difference between no load and full load, and post about temperature highs. Compile a kernel and run a graphics test program under windows & linux, and see how that goes. Report back numbers. My guess is that you don't use the potential of your pc at all in windows, and that's why you see cooler temperatures. If you have to pay up an arm and a leg for a compiler, I'm sure others will suggest cheaper cpu stress tests.

Although I don't know if it is available in windows, stress works well in linux for a quick and dirty test at near 100% load of the cpu.

speaking of stress tests, mksquashfs worked my 2 core, 4 thread cpu at near 400%. All you need there is a graphics test in palallel. Compiling and most things only give me ≅200%. If it's an APU, you want to stress both cpu & gpu together