Well, you're not alone:
Seems like a common problem with that jmicron card reader.
I still think something is interfering with the pciehp module when a card is not present at boot time. Let's test the theory by running lsmod. With the card present, boot up and run as root:
# lsmod > lsmod.txt
and reboot without a card and run:
Compare the output and see what modules are different in the two cases. My guess is that with card present, you will see the pciehp module loaded and without the card you will see sdhci-pci loaded. The sdhci-pci module is a driver for certain card readers but from everything I see your jmicron reader should use the pciehp module. If I'm right, you may be able to work around the problem by blacklisting sdhci-pci and forcing the pciehp module to load instead.