I've been trying to get Android x86 Marshmallow installed in a VirtualBox VM. The installation finishes and I get to boot into the OS. The problem is that whenever I reboot the system, I end up booting back to the installation menu where I choose to start as a live user or install the OS... as if I didn't install it in the first place.

I had this problem with Fedora 25. Same issue, but I was able to resolve it. Under "storage" in VirtualBox, the tree had two storage types: Controller IDE and Controller SATA. The .iso was under the IDE and the .vdi was under the SATA. I deleted the .iso and I was able to boot into Fedora.

However, the Android x86 system had both the .vdi and .iso under Controller IDE. I tried deleting the .iso after installing, but the system fails to boot up after that.

You need to "manually" eject the ISO file from the virtual CD drive.

If you click on the ISO in the storage tree window that selects the Virtual CD drive. Then in the attributes window you should see optical drive then a disk image icon with a triangle. That icon is a pull down menu. Click on the icon then select remove disk from virtual drive. The VM should now boot from the VDI.