I am planning to run an email server, but instead of installing it on bare metal like everybody else, I want to install and run it on a VirtualBox virtual machine.

I am wondering what kind of arrangements I will have to do to make it work.

When the server that is exposed to the Internet and responds for a certain domain receives mail, how does the mail server kick in from within the virtual machine and take over?

A web search for "linux email server virtual machine" will turn up many articles and tutorials.

You will want to use a "bridged" network adapter in the VM so the VM is in the same subnet as your LAN and you will need to harden the VM as you would a physical computer (firewall, etc.). As far as the LAN is concerned, the VM will be just another computer on the network and will be treated as such.

You should also check your ISP's terms of service. In the U. S., most ISPs prohibit public-facing servers on less than business-level accounts; that would include a mail server (as an aside to inconvenience wannabe spammers, not to force users to business-level accounts, as the cynical might think).

1 members found this post helpful.

I don't know if everybody else runs mail servers on bare metal. Most probably that is not the case.
The same arrangements as for a bare metal server. Plus the Virtualbox host must be configured to pass traffic to the VM, but I believe that this is configured automatically when you install Virtualbox. Also, you probably want bridge networking, as suggested by Frank Bell.
I am not sure what you mean by "responds for a certain domain". Do you mean the VM's host also runs a mail server?

If the host also runs a mail server, you need to configure the two mail servers so that they collaborate. Again, this is not different from bare metal configuration.

If the host does not run a mail server, it will forward traffic to the VM that is destined to the VM's IP address(es). And the other direction, too. If, for example, the VM has IP address 10.11.12.13, traffic to 10.11.12.13 and port 25 will be sent to the VM. The Virtualbox host just forwards it. If a mail server is then configured on the VM, it will pick up that traffic.

1 members found this post helpful.

The host doesn't run a mail server, but when I configure my domain at the registrar's control panel, I will of course use the host's IP address, but since the email server will be running inside a VM, it will have another IP address, an internal IP address, and that is the part that confuses me. I picture email coming to the host server and the host server shrugging like, "What the hell am I supposed to do with this? I am not an email server!"

Sorry if that last part sounds like a scene from a cartoon. But I'm sure you get what I mean.

Note: I am answering this but I haven't investigated web searches as recommended by frankbell yet.

In other words, you are not using bridged networking. Insted, the VM resides behind a NAT router. This is standard for Virtualbox.
Your mental picture is correct, but you can configure port forwarding on the host. Traffic to email ports like 25 gets forwarded to the VM. Easy to achieve with Virtualbox.

But that is not sufficient. You also have to configure your mail server so that it can work behind the NAT router. I don't know how to do that, but it is possible, for example see Postfix documentation.

As above, you should use Bridge Mode networking for the guest so that the guest can function as an independent machine on your network. This will enable it to query the router and obtain its own IP address completely independent of the host.

Once this is done, you can set up a port forward on your router directly to the VM's IP address.

1 members found this post helpful.

There is no router. I am going to do it on a VPS or dedicated server hosting account.

I am using neither NAT nor bridged. It isn't installed yet. I can use bridged, but if the guest is going to function as an independent machine, how is the host server supposed to know that some creature inside it handles email and forward anything?

It isn't supposed to know. The host doesn't care what's inside the VM. This is true for any network protocol, HTTP, SSH, ... The host just forwards all traffic that is destined to the VM, to the VM. Not only email traffic.

By default, Virtualbox sets up NAT networking. In this case, Virtualbox creates a NAT router and connects the VM to it. When you opt for bridged networking, there is no router, indeed.

Hmm. Is there any difference between a VPS and a VM? Isn't a Virtual Private Server just a Virtual Machine within a container? A VPS isn't "bare metal," as I understand it. (Someone please correct me if I'm wrong)

I'm wondering why you want to put the mail server in a VM? We provide email services on a different IP than a domain's web services by using an alias in the network configuration and configuring the servers to listen on different IPs. e.g.
...of course, that's just a guess at what you're wanting to accomplish...again, why use a VM?

I will be using a VM because I hate installing and configuring email, so I want to do it ONCE in a VM that I can take anywhere in case I move to another host provider and everything is just ready to go.

Can you do it in Docker? I've really come to appreciate the mobility of it. I rebuilt my home server a few days ago for 2 reasons. One to switch from the docker official package to the snap (not necessary to rebuild), and the other was just to see how long it took to rebuild. Before I moved to Docker it would take me 4-5 hours to get everything sorted. With Docker I had it fully functional and online doing it's job in an hour or so, most of which was waiting for download and the docker build to complete namely squid with ssl support.

That makes sense, but I accomplish that with extensive and complete backups of configuration files, etc*....but then I’ve yet to learn how to “do” VMs...I (think) I understand the concepts, but haven’t learnt the specifics. OTOH I’m very familiar with how to setup my mail server (I use qmail)

Do you already know how to create and administer a VM?
What MTA are you using that is so hateful?

*I should be able to recover my entire setup to the last backup on a new server in a few hours.

I am not familiar with Docker, but a virtual machine seems to be easier.

You create a VM, start with an empty hard disk, install the OS on it just like on bare metal, and use it.

On another machine, you transfer the existing VM into it, install VirtualBox, open the existing VM and run it.

Couldn't be any easier. Even I can do it.

The hateful MTA will probably be qmail because it's the only one I have installed and maintained, many many years ago. It is hateful, but all MTAs are.

My go to for qmail

1 members found this post helpful.