Why can't the tunnel mode of IPsec simply be another layer providing encapsulation? For example, if I have:

1. LAN#1 with subnet 172.24.0.0/16
2. Router#1 on LAN#1 with public IP 198.51.100.1
3. Router#2 on LAN#2 with public IP 203.0.113.1
4. LAN#2 with subnet 10.20.30.0/24

I cannot just route the 2 private subnets over the internet. I need to encapsulate them so that real routable IPs are the face of the packet, while the private IPs are just payload from the internet point of view. That's encapsulation.

Now all I need to do is have IPsec recognize that I want all traffic between 198.51.100.1 and 203.0.113.1 to be encrypted.

How is this not 2 distinct layers?

IPsec tunnel mode IS a layer of encapsulation. The original datagram is encrypted, header and all, and encapsulated in an IP protocol 50 packet (IPsec ESP).

You could combine something like GRE and IPsec transport mode and achieve basically the same result.

Why isn't the encapsulation that IPsec has simply a separately defined protocol? Why is it "a part of" IPsec? Is it considered superior to other encapsulation protocols? Or is it specifically optimal for IPsec? Why is it considered a "mode" (transport vs. tunnel)? Can I have BOTH transported traffic (using public IP to public IP) as well as tunneled traffic (using private IP to private IP) on the same IPsec path?

I am interested in determining if it is possible to set up an IPsec encrypted tunnel for a VPN, where the point that IPsec encryption takes place is separate from where the tunnel endpoint (where encapsulation and decapsulation takes place). The following reprents 4 hosts, 2 LANs, and the Internet:

LAN#1----Encapsulator#1----Encryptor#1----(Internet)----Decryptor#2----Decapsulator#2----LAN#2