Okay then, fair enough.
First, you need to research the big picture. Understand what the various pieces of VPN are doing. There are two basic systems that you might use,
ipsec-tools and
OpenSWAN. They are, in fact, very similar. But you must first understand, clearly, the context.
Let me give you a very high-level flight over this strange forest... (I happen to use
ipsec-tools, most of the time, so my references to tools and commands will be consistent with that.)
(1) An encrypted VPN tunnel is actually implemented, when all the dust settles, by code in the Linux kernel. Study the
setkey command. When you set up what is called a
Security Policy Definition (SPD), you are actually defining rules as to what TCP/IP traffic needs to be grabbed and stuffed through the tunnel. Traffic that matches the specified "source" and "destination" addresses, and any other criteria, is magically re-routed through the tunnel. (This is separate and distinct, by the way, from "routing!" Don't look for a routing-table entry because you won't find one.)
(2) A security-policy isn't a tunnel. A
policy is what causes a particular item of traffic to be snarfed-up and shoved through a tunnel. But a policy isn't a tunnel. "An instance of a tunnel, proper," is what's called a
Security Association (SA). Each conversation gets its own unique, and uniquely-negotiated, tunnel (SA). When Linux sees a policy that says that traffic must be encrypted, but there isn't yet a SA that can be used, Linux sends a request, to
racoon (ipsec-tools) or
pluto (OpenSWAN), to make one. When they have accomplished this, the traffic goes through the tunnel. Once the negotiations succeed, the SA exists and no further negotiations will need to occur until it needs to be renewed.
(3) The purpose of a tunnel is to make a
range of IP-addresses "over there" available "over here." The actual traffic is piped between two specific
gateway addresses. Although the concept is the same as "routing," once again, VPN uses a separate mechanism.
(4) Security associations are built using clever, automatic negotiations, which actually happen in two
phases. In phase-one, the
gateways talk to one another and they agree upon how to establish a secure communication channel between themselves... which they can then use to create tunnels. As each tunnel is requested (see
(2)),
phase-two negotiations occur across that already-encrypted link to establish the actual SA parameters for a particular tunnel. The
racoon/pluto (so-called "ISAKMP") daemons do all sorts of clever things to identify themselves, to coin random-used-once encryption keys, and even to renegotiate new keys from time to time. All uber-magic.
(5) The Linux kernel does all the work of encryption and decryption, once the SA is established. Until such time as a link needs to be re-negotiated, racoon/pluto are not further involved. So the process of transferring the packets is very fast.
(6) Negotiations work by means of proposals. In other words, "Hi there! Here's ten different ways that we could protect this conversation, using cipher and message-digest algorithms that I know. Pick one." The other side looks over the proposals, chooses one, or says NO_PROPOSAL_CHOSEN. Usually it picks the "strongest" method that the two of them mutually support. During all of these negotiations, the answering side is programmed
not to reveal information because it does
not "trust" the party with which it is negotiating. This can make it very hard to debug connections: the party will say "the answer is 'No!'" and will
not explain why. This is by design.
(6b) During phase
one, the
gateways (which could be "your Linux box") are trying to identify themselves and to establish one another's identity. They're also thinking about things like "man in the middle attacks." This happens "once." During phase
two, which happens per-conversation (per-SA) both at the beginning and periodically thereafter, they are negotiating the details of
a particular conversation (SA). And they're doing it by sending coded messages across the encrypted link established in phase-one... which is basically a secure, separate, control-channel devised expressly for that one purpose.
(7) VPN was designed by committee. Lots of committees. Lots of choices; some esoteric, some not. You could spend many un-successful months trying to "understand it all." That's why it is so important to "understand the forest, first." You need to be able to
zero in on what you need to know and do, and exclude what you don't.
(*) You betcha...! Why, I simplified the heck out of this high-level explanation! I waved my hands all over the place. I took all sorts of details and stuffed them down a rabbit-hole. They'll all come hopping back up soon, as you get into it.
(!) You will lose hair! But you are not alone! (Show of hands, here?
That many? I believe it.) Yeah, so you're among friends here. But when you finally get it working, you get to wear a pointy wizard-hat.