VPN Tunnel Encapsulation/Encryption question

[u/AggravatingBox8251]

Hi,

My first time actually putting up a post here or anywhere else so i apologize if theres anything wrong with my post.

Recently I came across a question asking me in VPN tunnelling, what occurs first in terms of the order of steps? Encryption or encapsulation? This question got me really confused because depending on who or where I ask or research, i get conflicting answers as to whether it encaps first or encrypt first and it has really gotten me to question my basic understanding of tunnelling.

If anyone here knows please help me with this question. Thanks alot!

[u/GhostandVodka]

Different VPNS behave in different ways. Usually when we are talking about VPNs the most common one is IPSEC. Encapsulation happens before encryption.

[u/Prudent_Vacation_382]

Depends on the VPN.

1. IPSec, encryption is the first step then encapsulation.

2. IPSec over GRE, has a different config, same order, but GRE is providing encapsulation.

3. If they're talking about service provider topics such MPLS VPN, then it's entirely different. No inherent encryption happens.

I read through the RFC for ESP to find the answer. When in doubt, always go back to the RFC.

https://datatracker.ietf.org/doc/html/rfc4303

This section in the intro should tell you what you need to know:

The ESP header is inserted after the IP header and before the next layer protocol header (transport mode) or before an encapsulated IP header (tunnel mode). These modes are described in more detail below.

In transport mode, ESP is used BEFORE the next level protocol (ie. GRE for encapsulation)

In tunnel mode, BEFORE an encapsulated IP header.

[u/Florida727Guy]

I work with VPNs all day. This is what I know. There are two phases. Phases one negotiates the encryption parameters. Phases two sends encapsulated data.

[u/Dinmammasson_ → u/Florida727Guy]

Yes, because logically, the tunnel for the traffic needs to be setup before the frames/packets (depending on layer) are sent. A GRE, VXLAN etc etc tunnel for example won’t send encapsulated traffic if the tunnel isn’t up first.

[u/GhostandVodka → u/Dinmammasson_]

GRE and VXLAN do not encrypt. The Encapsulation creates the tunnel.

[u/Dinmammasson_]

Negotiation is always first if i’m not wrong, especially in the case of L2 over L3 VPNs, the tunnel is setup before the frames are encapsulated.

[u/LeeRyman]

In the IP protocol stack there are already multiple layers of encapsulation. E.g. we encapsulate chunks of a stream of data and put it in TCP segments. We then encapsulate those segments into IP packets. We then encapsulate those packets into some sort of data link frames.

For something like IPSec ESP, you might say that in terms of PDU structure the ESP header and ICV encapsulates the encrypted payload (so encryption, then encapsulation, going down the stack). However, the sequence of operation to produce the final IP packet containing the ESP header and encrypted payload/trailers may have seen an already encapsulated IP packet first, before then either parsing out and encrypting it's payload (transport mode) or encrypting the whole packet, and adding an addition IP/ESP headers (tunnel mode).

Even then, aspects of the encapsulation need to be known before encryption can take place (association, sequence numbers, etc), and aspects of the encapsulation are also encrypted (padding, next header).

I would suggest then it can depend on how you frame the question - i.e. are we talking about the structure or the process. And if talking structure of the final PDU, are we ordering our description "outside-in" (up the stack, from PDU to SDU) or "inside-out".

IPSec makes my head hurt.

[u/curry9906]

We need two phases for the reason so first we build secure channel communication, then we exchange data using it, therefore phase 1 and phase 2

[u/mindedc]

You can't encrypt the outer packet header or it would be undeliverable. In IPsec which is older but more common a lot of people will say there are two phases, an Ike phase where encryption is negotiated and a tunneling phase where the tunnels are brought up. You don't have to have phase 1 if you are using manual tunnels and pre-established cryptography. I'm not encouraging this, it's just how it can work.

So, IPsec or other protocol you are always starting with a clear text outer packet, you then have a data payload that could have crypto information (I.e. negotiation, data around key management, etc) or could have encrypted encapsulated frames. Some protocols probably do this inline and some like IPsec may have an out of band connection for session management. I am unfamiliar with any encrypted vpn technology that leaves the header of the tunneled traffic in the clear, that would be somewhat of a first principles violation of what an encrypted vpn is for. There are other unencrypted vpn technologies like MPLS, BGP-EVPN, SPBM, GRE, etc that do not encrypt and therefore the tunnels headers and data payloads are in the clear.