Port striping v1

When cryptostorm was first created, one of the first things we realized customers would need was the ability to connect to the VPN on non-standard ports. At the time, other VPN providers offered this by simply running extra OpenVPN instances on their servers, each listening on different ports that were usually allowed through simple firewalls (UDP 53 for DNS, T***** 80 for HTTP, T***** 443 for HTTPS, etc.).

That didn't seem like an efficient method since you would need to run 65,535 OpenVPN instances if you wanted clients to be able to connect to any valid port (or 131,070 instances if you wanted to provide both UDP and T***** access).

So we started looking for another way to do it. What we came up with was two fairly simple iptables rules:

iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 1:65534 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 1:65534 -j DNAT --to-destination :443

Basically, these two rules forward any traffic to any port on 195.154.33.73 to the OpenVPN instance listening on port 443. Doing it this way, the server only needs to run two OpenVPN instances, one for UDP and one for T*****. We decided to call this feature "port striping".

Fast forward to a few years later. OpenVPN 2.4.0 finally gets released, and with it came support for Elliptic Curve Cryptography. We wanted to offer this to our customers, but because of our "port striping" rules the only way to do that would have been to buy twice as many IP addresses, or set aside a port just for these new ECC instances.

We decided to go with the latter option, reserving a single port just for ECC. We went with port 5060 because it's used by the SIP protocol, which is used by some VoIP applications. A lot of residential ISPs will apply QoS to VoIP traffic so that it receives a higher priority than other traffic, because ISPs don't want VoIP calls to be choppy. Some of them would apply these QoS rules by simply checking if the traffic is on port 5060, so for some people, connecting to our ECC instances on port 5060 would result in increased speeds.

The two iptables port striping rules became the six rules:

iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 1:5059 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5060 -j DNAT --to-destination :5060
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5061:65534 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 1:5059 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5060 -j DNAT --to-destination :5060
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5061:65534 -j DNAT --to-destination :443

These new rules did pretty much the same thing as the other two, except this time traffic to port 5060 got sent to the ECC OpenVPN instance listening on port 5060.

A few months after implementing this, several customers were asking for us to add port forwarding to the service so that they could get better torrent speeds. We did, but we ran into the same issue as with adding ECC. We needed to either buy twice as many IPs, or set aside some ports for port forwarding.

Again, the latter seemed like the best way, so we reserved ports 30000 through 65535 for port forwarding. So in the above 6 rules, "65534" got changed to "29999".

Shortly after adding support for port forwarding, we realized that our default non-ECC OpenVPN configs were still using older 2048-bit RSA keypairs, which wasn't the most secure option available. The only reason we kept using that for so long was to keep some backwards compatibility for customers who were stuck using OpenVPN versions older than 2.4.0.

So we began testing to see exactly how strong we could get the cryptography while still supporting those customers. During these tests, we realized that support for Elliptic Curve Cryptography was actually added to OpenSSL way back in 2005. OpenVPN didn't support it back then, but that only applied to the server certificate. Whenever OpenVPN verifies a CA certificate, pretty much all of that processing is handled directly by OpenSSL.

That means even for customers using ancient versions of OpenVPN, we could use an ECC CA certificate, even if the server certificate had to be RSA.

After some testing, we implemented these new cryptographic features. The new configs were rewritten to be OS independent, and the default RSA configs were to use 8192-bit RSA server certificates, along with se*****521r1 (521-bit EC) CA certificates, and 8192-bit DH parameters. The ECC configs were also set to use se*****521r1, but for both the server and CA certificate.

During these upgrades, OpenSSL 1.1.1 was released, and with it came support for Ed25519 and Ed448.
We knew a lot of customers would want to start using those two as soon as possible, so we put off the upgrade a little longer while we figured out how to go about adding support for those two.

OpenVPN wouldn't allow us to offer both se*****521r1 and Ed25519 and Ed448 on the same instance, so we ended up back at the port striping problem. Again, we decided to simply set aside port 5061 for Ed25519, and port 5062 for port Ed448.

The above 6 rules turned into the 10 rules:

iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 1:5059 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5060 -j DNAT --to-destination :5060
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5061 -j DNAT --to-destination :5061
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5062 -j DNAT --to-destination :5062
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5063:29999 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 1:5059 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5060 -j DNAT --to-destination :5060
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5061 -j DNAT --to-destination :5061
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5062 -j DNAT --to-destination :5062
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5061:29999 -j DNAT --to-destination :443

This new setup seemed to be working fine for most people. Unfortunately, some networks have gotten increasingly restrictive thanks to DPI-capable firewalls and routers becoming more affordable, and as a result our RSA OpenVPN instances was getting blocked more often.

To help bypass these DPI-capable firewalls, our ECC instances used OpenVPN's new
--tls-crypt feature, which encrypts the handshake so that it's difficult to tell that the traffic is OpenVPN traffic.
That worked for most people, but not for those who were behind such restrictive firewalls that even port 5060 isn't allowed out.

So we started looking for a way to offer more ports for the ECC instances, without conflicting with all the ports offered by the RSA instances.

Port striping v2

For testing purposes, two virtual machines were set up, one configured pretty much the same as a cryptostorm server, the other to act as a VPN client.

The goal was to monitor the traffic between the two to see if there was some sort of pattern that could be used to differentiate RSA traffic from ECC traffic, if it were to happen on the same port.
To begin with, we only looked at UDP OpenVPN traffic, since UDP is a stateless protocol it seemed like it would be easier to work with.

After staring at `t*****dump` output for a while, we did notice some patterns that appeared useful.

For the ECC instance, whenever a client would initiate the OpenVPN connection, the first packet looked something like:

15:36:01.251877  In 08:00:27:45:f5:87 ethertype IPv4 (0x0800), length 98: 192.168.1.61.1194 > 192.168.1.90.5060: SIP, length: 54
        0x0000:  4500 0052 0000 4000 4011 b6b3 c0a8 013d  E..R..@.@......=
        0x0010:  c0a8 015a 04aa 13c4 003e 6196 385f c40f  ...Z.....>a.8_..
        0x0020:  6058 b0dc 5e00 0000 015b bd11 30df 7a38  `X..^....[..0.z8
        0x0030:  01b4 e22b e3f6 1531 1df8 e608 8d95 5898  ...+...1......X.
        0x0040:  22ac da92 c402 f296 72b1 9b7d 6ad1 d8bb  ".......r..}j...
        0x0050:  5f91                                     _.

After reconnecting several times, we saw that some of the bytes in this initial packet were always the same. 
For ECC connections, that 4500 0052 at the beginning was always the same. We know from RFC 791 that the "45" refers to the first two fields in the header, the version (IPv4, so "4") and the header length.
The "5" is the Internet Header Length (IHL) telling the number of 32-bit words in the header. 
That would be 5×32 = 160 bits = 20 bytes, which is the minimum for the IHL.
The "00" after the "45" corresponds to the ToS or Type of Service. A "00" here indicates normal operation.

The "0052" is the total length of the IP header. Converted from hex to decimal it would be 82.
So for ECC connections, the IP header length in the initial packet is always 82 bytes.
Not sure exactly why that is, but it's probably due to the ECC instances wrapping everything up in the key provided by --tls-crypt.

For RSA instances, the first bytes were always "4500 0072". Again, the header length here is probably always the same because of a lack of --tls-crypt, or maybe because the RSA instances instead use --tls-auth.

But these four bytes aren't a specific enough pattern. There's too great of a chance that a packet would match that pattern and not be an initial OpenVPN packet.

Another byte that stayed the same for both RSA and ECC was that 38 at the byte offset 28 (that's the first byte of the second to last column in the second row, the 38 in 385f).

We dug through the OpenVPN 2.4.6 source code to see what that 0x38 byte was, to make sure it really was consistent. That byte is created by OpenVPN in src/openvpn/ssl.c on line 1469:

uint8_t header = ks->key_id | (opcode << P_OPCODE_SHIFT);

On the page https://openvpn.net/community-resources/openvpn-protocol/ they explain that this is the packet opcode/key_id (8 bits) which consists of:

packet message type, a P_* constant (high 5 bits)

and:

key_id (low 3 bits)

Since this is the initial packet, the key_id is zero.
The message type is defined by the constant on line 65 of src/openvpn/ssl.h:

#define P_CONTROL_HARD_RESET_CLIENT_V2 7

and P_OPCODE_SHIFT is defined on line 53 of the same file:

#define P_OPCODE_SHIFT 3

Rewriting the above code with those values, we get:

uint8_t value = 0 | (7 << 3);

Running that C code, the 'value' variable would equal to 56, which is 0x38 in hex.
So that's where that "38" comes from in the t*****dump output from above.

Now that we have a unique enough pattern that's consistent, and can be used to differentiate RSA and ECC traffic, how to go about actually using it? The iptables u32 module!

From https://www.stearns.org/doc/iptables-u32.current.html:

In it's simplest form, u32 grabs a block of 4 bytes starting at Start, applies a mask of Mask to it, and compares the result to Range. Here's the syntax we'll use for our first examples:

iptables -m u32 --u32 "Start&Mask=Range"

Since u32 grabs blocks of 4 bytes at a time, to match the first 4 bytes of our t*****dump output, we don't need a mask. So to define a pattern of "the first 4 bytes are 4500 0052", we would use the rule:

iptables -m u32 --u32 "0=0x45000052"

That would match the beginning of an initial ECC packet. For RSA, it would be "0=0x45000072". 

As for the 0x38 at byte offset 28, that one will require a mask. Again, since u32 works in 4-byte blocks, but we only require the first byte of a 4-byte block, we're going to subtract 3 from the offset, so 28-3=25. 
For the above iptables rule to also match that 0x38 byte, we'll add to it:

iptables -m u32 --u32 "0=0x45000052 && 25&0xFF=0x38"

According to the tutorial, instead of using the mask 0x000000FF we can just abbreviate to 0xFF, which makes the rule slightly shorter but still does what we want.

Now that we have a method of differentiating ECC and RSA traffic, for UDP at least, we can modify the "port striping v1" rules from above to:

iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 1:29999 -m u32 --u32 "0=0x45000072 && 25&0xFF=0x38" -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 1:5060 -m u32 --u32 "0=0x45000052 && 25&0xFF=0x38" -j DNAT --to-destination :5060
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5061 -j DNAT --to-destination :5061
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5062 -j DNAT --to-destination :5062
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p udp -m udp --dport 5063:29999 -m u32 --u32 "0=0x45000052 && 25&0xFF=0x38" -j DNAT --to-destination :5060


The first rule would match RSA, and since it's it's not going to conflict with Ed25519 or Ed448, we can include the whole port range of 1 through 29999.
ECC will conflict with Ed25519/Ed448, so those rules are only applied to ports 1-5060 and 5063-29999.
If something goes to port 5061 and doesn't match RSA, it goes to the Ed25519 OpenVPN instance on 5061.
If something goes to port 5062 and doesn't match RSA, it goes to the Ed448 OpenVPN instance on 5062.

That covers UDP, but what about T*****?

Unlike UDP, T***** is not a stateless protocol. A T***** connection is established through a three-way handshake, which means the initial packet we receive from the client will be a simple SYN packet and outside of the port number won't have any information related to OpenVPN in it. So u32 would be a difficult method of going about this, mainly because there's not anything actually listening on these ports. So a combination of different things would need to be done in order for the server to respond with a valid SYN-ACK as well as redirect the client to the appropriate OpenVPN instance.

Instead of dealing with all that, we decided to use HAProxy instead. HAProxy is a T*****/HTTP load balancer that's used by some of the busiest websites on the internet. But it isn't just for websites. It's configuration can be very flexible even for specific applications like this.

To see if the length of the initial packet (after the T***** handshake) stays the same for ECC and stays at a different value for RSA, we setup our test HAProxy config to log the length of the request:

log /var/lib/haproxy/dev/log local0 debug
t*****-request content capture req.len len 4096
log-format %[capture.req.hdr(0)]

After creating the UNIX socket /var/lib/haproxy/dev/log, we start up our test HAProxy instance and connected to it using a T***** RSA OpenVPN config, then again with a T***** ECC OpenVPN config. Just as expected, and also probably due to ECC's --tls-crypt and RSA's lack of it, the "req.len" variable was different for each but consistent across reconnects. 

For ECC, the first request's length was always 56, and for RSA it was 88.
Knowing this, we would rewrite the configuration file as:

frontend portstripev2
mode t*****
bind 195.154.33.73:443 name frontend-ssl
t*****-request inspect-delay 2s
t*****-request content accept if { req.ssl_hello_type 1 }
use_backend openvpn_rsa if !{ req.ssl_hello_type 1 } { req.len 88 }
use_backend openvpn_ecc if !{ req.ssl_hello_type 1 } { req.len 56 }

backend openvpn_rsa
mode t*****
server openvpn-localhost 127.0.0.1:1194

backend openvpn_ecc
mode t*****
server openvpn-localhost 127.0.0.1:5060


With HAProxy running this config on T***** port 443 of 195.154.33.73, we would then start up an RSA T***** OpenVPN server instance on 127.0.0.1 port 1194, then an ECC T***** one on port 5060.

To get other T***** ports to this HAProxy instance, we would use rules from port striping v1:

iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 1:5060 -j DNAT --to-destination :443
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5061 -j DNAT --to-destination :5061
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5062 -j DNAT --to-destination :5062
iptables -t nat -A PREROUTING -d 195.154.33.73/32 -p t***** -m t***** --dport 5063:29999 -j DNAT --to-destination :443

With that, RSA and ECC traffic would get sent to HAProxy, then sent to their appropriate back end OpenVPN server, and Ed25519/Ed448 would go directly to their OpenVPN instance.

Another benefit to including HAProxy in the mix is that it allows us to detect other non-OpenVPN protocols in this initial request. That means adding something like SSH support for SSH tunneling, or HTTPS support via stunnel is a lot easier now with HAProxy already in place. We've already implemented tunneling/obfuscation for SSH described here, and HTTPS described here.

UPDATE

It turns out a few people were behind network devices that for whatever reason were changing the ToS of their UDP packets, causing the first two bytes of their packets to be 0x45 and 0x48 instead of the 0x45 and 0x00 that the above u32 rules expected.

Since it's possible for that second byte (the ToS) to be something else, and we really don't need to check if the first two bytes are 45 00, we've changed the u32 rules to only check for the 4th byte (the 0x52 or 0x72) since those two are what differentiates ECC from RSA.

So the new rules look like:

iptables -m u32 --u32 "0&0xFF=0x52 && 25&0xFF=0x38"

for ECC, and

iptables -m u32 --u32 "0&0xFF=0x72 && 25&0xFF=0x38"

for RSA.
The updated rules have been applied to all the servers, so if you've been having trouble connecting to the UDP instances, try again.