TCP Fast Open

With TCP Fast Open, Google has presented a protocol extension for reducing unnecessary latencies in network traffic. The proposal was originally presented in 2011 and was published as RFC 7413 in December 2014.

The majority of Internet traffic is made up of pretty short-lived data flows, such as the typical call to a website: The servers set up several, not infrequently several dozen simultaneous TCP sessions, only to then transfer relatively little data of many small elements (HTML code, Javascript, mini-graphics, etc.). The (time) effort required to establish the connection is not in good proportion to the transferred content. According to studies, one-third of all requests initiate new connections, and one-quarter of all waiting time caused by latency comes from the connection setup.

To prevent new connections from being established, some browsers try to keep unused TCP connections open after the initial page request (so-called HTTP Persistent Connections). However, this ties up resources and annoys admins: especially with heavily frequented servers. They now set the timeout adjusters to very short periods. Better approaches are offered by the Google project: TCP Fast Open.

What is TCP Fast Open?

TCP Fast Open (TFO) is an extension to the transmission control protocol (TCP) that helps reduce network latency by enabling data to be exchanged during the sender's initial TCP SYN. A traditional TCP handshake is a three-step process carried out as follows.

1. The sender sends the SYN packet to the receiver to initiate the connection
2. The receiver sends the SYN-ACK packet back to the sender to let it know that it is ready to start transmitting data
3. The sender sends an ACK packet to the receiver

Once this process is complete, the sender and receiver can exchange data. However, performing these three steps increases network latency time, decreasing overall page load speed. TCP Fast Open helps reduce the additional network latency time this handshake incurs by having the client send data during the initial SYN, thus allowing connections to take place during the handshake.

How does it work?

The TCP Fast Open extension sends data to the receiver upon the initial SYN from the sender. This allows for data transfer to begin immediately instead of waiting for the entire handshake process to take place. However, TFO only works once a normal TCP handshake has taken place, as the sender not only sends the SYN packet to the receiver but also sends a cookie request.

Therefore the process which takes place during the first TCP Fast Open request between a sender and receiver is as follows:

1. The sender sends the SYN packet along with a cookie request
2. The receiver generates the requested cookie
3. The receiver sends the SYN-ACK along with the cookie back to the sender
4. The client caches the cookie for the particular server's IP

Now that the client has cached the cookie generated by the server, subsequent sender and receiver handshake communication will look like this:

1. The sender sends a packet containing the SYN, TFO cookie, and data
2. The receiver validates the TFO cookie, and data is made available to the application
3. The receiver sends back the SYN-ACK and continues to send more data packets to the client
4. The client sends the ACK back to the receiver

Once this process is complete, the TCP continues on normally. By using the TFO extension, the client is able to start sending data immediately, and communication can begin earlier.

TCP Fast Open Nginx configuration

In order to use the TCP Fast Open extension with Nginx, your kernel settings must be configured to support TFO. As of Linux version 3.7, TFO support was included for clients and servers and as of version 3.13, it should be enabled by default (check your version with uname -r). Use the following command to see if TCP Fast Open is running on your server.

cat /proc/sys/net/ipv4/tcp_fastopen

If this returns a value of 0, it is disabled. Additionally, if it returns a value of 1, this means TFO is only enabled on outgoing connections (client), and a value of 2 indicates it is only available on listening sockets (server). However, you'll want your tcp_fastopen set to 3, which enables both.

If the tcp_fastopen value is not equal to 3, you can modify this by running the following command.

echo "3" > /proc/sys/net/ipv4/tcp_fastopen

This will modify the tcp_fastopen value until the system reboots. However, if you would like to avoid having to change the value each time the system reboots, you can create a tcp_fastopen.conf file within the /etc/sysctl.d directory as follows:

echo "net.ipv4.tcp_fastopen=3" > /etc/sysctl.d/30-tcp_fastopen.conf

We now have a tcp_fastopen configuration file which specifies a value of 3 stored in the sysctl.d directory. We can verify the contents of this file by running cat /etc/sysctl.d/30-tcp_fastopen.conf

As for Nginx TFO support, most Nginx packages do not include this by default. However, it can be specified if you build Nginx from source by adding the -DTCP_FASTOPEN=23 compiler flag to NGINX's configure script. Once your Nginx server is properly configured and supports TFO, adding the TCP Fast Open option is as simple as opening up your nginx.conf file and adding it to listen directive.

listen 80 fastopen=256

Summary

As previously mentioned, using the TCP Fast Open extension can help further reduce network latency by allowing data to be sent earlier. This is especially useful for website visitors who are far from the origin server, increasing round trip times. Although some configuration will likely be required at the server level, this may prove beneficial if you are experiencing network latency issues or need another method to help improve page load times for returning users.

If you're interested in learning more about TCP Fast Open, take a read through the specification.