I've been quite confused about the total number of open TCP
connections a host can have. Most articles on the web seem to
only partly answer this question. Most emphasise the need of
increasing the number of file handles a user process can have
open (ulimit -u) or the total number of files the
kernel can have open (viewable
in /proc/sys/fs/file-max).
However, the real problem is the limit of ports that can be
open. This is
because the TCP
specification states that the number of ports the host can
handles is defined by a 16bit number. This gives a theoretical
maximum of 65535 open connections. When setting up a connection,
the host uses a (anonymous) local port and binds it to the
requesting host: localhost:1212 ->
otherhost:2332
Luckily, the story doesn't stop there. Albeit the number of
ports are limited to this number, a local port can be used to
connect to multiple IPs. Hence, the number of theoretical
connections is (65535 - open normal files) * number of
connecting IPs
This implies that scaling any host that reaches the celing of 65535 incoming connections, is to add a(nother) reverse proxy in front of it so that there are more incoming IPs that can generate more TCP connections (which must all have a unique combination of origin server, origin port, destination server and destination port.
cacheserver:80 -> proxy1:1024
cacheserver:80 -> proxy1:1025
[..]
proxy1:655535
Another way to solve this, is to create multiple interfaces on the host and make the service listen to these (this might feel more like a hack than a proper solution).
In practice, this problem will only occur to the
server layer receiving traffic from the outside world; your
cache or web server. In multi server architectures, the servers
behind the front layer, e.g. the database in a internet
-> web server -> database setup, will never have a
need of scaling the number of TCP connection.
It is also worth noticing, that the load balancer in front of your first/top layer must be fully transparent, so that it doesn't expose its IP to the web or cache servers serving the requests. With hardware load balancers, this is less of a problem, but with software balancers, such as HA Proxy, heed must be taken as outlined in this article.
Good articles on the subject: