If a server accepts TCP connections, does it ALWAYS respond to a SYN message by sending a SYN/ACK message or does it ever have a blacklist of IP addresses that it does not respond to? That is, does the TCP protocol state that all open and listening ports must respond to a SYN message?
Any clarification would be much appreciated
No. Sometimes there is no IP connectivity to the target so a response never arrives, or there may be no TCP protocol stack on the target, or there may be other protocols/services in place to disallow a particular connection. However, if there is IP connectivity, a TCP protocol stack on both hosts, and a default implementation then yes:
To establish a connection, TCP uses a three-way handshake. Before a client attempts to connect with a server, the server must first bind to and listen at a port to open it up for connections: this is called a passive open. Once the passive open is established, a client may initiate an active open. To establish a connection, the three-way (or 3-step) handshake occurs:
SYN: The active open is performed by the client sending a SYN to the server. The client sets the segment's sequence number to a random value A. SYN-ACK: In response, the server replies with a SYN-ACK. The acknowledgment number is set to one more than the received sequence number i.e. A+1, and the sequence number that the server chooses for the packet is another random number, B.
ACK: Finally, the client sends an ACK back to the server. The sequence number is set to the received acknowledgement value i.e. A+1, and the acknowledgement number is set to one more than the received sequence number i.e. B+1. At this point, both the client and server have received an acknowledgment of the connection. The steps 1, 2 establish the connection parameter (sequence number) for one direction and it is acknowledged. The steps 2, 3 establish the connection parameter (sequence number) for the other direction and it is acknowledged. With these, a full-duplex communication is established.
TCP is at layer-4, but IP is at layer-3. TCP doesn't have any sort of a blacklist, nor does it particularly care about IP addresses. Don't confuse the TCP in a host with how a firewall protects the host. You should read RFC 793, TRANSMISSION CONTROL PROTOCOL, which defines TCP, to learn how TCP works. Section 3.4, Establishing a connection, goes into detail about what happens, including anomalies.
In so far as TCP, yes, a SYN will always be followed with a SYN/ACK response.
When a Server or Firewall (or software Firewall) is employed, and a particularly blacklisted IP is sending the SYN, then in that case, it is like if TCP (for that IP address) is turned off. So the SYN will "fall on deaf ears", per say, and no response will be generated.
But note that this is an outside entity acting on TCP's behalf. TCP by itself does not exist for security, its job is to simply create a reliable data stream. If Something else wants to decide who the reliable data stream should be formed with, then that is outside the scope of TCP.
A TCP Segment send with SYN flag set can be replied with TCP Segment with RST flag set. This can happen if the server is not listening to that particular destination port mentioned in the TCP segment with SYN flag set. This means the connection is aborted from the server side.
Hope this will help.
