Tcp Ingress Flow

John Dykstra's Blog

1. Ingress packets are delivered to TCP, in softIRQ context, through tcp_v4_rcv().
2. If the sock is currently owned by a task, we can't touch most of its contents, so the packet is stashed on the backlog queue.
3. If the sock is not owned, but there is a task blocked in tcp_recvmsg() on this socket, the packet goes into the prequeue and the task is unblocked.
4. Otherwise, we call tcp_v4_do_rcv() immediately, and the data contents of the packet are going to end up in the receive queue.
5. The backlog queue is emptied in user context when the task that had locked the sock gets around to releasing it. The packets go to tcp_v4_do_rcv() and thence to the receive queue.
6. Once the task in tcp_recvmsg() wakes up, it calls tcp_prequeue_process(), which pulls each packet in turn off the prequeue and passes it to tcp_v4_do_recv(). That function recognizes it is in the context of the receiving task, and moves the data contents of the packet directly to the application's buffer.
7. Data that makes it into the receive queue gets pulled off by tcp_recvmsg() and written into the application buffer. Note that this code must preserve packet ordering--the receive queue must be empty before packets are pulled off the prequeue.
8. Also note that packets in the backlog and prequeues have not yet been seen by the TCP state machine in tcp_v4_do_rcv(), so for the length of time they sit there, they are unacknowledged. This latency gets wrapped up in the bandwidth-delay product for big transfers, but it can be an issue for short connections stuck in slow-start.