Overview

In this post, I’m going to talk about what IP fragmentation is, how it works and why it’s needed. And while learning that, we’re going to touch on subjects like OSI Layers / PDU / MTU / MRU and PMTUD.

Knowledge about them is required for truly understanding IP fragmentation process and troubleshooting network connection issues in general.

OSI model

This is basically a standardized model for network communications which breaks them into what’s called abstraction layers.

The original model consists of 7 layers: Physical, Data link, Network, Transport, Session, Presentation, and Application .

Going up the OSI model hierarchy, layers stack up on top of one another.

Each layer would add an overhead to the final size (Yes, even the physical layer!)¹

More info can be found Here.

PDU

In the OSI model, each layer consists of units. They are called Protocol Data Units.

Any transmission between 2 entities on a layer, would be be done by sending PDU of that layer.

If a unit does not get fully received, it must be dropped (provided that it couldn’t be reconstructed somehow) and if the carried information was important, there must be a way to signal the other end to re-send it.

PDU size

Depending on the application, each PDU be could be hundreds (or even thousands) of bytes in size.

Not counting the hardware limitations (which is usually the bottleneck), there is a balance that needs to be made between the maximum acceptable unit size, resource usage and links stability (As I said earlier, if a unit does not get fully received, it is lost. Bigger the unit, bigger the re-sent data going to be).

As a rule of thumb, assuming enough stability, the bigger the unit size, the lesser the overhead (each unit is bundled with its own header), and greater the speed. It also likely has less impact on resources (for each unit the header might be validated which depending on the used application, could also mean verifying the unit’s data as well).

MTU/MRU

Maximum Transmission Unit is the maximum size of a unit that can be successfully sent over a network to the other end.

likewise, Maximum Receive Unit, is the maximum size of a unit can be received from the other end².

Packets

Packets are the PDUs that travel over a link on the netwrok layer.

IP is the most widely used network layer and also the protocol in which TCP (The most widely used transposrt layer) relies on.

In our secure-webpage-loading example, everything above IP (which would be TCP, TLS and HTTP) should be packed in an IP packet with some added IP related header.

Packets size

Minimum acceptable packet size is:

• IPv4: 576 bytes
• IPv6: 1280 bytes

According to standard, all devices along the way, must be able to accept and route those packet sizes.

Absolute maximum packet size is:

• IPv4: 64KiB³
• IPv6: 4 freaking GiB!⁴

Note: The practical packets MTU/MRU, is much less than that. In a normal LAN, the maximum size of a packet is likely 1500 bytes. Network nodes (including NIC interfaces of each ends as well as switches/routers along the way) might not support bigger packets than that at all.

Now in a network like WAN (or basically your typical Internet connection), there is absolutely no guarantee that your MTU between different endpoints would be consistent. For example, connecting to one server, your MTU might be 1492 and for another one might be 1400. To make matter even worse, due to things like Dynamic Routing even your MTU to a single server might change over time.

As a client, your ISP has set an upper limit to your links MTU/MRU and you can not send/receive any packets successfully if it gets bigger than that size. In an ideal link, that upper limit is 1500 bytes (which is also the default for most OS’s and NIC drivers).

But things aren’t always that simple. Sometimes lower layers (e.g data link), append some extra headers to their own unit, making less space available to their payload (the upper layers data embedded in). As an example, your typical PPPOE connection adds 8 bytes of extra header, limiting your packets MTU/MRU to only 1492 (1500 - 8).

IP Fragmentation

But what if the data you want to send, don’t fit in a single packet? That’s where IP Fragmentation comes into play.

For IPv4, once the MTU of a path has been determined by the router, if a packet exceeds that size, it seamlessly fragments the packet into smaller ones before sending them along the way. The first router that determines its next-hop⁵ could handle the full packet size, would defragment it on the spot.

For further clarification, take a look at these examples:

Now all these (de)fragmentation is costly. Specially for routers which usually do the work. They have to tear the packets apart, split their payloads and calculate and add headers for each new ones.

The work adds up quickly when a router is dealing with tens/hundreds of thousands of packets per second. This is one of the reasons why on IPv6, fragmentation on router level is not allowed anymore and hosts are solely responsible for (de)fragmenting their own packets.

DF flag

The Don’t Fragment flag, is a special bit in the IPv4 packets header. It’s sole purpose is to tell the routers along the way that the packet should not be fragmented.

Well-configured routers, have no choice but to drop the packet if its bigger than the MTU of their next-hop. This flag has 2 main uses:

• Well-designed applications/protocols can optimize the packet size themselves (instead of relying on routers IPv4 fragmentation capability). If a packet gets dropped because of violation of MTU size along the path, the said application/protocol use that to its own advantage to adjust its packets size in a way that they wouldn’t get fragmented anymore.

• As we will see shortly, DF flag also plays a huge role in actually identifying the MTU of a path.

PMTUD

So how would an OS know about the MTU/MRU of a path?

By now we know that its in our best interest to send as big of a packet as possible, but not bigger than our MTU. Also because of different factors in play, we can’t usually know the MTU of a path before hand.

Path MTU Discovery is here to the rescue! Originally intended for routers (since they were traditionally responsible for fragmenting IPv4), It is now a needed feature for every client OS.

When the DF flag is set on an IPv4 packet and a router along the way needs to drop it (because the packet size is higher than its next-hop MTU), it should also do something else…

rfc1191 explains:

When a router is unable to forward a datagram because it exceeds the MTU of the next-hop network and its Don’t Fragment bit is set, the router is required to return an ICMP Destination Unreachable message to the source of the datagram, with the Code indicating “fragmentation needed and DF set”. To support the Path MTU Discovery technique specified in this memo, the router MUST include the MTU of that next-hop network in the low-order 16 bits of the ICMP header field that is labelled “unused” in the ICMP specification. The high-order 16 bits remain unused, and MUST be set to zero.

So when an IPv4 packet with the DF flag set is dropped by the router, the router is required to send a special ICMP Type 3, Code 4 (“fragmentation needed and DF set”) to alert the host initiating it, about the packet loss.

The ICMP message should also include the acceptable MTU size for the said router’s next-hop.

As you can see from the above example, not only the MTU of a path can be obtained like this, it can also be optimized in an ever changing environment.

For IPv6, the burden of fragmenting packets is completely on the hosts and PMTUD plays a huge role in that. Routers only drop the packets and notify the hosts via ICMPv6 Type 2 (“Packet Too Big”) which again also includes the MTU size of their next-hop.

PMTUD Black hole

Some system administrators (whether not realizing the importance of these ICMP messages, not knowing about them at all, or maybe to protect their network from probing, dos attacks, etc), opt to disable the required ICMP generation and silently just drop the bigger-than-MTU packets that have DF flag set⁶. Also inadequate firewall setup on the host might filter those ICMP messages.

This, effectively cripples the host’s ability to correctly determine MTU of a path and is referred to as PMTUD black hole.

Typical PMTUD black hole symptoms are as follows (You should usually experience more than one):

• Some connections (like SSH) establish successfully but stall right when you start using them even though hosts are pingable all the time.
• Users complaining about not being able to visit “some sites” either completely or not fully. (usually as the result of SSL connections stalling).
• Connections may work fine (with some initial delay) in a device (e.g., an android phone), while failing on other ones (a desktop PC).
• Overall slow network experience even though the link to the ISP is reliable and it should work faster but it doesn’t.

Although some workarounds have been introduced by different OS’s to ease the effect, because the nature of the issue, they are not without their drawbacks.

And as always, your comments would be most appreciated!