FWIW.CO.ZA

Blog of Christiaan Rademan.

Datagrams, Packets and MTU

The Maximum Transmission Unit (MTU) is the largest number of bytes an individual datagram can have on a particular communication link.

When encapsulation, encryption or overlay network protocols are used the end-to-end MTU size is reduced. Some applications may not work well with the reduced MTU and fail to perform Path MTU Discovery PMTUD (rfc1191 or rfc1981).

Fragmentation

Routers are capable of performing fragmentation of datagrams to slice them into smaller datagrams, but this is not optimal. When an incoming datagram to a network device gets its size increased due to encapsulation the datagram then gets sent through the outgoing interface on its way toward the destination. However, if the new total datagram size exceeds the MTU of the outgoing interface, the network device may fragment the datagram into smaller datagrams before being able to forward the datagram. A router will fragment and forward the datagram, but also send back to the source an ICMP "datagram too big" error message to inform the source that it should use a smaller MTU size. The information in this message is used as part of the Path MTU Discovery process. There is no guarantee that the source will send a smaller datagram.

Known issues

Importantly if transit routers only provide a layer2 service such as VPLS they will not fragment the datagrams. These datagrams will silently be dropped. This is the same behavior you can expect from a switch when sending larger datagrams than supported.

An IP datagram can be marked "don't fragment." Any datagram so marked is not to be fragmented under any circumstances. If datagram marked don't fragment cannot be delivered to its destination without fragmenting it, it is to be discarded instead.

Some nodes may not be performing Path MTU Discovery (PMTUD) as prescribed by the IETF rfc1191 or rfc1981 and therefor relying on the routers to perform the fragmentation on behalf of the source host. RFC 2923 also covers the topic of "TCP Problems with Path MTU Discovery".

Datagram Sizes

The minimum length of the data field/payload of a datagram sent over an Ethernet is 1500 bytes, thus the maximum length of an IP datagram sent over an Ethernet is 1500 bytes. (rfc 894)

Ethernet 2 networks have a standard default MTU of 1518. 14 bytes for Header and 4bytes for Frame Check Sequence FCS. Some other Layer 2 protocols have different MTU sizes.

datagram header size

ETHERNET 14 bytes
802.1Q 4 bytes (1vlan)
802.1Q Q in Q 8 bytes (2vlans)
MPLS Label 4 bytes
MPLS L2 CONTROL-WORD 4 bytes
VXLAN 50 bytes
GRE 4 bytes
OTV 42 bytes
NVGRE 42 bytes
IPv4 20 bytes
Ipv6 40 bytes
TCP Header min 20 bytes max 60 bytes
UDP Header 8 bytes
ICMPv4 & V6 8 bytes
6in4 encapsulation 20 bytes
4in6 encapsulation 40 bytes

MPLS L2 Control-Word

In the Layer 2 circuit IETF drafts, the control word is optional for most Layer 2 protocols, except Frame Relay and ATM AAL5 where it is required, The control word is 32-bit that is inserted between the VC label and the transported layer 2 frame in case of AToM, it’s used to carries extra information such as protocol control information and sequence number, this information is needed to correctly and efficiently carry the layer 2 protocol across the MPLS network.

Testing

One method to test and detect a reduced MTU size is to use a ping with a large packet size and setting the DF (Do Not Fragment) bit.

For IPv4 you should effectively be able to get 1472bytes icmp ping on IP MTU of 1500bytes. To calculate max icmp packet it is IP MTU (1500Bytes) - IP HEADER (20bytes) - ICMP Packet header (8bytes) = 1472Bytes.