Events in the Packet World

When connecting a network monitoring tool (aka Sniffer) into a network environment through mirror or span ports, it is very likely that duplicated packets will be captured.

The duplicated packets generated usually due to incorrect switch configuration, e.g. both ingress and egress packets are captured however the inter-VLAN traffic will result packets record double for those packets leave and enter both monitored VLAN.

Some older equipments like Cisco 6509 Catalyst OS are reported, they will always get duplicated packets mirrored to the monitor tool.

From the Ethernet level, it is hard to distinguish whether the frame is duplicated or not. Let’s focus to a level 2 packet, such as ARP request. There is neither sequence number nor packet ID to identify the difference of 2 packets with same content. However, the packet may still be correct, we cannot simply filter them out, because those similar ARP packets may point out an ARP storm, if we remove those duplication, mistakes will easily come.

The duplicated packets are a nightmare to analyzers. All of the response time, TCP round-trip, retransmission detection, and application level responses are messed up regarding on the packets duplicated condition. So to identify packet duplication and remove that not important duplication is very important for the analysis procedure.

To assess packet duplication, the best way is go to the layer 3. When I talk about Layer 3, actually I meant IP Layer. The IP layer provides a very good field to make sure passive packet duplication. The key is the IPID field, when a machine sends IP packets; the OS will automatically increase the IPID count and put that number into the IP Packet.  This number will not be changed by the inter-path routers or switches. So if the packets with the same IPID and same pattern, normally we can confirm, it is duplication. However, the IPID field has only 2 bytes which means only 65,536 numbers can be a potential candidate of the IPID value. So if there is a very busy server, in a single second, more than 100K packets may be sent. So the IPID verification must go together with the content and packet length validation.

The real world is more complicated, the duplicated packets may be not that the SAME with each other. Let’s consider such a case. A Packet received in Port A, this packet has no VLAN tag at all and when this packet being forwarded to Port B, an 802.1Q/ISL tag is tagged into this packet by the switch to adapt to Port B Vlan settings. So if there are some reasons, the switch mirrors both Packets at Port A and Packet at Port B to the monitor port, the monitor tool will see a duplicated packet pair with different length and byte-to-byte content match.

IPID still work in this case, and we need to take all of the IP layer data to compare the content and length despite of the difference at frame level.

Another case, packets transferred from a router to a firewall, the firewall and router both are working under load balancing/dynamic routing mode. So the packet received and sent by the firewall/routers will usually from and to different Mac address, although they are the same equipment and even same port.  So this cause another kind of packet duplication, the MAC address are different however all others are the same.  The good news is the IPID algorithm with IP Layer byte match is still working under such condition.

The IPID measurement has its limitation, e.g. difficult when handling the NAT packets; and it is difficult to handle the fragmented vs. non-fragmented packets for duplication detection purpose. On some heavy load environment, the algorithm might result faulty. Reducing fault can be easily fine tuned by a continuous detection and duplication count limit algorithm. So basically, the IPID detection works very well in most cases. Someone may ask whether the IPID based packet duplication will remove the TCP retransmission? Definitely not, the TCP retransmission is actively generated by communication hosts, so the IPID will be increased for each TCP transmission.

P.S. These information can be also found on packetbone website.