Sometimes, you may run into a situation where traffic enters a routed interface and then per the routing table, exits the same interface. For instance, take a look at the topology below:
In that topology for instance, let’s say that FW3 has a default route pointing it to the gateway which is located on the layer 3 switch at the bottom. From there, the layer 3 switch has a default route pointing to FW1. All 3 firewalls and the layer 3 switch share interfaces on the same subnet. In our case, the traffic would enter and exit the layer 3 switch on the same interface. That’s where the fun begins.
Because the traffic is entering and existing the layer 3 switch on the same interface, the switch will send ICMP redirect messages to the originating router or in this case, firewall. These would let the originating device know that there is a more optimal route that it could be taking. This ICMP redirect points the originating device to the other, better device in routing terms (in our case FW1). The goal is to have the originating device use this better route next time instead. In my case, this redirect message was ignored and the routing table was followed next time as well.
This sounds good in practice but can cause issues. It forces the processing of the packet to be done via the software layer when this happens versus at the hardware layer. The excessive redirecting as well can cause CPU punting issues as well. Both things can lead to packet loss and latency in the overall scope of the traffic.
So how do we fix it? It’s easy! On the layer 3 switch that is acting as the gateway, simply disable IP redirects on the interface by entering “no ip redirects”. When that is done, the gateway will now follow its routing table and not worry about trying to inform the originating device about a better route. The processing of the packet is also done at the hardware level as it should be, avoiding any punting issues too. If that is the root cause of your issue, you should notice instance resolution to your issue.