It figures. 4:30pm Friday and I’m working on a customer’s Nexus 5672 to enable jumbo-frames. As soon as I entered the “system qos” command, all of the customer’s virtual servers stop responding. Great. A quick look at CCO revealed nothing (not surprising), and Google was silent. I called a colleague for emergency help and he said “oh yeah, that’ll change the MTU on your fiber-channel interfaces, too.” Great. Fortunately, since he had been through the same exercise, he had the fix handy (thank you, Joey!). So, for all of you getting ready to enable jumbo MTU on your 5672, if you are running fiber-channel on the same switch, please look at the following script. It may save a little frustration on your part:
policy-map type network-qos fcoe-default-nq-policy-plus-jumbo class type network-qos class-fcoe pause no-drop mtu 2158 class type network-qos class-default mtu 9216 ! system qos service-policy type network-qos fcoe-default-nq-policy-plus-jumbo !
What this does is ensure that the FCOE MTU is not changed when you change the system MTU. I can hear someone out there saying “but I’m not running FCOE.” It doesn’t matter. If you change the system MTU, it applies to EVERYTHING. Apparently, the FC interfaces derive their MTU from the same MTU setting as the FCOE policy.
Live and learn.
UPDATE 9/26/15: I finally ran across a small line in the Cisco documentation that explained how the system MTU is applied to all of the interfaces, including FCOE and FC. It was buried in the middle of a document somewhere with no particular emphasis. As the Inspector said in Monty Python’s “Crunchy Frog”, “I hardly think that’s good enough! I think it’s be more appropriate if the box bore a great red label: ‘WARNING: LARK’S VOMIT!!!'” Or “WARNING: THIS WILL CRASH YOUR VM ENVIRONMENT IF YOU MESS WITH IT!!!”, or something to that effect.