We recently had to upgrade the image on our Cisco Nexus 7010s, the core switches in our data center. They had been running images that were demoted to “deferred” status and were unstable. As these are our core switches and we wanted to reduce the risk of downtime, we decided to follow the in-service-stateful-upgrade (ISSU) process. For a while, everything seemed to go well. We did not drop a single packet during the upgrade, that we saw. We tested our core application environment, Internet access, WAN access; everything looked normal. We patted ourselves on the back for a job well done and called it a night.
It turns out we missed one environment in our testing and, would’t you know it, that was the environment that went down. “And why did it go down?” I hear you ask. Let me tell you.
The ISSU process for the 7K is not as thorough an upgrade as some documentation might lead you to believe. That is because the previous version of code is not fully cleared during the upgrade process. It takes a full reload of the CHASSIS to accomplish that. Yes, that’s right, you must completely shut down all power to the chassis in order to clear the old image and any residual bugs. And that was the problem we were facing. One of the bugs in the old image had an impact on forwarding at layer 2 that only affected one of the environments in a VDC. It was only after ten hours of troubleshooting with TAC that an engineer finally conveyed that information to us. When the switches were reloaded, they began forwarding traffic normally.
So, if anyone reading this post is considering an ISSU on the 7K (and probably other platforms), you may want to keep this information in mind.
By the way: I must say this was a huge disappointment for us. We made a substantial investment in this platform to help ensure near continuous uptime in almost any situation. The ISSU feature was one of the selling points. Though I’m fairly satisfied with the 7K in general, Cisco has some work ahead of them before that particular feature is ready for prime time.