Feeds:
Posts
Comments

Archive for the ‘Cisco’ Category

Ran into an interesting issue the other day.  A customer was trying to run an application called AirServer in their environment.  AirServer extends Apple’s wireless Airplay feature to wired devices.  By default, AirServer works only on one subnet, so wired and wireless users must be in the same IP subnet.  By using the Bonjour service, AirServer can extend Airplay across multiple IP segments.

AirServer is installed on target desktops and laptops, which then send video streams to client devices (iPads and iPhones) using multicast addresses in the 239.0.208.0/24 range.  It also uses mDNS to advertise server names to the clients.  The customer’s network infrastructure, both wired and wireless, is built using only Cisco devices, particularly WLC5508s for the wireless controllers.

The issue we rant into was this: after configuring PIM and IGMP on the layer 3 and 2 devices respectively, and enabling multicast and IGMP snooping on the WLCs, the clients could not see the servers.  We could see the mroutes in the table and the members in the IGMP table, but the multicast streams never got beyond  the core layer 3 switch.

After a lot of troubleshooting and a couple of go-rounds with Cisco TAC, we stumbled on the solution.  Because the multicast streams are not sourced from the wireless environment, all of the wired VLANs had to be trunked to the WLC5508s, after which SVIs had to be created on the WLCs for each of the wired VLANs.  Once we did that, everything worked like a charm.

My guess at the root cause is that the WLCs do not snoop IGMP traffic on the trunk links, so they never took the multicast streams in to send out to the wireless clients.  If anyone out there can confirm or correct that, I’d appreciate it.

Advertisements

Read Full Post »

It figures.  4:30pm Friday and I’m working on a customer’s Nexus 5672 to enable jumbo-frames.  As soon as I entered the “system qos” command, all of the customer’s virtual servers stop responding.  Great.  A quick look at CCO revealed nothing (not surprising), and Google was silent.  I called a colleague for emergency help and he said “oh yeah, that’ll change the MTU on your fiber-channel interfaces, too.”  Great.  Fortunately, since he had been through the same exercise, he had the fix handy (thank you, Joey!).  So, for all of you getting ready to enable jumbo MTU on your 5672, if you are running fiber-channel on the same switch, please look at the following script.  It may save a little frustration on your part:

policy-map type network-qos fcoe-default-nq-policy-plus-jumbo
     class type network-qos class-fcoe
          pause no-drop
          mtu 2158
     class type network-qos class-default
          mtu 9216
 !
 system qos
     service-policy type network-qos fcoe-default-nq-policy-plus-jumbo
 !

What this does is ensure that the FCOE MTU is not changed when you change the system MTU.  I can hear someone out there saying “but I’m not running FCOE.”  It doesn’t matter.  If you change the system MTU, it applies to EVERYTHING. Apparently, the FC interfaces derive their MTU from the same MTU setting as the FCOE policy.

Live and learn.

UPDATE 9/26/15: I finally ran across a small line in the Cisco documentation that explained how the system MTU is applied to all of the interfaces, including FCOE and FC.  It was buried in the middle of a document somewhere with no particular emphasis.  As the Inspector said in Monty Python’s “Crunchy Frog”, “I hardly think that’s good enough! I think it’s be more appropriate if the box bore a great red label: ‘WARNING: LARK’S VOMIT!!!'”  Or “WARNING: THIS WILL CRASH YOUR VM ENVIRONMENT IF YOU MESS WITH IT!!!”, or something to that effect.

Read Full Post »

Awhile back, I posted a simple expect script for automating SSH sessions on a Mac.  Here is a similar script for automating telnet sessions.

As with the SSH script, create a text file using the naming convention of your choice and .sh for the extension.  In the file, type the following:

expect -c  ‘spawn telnet <ip-address>
expect “Username:”
send “<username>\r”
expect “password:”
send “<password>\r”

That is all you need to log in.  If you need to add an enable password, add the following to the script:

expect “>”
send “enable\r”
expect “password:”
send “<password>\r”
interact’

And there you have it.  If you want to log the output to a file, create a file to receive the output and then append the following to the last line:

| tee -a /path/filename.txt

Let me know if you find this helpful.

Read Full Post »

As a network engineer, I’ve discovered that one of the many great things about Apple’s OSX (possibly the best thing)  is that it is built on UNIX.  This means that many of the tools that come with standard UNIX distributions are included in OSX.  This, in turn, means that engineers don’t need to go buy additional tools for everyday tasks.  One of those tools is an SSL client.  OSX comes with OpenSSH already installed.  Many engineers probably know this, but they may be put off by the manual aspect of working with OpenSSH.  You have to know the IP address of the host you want to manage and type in the SSH command and all the credentials.  This is not necessary with clients like Putty, Secure CRT or TeraTerm.  Sessions can be created and saved along with credentials, making your management sessions only a mouse click away.  I’m here to tell you that you can do all of this from the terminal without having to reach for the mouse or touchpad.

You may already know that you can create a simple shell script that will launch OpenSSH and connect to a host just by executing the script.  This is done by:

1. Creating a file with a .sh suffix using either touch, or vi or your favorite text editing tool as follows: vi remote host.sh.

2. In your script, type the command you want to execute: ssh <username>@<ip_address>

3. Change the permissions on the script to make it executable: sudo chmod +x remotehost.sh

4. Execute the script using a ./, as in: ./remotehost.sh

And that’s it.  Now, if you want to automate the login process, you need only take advantage of the expect feature.  Expect is a powerful tool that can be used in a number of ways.  In this case, we will use it to launch the SSH client, tell it to expect a prompt, and how to respond to that prompt:

expect -c ‘spawn ssh <username>@<ip_address> ; expect password ; send “<your_password>\n” ; interact’ 

Voila, you’re done.  If you’re hyper organized, create a Sessions folder with as many sub-folders as you like and organize your scripts in any way you like.  You can launch your script from the terminal no matter where you are in the directory structure.  Just use the ./ and add the path to your scripts.  I’m sure there is a shortcut that will allow you to cut out the full path somehow, but I haven’t got to that yet.

So don’t go out and buy SecureCRT for Mac, and skip the freeware.  Just use the command above, or any variant that works for you, and save yourself the money and a few mouse clicks.  Leave a comment and let me know if you found this useful.

UPDATE: If you would like to log the output from your terminal session to a file, you need only append the following to your script:

| tee -a /<path>/<filename>

This will create a file and log the output automatically.  The -a will tell tee to append future text, so you will always have a running log.  I use the path that leads to the connection script and I give the file the same name as the script, with the exception of using a .log or .txt suffix.

I’m working on adding a timestamp so that I will know what changes were made when.  As soon as I figure out how to do that, I’ll post it here.

Read Full Post »

I have been putting together a complex lab in Dynamips/Dynagen and running into an error.  The error message on the dynamips console read something to the effect of dynamips error: 203-Bad number of parameters (1 with min/max=2/2).  I searched the web for an explanation, but could not find one.  Finally, I think I have stumbled across the root cause.  If you have undefined parameters in your .net file, dynamips will throw this error for each undefined parameter.  For example, the config and idlepc values were undefined in my .net file, as seen here:

[[router R1]]
s0/0 = R7 s0/0
s0/1 = R2 s0/1
f0/0 = SW1 1
f0/1 = SW2 1
model = 3745
console = 2001
config =
idlepc =
[[router R2]]
s0/0 = R7 s0/1
s0/2 = R6 s0/2
f0/0 = SW1 2
f0/1 = SW2 2
model = 3745
console = 2002
config =
idlepc =

This is the .net file that caused the errors.  I could have sworn that I had run .net files without defining all the parameters before, but it did not work this time around. As soon as I removed those parameters, the .net file worked perfectly, as follows:

[[router R1]]
s0/0 = R7 s0/0
s0/1 = R2 s0/1
f0/0 = SW1 1
f0/1 = SW2 1
model = 3745
console = 2001
[[router R2]]
s0/0 = R7 s0/1
s0/2 = R6 s0/2
f0/0 = SW1 2
f0/1 = SW2 2
model = 3745
console = 2002

So, if you happen to be running into that error, check your .net file to see if there are any undefined parameters, then try deleting them.  Let me know if this is a help to anyone.

Read Full Post »

The Nexus 7K and ISSU

We recently had to upgrade the image on our Cisco Nexus 7010s, the core switches in our data center.  They had been running images that were demoted to “deferred” status and were unstable.  As these are our core switches and we wanted to reduce the risk of downtime, we decided to follow the in-service-stateful-upgrade (ISSU) process.  For a while, everything seemed to go well.  We did not drop a single packet during the upgrade, that we saw.  We tested our core application environment, Internet access, WAN access; everything looked normal.  We patted ourselves on the back for a job well done and called it a night.

It turns out we missed one environment in our testing and, would’t you know it, that was the environment that went down.  “And why did it go down?” I hear you ask.  Let me tell you.

The ISSU process for the 7K is not as thorough an upgrade as some documentation might lead you to believe.  That is because the  previous version of code is not fully cleared during the upgrade process.  It takes a full reload of the CHASSIS to accomplish that.  Yes, that’s right, you must completely shut down all power to the chassis in order to clear the old image and any residual bugs.  And that was the problem we were facing.  One of the bugs in the old image had an impact on forwarding at layer 2 that only affected one of the environments in a VDC.  It was only after ten hours of troubleshooting with TAC that an engineer finally conveyed that information to us.  When the switches were reloaded, they began forwarding traffic normally.

So, if anyone reading this post is considering an ISSU on the 7K (and probably other platforms), you may want to keep this information in mind.

By the way:  I must say this was a huge disappointment for us.  We made a substantial investment in this platform to help ensure near continuous uptime in almost any situation.  The ISSU feature was one of the selling points.  Though I’m fairly satisfied with the 7K in general, Cisco has some work ahead of them before that particular feature is ready for prime time.

Read Full Post »

I was recently tasked with deploying two Cisco 5508 WLCs in a fault tolerant configuration for our enterprise WLAN.  I searched CCO for documentation that clearly explained configuring mobility groups and/or HA to provide fault tolerance, and a means by which to test the config, but found none (feel free to post a link to any doc with which you might be familiar).  I understood the concept behind mobility groups, but did not know how they would provide fault tolerance should a controller fail.  I could see that the mobility group was configured correctly, but none of the access points in our environment were registered to the secondary WLC.  My question, specifically, was “what happens to the access points when one of the WLCs in a mobility group fails?”

The answer turned out to be easy enough.  When a secondary WLC is configured on the HA tab of the wireless access point configuration section, an access point will register with the secondary WLC, but only when the primary fails.  That’s all there is to it.  To test this, all that is required is to swap the order of the primary and secondary controllers on the HA tab of the wireless access point configuration page.  As soon as that is done, the AP disappears from one WLC and appears on the other.  It takes about 60-90 seconds for the registration process to complete.  Provided that the configuration of the secondary matches the first (with the exception of the IP addresses of the interfaces), connectivity is restored to wireless users in less than two minutes.  Not the fastest failover in the world, but certainly within acceptable tolerances.

From all that I’ve read and seen so far, mobility groups are not necessary to provide protection from hardware failures.  The mobility groups provide a different type of high availability; the ability to roam through a wireless network without losing connectivity.  In other words, HA and mobility groups are complementary technologies necessary to provide true high availability in a wireless environment.  There is no overlap of functionality between the two.  I intend to keep testing to learn more about HA and mobility groups, and would be happy to learn from others out there working on similar projects.

Read Full Post »

Older Posts »

%d bloggers like this: