Upgrading Redundant Supervisors on 6500’s

I need to know how to upgrade the IOS on CISCO 6500’s, So here is the outline along with a download of the official PDF from CISCO.

First confirm what IOS image is running, which supervisor is running as active and the redundancy mode that is active. This can be achieved with the following commands.

Router#show version
Router#show module
Router#show redundancy 

You should now know the current IOS version and the module numbers of the supervisor units and which one is active.
Now we need to set up the boot variable (so the correct IOS boots) and copy the new IOS over if necessary.

Router#show bootvar  (Find out the current IOS booting)

Router#copy start tftp (copy the current start up configuration to TFTP for safe keeping)

Router#copy tftp disk0:          (Copy the new IOS to the two supervisors)
Router#copy tftp slavedisk0:

Router#no boot system disk0:old_IOS.bin
Router#boot system disk0:new_IOS.bin   (set up the new boot variable on the active supervisor)

Router#copy run start  (syncs running configuration on both supervisor units)

Now the boot variables are set up and the new IOS images are on the supervisors we can upgrade each in turn, starting with the one currently set as the standby.

Router#hw-module module  reset  (chose standby supervisor to reset)

Standby supervisor will now reboot and restart running the new IOS image. During this time the redundancy mode will fall back to RPR mode due to the mismatch in IOS images running. You can see this by running the “show” commands above.

Once the standby is fully back up and running you can now update the active module, this is done by forcing the standby to become active, this will result in a reload of the current active supervisor.

Router#redundancy force-switchover (switches standby to become active and reloads current active)

(you will need to swap console cables over to the standby supervisor as only the currently active one can be consoled in to.)

Again use the above “show” commands to check the modules have reloaded correctly, both supervisors are running the same IOS version, and the correct redundancy method. Now the original active router will be the standby. If you wish you apply the last command above to force a second switch over back to the original.

And there it is, upgrade of IOS with out taking the switch off line. Although it is suggested it is done out of hours as there may be some minor traffic interruptions during the switch over.

Here is the CISCO PDF, with more detail and outputs of the commands..


CCNP ROUTE (Part 9 EIGRP Authentication)

Seeing as we just finished up a simple EIGRP lab, it seems a good opportunity to add one more simple thing in to the mix.

At the moment any one could in theory add a router in to the network, sniff for packets to determined the AS number we are running EIGRP on, and start advertising routes and forming neighbours. This is not something we want to happen, even if not a malicious attack a rogue router sending EIGRP hellos and updates could cause havoc with a network.

So like all good network administrators it is important to secure EIGRP against such happenings. This is achieved in EIGRP by means of md5 authentication and key-chains.

The theory works some thing like this. All routers must be in time with each other, if possible a time protocol such as NTP should be used, but you could also set the clocks manually (just remember to redo this after a reboot as the router will lose its time). One they are in sync we can set up the key-chains. Each key chain has a number, time frame in which it will be sent and a time frame in which it will be accepted, along with the actual key value its self.

For a key to be accepted as valid by a router, when it receives it, the key-chain number and the key value must match on both devices, and it must be revived with in the accepted time frame. Below is a generic template for setting up a key-chain.

Router(config)#key chain

The idea is that you may use one key each month for example, with the accept and send time of the next key in the chain over lapping with the last slightly (if you have NTP the over lap can be a matter of seconds due to the increased sync of the routers clocks), to insure the neighbours do not get dropped during the change over of keys.

Once the keys have been set up you apply them to the interface which is sending out EIGRP updates as below.

interface FastEthernet0/0
 ip authentication mode eigrp <AS> md5
 ip authentication key-chain eigrp <AS> <keychain name>

I have set this up in the GNS3 lab here.. to get it working you will need to set the time on router 1 to 00:00:00 24th october 2010 (#clock set 00:00:00 24 october 2010), and then on router 2 remove and re-add the NTP server. This will sync up the clocks to the correct time for the configured key chains. You should then see the neighbours come up. Running a #Debug eigrp packets, and you will see the hellos and updated getting sent with the md5 authentication.


Removing old keys from the SSH known host file

When connecting to a device via SSH using Linux (Unbuntu), you may come across the following error message.

The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /xxxx/xxxx/.ssh/known_hosts to get rid of this message.
Offending key in /xxxx/xxxx/.ssh/known_hosts:15
RSA host key for has changed and you have requested strict checking.
Host key verification failed.
lost connection

This happens when the device you are connecting to has changed it’s SSH keys, but the client you are connecting from has the original key saved. By default Linux does not update these changes (assuming a security exploit) and will just drop the connection .

Stored SSH keys are by default saved in the /home/[user]/.ssh/known_host file. So to connect to the device you need to update this file.

You can achieve this a number of ways, including using the information above and manually editing the file to remove the offending key. However it can be difficult to work out what entry in the file you need to remove.

However I came across this the other day after having use the manual method for ages. In the error message it give the line number of the key you need to change. In the case above it would be 15.

(Offending key in /xxxx/xxxx/.ssh/known_hosts:15)

You can use the following command to delete that line with out having to manually open and edit the file.

sed -i 15d ~/.ssh/known_hosts

or use

ssh-keygen -R

Both will delete the line from the known_host file that relate to the error message above. (might need a SUDO, or run as root)

Now the next time you connect to the host device, your system will behave as if it is the first time it has seen the host and ask if you want to add the key in.

It is also possible to disable key checking, however I would not suggest this as it is a security risk.


CISCO commands

Just lately while cleaning up things at work, and on the web I have come across some CISCO commands that are usefully but often over looked, or forgotten. So I thought I would write them up here and attach them to the Tips and Tricks page so I would always have them to hand. IT might start of a small list but I hope to increase it gradually as I remember/find more. Think of it as a work in progress which you can find here.

I also though as well as the useful ones I would create a common list as well. These are things like the #show IP interface brief, and show Interface status. Again a work in progress and found under the tips and tricks page. If you have any ides suggestions for things that should be included let me know.

I don’t want to have a list of every command on CISCO, but the common ones we all use daily and take for granted. Or ones that are not quite so well known but very useful nevertheless

As I say the lists are no way completed, but I have put up the pages so I can start adding thing on, as and when I think of them.


CCNP SWITCH (retake)

Well one more update. I have just checked my email and it seems that CISCO have responded to my comments about the SWITCH exam, and provided me with a re-sit voucher!

I think this says something about the quality of this exam as this is the second voucher I have received. The first was after the exam crashed on me. So hats of to CISCO for that. But this time no excuses, I still don’t think the course material if fit for its purpose, and does not really prepare you for the exam. But having sat it twice, I can’t claim I don’t know the focus, or where my weaknesses lay.

So my intention is to start my new job, get hold of some lab equipment and do some hard studying on the topics I need to. Then I think I should be able to pass this no problem. I did notice the First chapter of the ROUTE foundation seems to cover some of the planning topics nicely so I might flick through it a little more to see if there are any other relevant bits.


Applying a configuration to a CISCO device using xmodem.

Last week at work now, so been rushing to get things sorted out. I have still been doing a bit of study and planing some more ROUTE posts, but with a broken down car and house sale looking like its falling through haven’t had time to do any actually real posting.

However I came across something today, that I have known about for a while but never really used much. One of the things every one seems to love about CISCO is the fact you can simple copy and past configurations in to the terminal emulator window.  And this is indeed great. set up one interface, copy the config to notepad, update it as you wish and past it back in… A real time saver and why we all love CISCO more than Microsoft ;).

In the past this is also how I have always copied backed up configurations on to a new switch/router. Simply open the saved config in notepad. Ctrl-A to select it all, copy and paste to the device. However I was doing this today and hit an issues. With really large configuration files (500+ lines of configuration), I was watching the console windows and could see it was skipping some of the configuration when doing this connected through the serial port. I could see that while things like VLAN’s where being created and the device was pausing, the following lines would some times get lost or corrupted. Now while if you only have a small size configuration file this is not an issue as it is quite easy to check, hundreds of lines become very hard to validate.

I found the best way around this problem was to set up the device with an IP address, put it on a limited access network that has a TFTP sever and copy over the configuration file, either to the startup-config or running-config. This works fine but it is a bit of a hassle going to all that trouble and it means you have to connect the switch to the network, so you have to be very careful with things like VTP and spanning tree. What I really wanted was a way to send the configuration file through the console port.

This made me think of how to recover a corrupted IOS image (which you can fine in the tips and tricks link above). Where boot the device in to ROMmon mode, and then copy the IOS over using the xmodem protocol. Almost all the mainstream terminual emulators have this built in, and while for recovering the IOS you need to increase  the baud speed of the console port to speed up the copying process, as the configuration file is only 20-30kb max for most people, the standard speed will move that across in a few seconds.

So then it is just a case of knowing the command to achieve the goal, and I was happy to see it is as simple as it should be. On the device simple type the following from the enable prompt.

router#copy xmodem: startup-config

That’s it, no file names or anything, the device will now wait to receive the file(if you do not start the transfer within a few minutes the device will time out waiting). Then in your terminal emulation program start the transfer. In teraterm it is under the file menu, while secure CRT has a whole menu structure dedicated to various methods to transfer files. Simple chose the xmodem protocol (I found selecting the 1K option was more reliable), and browse to the configuration file, and away it goes. A few moments later the configuration will be on the device (#show Flash: to confirm), and a reboot will have it all up and running.

To me this is a far more reliable way of copying large configurations across, and allows you to easily set up the device from any client, this can be very useful if you are out on site and don’t have access to a limited access network, or the TFTP server to use to copy the files via TFTP or FTP using the network.


PS. Some older routers don’t seem to like you copying from xmodem to nvram, or require you to give a source file name. But you can still achieve the same by copying the file to Flash: .

PPS. Although I prefer the xmodem method, you can improve the reliability of the copy/paste method by increasing the line/character delay in you terminal emulation program. A 5msec delay per character seems to help, although with a 1000+ lines of configuration you may get from a complex configuration, you may find the paste takes a little time, and you may still get errors.

New Job,

Well the reason I haven’t been posting as much, and have moved the blog to an on-line host. Is that in a few weeks I will be starting a new role working for Dimension Data.

Seems like the hard work over the last few years is paying off now, and really excited to be moving to such a well respected company in the networking field. Hopefully with a bit of hard work I will be achieving many things over the coming years. I think this will be a great opportunity, and goes to show that working hard, and taking your time to look for the right openings does pay back in the end.

So my plans for the next moth or two go some thing like this..

Move house, buy new car, carry on raising the devil child (I mean my gorgeous daughter), start the new job, and carry on studying for my CCNP. I have decided that until I am settled in the new house and new job I am not going to think about taking the exams. But hopefully by Christmas or early next year, I will be ready to take both the switch and route exam close together, with trouble shoot following shortly after.

So if posts seems slow for the next month or two that’s the reasons. I have a list of posts I want to cover, including about 3 or 4 more on EIGRP before moving on.

Hopefully I will get a few more completed over the weekend.


Tagging the Oceans

As a young boy I was fascinated by wildlife and had piles of videos that I would watch over and over. I always believed I would end up as a wildlife researcher tracking cheaters across the plains with a faithful rescued tiger as my companion. Well it did not turn out quite that way, but I am still in love with wildlife. You may believe in a creator or you may believe in science. But what we must all see is the wonder and complexity of nature and our existence.

One of humans worst qualities is that the further removed we are from something the less we seem to care. Weather this is removed in the sense of distance or time. And with technology of the 20st century putting more and more distance between us and nature, it is unsurprising that we take it for granted and don’t always treat it with the respect it deserves.

But what is great to see is how technology in the 21st century is starting to bring us back in touch with nature, to allow us to see the complex interactions and how us as humans fit in to the picture. People talk about the need to go back to simpler times to “save” the planet.. But it is not back we need to go but forward, to use technology and develop it so we understand and have the tools to make the changes we need, so we can both live as 21st century humans, while looking after the planet that sustained us, including the other living organisms that share it with us.

today I came across a video that shows how we are developing these technologies and how they can be used to reach out and make people interested. Because once people are interested the “distance” decreases, and as that goes down the more we will care.

Barbara Block: Tagging tuna in the deep ocean