The mystery of Cisco 2960s and strange ARP cache

This all started when I introduced a new network monitoring tool on to the network, the tool was “cisco prime”, but before I say any more lets be clear the issue here has nothing to do with prime which is a great tool for managing cisco devices. I noticed that when pushing new IOS files and backing up switch configuration that some time they would seem to lose network connectivity. I was able to ping them and ssh to them from my desktop, but they would simple not speak to the prime server.

So lets start with the set up (ip address modified of course)

Prime server – a vmware guest , in vlan 1, ip address 10.10.224.98/21

my desktop – physical machine, in vlan 1 also, ip address 10.10.226.46/21

Switch – management interface in vlan 666, ip address 20.20.255.6/24

interface Vlan666
ip address 20.20.255.6 255.255.255.0
no ip proxy-arp
end
switch#sh ru int vlan 1
Building configuration…
Current configuration : 65 bytes
!
interface Vlan1
no ip address
no ip proxy-arp
shutdown
end

Router – 6506 with a live interface in both vlan 1 and 666 set as DFGW on clients.

So to start with I can ping every thing and every thing can ping every thing else, and on the switch with a show arp I see

switch#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 20.20.255.1 0 4403.a754.8300 ARPA Vlan666
Internet 20.20.255.6 – 7010.5c99.f241 ARPA Vlan666
switch#

So all looking good, I can see the switch IP address and that of the DFGW

switch#ping 10.10.224.98
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.224.98, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/6 ms

But then the ping stops working? and the switch can no longer contact the prime server, however I can still see it from my desktop? Logging back on to the switch I again look at the ARP cache..

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.224.99 2 0050.5683.61ce ARPA Vlan1
Internet 20.20.255.1 0 4403.a754.8300 ARPA Vlan666
Internet 20.20.255.6 – 7010.5c99.f241 ARPA Vlan666

What?? why has the switch learnt a mac address on vlan 1? I have two issues with it doing this, first Prime is still trying to contact it via the DFGW ( I can see this in a packet trace) so the switch is not seeing the packets coming in on vlan 1, second the interface for vlan 1 is shut down so there should be no ARP entries on it! to get it working I clear the ARP cache of this entry and all is good… well for a few days / hours and then it happens again, but the time between issues seems very random. Keeping an eye on it I see it happen with anther monitoring server, and then another? The one thing I notice is that all the times it happens it is always a server on vmware. Physical servers/appliance and desktops never seem to have this issue. This is the first piece of the puzzle, what does Vmware do different to other servers? They Migrate! And when they migrate between the physical hosts the vmware system sends a gratuitous arp on to the network to alert switches what port in the network to now find the server on? And some switches that have “ip arp gleaning” switched on which is the default hear this and place an entry in to there ARP table. Even though the switch had the interface vlan 1 shut down, it still passed vlan1 traffic through the switch at layer 2 and this seemed to be enough that it saw the ARP packet, and added the entry to its ARP table. Then of course the it try’s to use this entry for communication but as the interface is indeed shut it will not work!

A little bit of time with CISCO TAC and the solution was to disable IP arp gleaning on all the access switches, it might be useful for provisioning switches but as I found it can cause issues.

no ip arp gleaning tftp
no ip arp gleaning udp

The fact it was learning on a disabled interface is a bug and something CISCO are looking in to.

However that’s not quite the end of the story, disabling “IP ARP Gleaning” did not work! and this had us scratching out heads for a while, until one of the cisco engineers I was talking to noticed this..

Cisco Bug CSCun38166

“no ip arp gleaning tftp and udp” doesn’t work

CSCun38166

 Description
Symptom:
On 2960,when we configure “no ip arp gleaning tftp” and “no ip arp gleaning udp” then do
tftp use command”copy flash:teset tftp:”
It is expected following behavior:1.do not learn a IP address from other segment.
2.add a arp entry corresponding with HSRP virtual mac addressHowever in our case,it turns out like below:
1.generate a arp entry of IP address from other segment
2.add a arp entry corresponding with HSRP PHY mac address.

Conditions:
1.no ip arp gleaning tftp
OR
no ip arp gleaning udp
2.do tftp use following command
copy flash:teset tftp:

And looking at the version code, oh yes I would be running one of the affected versions. 15.0(2)SE6, a quick upgrade to version SE7 and all is good.

I am impressed though, hitting 2 cisco bugs in one issue.

And in the end it was prime that sorted it all out, a single click and it pushed the IOS update and the configuration for “no ip arp gleaning” to all 2960s affected devices, not going to ake this a post for plugging prime, but it does have its good points.

Notepad++ Editing Linux with SFTP.

Now if you haven’t tried Notepad++ get over to http://notepad-plus-plus.org/ and give it a go. It is once of the best text editors I have come across, not only is is light weight and gives you tabs for you files, it also has syntax highlighting for a good number of programming and scripting languages, can be run from a USB drive with out installing and On top of all that its a free download, so perfect in every respect.

Nopepad++ screen shot

Then there is the standard “compare” functions, ability to write macros, and a whole host tool for editing text.

But just a few days ago I found another function that I had not noticed before. Under the plugins one of the defaults is “Nppftp” clicking on this brings up a side panel that allows you to set up profiles for FTP and SFTP connections to remote servers.

Once you have set one up (SFTP for Linux ) and connected the remote file system will appear in a tree structure in this pane, allow you top open the files in Notepad++ edit and re-save them. And you get all the standard benefits of the application such as highlighting of code and copy paste between multiply files.

One thing to be careful of is the EOL settings, when creating new files, these defaults to windows format that will be miss read by the Linux system (look up EOL in google for more ifo on this). This can be changed either for an individual file under “Edit >> ELO Conversion >> Unix Format” on the menu bar. Or under “Settings >> preferences >> New Documents/….. ” to change the default.

So that’s one bonus point about Notepad++, but honestly even if this is not something you would use, give it a try, it has so many features over the basic notepad that comes with windows that there is something there for every one.

More Generating Test Data.

So Last time I mentioned you can generate traffice between two routers using TCP small Servers. This does work fine but there are some limitations, it can’t genetrate large amounts of traffic, it puts a high load on the CPU and it does not tell you much once it has completed.

A second method I came across is “TTCP” (Test TCP) which is avalible on many of the more recent IOS versions (11.2 upwards). This method not only gives you more control over the data that is sent, but also will provided you with infomation on how the trasfer of data went once it is complete. TTCP is also avalible for Windows and Linux/Unix, which means you can test between various end stations on the network.

It is all very simple to set up, simply at the “>” or “#” prompt type “TTCP”, and then follow the prompts. You will need to set up one router as the sender first, and then the second as the reciver. You can leave all the settings as default (you may want to reduce the “nbuf” setting as the default of 2048 can take some time to complete, espicaly on a slow link).

Once complete you will get an out put of time taken and bandwith achived among other stats. Again this is a very simple tool to give you a indication as to the state of a link, you can find more details in the below link to the CISCO site.

Cisco TTCP Document

SecureCRT sending commands to multiple sessions.

I came across this in secureCRT and thought I would share it.

When labing things up (and indeed on real networks), there are times when you need to send the same command to multiple devices. you can of course copy and paste between the sessions but what about if you want to past the exact same block of configuration to 20 devices, or just want to do something simple like save the running configuration on your devices in you lab before you close down?

Well SecureCRT has a nice little feature to do this, so before enabled secure CRT looks much like below, as you can see I have several tabs open.

Default SecureCRT Window

However by going in to the view menu up the top there is a option to enable the “chat window”, this will bring up an extra panel at the bottom of the screen. Then by right clicking in this new panel you can enable the “send chat to all tabs” option as shown below.

Chat window enabled

Now any command typed in the chat window will be sent to all devices. Commands typed in the main terminal are still only sent to a single device.

What would be even nicer is if you could highlight multiple tabs and have the commands only sent to those terminal sessions. At the moment it is an all or nothing solution, maybe I will go suggest it to them as an improvement for future versions :)

The more I use CRT the more I like it, written quite a few scripts for it now, if you know any VB script or Java you can pretty much do what ever you like as SecureCRT has a nice simple API in to it.

I am finally moving house this week, so after that should have more time to post on here, and will take some of the script I have and tidy them up and post them for people.

Take care

DevilWAH

Raspberry Pi

I am sure a lot of you have hear about these, I have been trying to get one since the day it was released but still in the queue. http://www.raspberrypi.org/

Response was overwhelming and took down the sites of distributes, and this is what it is all about!

While it may not look much this is a 700mhz ARM processor, can play back Video at HD, with USB, Ethernet, HDMI output + more all on a board the size of a credit card, and for the sum of £22 ($30ish).  Perfect for a home brew media server or web server. or indeed many other hobby projects.

But it gets better the whole purpose behind this is to encourage kids to get in to programming. There are a lot of kids around that can use computers but as computers get easier to use, less of them a delving behind the screens to see how they work and more importantly how to make them work. So later in the year they will be releasing them in to the schools and collages. pre installed with tools and languages for the kids to have a play on.

One thing that hasn’t been widely covered is that when you purchase one you will be able to chose to purchase an additional unit that will be donated on your behalf to a school or other good cause. I think its a great idea, and while I will be getting one or two to play with over the next few weeks I hope for my own enjoyment and leaning experience. I hope the developers do see there dream bear fruit and get taken up by the schools and colleges to encourage the next round of IT stars.

I would encourage any one who is thinking of purchasing one to remember that they are a charity, and while the have produced something that many of us will get enjoyment from. That where possible we should try to feed back and encourage there idea of getting them in every school and collage in the country.

As soon as I get my hands on one I will be updating this blog, who knows in a month or two it might well be running from one :)

DevilWAH

Using Syslog while Studying in GNS3 (or indeed and cisco Lab)

I have been getting back in to my studying a lot lately and one thing I have found is the need to use a lot of debug commands so I can watch what is happening during things like routing updates and neighbour formation. One thing I do find though is that I am forever having to turn debug on and off, forgetting to do one or the other, and when it is on it clutters up the screen a breaks up the config I am entering making it difficult to read back.

Which got me thinking, I have used syslog servers a lot in the past, so why not send all the debugging out put to a syslog server and turn of logging to the console? This way I can have all the debugs in one place, and keep the console of the devices tidy so I can see what I am doing.

Now if you are doing this through GNS3 you will need a cloud connection so your PC can talk to your GNS3 network. If you are not sure how to do this there are lots of videos and walk though on the net, however the one below is one of the best I have found, very clear and complete.

How-To: Using the Cloud in GNS3 to Provide Internet Access from Matthew on Vimeo.

So once you have your cloud set up you then need to set up a simple GNS3 topology, Here I have set up 4 routers running OSPF connected through a switch as I am looking at the DR and BDR election process.

I have given R1 and R2 F0/1 address 192.168.10.10 and 192.168.10.20, and the loopback adapter used by the cloud is 192.168.10.254. Once the routers are booted and connected to the cloud, check they can ping the loop-back address (you may need to disable your fire wall on the loop-back connection.)

then of course you will need a SFTP server, in windows there are two good free choises, for a realy simple server that can run with out install try, http://tftpd32.jounin.net/tftpd32.html simple but does all you need, just make sure you disable dhcp and other none necessaries services in the settings. For a more complete tool try http://kiwisyslog.com/, they have a free syslog server offering that allows filtering and more.

In either case set it up and insure it is listing on the loopback interface, in the case of TFTP32d this is simple a case of choosing the interface from the drop down list.

Finale we need to change the logging setting of R1 and R2 to direct debugging message to the syslog server and not to the console. Remember debug messages are level 7 so we need to set console logging to level 6 or lower and trap logging to level 7. the following code will do just this from global config mode.

#logging 192.168.10.254
#logging console 6
#logging trap 7

So now we can enable the debugging and reset the neighbour relation ships to see what it looks like.

From the console

So not much there apart from we see the neighbours bounce as I clear the OSPF process.

So how about on the syslog server?? 

So here are all our debug messages, for us to scroll through and review at our leisure, If you have something like Kiwicat syslog server you could filer them in to views, based on device that sent it, or text with in message, ect.

You need to make sure of course that you either have the device connected directly to the syslog server network, or it has a route to get there. Directly connected is always best of course as you will insure that as long as that interface on the device is up you will catch all messages. On real hardware simply use a spare switch or create a separate VLAN and do exactly the same thing.

I have found for large labs this works great, indeed for testing setups for clients its great as well. once you have insured the correct debugging is enabled you can walk though test scripts and plans, safe in the knowledge that you have a full detailed log of every thing that has happened.

Simple to set up and hopefully some of you will find it useful.

DevilWAH

Where is Microsoft Headed with Ballmer????

What ever people may think of Bill Gates, it is hard to argue against the fact that he was and still is an amazing business man. Taking Microsoft from its humble beginnings, and lets not argue about did he or didn’t he steal the idea. if he did he was not the first and all the others who claim he stole from them are hardly whiter than white them selves. To with out dispute the most well known and dominate OS producer we have yet seen.

Now after leaving Microsoft he has gone on to do more things, and again despite personal feeling you may have about him, he does more to further the causes of people in need than any other past or present director of the major IT industry leaders and not just in terms of money but also applying the skills he developed whilst at Microsoft.

Bill on mosquitos, malaria and education

Bill on energy: Innovating to zero!

I think we can say that Gates left Microsoft with a very firm footing after years of innovation and drive. So what of it’s future??

My personal view is that what Microsoft lost in Gates has not be replaced by Steve Ballmer. Where once they where pushing the boundaries and bringing new and exciting things to the IT world, suddenly they seem to be the ones playing catch up. Now while it is not unusual for a company to appear to slow while ramping up for the next “big thing”, in the case of Microsoft that “big thing” has been an awful long time in coming. Yes we have had windows 7, but that was hardly anything ground breaking, a good OS yes but nothing to write home about. Latley we have had huge publicity around Halo Reach, and again nothing denying its a good game, but still this is the 4th instalment, where is the innovation here?

Maybe even more worrying is Steve’s attitude to the competition. the following are two videos of his response to news of apples I-phone and Google Crome OS.

Laugh at the iphone.

Laugh at Google Crome OS.

Although we can forgive him for laughing at the iphone, as no one new how it was going to take off. However repeating the process with Google seems a little foolish. Neither Google nor indeed Apple are any longer small players in IT any longer, they have the backing, the following and plenty of money to be dipping there fingers in to lots of pies that previously only Microsoft where making.

So where is he taking Microsoft now? Windows 7 is just released, Server 2008 still quite young, and I haven’t heard about any plans for much development of the Xbox. What new and novel product will they launch. What is going to keep them on the map?

VMware have got on with Virtulisation, which seems to be the “next big thing”.

Google are so far ahead in terms of search and web they will be hard to catch,

Apple have the mobile market, (and if they falter Google with android is snapping at there heals)

Net-books and tablets are taken by the apple IPAD, blackberry’s playbook among others.

With only there desktop and server OS going for them, and as I said both a few years away until people will be ready to buy the next instalment. Has Microsoft missed the boat and starting on a slippery path to obsolation in the next 10  to 20 years? Will Steve actuly manage to take a hold of the company and point them back in the right direction? Or will he continue to simply laugh at the opposition in foolish hope. My advice to him would be if you are going to laugh, make dam sure that when it comes to crunch time you have the product to match.

I also noticed that his body language in the last clip, (Google) show him as being very uncomfortable, shifting on his chair and his gaze flicking around. Is this man really the person to lead?

Well just my thoughts, I think we are in a great time with they way things are developing, lots of major players all pushing each other. I think the next 10 year are going to show some great development in IT. I can’t wait to get involved in them.

DevilWAH

PS. I have added some more materials to the CCNP ROUTE course materials and a number of flash cards to the ROUTE pack.

Edit: and it seems I am not the only one seeing the cracks.. Microsoft staff survey.

And now for some thing completely different…. (VBscript to show logged on users)

Now let me start by saying I am no scripting guru, in fact apart from a few batch files and a C++ course many years ago I am a novice. However I have the ideas of how scripting / programming works and over the years have managed to throw to gather a few basic scripts.

I did not intend this blog to ever contain scripting (or at least not for a few years yet), but I came across this in work and thought others might find it useful. Now before we start this is a rehash of others work, links to there sites are at the bottom of the post and I would like to thank them for posting there work to the public.

The background to this is that I am currently working on a project to implement grid computing to process long running mathematical modelling jobs. The software CONDOR is a job scheduling application running on a server. The mathematician break there jobs up in to small chunks (maybe severely hundred jobs in total), and the CONDOR server finds available desktop PC’s they are inactive and sends the job to run on them. IF you have used “folding at home” or “Boinc” you will have been the client in the same kind of system.

Now while CONDOR can determine the user activity in terms of keyboard/mouse activity and CPU usage to decided if a computer is free to run jobs, there is no built in way to use the logged in status of the PC to control jobs. The idea is we want jobs to run when users log out of there PC’s, but stop and be prevented from running once a user logs in. To do this I needed to create a script that could return a “True” or “False” result to the question “is any one logged in to this computer”.

My first Idea was PSloggedon from Sysinternals, however this does not return a true false value and would require wrapping up in a second script to parse the output. So I decided to search around for a VB script that could do it all in one step.

This lead me to the following code,

strComputer = "servername"   ' " use "." for local computer 

Set objWMI = GetObject("winmgmts:" _ 
              & "{impersonationLevel=impersonate}!\\" _ 
              & strComputer & "\root\cimv2") 

Set colSessions = objWMI.ExecQuery _ 
    ("Select * from Win32_LogonSession Where LogonType = 10") 

If colSessions.Count = 0 Then 
   Wscript.Echo "No interactive users found" 
Else 
   WScript.Echo "RDP Sessions:"
   For Each objSession in colSessions 
     
     Set colList = objWMI.ExecQuery("Associators of " _ 
         & "{Win32_LogonSession.LogonId=" & objSession.LogonId & "} " _ 
         & "Where AssocClass=Win32_LoggedOnUser Role=Dependent" ) 
     For Each objItem in colList 
       WScript.Echo "Username: " & objItem.Name & " FullName: " & objItem.FullName 
     Next 
   Next 
End If 

Not exactly what I need but at least it will find and display the logged in users, so seems a good starting place, and easy to tidy up.

First of all we don’t want it to tell us a list of users, just return a true of false statement. This is just a case of replacing Wscript.Echo "No interactive users found" with Wscript.Echo "False" , and replacing the whole of the code under the else statment with simply Wscript.Echo "True".

Secondly this script uses the statment “LogonType = 10″ from windowsecurity.com we can see that this will give us remote connections, as we want local logged on users we need to replace that with “LogonType = 2″. Ending up with the revised code below.


strComputer = "."   ' " use "." for local computer 
Set objWMI = GetObject("winmgmts:" _ 
           & "{impersonationLevel=impersonate}!\\" _ 
           & strComputer & "\root\cimv2")
Set colSessions = objWMI.ExecQuery _ 
  ("Select * from Win32_LogonSession Where LogonType = 2") 
     
If colSessions.Count = 0 Then     
  Wscript.Echo "FALSE" 
Else
  WScript.Echo "True"
  
End If

Oh and notice we needed to replace the servername with “.” as suggested. Saving this as a .vbs file and running it and a pop up box will display “True” (after all you are logged on to the PC 😉 )

And that was my script done..

Apart from the fact for it to work in CONDOR I needed it to be able to return out put to stdout, which requires it to be running under Cscript.exe (cscrfipt is command line, Wsccript will interact with windows, in this script one will out put to the command line, one will pop up the same out put but in a small window). Now there are several ways to do this, simple run it from the command line by calling it using cscript.exe c:\csccript.exe script.vbs , or you can wrap it up in a batch file. Howevver I came across a nice little trick to enable the script to check if it is run as cscript and if not switch to csccript.


Public objShell : Set objShell = CreateObject("WScript.Shell")
If Instr(1, WScript.FullName, "CScript", vbTextCompare) = 0 Then
    objShell.Run "cscript " & chr(34) & WScript.ScriptFullName & chr(34) & " //Nologo" & chr(34), 0, False
 
    WScript.Quit
End If

Simply pasting this code above the main script, (And I also updated the Wscript.echo to be

Set objStdOut = WScript.StdOut
  objStdOut.Write "UserLoggedIn = True"  

for CONDOR reasons, and we end up with the complete code below.


Public objShell : Set objShell = CreateObject("WScript.Shell")
If Instr(1, WScript.FullName, "CScript", vbTextCompare) = 0 Then
    objShell.Run "cscript " & chr(34) & WScript.ScriptFullName & chr(34) & " //Nologo" & chr(34), 0, False
 
    WScript.Quit
End If
strComputer = "."   ' " use "." for local computer 
Set objWMI = GetObject("winmgmts:" _ 
           & "{impersonationLevel=impersonate}!\\" _ 
           & strComputer & "\root\cimv2")
set colSessions = objWMI.ExecQuery _ 
  ("Select * from Win32_LogonSession Where LogonType = 2") 
     
If colSessions.Count = 0 Then     
  Set objStdOut = WScript.StdOut
  objStdOut.Write "UserLoggedIn = False"
   
  
Else
  Set objStdOut = WScript.StdOut
  objStdOut.Write "UserLoggedIn = True"  
End If

Now run that and how ever you start it, if you don’t use Cscript.exe it will resubmit it’s self using Cscript.exe and output the result to stdout.

Now by playing around with the logonType and other bits of this code we could make it quite useful, it will return the logged on status of a remote machine if you replace the “.” with a valid remote computer name.

Well I hope that may be useful to some, and my thanks goes to those who came before me and provided the foundations, Please visit there sites to see many other great scripting examples.

VBScript – How to show logged on users?

Force Cscripe

DevilWAH

CCNP Route

Well I Official started my ROUTE studies last night. :) Back when I started this blog I was already well in to my study of the SWITCH material, and like wise creating Flashcard pack I made up with Anki was a rushed job as I went over the material. So I know neither the blog or the cards are really suitable for any one but me to use for studying.However for the ROUTE I want to do things correctly, even if this means it take me a little longer to achieve. I hope (and please let me know if I don’t) to post around two or three Lab based posts for each of the major area’s of the ROUTE material. Including EIGRP, OSPF, IPv4, BGP, Redistribution and IPv6. These will be tagged in category with a quick link on the menu bar for easy access. I also will be attempting to make a much better set of Flash cards this time round, that are more suitable for using as revision / memory aid. I hope combined these will make useful addition materials to people studying for there ROUTE exam.

Of course along side the ROUTE posts, I will also be trying to keep up with other interesting things I come across both inside and outside of IT. We shall have to see how things go. But I hope over the next few months I can really start to fill out this blog and produce some thing that other people will find useful. After all half the fun in learning things is to be able to pass it on to others.

On a side note I came across this today, EIGRP disababled by default. This is interesting for two reasons. The first of course being that it is nice to see CISCO have listened and have set the default to what 95% of us have to manualy set it to any way :). And secondly this comes from www.tekcert.com. Which is a new blog that Jeremy Cioara of CBT nuggets Fame is now posting on along with a fellow IT tec guru Adam. Another site to add to the Favourite bar, don’t worry if you forget it will be in the links above. 😉

DevilWAH