Trouble Shooting with ACL’s (part 2, naughty CISCO and there firewall)

OK so following on from here  Trouble shooting with ACL’s (part 1).

To recap for un-know reasons packets had begun to get lost on one of my firewalls, and by using a combination of ACL’s applied to interfaces, logging commands and debug commands, I had established that while icmp packets sent from the router to the inside network where coming back in the interface. they where then some how getting lost with out any notifications.

Fig 1

So the last think I had done was enable the “#debug ip packets 150” on the router where 150 was an access list to capture any traffic to or from the address. From this I was receiving (after a display of the packet going and coming) the following last line from the debug.

000801: Sep 13 12:40:35.452 UTC: pak 64A7D05C consumed in enqueue feature , packet consumed, CCE Firewall(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE.

This didn’t really help to much to start with, as any google searches on various parts of that got me no where. I then spent several moments looking at the firewall policy’s. I knew that the router set up as a fire wall, places any connected interfaces in to the “self” zone.  So although the interfaces and are servicing two manually configured zones (“inside2 and “management”), the ports them selves are actual part of the “self” zone. So I was looking for any policies between the “inside” and “self” and the “management” and “self” zones.

All I could find was a single policy that was assigned between the “inside” and “self” zone. However the direction for this was from the “inside” to “self“, that allowed ICMP and denied every thing else (so inside network can’t manage the router). So this still did not seem to explain the issue I was seeing, as the default policy unless configured is “self” is allowed to talk to any thing.

However after much searching on the internet I finally came across this.

"Although the router offers a default-allow policy between all zones and the self zone, if a policy is configured from any zone to the self zone, and no policy is configured from self to the router’s user-configurable interface-connected zones, all router-originated traffic encounters the connected-zone to self-zone policy on its return the router and is blocked. Thus, router-originated traffic must be inspected to allow its return to the self zone."

From Cisco’s documentation.

It goes on to describe how if a policy is applied in to “self“, then a policy must also be applied outgoing from self to the zone to allow return traffic to be inspected… So yes that little policy I had noticed above really was causing all the trouble. And guess how it got there?

Well it had originally been an ACL applied to the interface. But when I ran CISCO SDM to help configure Easy-VPN, it had asked to make changes to the fire wall to insure still worked. And created the policy for me and applied it.. Which is the reason for the title of the post. I don’t generally like to use the SDM, but for learning it is useful. However this just shows how important it is to check the configs first and insure you keep  record of exactly what it is doing in case problems arise.

Solution was simple, either remove the policy above and replace it as an ACL assigned to the interface, or others wise set up an out going policy from “self” to “inside“, to either allow all traffic and inspect (or just allow the traffic you want to go to self).

In my view you don’t want any traffic from “inside” to “self“, apart from ICMP. This allows you to check a user can see the DFGW, but prevents any management traffic, so stops any attacks on the router from users or compromised systems inside your network. (Oh if you use IP helper address for DHCP the router must also be able to see these through your policy).

But yes all working fine now and lots more learnt about fire wall policies. Been a slight distraction from my CCNP switch studies but these are still going well. Just 7 points to go over before the exam, all simple ones just want to go through configuring them once more. Wish me luck!!


Trouble shooting with ACL’s

We all know of ACL’s for use in restricting traffic when applied to an interface, and also for classing traffic such as when used in NAT to chose the ranges to apply NATing to. But they can also be very useful in trouble shooting you network, and the last few days brought this back to me.

It all started with what seemed like a simple problem. On one of my networks the DHCP helper function had stopped working, and clients could no longer get an IP address. However a quick check of he DHCP server and a glance over the config on the network devices and it all seemed fine.

Now the set up is quite simple, your standard basic router on a stick set up. With a CISCO 1841 as the router, which as well as working as the router also is set up as one of the network firewalls. With one interface pointing to the internet (not shown) and the other to the internal network.


We can imagen that the DHCP server is sitting in VLAN 200 and the clients that have stopped working are in VLAN 100. So what’s going on?

Well first move was to look at the DHCP logs on the server to see any sign of requests eing received. Nothing there suggesting the packets whegetting stopped before they gotthere.

Check the router config for the “ip-helper” command. This all looked fine and a quick ping from the router to the DHCP server shows that there is not issue with the router forwarding packets to it. Net step ping the Client PC from the router….. OK here’s an issue router can’t ping the Client? But the client can reach the internet through the router? And stranger still the Client CAN ping the router interface of

To bypass any other part of the network, I set up two SVI on vlan 100 and 200 on the switch directly connected to the router and checked the trunk was carrying both. Again the switch could ping both the interface on the router, but the router could only ping the IP address assigned to the SVI for vlan 200?

Well the first step was to work out if the router was indeed sending a packet out, as I mentioned the Router also acts as a fire wall so could a policy update be causing the issue?

Here is the first use for ACL’s in trouble shooting. Debug commands in cisco are very useful as we know, and one I have used often is the “debug ip packet detail”. But before you go typing it in to a router to test, be aware it will have a massive hit on the CPU and you will be over whelmed with information as the detail of every packet crossing your router is displayed to you.

Before you start debugging create an access list that will permit all the traffic you are interested in. In this case I only want to see traffic to and from, so logging on the the router create the access list.

ip acccess-list extended 150

permit ip any host

permit ip host any

Then you can run the debug command and only view the details about packets covered by this access list.

debug ip packets 150 detail

Enabling this on the Router and again pinging the address and the debug output show the packets sent out on vlan 100, and to be sure enabling the same debug on the switch and I could see the packets both received from the router and being sent back out the same vlan interface. Yet the router logs show no sign of packets getting dropped or even being received. Neither dose this debug show any sign of the packets this is not surprising as debugging IP packets shows packets that are crossing the control plane of the router and if an  ACL or the fire wall are blocking them they will not reach this.

So here is the next use for a ACL in trouble shooting. One of the first steps a packet takes when received on an interface is getting checked by any applied ACL. This is a reasonable step as for security reasons you want to drop any rogue packets ASAP.  So by creating adding the line “permit ip any any” to the end of the above ACL, and the command “log” to the first two line. I then applied this ACL to the interface in the inward direction.

Now repeating the ping to from the router and I see in the logs packets being transmitted and getting received. Now I know that the issue is with in the firewall policy’s on the router.

So yes ACL’s are not only great for security and for managing live data flows across the network. But they are also useful in trouble shooting, especially when used to filter outputs of show and debug commands to  useful information. And using the log function you can capture sporadic issues with out having to be logged on the whole time watching for it.


PS. There is also the “debug packet” command to capture traffic received on an interface, but I like the simplicity and logging ability of using an ACL.

Filtering the VLAN Traffic

So it ended up I decided to do a recap on VLAN access control lists (ACL’s) before I got back into Spanning Tree. I also covered Private VLAN’s tonight but will come back to them some other time for the blog.

Over the years I have had lots of dealing with port and router based ACL’s, but VLAN based ACL’s I only came across when I started studying for my CCNP. And I already have plans to use them to limit the traffic on some of our more sensitive network segments.

Now if you know you VACL set up, here is the point to stop reading, what follows is a run through of the config, with some description of the steps.

Still with me? OK lets get to it.

The first step in creating a VACL’s is in fact to create some “standard” ACL’s first, these will be used to classify what traffic is filtered once the VACL is applied. the VACL will accept two types of access lists as arguments IP and MAC, so lets set some up.

(config)#access-list 100 permit ip host any

(config)#mac access-list extended MAC-ACL
(config-ext-mac)#permit any host b7d4.5f6d.8e31

So two simple ACL’s created, now you can you the IP access list command and create named access lists as will if you wish.

So now we need to create the VACL and add these lists to it.

(config)#vlan access-map <name> 10
(config-map)#match ip address 100
(config-map)#action drop
(config-map)#vlan access-map <name> 20
(config-map)#match mac-address MAC-ACL
(config-map)#action drop
(config-map)#vlan access-map <name>30
(config-map)#action forward

Notice by default if a VACL is configured on a VLAN is a packet does not match the VACL it will be dropped. As we can see each section in the VACL has a sequence number, a match statement (can have more than one) and an action to take. In this set up any traffic that matches the two ACL’s we set up will be dropped. By adding a sequence with out any match statement and only an action, we have set up a “catch all”  situation, just like you may do with a “standard ACL when you enter “permit any any”.

So there we have it the VACL all set up and ready to go, now its just a case of applying it to a VLAN or two.

(config)#vlan filter <name> vlan-list 10

And there you have it, now any traffic passing across the switch on the configured VLAN’s will be subject to the statements in you VACL. I think there great for adding that extra layer of security to your network, and keeping traffic where it should be.

OK so not an exciting post tonight, but I will get back to STP tomorrow and I can tell you from past experience how not to configure it.

Night all and take care.