Fortigate Site to Site VPN

Fortigate Site to Site VPN

I would like to add some more simple configuration, this time VPN Site to Site between a Fortigate and a CheckPoint firewalls, 

For an ease of access I'll split this to two parts, so let's start with the easier, the Forti :

So, first thing – create an address object for the LAN of the NGX

With that out of the way create Phase 1, I used DES for encryption and MD5 for authentication, feel free to change it as you wish, I matched the keylife to Checkpoint's default settings

Now, phase 2 make sure to specify the source and destination for the tunnel – may cause problems if it's set to any.

And last thing to do is a rule to allow the actual tunnel,

 from internal to external action "ipsec" and choose the tunnel.

And that's it for the fortigate side.

Firewall has a 3-way connection establishment handshake

Firewall has a 3-way connection establishment handshake, and a pair of 2-way close handshakes.

The PIX, like many other professional firewalls, is a stateful firewall. Essentially, this means that the PIX actively monitors all connection oriented traffic (mainly TCP), and builds a connection table (or database) that it uses to keep track of active sessions. Depending on where you set your buffer logging, you can see both the building and teardown of TCP connections that traverse the firewall. For example, you may have a user opening up an FTP session from the inside of your network out to the internet somewhere. In this case, you will see the PIX log a message like:

Built outbound tcp connection for IP address/source port to IP Address/21... (word order not exact, but you get the idea)

This log message indicates the start of the 3-way handshake process (it does not mean that it worked, however...)

You can now do a "show conn" on the PIX command line, and you should see a line in there for this particular TCP session. The destination port would be 21, since this is an FTP. Also, there are some flags at the end of the line. If the connection was successful, you will see a capital U (for Up, meaning the 3-way handshake was successful). If you have any a's or s's (small or upper case), then something didn't work, and you are most likely in a SYN-Sent state, waiting for the remote Ack-SYN. That's a whole other conversation though...

Back to the topic - let's say that this connection failed, perhaps the remote FTP server is down. The PIX has some Global connection timeout values. The default value for a half-open, (embryonic) connection is 2 minutes. If the remote server doesn't send the ACK-SYN back to the initial connection establishment, then the PIX will clear the connection from it's table, and log a connection teardown message.

There are about 3 or 4 types of teardown messages that can be logged (if memory serves me). At the very end of the log entry there is a reason. Some of them are:
FINs
TCP-Reset-O
TCP-Reset-I
Conn-Timeout
Deny

There may be others, but you'd have to check the PIX docs to see (if it's even in there...)

I know you're going to ask, so here is what each one means:

FINs - this represents a normal close - in other words the two systems talking to each other mutually agreed to close the session. This is normal behavior.

TCP-Reset-O (or I): Another way that a TCP session can be closed is for one of the parties to send a TCP-Reset. This is essentially a packet that tells the other side that "I'm hanging up on you". It's the TCP/IP equivalent to when you really piss off your girlfriend, and she slams down the phone in your ear. The difference between the I and the O has to do with the interface the reset was heard from. If it's an I, it means inside - but not necessarily the actual inside interface... it's the interface that has the higher security value of the two interfaces being used to build the connection in question. If it's an O, then it came from the lower security interface...

Conn-Timeout - just what you think - this refers back to the scenario I mentioned above with the FTP session and the remote server being down. After that 2 minute timer expires, the PIX will log the teardown with this as the reason. You will also see this if the PIX active connection idle timer expires too (default is 1 hour for that). And lastly, you can see this if the two systems fail to exchange the TCP-FIN process properly (normal close process). The normal close process, by the way, is simply a FIN packet from the system who is initiating the close request, and the other side is supposed to ACK the FIN (that's one pair). Now the session is 'half-closed'. Once the remote side is ready to close the connection too, it sends a FIN, and the other side should ACK it. Now it's fully closed. If part of this pair of 2-way handshake fail, then the connection according to the PIX is half closed (you'll see F flags in the show conn command output). The half-closed timer I believe is 5 minutes. If the connection doesn't close normally after that 5 minute counter expires, the PIX clears the connection from it's table and logs this conn-timeout reason in the teardown.

Lastly, there is 'Deny'. This one is seldomly observed. Generally, you would see it if a connection that was being application inspected (like FTP or http, for example), violated the fixup rulebase. The PIX would clear the connection and log the deny as the reason in the teardown message. I've seen really old FTP servers do this.

Anyway - I hope this clears things up a bit for you. If you are using the PIX for your business and you have a lot of traffic going through it, I highly recommend you learn more about the connection table, especially the flags at the end. The PIX can be a very powerful troubleshooting tool when the application clowns are telling you there is a network problem (the story of my life!). Between the logs and the connection flags, you can pretty much see what's going on at any given moment in time - excellent for real time troubleshooting - and tossing the problem back in their laps!

Juniper (SRX) Firewall Commands

Juniper SRX Firewalls

**********************
run = used in configure mode to use operational mode commands

//Show Routes
show route brief
show route best x.x.x.x
set routing-options static route 10.2.2.0/24 next-hop 10.1.1.254
//Forwarding Table
run show route forwarding-table destination x.x.x.x/24

//TraceOptions settings
root@fw1# show security flow | display set
set security flow traceoptions file matt_trace
set security flow traceoptions file files 3
set security flow traceoptions file size 100000
set security flow traceoptions flag basic-datapath
set security flow traceoptions packet-filter f0 source-prefix 10.0.0.1/32 destination-prefix 200.1.2.3/32
set security flow traceoptions packet-filter f1 source-prefix 10.0.0.1/32 destination-prefix 200.1.2.3/32
activate security flow traceoptions
commit
monitor start matt_trace
monitor list

!! Kill the capture
monitor stop
clear log             !! Clear the log file
delete security flow traceoptions
commit
file delete

//Show Traceoptions
show security flow session source-prefix 10.124.80.42 destination-prefix 117.1.1.25
start shell

egrep ‘matched filter|(ge|fe|reth)-.*->.*|session found|create session|dst_xlate|routed|search|denied|src_xlate|outgoing phy if’ /var/log/matt_trace | sed -e ‘s/.*RT://g’ | sed -e ‘s/tcp, flag 2 syn/–TCP SYN–/g’ | sed -e ‘s/tcp, flag 12 syn ack/–TCP SYN\/ACK–/g’ | sed -e ‘s/tcp, flag 10/–TCP ACK–/g’ | sed -e ‘s/tcp, flag 4 rst/–TCP RST–/g’ | sed -e ‘s/tcp, flag 14 rst/–TCP RST\/ACK–/g’ | sed -e ‘s/tcp, flag 18/–TCP PUSH\/ACK–/g’ | sed -e ‘s/tcp, flag 11 fin/–TCP FIN\/ACK–/g’ | sed -e ‘s/tcp, flag 5/–TCP FIN\/RST–/g’ | sed -e ‘s/icmp, (0\/0)/–ICMP Echo Reply–/g’ | sed -e ‘s/icmp, (8\/0)/–ICMP Echo Request–/g’ | sed -e ‘s/icmp, (3\/0)/–ICMP Destination Unreachable–/g’ | sed -e ‘s/icmp, (11\/0)/–ICMP Time Exceeded–/g’ | awk ‘/matched/ {print “\n\t\t\t=== PACKET START ===”}; {print};’

//Show Sessions
run show security flow session destination-prefix x.x.x.x

//Match Policy
run show security match-policies from-zone zonea to-zone zoneb source-ip x.x.x.x destination-ip x.x.x.x protocol tcp source-port 1024 destination-port xx

//Check for Block Group
show security policies from-zone untrust to-zone trust | display set | grep deny

//Find Syntax for an Existing Command
show | display set | xxxxxxxxx

//VPN Troubleshooting
show security ike security-associations [index ] [detail]
show security ipsec security-associations [index ] [detail]
show security ipsec statistics [index ]

//VPN
//Set proxy ID’s for a route based tunnel
set security ipsec vpn vpn-name ike proxy-identity local 10.0.0.0/8 remote 192.168.1.0/24 service any

//Packet Capture
set security datapath-debug capture-file my-capture
set security datapath-debug capture-file format pcap
set security datapath-debug capture-file size 1m
set security datapath-debug capture-file files 5
set security datapath-debug maximum-capture-size 400
set security datapath-debug action-profile do-capture event np-ingress packet-dump
set security datapath-debug packet-filter my-filter action-profile do-capture
set security datapath-debug packet-filter my-filter source-prefix 1.2.3.4/32

//Super SRX Packet Capture Filter
egrep ‘matched filter|(ge|fe|reth ) -.*- > .*|session found|Session \(id|session id|create|dst_nat|chose interface|dst_xlate|routed|search|denied|src_xlate|dip id|outgoing phy if|route to|DEST|post’ /var/log/mchtrace | uniq | sed -e ‘s/.*RT://g’ | awk ‘/matched/ {print “\n\t\t\t=== PACKET START ===”} ; {print} ;’ | awk ‘/^$/ {print “\t\t\t=== PACKET END ===”}; {print};’ ; echo | awk ‘/^$/ {print “\t\t\t=== PACKET END ===”}; {print};’

// Policy commands

show | display set (shows policy)
set system syslog
set security log
set interfaces ge-0/0/3 gigether-options auto-negotation (redundant-parent)
set security policies from-zone xxx to-zone xxx policy policy_name match
set security zones security-zone untrust address-book address
set security nat source rule-set zone-to-zone rule rule-source-nat match source-address 10.0.0.0
set routing-instances
set applications

set security ike proposal
set security ike policy
set security ike gateway
set security ipsec proposal
set security ipsec policy
set security ipsec vpn

show|compare
commit check
commit comments ticket#2222 and-quit

set security policies from-zone dmz to-zone trust policy 12 match source-address h_10.124.0.1 destination-address h_1.2.3.4 application tcp_22
set security policies from-zone dmz to-zone trust policy 12 then permit
set security policies from-zone dmz to-zone trust policy 12 then log session-init session-close

+         match {
+             source-address h_10.124.0.1;
+             destination-address h_1.2.3.4;
+             application tcp_22;
+         }
+         then {
+             permit;
+             log {
+                 session-init;
+                 session-close;
+             }
+         }
+     }

Various:

show system uptime	Uptime
show version	Version of platform (host/model)
show chassis firmware	Firmware loaded on FPCs
show system software detail
show chassis routing-engine	CPU, Memory for Routing-Engine
show chassis fan	Speed and status of fans
show chassis environment	Temperature status of components
show chassis hardware detail	Hardware inventory (backplane)
show system core-dumps	Core-dumps
show system alarms	System alarms
show chassis alarms	Alarms for hardware and chassis
show system boot-messages	Logs from boot sequence
show log chassisd	Logs for SRX chassis (Cards)
show log messages	Recent system messages
show configuration security log	Syslog configuration
show system buffers	Utilization of memory buffers
show system virtual-memory	Virtual memory utilization
show system processes	Processes running on system
show security idp memory	IDP memory statistics
show security monitoring performance session	Session counts on each FPC

Palo Alto Firewall Commands (4.0 – 5.0)

Palo Alto

// Base Help

lab@infostruction> ?
clear Clear runtime parameters
configure Manipulate software configuration information
debug Debug and diagnose
exit Exit this session
grep Searches file for lines containing a pattern match
less Examine debug file content
ping Ping hosts and networks
quit Exit this session
request Make system-level requests
scp Use ssh to copy file to another host
set Set operational parameters
show Show operational parameters
ssh Start a secure shell to another host
tail Print the last 10 lines of debug file content
username@hostname>

lab@infostruction# set ?
> deviceconfig deviceconfig
> mgt-config mgt-config
> network network configuration
> shared shared
> vsys vsys

// Device configuration and RMA
default login & password is admin/admin
scp import configuration 22 user@host:/path/to/file
set deviceconfig system hostname ip-address netmask default-gateway dns-setting server primary
set password

// Administration
check pending-changes **Check for uncommitted changes
show config
show config-locks
request commit-lock remove admin
save/load (saved config) -> set (candidate config) -> commit (active config)
edit, up, top **Change hierarchy

// Privilege Levels
superreader Has complete read-only access to the firewall.
vsysadmin Has full access to a selected virtual system on the firewall.
vsysreader Has read-only access to a selected virtual system on the firewall.
deviceadmin Has full access to a selected device, except for defining new accounts or virtual systems.
devicereader Has read-only access to a selected device.

// Health Commands
show counter global | match drop
show interface ethernetX/X
show system state filter * | match over **Packet overruns.
show session info **High concurrent sessions
show running resource-monitor **CPU (app-id, decoders, session setup teardown)
debug dataplane pool statistics **Work Queue and Segment reassembly.
show counter global filter aspect resource **Resource counters, TCP window information.
show system statistics **System Counters
show system info **Show system information
show network interface ethernet **Show Interface configuration
show arp
netstat all

// Failover – High Availability
request high-availability state suspend                     // Fail the master firewall and set to ineligible.
request high-availability state functional                    // Set current device to functional/eligible.
request high-availability sync-to-remote                  // Force config and session sync to peer
show high-availability state                        // Show current H/A status
show high-availability all                             // Show high-availability information on current device.
show high-availability link                            // Show H/A state of devices
show high-availability state-synchronisation         // View the sync state to the peer device
show high-availability control-link                     // Show H/A link statistics

// HA Upgrade Process
request high-availability state suspend – passive firewall
Upgrade passive to 4.1.7
request high-availability state suspend – Current old version active firewall
request high-availability state functional – Newly upgraded firewall (Outage until this command completes)
Upgrade old active firewall to 4.1.7
request high-availability state functional – Newly upgraded firewall

Notes:

HA processes can take up to 5 minutes to start up after reboot

// Troubleshooting
test nat-policy-match source destination protocol 6 destination-port
test security-policy-match source destination protocol 6 destination-port
test url **Provides URL category
show routing
show arp
show session

// Packet Capture
From the CLI:

Packet Filter

debug dataplane packet-diag set filter match source
debug dataplane packet-diag set filter match destination
debug dataplane packet-diag set filter on
debug dataplane packet-diag show setting

Packet Capture

debug dataplane packet-diag set capture stage drop file
debug dataplane packet-diag set capture stage transmit file
debug dataplane packet-diag set capture stage receive file
debug dataplane packet-diag set capture stage firewall file
debug dataplane packet-diag set capture on

Clear Packer Filter or capture

debug dataplane packet-diag set filter off
debug dataplane packet-diag set capture off
debug dataplane packet-diag clear filter all
debug dataplane packet-diag clear capture all

View PCAPs

view-pcap follow yes filter-pcap

Export the PCAPs

scp export filter-pcap from to
= user1@192.168.1.1:c:/1/scp
tftp export filter-pcap from to

// VPN Troubleshooting
show vpn flow  **View active tunnels
show vpn flow tunnel-id   **View additional information on tunnels based on ID
show vpn ike-sa
show vpn ipsec-sa
clear vpn ike-sa
clear vpn ipsec-sa
test vpn ike-sa gateway
test vpn ipsec-sa tunnel

// Debug Commands
debug authd
debug cli
debug dhcpd
debug high-availability agent
debug ike
debug log-collector
debug management-server
debug master-service
debug ssl-vpn
debug user-id

Cisco ASA Fail over Active / Standby

You want to deploy 2 Cisco ASA 55xx Series firewalls in an Active/Standby failover configuration.



Solution


Assumptions.

Hardware on both ASA firewalls is identical.
The correct licence's for failover are installed on both firewalls.
The same software versions are installed on both firewalls.
You have your PRIMARY firewall set up and running correctly (Everything works!).

In this example the firewalls were ASA5510's and all interfaces were being used, so the Management port was used as the "Failover Link" (That needs a security plus licence!).
This Link will use a crossover cable (Only available after version 7.0(2) before that you had to use a switch - I think!).

Also I'm using the same link for LAN Based failover (heartbeat) AND Statefull replication.

IP Addresses

Each interface will need its existing IP address, and an address to use whilst in "Standby". In this example I will use the following,


Click For Larger Image

Outside Interface (Ethernet 0/0) 123.123.123.123 255.255.255.0
Outside Interface STANDBY 123.123.123.124 255.255.255.0
DMZ1 Interface (Ethernet0/1) 192.168.1.1 255.255.255.0
DMZ1 Interface STANDBY 192.168.1.254 255.255.255.0
DMZ2 Interface (Ethernet0/2) 192.168.2.1 255.255.255.0
DMZ2 Interface STANDBY 192.168.2.254 255.255.255.0
Inside Interface (Ethernet 0/3) 172.16.1.1 255.255.255.0
Inside Interface (STANDBY) 172.16.1.254 255.255.255.0
Failover Interface (Management0/0) 172.16.254.254 255.255.255.0
Failover Interface STANDBY 172.16.254.250 255.255.255.0


Step 1 Carry Out this procedure on the PRIMARY (Already configured and working) firewall.

1. Backup the running config on the primary firewall.
PetesASA# copy run flash:/before_failover.cfg

Source filename [running-config]?

Destination filename [before_failover.cfg]?
Cryptochecksum: babed83d 62a5fba7 e5ea368d 642157bd

8549 bytes copied in 3.670 secs (2849 bytes/sec)
PetesASA#

2. Blow away the config on the interface you are going to use for failover.
PetesASA(config)# clear configure interface m0/0
PetesASA(config)# int m0/0
PetesASA(config-if)# no shut
PetesASA(config)#
3. Change the interface IP addresses – (to add the standby addresses for each interface).
PetesASA(config)#
PetesASA(config)# interface Ethernet0/0
PetesASA(config-if)# speed 100
PetesASA(config-if)# duplex full
PetesASA(config-if)# nameif Outside
PetesASA(config-if)# security-level 0
PetesASA(config-if)# ip address 123.123.123.123 255.255.255.0 standby 123.123.123.124
PetesASA(config-if)# interface Ethernet0/1
PetesASA(config-if)# speed 100
PetesASA(config-if)# duplex full
PetesASA(config-if)# nameif DMZ1
PetesASA(config-if)# security-level 50
PetesASA(config-if)# ip address 192.168.1.1 255.255.255.0 standby 192.168.1.254
PetesASA(config-if)# interface Ethernet0/2
PetesASA(config-if)# speed 100
PetesASA(config-if)# duplex full
PetesASA(config-if)# nameif DMZ2
PetesASA(config-if)# security-level 55
PetesASA(config-if)# ip address 192.168.2.1 255.255.255.0 standby 192.168.2.254
PetesASA(config-if)# interface Ethernet0/3
PetesASA(config-if)# speed 100
PetesASA(config-if)# duplex full
PetesASA(config-if)# nameif Inside
PetesASA(config-if)# security-level 100
PetesASA(config-if)# ip address 172.16.1.1 255.255.255.0 standby 172.16.1.254
PetesASA(config-if)# exit
PetesASA(config)#
4. Set up the failover LAN interface (In config mode!).
PetesASA(config)#
PetesASA(config)# failover lan interface failover m0/0
INFO: Non-failover interface config is cleared on Management0/0 and its sub-interfaces
PetesASA(config)#
5. Setup failover link IP address.
PetesASA(config)#
PetesASA(config)# failover interface ip failover 172.16.254.254 255.255.255.0 standby 172.16.254.250
PetesASA(config)#
6. Setup a shared key.
PetesASA(config)#
PetesASA(config)# failover lan key 666999
PetesASA(config)#
7. Set it as the primary firewall.
PetesASA(config)#
PetesASA(config)# failover lan unit primary
PetesASA(config)#
8. Turn on failover.
PetesASA(config)#
PetesASA(config)# failover
PetesASA(config)#
9. Now we need to enable statefull failover.
PetesASA(config)#
PetesASA(config)# failover link failover Management0/0
PetesASA(config)#
10. Save the config.
PetesASA(config)#
PetesASA(config)# write mem
Building configuration...
Cryptochecksum: 5c8dfc45 ee6496db 8731d2d5 fa945425

8695 bytes copied in 3.670 secs (2898 bytes/sec)
[OK]
PetesASA(config)#


NOW CONFIGURATION IS FINISHED ON THE PRIMARY FIREWALL, ENSURE THE CABLING IS IN PLACE ON BOTH FIREWALLS THEN CONNECT TO THE STANDBY FIREWALL

Step 2 Carry Out this procedure on the Standby Firewall.

11. Enter enable mode .
ciscoasa>
ciscoasa> en
Password:
ciscoasa#
12. Open the failover link and issue a “no shut” command.
ciscoasa#
ciscoasa# conf t
ciscoasa(config)# interface m0/0
ciscoasa(config-if)# no shut
ciscoasa(config-if)# exit
ciscoasa(config)#
13. Turn on LAN interface for failover.
ciscoasa(config)#
ciscoasa(config)# failover lan interface failover m0/0
INFO: Non-failover interface config is cleared on Management0/0 and its sub-interfaces
ciscoasa(config)#
14. Give it an IP address (YES: that’s the same as the primary firewall there WON’T be a conflict).
ciscoasa(config)#
ciscoasa(config)# failover interface ip failover 172.16.254.254 255.255.255.0 standby 172.16.254.250
ciscoasa(config)#
15. Give it the same key you used above (In step 6).
ciscoasa(config)#
ciscoasa(config)# failover lan key 666999
ciscoasa(config)#
16. Set it as the secondary (standby firewall).
ciscoasa(config)#
ciscoasa(config)# failover lan unit secondary
ciscoasa(config)#
17. Turn on failover.
ciscoasa(config)#
ciscoasa(config)# failover
ciscoasa(config)#
18. You should see......

Detected an Active mate
Beginning configuration replication from mate.

19. When is says that is has ended replication On the secondary firewall, issue a "show failover" (Note: the hostname will have changed to the one on the primary firewall).

PetesASA(config)#
PetesASA(config)# show failover
Failover On
Failover unit Secondary
Failover LAN Interface: failover Management0/0 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 250 maximum
Version: Ours 7.2(2), Mate 7.0(5)
Last Failover at: 14:49:43 UTC May 4 2007
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5510 hw/sw rev (1.1/7.2(2)) status (Up Sys)
Interface Outside (62.254.170.254): Link Down (Waiting)
Interface DMZ1 (172.31.5.254): Link Down (Waiting)
Interface DMZ2 (172.31.4.254): Link Down (Waiting)
Interface Inside (172.31.3.254): Link Down (Waiting)
slot 1: empty
Other host: Primary - Active
Active time: 514 (sec)
slot 0: ASA5510 hw/sw rev (1.1/7.0(5)) status (Up Sys)
Interface Outside (62.254.170.225): Link Down (Waiting)
Interface DMZ1 (172.31.5.1): Link Down (Waiting)
Interface DMZ2 (172.31.4.1): Link Down (Waiting)
Interface Inside (172.31.3.3): Link Down (Waiting)
slot 1: empty
20. To double check go back to the PRIMARY firewall and issue the same command.
PetesASA(config)# show failover
Failover On
Failover unit Primary
Failover LAN Interface: failover Management0/0 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 15 seconds
Interface Policy 1
Monitored Interfaces 4 of 250 maximum
Version: Ours 7.0(5), Mate 7.2(2)
Last Failover at: 13:21:42 UTC May 4 2007
This host: Primary - Active
Active time: 616 (sec)
slot 0: ASA5510 hw/sw rev (1.1/7.0(5)) status (Up Sys)
slot 1: empty
Interface Outside (62.254.170.225): Link Down (Waiting)
Interface DMZ1 (172.31.5.1): Link Down (Waiting)
Interface DMZ2 (172.31.4.1): Link Down (Waiting)
Interface Inside (172.31.3.3): Link Down (Waiting)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5510 hw/sw rev (1.1/7.2(2)) status (Up Sys)
slot 1: empty
Interface Outside (62.254.170.254): Link Down (Waiting)
Interface DMZ1 (172.31.5.254): Link Down (Waiting)
Interface DMZ2 (172.31.4.254): Link Down (Waiting)
Interface Inside (172.31.3.254): Link Down (Waiting)

21. The failover time out of the box is a bit pants, to nail it down a little, on the PRIMARY ASA
PetesASA(config)#
PetesASA(config)# failover poll 1 hol 3
PetesASA(config)# failover poll interface 3
PetesASA(config)# int m0/0
PetesASA(config-if)# failover poll interface 3
PetesASA(config)#
22. Save the config. (Note: config changed WILL be replicated to the standby firewall).
PetesASA(config)#
PetesASA(config)# write mem
Building configuration...
Cryptochecksum: 6650f6c9 09bbb5f0 0dafa0d1 8fc08aba

8756 bytes copied in 3.680 secs (2918 bytes/sec)
[OK]
PetesASA(config)#

23. When done pull the power on ASA 1 to fail. With a constant ping running you usually will only lose 1 ping packet.

CEF and load sharing

Load-sharing is one of the clumsy areas that is full of confusing parts. In this post we should be covering its ABCs, and latter on we should be covering more parts in details. We chose the name “CEF and load sharing” as the post name due to the main role that CEF plays when talking about load sharing.

In IP routing context the forwarding/switching mechanism that the router uses is the actual controller of the load sharing process (data/forwarding plane operation), having multiple routes in the routing table has no significance on how exactly will load sharing be done, you might be left with poor load sharing or no load sharing at all, although you have multiple routes for a certain destination in the routing table.

The routing protocols are responsible for placing multiple paths in the routing table in the first place (control plane operation), by default all the IGPs are capable of inserting 4 equal cost paths, while BGP defaults to only 1 (BGP behaves completely different than the IGPs, we should be covering load-sharing with BGP in details in a later post). To control the maximum paths allowed per routing protocol we can use the maximum-paths command (The maximum was 4 in IOS releases earlier than 11.0, 8 with IOS Release 12.0S based software, 16 with IOS Release 12.3T based software, and 32 with IOS Release 12.2S based software.

NOTE This post is not meant to explain CEF operation, we’ll only be focusing on CEF load-sharing, however we might consider to have a dedicated CEF inside out post later.

The most popular forwarding/switching mechanisms with Cisco routers are; Process switching (performs per-packet load-sharing), fast switching (performs per-destination load-sharing) and CEF (can do both per-packet and per-destination (completely different than fast switching per-destination load-sharing), plus also a new flavor which is per-port load-sharing).

NOTE According to Cisco, IPv4 fast switching is removed with the implementation of the Cisco Express Forwarding infrastructure enhancements for Cisco IOS 12.2(25)S-based releases and Cisco IOS Release 12.4(20)T. For these and later Cisco IOS releases, switching path are Cisco Express Forwarding switched or process switched. This makes the switching decision easier for future development of software features. Starting with the implementation of the Cisco Express Forwarding enhancements and the removal of IPv4 fast switching, components that do not support Cisco Express Forwarding will work only in process switched mode.

Load-sharing with CEF

For each destination with multiple equal cost paths (or unequal-cost in the case of EIGRP using variance, or with BGP using the BGP Link Bandwidth feature and also in the case of MPLS-TE) the router creates a 16 hash buckets, each pointing to one of the available paths.

The load sharing is controlled by the ratio of the number of buckets pointing to each path (outgoing interface), with equal-cost paths the buckets are fairly distributed (two equal cost paths results in 8 buckets per each path, three equal cost paths results in 5 per each (yes, one bucket is omitted), 4 equal cost paths results in 4 per each, and so on). While with unequal-cost scenarios each path will be associated with different number of buckets (according to the load sharing ratio).

CEF has three load-sharing options:
•per-destination (per-session):

I prefer to name it per-session – as stated in the show ip cef x.x.x.x internal command output – since it is actually done based on both the source and the destination IP addresses in the IP packet rather than solely the destination, by hashing both into a 4-bit hash value that is used to select the outgoing interface) – This is the default CEF load sharing option.


Easy AdSense by Unreal

It is clear that per-destination load-sharing performs statistical distribution of traffic, and accordingly load sharing becomes more effective as the number of source/destination pairs increases as compared to lower number of source/destination pairs. Obviously this might result in having one link overloaded while the other(s) underutilized, if a relatively heavy session flows between a certain source/destination pair over this link.

The hash calculation depends on the algorithm used. The original algorithm uses only the source and destination IP addresses to compute a 4-bit hash value, giving 16 probabilities, and thus choosing an outgoing bucket from the 16 available buckets pointing to one of the outgoing paths, this results in all the routers in the network running the same algorithm with the same results, which introduced a load sharing hitch called CEF Load-Sharing Polarization (you can see a good example for this in Cisco press book “Cisco Express Forwarding”). To circumvent this behavior the universal algorithm (the default in current IOS versions) adds a 32-bit router-specific value to the hash function (called Fixed ID, which can be manually controlled – a router uses its highest loopback IP address as this value when booting) and thus seeding the hash function on each router with a unique ID, ensuring that the same source/destination pair will hash into a different 4-bit value on different routers along the path and thus provides a better network wide load sharing and circumvent the Polarization issue.

NOTE There is a third available algorithm called the tunnel algorithm, I couldn’t find or understand its anatomy, but Cisco stated that this algorithm is meant to solve load sharing when tunneling techniques such as MPLS, GRE and L2TP are in operation, since with tunneling the traffic pattern is taken down to a small number of sessions (between the tunnel head/tail ends) which will introduce another form of traffic polarization. This algorithm also uses a unique per-router ID to work around this issue, again I can’t find more details about this algorithm, but if I do I’ll let you know.
•per-packet

Packets are handled in a round-robin fashion, ensuring that the traffic is balanced over multiple links. However, using Per-packet load sharing is not generally recommended, because it most commonly results in out-of-order packets, affecting TCP traffic throughput (since TCP will bother to fix the out-of-order) and UDP data loss (since UDP will not bother to fix the out-of-order) and to make things more scary out-of-order packets might be interpreted as an attack by firewalls.

The default CEF load sharing mode is per-destination, and we can change this using the ip load-sharing per-packet interface command on the outgoing interfaces involved.

NOTE Since load sharing decisions are made on the outbound interfaces, thus either choosing to do per-packet or per-destination load sharing should be done on the outbound interfaces.
•per-port (per-flow)

This is the most adequate option (was introduced with IOS 12.4(11)T release) with networks with low number of sources/destinations with the majority of the traffic between hosts that use different port numbers, commonly seen with Real-Time Protocol (RTP) streams, it simply adds the layer 4 source or destination ports or both in the CEF hashing function. This option is enabled via the ip cef load-sharing algorithm include-ports command in the global configuration.

The most common scenario with this option as the only effective solution is when having a subnet of hosts NATed to a single IP then having a router with multiple paths in the path to their traffic destination, per-destination option is obviously useless in this case if all the hosts are communicating with a single destination, since it is always a single source/destination pair, and accordingly if the layer 4 ports are involved in the hashing function this would enhance the load sharing process.

I hope that I’ve been informative.

Regards
Mirza Mukaram Baig

ARP Caching and TimeOut

From time to time I find myself craving to the fundamentals; I do this for two main reasons, the first one is that fundamentals are the building blocks of all complex networking topics and deeply understanding them makes a better engineer, the second one is longing to simplicity after doing some complex tasks.

One of these fundamentals that is worth reviewing is the Address Resolution Protocol, this protocol is one of the main building blocks of any network existing on earth today.



Every time a network device is sending an Ethernet frame to another device, it constructs a frame and to construct the frame it needs to find the hardware address mapping of the IP address. ARP is responsible for doing this job.

Each time a device sends an ARP message, network resources are consumed. This means that for two hosts to communicate; ARP messages should be exchanged between them and repeated for every packet. Imagine how ugly is this when transferring large data streams like large file exchange via FTP.

ARP caching provides the solution for this efficiency problem as explained below.

ARP Caching

If you know you are going to send many emails to a friend; is it effective to call him every time asking for his email address?. I think the answer is no unless you are fascinated by listening to his voice. Simply you call him one time asking for the address and cache the information somewhere for future uses and that’s exactly what ARP does.

When a host sends an ARP request to another host and a reply is received the sender caches the received information is a table for later use.


Easy AdSense by Unreal

Going back to our analogy of the email sender, what if you know that you are not going to send any more emails to your friend “God keep you friends ” Is it still effective to keep his address in your cache table ?. I think not, you have to timeout unused information. Again this is exactly what ARP does.

If an ARP entry is not used a specific amount of time called the ARP timeout the entry is removed from the caching table.

There is no standard value for this amount of time and it varies from one vendor to another. I will limit my discussion to Cisco devices to clear up the idea.

One more point to mention here is that entries in the ARP table can be static; created by manual configuration or dynamic; created automatically by the normal operation of the protocol. Static entries remain in the table forever and are not timed out.

The default timeout timer for is 4 hours for Cisco devices, this means that a dynamic ARP entry will remain for 4 hours in the cache table before the router attempt to refresh the entry. If the entry is no longer needed it will be removed.

You can show the ARP table using the command show arp and change the timeout timer for a specific interface using the interface level command arp timeout seconds.

Configuration
R1#show ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.11.1 - sa00.0a11.0001 ARPA FastEthernet0/0
Internet 10.10.11.3 97 sa02.0a11.0002 ARPA FastEthernet0/0
Internet 10.10.11.1 8 sa00.0a11.0003 ARPA FastEthernet0/5
Internet 10.10.11.5 136 sa04.0a11.0004 ARPA FastEthernet0/2

!-- setting the timeout for 10 seconds
R1(config-if)#int f0/2
R1(config-if)#arp timeout 15

!-- see the debug output, shows 15 seconds difference between replies
R1#
Jan 1 00:01:14: IP ARP: sent req src 10.10.10.1 sa00.0a74.0005,
dst 13.13.13.3 ca02.0a74.0008 FastEthernet0/0
Jan 1 00:01:14: IP ARP: arp_process_request: 10.10.10.1, hw: sa02.0a74.0008; rc: 3
Jan 1 00:01:14: IP ARP: rcvd rep src 13.13.13.3 sa02.0a74.0008, dst 13.13.13.1 FastEthernet0/0
Jan 1 00:01:14: IP ARP: creating entry for IP address: 13.13.13.3, hw: sa02.0a74.0008
R1#
Jan 1 00:01:24: IP ARP: sent req src 13.13.13.1 ca00.0a74.0008,
dst 13.13.13.3 ca02.0a74.0008 FastEthernet0/0
Jan 1 00:01:24: IP ARP: arp_process_request: 13.13.13.3, hw: ca02.0a74.0008; rc: 3
Jan 1 00:01:24: IP ARP: rcvd rep src 13.13.13.3 ca02.0a74.0008, dst 13.13.13.1 FastEthernet0/0
Jan 1 00:01:24: IP ARP: creating entry for IP address: 13.13.13.3, hw: ca02.0a74.0008
Note: ARP cache table is not the same as MAC address table used by switches and each one has its own different timers.

Thank you once again.

Regards