Planet Ergo

I was setting up a network infrastructure the other day, and needed trunking over bonded (Linux) network interfaces connected to a Cisco switch for a virtualization <-> storage network (Network #1), and "the rest" (production network, management network, etc.). Here's just some quick notes:

• My rule of thumb was: The higher the VLAN number or IP space network number, the more private the network is.
• I decided the Management network was to be VLAN 666 (how appropriate) with IP space 10.66.6.0/24. The storage network inherently became 10.66.7.0/24 (VLAN 667) because it was even more private then the management network.
• I decided that any public servers would be in VLAN 2 (very public, very close to the Internet, etc.).

Let's suppose these were all the networks I needed.

On the Hypervisor, I configured eth0 and eth1 as slave interfaces for bond0. bond0 itself though was not supposed to have any IP address configuration.

# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=<some-mac-address>
ONBOOT=yes
MASTER=bond0
SLAVE=yes
# cat ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
HWADDR=<some-mac-address>
ONBOOT=yes
MASTER=bond0
SLAVE=yes
# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes

For a bonded interface, you need to choose the exact mode of operation, and because the interfaces were bonded to increase the throughput, I chose 802.3ad. For it to actually happen, you need to configure the bonding kernel module through /etc/modprobe.conf:

alias eth0 bnx2
alias eth1 bnx2
alias bond0 bonding
options bond0 mode=4 miimon=100

NOTE: Listing the physical interfaces first is mandatory.

Still, we have no network configuration. The only network configuration I needed was a semi-physical interface in the storage network, and a set of 802.1q encapsulated interfaces for the rest of the network communications. Ergo, I created the following interfaces:

• bond0.2 (public)
• bond0.666 (management)
• bond0.667 (storage)

The only interface out of the three that would actually get an IP address though was the storage network interface bond0.667. The configuration would look as follows:

DEVICE=bond0.667
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
TYPE=Ethernet
IPADDR=10.66.7.1
NETMASK=255.255.255.0
NOZEROCONF=yes

The other two interfaces (bond0.2 for the internet and bond0.666 for the management) are a little more tricky. They needed to be bridged interfaces, in order to allow virtualized guest nodes to be positioned in either one of those two networks. The configuration for bond0.2 therefore looked as follows:

DEVICE=bond0.2
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
TYPE=Ethernet
BRIDGE=br2

Bridge interface br2 was to be used to connect the virtualized guest nodes to. Its configuration looks like:

DEVICE=br2
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
VLAN=yes
STP=yes
DELAY=5

Note that the bridge interface does not have its own IP address, or we would be connecting the Hypervisor directly to the Internet (and we don't want to, FWIW).

The management network interface though, which also needed to be bridged, does have its own IP address (in the management network, of course):

# cat ifcfg-bond0.666
DEVICE=bond0.666
BOOTPROTO=none
ONBOOT=yes
VLAN=yes
TYPE=Ethernet
BRIDGE=br666
# cat ifcfg-br666
DEVICE=br666
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
IPADDR=10.66.6.1
NETMASK=255.255.255.0
GATEWAY=10.66.6.254
NOZEROCONF=yes
STP=yes
DELAY=5

Now, we're done for the Linux Hypervisor part of the infrastructure. Lets get to the Cisco side of things!

All that a Cisco Catalyst 3560G really requires is that you:

• Create the VLANs you want to use. This used to be through enable mode's "vlan database" command, which is still a valid way of configuring the available VLANs, but will be (may already have been on newer equipment) deprecated for a series of commands in configure mode.
• Create the "channel-group" with the appropriate interfaces.
• Enable trunking on the channel-group interface.

Ergo, here we go (from enable mode):

conf t
  int range gi0/1-2
    no shut
    speed 1000
    duplex full
    switchport trunk encapsulation dot1q
    switchport mode trunk
    switchport trunk allowed vlan 2,666,667
    no switchport trunk native vlan
    description **some etherchannel**
    channel-group 1 mode active

You should now get an interface called Po1, with the following configuration:

show running-config interface Po1
Building configuration...

Current configuration : 142 bytes
!
interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,666,667
 switchport mode trunk
end

You should be good to go by now.