Find it

Thursday, March 18, 2010

Solaris Link Aggregation using dladm

A Solaris "Link Aggregation" is the act of bonding several interfaces together to create a single logical interface.


A link aggregation consists of several interfaces on a system that are configured together as a single, logical unit. Link aggregation, also referred to as Trunking, is defined in the IEEE 802.3ad Link Aggregation Standard.

The IEEE 802.3ad Link Aggregation Standard provides a method to combine the capacity of multiple full-duplex Ethernet links into a single logical link. This link aggregation group is then treated as though it were, in fact, a single link.

The following are features of link aggregations:

1. Increased bandwidth – The capacity of multiple links is combined into one logical link.

2. Automatic failover/failback – Traffic from a failed link is failed over to working links in the aggregation.

3. Load balancing – Both inbound and outbound traffic is distributed according to user selected load balancing policies, such as source and destination MAC or IP addresses.

4. Support for redundancy – Two systems can be configured with parallel aggregations.

5. Improved administration – All interfaces are administered as a single unit.

6. Less drain on the network address pool – The entire aggregation is assigned one IP address.

Before going toward HOW-TO, Your aggregation configuration is bound by the following requirements:

1. You must use the dladm command to configure aggregations.

2. An interface that has been plumbed cannot become a member of an aggregation.

3. Interfaces must be of the GLDv3 type: bge, e1000g, xge, nge, rge, ixgb.

4. All interfaces in the aggregation must run at the same speed and in full duplex mode.

5. “Legacy” data link provider interfaces (DLPI ), such as the ce interface do not support Solaris link aggregations. Instead, you must configure aggregations for legacy devices by using Sun Trunking. You cannot configure aggregations for legacy devices by using the dladm command.

How aggregation works?


The MAC layer, which is part of GLDv3, is the central point of access to Network Interface Cards (NICs) in the kernel. At the top, it provides a client interface that allows a client to send and receive packets to and from NICs, as well as configure, stop and start NICs. A the bottom, the MAC layer provides a provider interface which is used by NIC drivers to interface with the network stack. In the figure above, the client is the Data-Link Service (DLS) which provides SAP demultiplexing and VLAN support for the rest of the stack. The Data-Link Driver (DLD) provides a STREAMS interface between Nemo and DLPI consumers.


The core of the link aggregation feature is provided by the "aggr" kernel pseudo driver. This driver acts as both a MAC client and a MAC provider. The aggr driver implements a MAC provider interface so that it looks like any other MAC device, which allows us to manage aggregation devices as if they were a regular NIC from the rest of Solaris.

HOW-TO?
 
Make sure your eeprom’s local-mac-address? variable is set to true.


# eeprom local-mac-address?

local-mac-address=false

# eeprom local-mac-address? = true
# eeprom local-mac-address?

local-mac-address?=true

PS: Above step is not applicable for x86.

1. Unplumb the interfaces to be aggregated:


# ifconfig bge0 down unplumb
# ifconfig bge1 down unplumb


2. Create a link-aggregation group with key 1. key Is the number that identifies the aggregation. The lowest key number is 1. Zeroes are not allowed as keys.

passive mode by default:

# dladm create-aggr -d bge0 -d bge1 1



# mv /etc/hostname.bge0 /etc/hostname.aggr1

3. Perform reboot. Good to go for reconfigure reboot but not required.

# reboot -- -rv

After reboot –

You can check the status of the aggregation with the dladm command...


# dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:14:4f:2b:be:18 (auto)
device address speed duplex link state
bge0 0:14:4f:2b:be:18 1000 Mbps full up attached
bge2 0:14:4f:2b:be:1a 1000 Mbps full up attached


Hope that helps.

6 comments:

  1. Can you mix interface types? e.g. an aggregation using e1000g and nxge at the same time.

    ReplyDelete
    Replies
    1. Hi,

      I'm certain it's supported. It is possible to do Link Aggregation using mix of NIC's (Currently, interfaces must be of the GLDv3 type: xge, e1000g, and bge.) from different vendors. Only condition is to have identical link speed and full duplex.

      HTH

      Tx/Nilesh

      Delete
  2. After configuring the link aggregation in the e1000g with the equal speed.

    bash-3.00# dladm show-aggr
    key: 1 (0x0001) policy: L4 address: 0:14:4f:e:c1:32 (auto)
    device address speed duplex link state
    e1000g0 0:14:4f:e:c1:32 100 Mbps full up standby
    e1000g1 0:14:4f:e:c1:33 100 Mbps full up attached


    how to make that link state standby to attached for the HA feature

    can any one please help on this

    ReplyDelete
  3. What happens if you try to remove bge0 from the aggregation in your example? I notice the mac address of the trunk is using the mac of bge0 -- thus if you delete it what happens?

    ReplyDelete
    Replies
    1. Never got a chance to do it practically..... Not sure what will happen. If you have done this please let me know also.

      Tx/Nilesh

      Delete
  4. I set up link aggregation ok between two solaris on VirtualBox. Now my point is if this add failback support then why if I deactivate one of the interfaces, using "disconnect cable" on Virtual Box that its part of the aggregation key, why I cannot reach the other host anymore? Shouldnt this be one of its major characteristics? Your help is really appreciated

    Nico

    ReplyDelete