Sunday, November 26, 2006

Storage area networking

unixway logoStorage area networking

Introduction

A storage area network (SAN) can address several challenges faced by system administrators. Unlike direct-attached storage (DAS), sans allow the administrator to manage a central pool of storage and allocate it to individual hosts as needed. Furthermore, the optical nature of sans provide flexibility not available with direct-attached storage which typically uses electrical signaling. For example, one can unplug a fibre cable without having to worry about "blowing" an hba.

However, their size and complexity can be daunting, particularly when faced with the terminology:

  • World-wide name (wwn): Similar to an Ethernet address in the traditional networking world, this is a unique name assigned by the manufacturer to the HBA. This name is then used to grant and restrict access to other components of the san.
  • zone: Within a fabric, switches can segment the SAN into logical "zones". Only those elements in the same zone can "see" each other.

Topology of a simple SAN

In this exercise, we're going to configure a simple san composed of three nodes and one storage array. The three nodes are Solaris servers with Emulex or JNI fibre host bus adapters (hba). As we'll see, the configuration process for these hba models differs slightly, and the concepts are very similar. Our example uses Brocade switches and an EMC Symmetrix array, but the concepts would apply equally well to different vendors' products.

The following diagram illustrates several principals common to most san topologies:

  • There are two separate storage area networks ("san1" and "san2"). Though most switches support inter-switch links (ISLs), many vendors suggest isolation of fabrics for simple SANs as shown here.. Each switch therefore comprises an independent "fabric".

  • The array provides multiple I/O paths to every disk device. In the diagram below, for example, the disk device "2ad" is accessible via both fa4b and fa13a. Depending on the array, the two paths may be simultaneously active (active/active) or only one path may be valid at a time (active/passive).

  • Each host has a connection into each fabric. Host-based software load-balances the traffic between the two links if the array supports active/active disk access. Furthermore, the host should adjust to a single link failure by rerouting all traffic along the surviving path.

Take a moment to examine the diagram below and consider the host "pear". Assuming that it requires highly-available access to disk 2ad, note that there are two separate paths to that device:

  1. pear(fcaw0) -- san1(port2) -- san1(port15) -- fa13a -- 2ad

  2. pear(fcaw1) -- san2(port2) -- san2(port14) -- fa4b -- 2ad

Figure 1: Topology of a simple SAN:
Image of simple san

 

Our desired configuration requires the following:

  • The server pear requires access to two devices accessible via both fa4b and fa13a. We will assign EMC devices 2ad and 2b1 to pear.
  • The server apple requires access to four devices accessible via both fa4a and fa13b. We will assign EMC devices 15d, 161, 165, and 169 to apple.
  • The server banana requires access to three devices accessible via both fa4a and fa13b. We will assign EMC devices 16d, 171, and 175 to banana.
  • All servers require access to the VolumeLogix device 000 on each FA.

After the configuration is complete, the following devices will remain unassigned and available for future use: 2b5, 2b9, 2bd, 2c1, and 2c5.

Identify the WWN of devices

In the sequence below, we configure the host hba drivers to scan for the required luns, update the switch zoning, and finally configure the lun masking (VolumeLogix) in order to make the storage visible to the above three servers.

Determine world-wide names of all adapters in SAN configuration.

For host bus adapters, the world wide name is typically displayed in the /var/adm/messages file after the fibre card and software driver have been installed. An example of for an Emulex PCI fibre HBA is as follows. In this case, the relevant value the WWPN, the World-Wide Port Name:

Aug 26 09:26:28 banana lpfc: [ID 242157 kern.info] NOTICE:                  lpfc0:031:Link Up Event received Data: 1 1 0 0 Aug 26 09:26:31 banana lpfc: [ID 129691 kern.notice] NOTICE:                  lpfc0: Firmware Rev 3.20 (D2D3.20X3) Aug 26 09:26:31 banana lpfc: [ID 664688 kern.notice] NOTICE:                  lpfc0: WWPN:10:00:00:00:c9:28:22:38                  WWNN:20:00:00:00:c9:28:22:38 DID 0x210913  Aug 26 09:26:31 banana lpfc: [ID 494464 kern.info] NOTICE:                  lpfc1:031:Link Up Event received Data: 1 1 0 0 Aug 26 09:26:34 banana lpfc: [ID 129691 kern.notice] NOTICE:                  lpfc1: Firmware Rev 3.20 (D2D3.20X3) Aug 26 09:26:34 banana lpfc: [ID 664688 kern.notice] NOTICE:                  lpfc1: WWPN:10:00:00:00:c9:28:22:56                  WWNN:20:00:00:00:c9:28:22:56 DID 0x210913

Note that the HBA vendor also may provide a tool that allows the system administrator to query the wwn of the HBA (e.g. Emulex supplies the lputil application). The world-wide name of the symmetrix fibre adapters (FA) can be obtained from EMC. In the case of our sample san, the world wide names are as follows:

Host name port SCSI target World Wide Name
apple fcaw0 200000e069415402
apple fcaw1 200000e0694157a0
pear fcaw0 200000e069415773
pear fcaw1 200000e069415036
banana lpfc0 10000000c9282238
banana lpfc1 10000000c9282256
symmetrix fa4a target 20 500604872363ee43
symmetrix fa4b target 21 500604872363ee53
symmetrix fa13a target 22 500604872363ee4c
symmetrix fa13b target 23 500604872363ee5c

Update the host configuration

There are several vendors of host bus adapters, and the vendor's documentation is the best reference for the configuration process. In this section, we'll highlight the process of configuring the server for two popular vendors, Emulex and JNI. Please refer to the vendor's documentation for your specific card.

Typically, there are two configuration files that need to be updated once the vendor's hba software has been installed. The hba driver's configuration file typically resides in the /kernel/drv directory, and must be updated to support persistent binding and any other configuration requirements specified by the array vendor. Secondly, the Solaris "sd" driver configuration file sd.conf must be updated to tell the operating system to scan for more than the default list of scsi disk devices. The examples below describe the process for configuring Emulex and JNI cards in to support an EMC Symmetrix array.

  • Configure the /kernel/drv/fcaw.conf on servers apple and pear:

    fca_nport = 1; failover = 60; def_hba_binding  = "fas"; fca_verbose = 1;

    Also available for examination are the complete fcaw.conf files for PowerPath and non-PowerPath.

  • Configure the /kernel/drv/lpfc.conf on server banana. 
    fcp-bind-WWPN="500604872363ee43:lpfc0t20",               "500604872363ee5c:lpfc1t23";

    Also available for examination are the complete lpfc.conf files for PowerPath and non-PowerPath.

By default, the Solaris server will scan for a limited number of scsi devices. The administrator has to update the /kernel/drv/sd.conf file to tell the sd driver to scan for a broader range of scsi devices. In both cases, the target number associated with the WWN of the array adapter is arbitrary. In our case, we've assigned scsi targets 20, 21, 22, and 23 to the four array adapters. The following list describes the additions to the /kernel/drv/sd.conf file for each of the three hosts:

  • apple: 
    # Entries added for host apple to "see" lun 0-4 on fa4a and fa13b # fa4a = 500604872363ee43 name="sd" class="scsi" target=20 lun=0 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=1 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=2 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=3 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=4 hba="fcaw0" wwn="500604872363ee43";  # fa13b = 500604872363ee5c name="sd" class="scsi" target=23 lun=0 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=1 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=2 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=3 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=4 hba="fcaw1" wwn="500604872363ee5c";
  • pear: 
    # Entries added for host pear to "see" lun 0-2 on fa4b and fa13a # fa4b = 500604872363ee53 name="sd" class="scsi" target=21 lun=0 hba="fcaw0" wwn="500604872363ee53"; name="sd" class="scsi" target=21 lun=1 hba="fcaw0" wwn="500604872363ee53"; name="sd" class="scsi" target=21 lun=2 hba="fcaw0" wwn="500604872363ee53";  # fa13a = 500604872363ee4c name="sd" class="scsi" target=22 lun=0 hba="fcaw1" wwn="500604872363ee4c"; name="sd" class="scsi" target=22 lun=1 hba="fcaw1" wwn="500604872363ee4c"; name="sd" class="scsi" target=22 lun=2 hba="fcaw1" wwn="500604872363ee4c";
  • banana: 
    # Entries added for host banana to "see" lun 5-7 on fa4a and fa13b # fa4a = 500604872363ee43 name="sd" parent="lpfc" target=20 lun=0 hba="lpfc0"; name="sd" parent="lpfc" target=20 lun=5 hba="lpfc0"; name="sd" parent="lpfc" target=20 lun=6 hba="lpfc0"; name="sd" parent="lpfc" target=20 lun=7 hba="lpfc0";  # fa13b = 500604872363ee5c name="sd" parent="lpfc" target=23 lun=0 hba="lpfc1"; name="sd" parent="lpfc" target=23 lun=5 hba="lpfc1"; name="sd" parent="lpfc" target=23 lun=6 hba="lpfc1"; name="sd" parent="lpfc" target=23 lun=7 hba="lpfc1";
Update the /etc/system file as per EMC's requirements for the Symmetrix. 
set sd:sd_max_throttle=20 set scsi_options = 0x7F8  * Powerpath?   Enter * No           sd:sd_io_time=0x78 * Yes          sd:sd_io_time=0x3C set sd:sd_io_time=0x3C

Update the switch configuration

In the example session below, we configure the switch "san1" zoning so that one of the hbas in each host can "see" fa4a and fa13a of the Symmetrix. The switch "san2" is configured so that the other hba in each host can "see" fa4b and fa13b of the Symmetrix. We leave the configuration of san2 as an exercise for the reader.

This configuration illustrates a few principles that we have not discussed earlier. First, each host bus adapter is given only the access required to see the disks it needs. Secondly, each zone contains a single host bus adapter (i.e. a signal initiator) and a single array adapter.

Fabric OS (tm)  Release v2.5.1b                                 login: admin Password:
san1:admin> switchshow 
switchName:     san1   switchType:     6.1 switchState:    Online    switchRole:     Principal switchDomain:   2         switchId:       fffc02 switchWwn:      10:00:00:60:69:40:27:db switchBeacon:   OFF                     port  0: id  No_Light                                      port  1: id  Online        F-Port  20:00:00:e0:69:41:54:02 port  2: id  Online        F-Port  20:00:00:e0:69:41:57:73 port  3: id  Online        F-Port  10:00:00:00:c9:28:22:38 port  4: id  No_Light       port  5: id  No_Light       port  6: id  No_Light       port  7: id  No_Light       port  8: id  No_Light       port  9: id  No_Light       port 10: id  No_Light       port 11: id  No_Light       port 12: id  No_Light                                      port 13: id  No_Light   port 14: id  Online        F-Port  50:06:04:87:23:63:ee:43 port 15: id  Online        F-Port  50:06:04:87:23:63:ee:4c
san1:admin> aliCreate "APPLE_FCAW0", "20:00:00:e0:69:41:54:02";
san1:admin> aliCreate "PEAR_FCAW0", "20:00:00:e0:69:41:57:73";
san1:admin> aliCreate "BANANA_LPFC0", "10:00:00:00:c9:28:22:38";
san1:admin> aliCreate "SYMM_FA4A", "50:06:04:87:23:63:ee:43";
san1:admin> aliCreate "SYMM_FA13A", "50:06:04:87:23:63:ee:4c";
san1:admin> zoneCreate "Z_APPLE_FCAW0_SYMM_FA4A", "APPLE_FCAW0; SYMM_FA4A";
san1:admin> zoneCreate "Z_PEAR_FCAW0_SYMM_FA13A", "PEAR_FCAW0; SYMM_FA13A";
san1:admin> zoneCreate "Z_BANANA_LPFC0_SYMM_FA4A", "BANANA_LPFC0; SYMM_FA4A";
san1:admin> cfgAdd "FABRIC1", "Z_APPLE_FCAW0_SYMM_FA4A; Z_PEAR_FCAW0_SYMM_FA13A; Z_BANANA_LPFC0_SYMM_FA4A";
san1:admin> cfgSave
Updating flash ...
san1:admin> cfgEnable "FABRIC1";
zone config "FABRIC1" is in effect
san1:admin> zoneshow 
Defined configuration:  cfg:   FABRIC1                        Z_APPLE_FCAW0_SYMM_FA4A; Z_PEAR_FCAW0_SYMM_FA13A;                  Z_BANANA_LPFC0_SYMM_FA4A          zone:  Z_APPLE_FCAW0_SYMM_FA4A                              APPLE_FCAW0; SYMM_FA4A  zone:  Z_BANANA_LPFC0_SYMM_FA4A                         BANANA_LPFC0; SYMM_FA4A   zone:  Z_PEAR_FCAW0_SYMM_FA13A                          PEAR_FCAW0; SYMM_FA13A  alias: APPLE_FCAW0                                     20:00:00:e0:69:41:54:02  alias: BANANA_LPFC0                                    10:00:00:00:c9:28:22:38  alias: PEAR_FCAW0                                      20:00:00:e0:69:41:57:73  alias: SYMM_FA13A                                      50:06:04:87:23:63:ee:4c  alias: SYMM_FA4A                                       50:06:04:87:23:63:ee:43                                         Effective configuration:  cfg:   FABRIC1        zone:  Z_APPLE_FCAW0_SYMM_FA4A                         20:00:00:e0:69:41:54:02                 50:06:04:87:23:63:ee:43  zone:  Z_BANANA_LPFC0_SYMM_FA4A                         10:00:00:00:c9:28:22:38                  50:06:04:87:23:63:ee:43  zone:  Z_PEAR_FCAW0_SYMM_FA13A                          20:00:00:e0:69:41:57:73                 50:06:04:87:23:63:ee:4c                                         san1:admin> logout

Update the array's lun masking configuration

The zoning configuration defined above allows the two virtual endpoints of the fibre link to "see" each other. In the zoning example shown above, however, both apple and banana are both zoned to "see" fa4a (zones Z_APPLE_FCAW0_SYMM_FA4A and Z_BANANA_LPFC0_SYMM_FA4A). Often, the administrator wants to restrict the host visibility of disks for a single fibre adapter. The process of restricting access to disks based on the host WWN is referred to as lun masking.

In the example below, the administrator uses the EMC fpath command (recently, EMC has deprecated the fpath command in favour of the symmask/symmaskdb commands) to explicitly grant disk access to host WWNs. Without this access, the host would not be able to see the disks even though the zoning and hba configuration allowed it to see the fibre adapter on the array.

# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \                -w 200000e069415402 -f 4a -r "15d 161 165 169" # /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \                -w 200000e069415402 -n "apple/fcaw0" # /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \                -w 200000e0694157a0 -f 13b -r "15d 161 165 169" # /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \                -w 200000e0694157a0 -n "apple/fcaw1"  # /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \                -w 200000e069415773 -f 4b -r "2ad 2b1" # /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \                -w 200000e069415773 -n "pear/fcaw0" # /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \                -w 200000e069415036 -f 13a -r "2ad 2b1" # /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \                -w 200000e069415036 -n "pear/fcaw1"  # /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \                -w 10000000c9282238 -f 4a -r "16d 171 175" # /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \                -w 10000000c9282238 -n "banana/lpfc0" # /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \                -w 10000000c9282256 -f 13b -r "16d 171 175" # /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \                -w 10000000c9282256 -n "banana/lpfc1"

Refresh the VolumeLogix database:

# /usr/symcli/bin/fpath refresh -d /dev/rdsk/c1t20d0s2

Query the VolumeLogix database:

# /usr/symcli/bin/fpath lsdb -d /dev/rdsk/c1t20d0s2 -s on

Make a backup of the VolumeLogix database:

# /usr/symcli/bin/fpath backupdb -d /dev/rdsk/c1t20d0s2 \      -o /usr/emc/VolumeLogix/backup/vcmdb.`date'+%Y%m%d'`

Reboot servers

Perform a reconfiguration reboot (e.g. "reboot -- -r") on all three servers.

You should see the desired disks. Put a Sun label on them via the "format" command and the configuration is complete.

To summarize, four "layers" of configuration must be correct in order to grant a host access to SAN storage:

  1. physical: The physical cabling linking the hba, switch, and array must function.
  2. host configuration: The host bus adapter must be configured according to the manufacturer's documentation. Our example referred to Emulex and JNI cards, but the process for other vendors is usually similar. Furthermore, The Solaris sd.conf configuration file must be updated so that the host scans for the appropriate luns.
  3. san: The san fabric must be configured so that the host bus adapter and the array fibre adapter are in the same zone.
  4. lun masking: The array lun masking configuration must be updated to give the host permission to access the required disks.

References:

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)