Bandwidth Challenge
SC 2008

Planning
 

1. Equipment

Servers rack was comprised of a set of 1U Twin Motherboard Supermicro servers.

1U Servers  Qty 32
   2 x Supermicro  6015TC-10GB server
 Processor  2 x Dual Intel  L5420,  2.5 GHz quad-core
 RAM  2 x 16GB
 Hard Disks  4 x 1TB SATA-II
 NIC  2 x 10GbE CX4
 HBA
 
 1 x QLogic 2560 FC-8
 (in 16 sevrers only)

 Cisco 6509-E  Qty 2
 Supervisor  SUP VSS 720 with PFC3B
 LR Ports  19
 SR Ports  39
 CX4 Ports  27
 Cisco 3560-E  Qty2
 Cisco 3550  Qty2
 
 Force10 s2410  Qty 2 (Total 48 ports)
 FO Ports  2 x 4 MM SR
 CX4 Ports  2 x 20 CX4
 
 GlimmerGlass  System 100 48x48 ports

 XKL DXM10  Qty 2  - 10 Port DWDM
 MM SR Ports  6
 SM LR Ports  4


 Storage System  2 x Data Direct Networks S2A9900,
 8 port FiberChannel storage system,
 using 80 SAS disks in 5 trays
 HBA  16 x QLogic 2560, Single port 8Gb Fiber Channel

 Miscellaneous  
   
 Rack  2 x 42U
 FO PatchPanel  1
 KVM  1 x 16 port

 

2. Server Configuration

Each system ran the same custom Linux UltraLight 2.6.24 kernel, with patches for FAST TCP and other TCP performance tunings built in.

Based on the different hardware combination, systems were grouped into four main categories:

A) 1U Servers with Fiber Channel Storage System + onboard Intel CX4 10GE NIC

8 of  the 1U twin systems were outfitted with QLogic  fiberchannel HBAs connected to Data Direct Networks storage system. Maximum read/write speeds between ??.

B) 1U Servers with Local Storage + Myricom10 GbE NIC (dCache Head Nodes)

4 of the 1U systems were outfitted with Intel Oplin single port 10GbE NIC and local storage with RAID-0 was used to source/sink data.
Maximum read/write speeds at 200MB/s.

C) 1U Servers with Local Storage with on board 10GE CX4 NIC (dCache pool nodes / individual systems)

20 of 1the U systems were configured with Myricom 10GbE NIC while the 4 hard drives were placed in RAID-0 configuration. Maximum read/write speeds at 200MB/s.

D) Sun Thumpers with Local Storage + Myricom 10 GbE NIC (disk servers)

The 4 x 2U servers were whitebox systems with Myricom 10GbE NIC and each system having 12 hard disks connected through Areca 1231ML RAID Controller and placed in RAID-0. Maximum read/write speeds at 500MB/s.