As VMworld 2010 is right around the corner, I wanted to take a few minutes to make a plea to all attendees. 

This year, IF you receive a bag or backpack that you just don’t want, please don’t throw it away, but instead take it home, go to the dollar store and fill  the backpack with pencils, crayons, paper and erasers and donate it to your local school system.  You would be AMAZED to find out the numbers of children who don’t get backpacks and whose familes can not afford the costly school supplies that are required each year.  You will be making some family happy and you’ll get the name “VMware” marketed throughout the schools, getting the next generation of techno geeks ready to learn all about virtualization.

Thanks for the consideration!

(Updated 7/27/2010 – 11 am EST – added info on power and tower options)

 When you think about blade servers, you probably think, “they are too expensive.” When you think about doing a VMware project, you probably think, “my servers are too old” or “I can’t afford new servers.” For $8 per GB, you can have blade servers preloaded with VMware ESXi 4.1 AND 4TB of storage! Want to know how? Keep reading. 

No, I’m not smoking something. I’ve done the configuration and I can show you how to achieve a 4 TB SAN and 3 ESX hosts on blade servers with the IBM BladeCenter S. Before I can explain what I’ve done, let me give you the basics of the IBM BladeCenter S. 

Overview of the IBM BladeCenter S 

At 7U high, the chassis of the IBM BladeCenter S is the same height as the original IBM BladeCenter (now called the BladeCenter E). The chassis uses the same blade servers as the rest of the IBM blade chassis family, but the chassis holds only 6 blade servers – primarily due to the addition of locally attached storage drives. In addition, the chassis has the option to add a DVD drive for access to local media. 

In contrast, the SAS RAID Controller Module (IBM part #43W3584) allows for you to pool the storage and offer access to these arrays to each blade server that has the SAS Connectivity Card installed. Volumes that are created can be assigned to a specific blade or shared by several blade servers. The IBM SAS RAID Controller Module supports RAID levels 0, 1, 5 and 10 and each module also comes with RAID battery backup module. There are some caveats to be aware of: only SAS or NL SAS drives are supported (no SATA); the maximum volume size is currently limited to 2TB and the maximum number volumes each blade server can have is 8 (for a total of 48 volumes per chassis.) Another important thing to take note is that you must have 2 x SAS RAID Controller Modules, which sit in I/O Bays #3 and #4. This provides a redundant connection for each blade server with the SAS Connectivity Card. In fact, since I brought it up, let’s take a closer look at how the modules work in the IBM BladeCenter S. 

BladeCenter S a Look From BehindWhen you look at the back of the IBM BladeCenter S, it may look confusing, but don’t worry – it’s pretty straight forward. The chassis comes standard with 2 x 1450w power supplies standard and a single Advanced Management Module. If you are using high wattage blade servers or using the second DSM, you probably will need the 2nd set of power supplies (IBM part # 46C7438). If your budget is tight and you can’t afford to pony up the additional $599 U.S. list I recommend you take advantage of IBM’s Power Calculator prior to purchase to see if you need the 2nd set of power supplies. Following the design of the other IBM BladeCenter chassis, the IBM BladeCenter is cooled by a set of 4 redundant hot-swap blower modules. Don’t bother looking for any other fans or cooling devices, because you won’t find them. These four blowers cool the entire chassis, modules and blade servers. 

Management 
The Advanced Management Module (AMM) is the device that provides you with LOCAL keyboard, video and mouse connectivity (although only USB for keyboard and mouse) as well as an ethernet port to connect into your management network. The AMM gives you the ability to manage / monitor all of the chassis’ thermals as well as remotely control the blade servers and the I/O modules. In all honesty, the AMM is feature rich, so if you want to take a peek at what it can do, take at look at this IBM BladeCenter S Advanced Management Module simulator. Unlike the other IBM blade chassis, there is not an option for a redundant AMM, however in the event of a failure your blade servers, I/O modules, fans and blowers will continue to function without penalty. 

I/O Architecture 

The biggest technical confusion I see from engineers and customers alike is around the I/O layout of the blade chassis. The IBM BladeCenter S is a bit dissimilar to the other IBM chassis in the BladeCenter family so let me explain how it works. There are 4 I/O bays in the IBM BladeCenter S. The 1st I/O bay maps to the NICs that come on the motherboard of each blade server. If you are familiar with rack-mount servers, you know they typically have 2 x 1Gb Ethernet ports. The IBM Blade servers are no different – they also have 2 x 1Gb Ethernet NICs. In order for them to be “lit up” you need to have a module in bay 1 that can allow the signal from the blade server to extend out of the chassis. To simplify things – think of having a power outlet in the wall at home and connecting an extension cord to it so you can turn on a light that is a few feet away. The same rudimentary concept is how it works in the blade infrastructure. The only difference is, with the IBM BladeCenter S, both NIC ports 0 and 1 go to the I/O Module Bay #1. This means if that module has an issue, then those 2 NICs located on the motherboard of each blade server will be dead. There is no redundancy with the onboard NICs in the IBM BladeCenter S (unlike the other IBM BladeCenter chassis.) Why did IBM design it this way? Well, the original target market for the IBM BladeCenter S was small businesses and remote offices. When you look at those environments – how many have redundant NICs for the rack / tower server NICs? Odds are none. With that in mind, IBM designed the BladeCenter S to only have a single I/O module for both onboard NICs. Never fear, though. After a few months, IBM revised the design to allow for I/O Module #2 to provide an additional 2 NICs, using the 2/4 port Ethernet adapter (IBM part # 44W4479) on each blade server. The card is designed to provide 4 Ethernet Ports, however with the BladeCenter S, only 2 ports are connected. Therefore with network modules in I/O Module Bay 1 and 2 you can get 4 NICs. Add this to the 2 x SAS storage cards that we mentioned in the sections above and you “should” have adequate architecture to provide a VMware environment. 

The $32,000 Design 

Now that I’ve spent a few moments telling you what the IBM BladeCenter S is all about, perhaps you understand the potential. So how did I get to the $32,000 design that enables you to have 4 TB and 3 ESX hosts? I won’t devulge in the actual bill of materials, but here’s what I came up with: 

• 1 x IBM BladeCenter S chassis
• 1 x Disk Storage Module
• 4 x 1 TB Near Line Storage Disk Drives
• 1 set of 1450W Power Supplies
• 2 x Server Connectivity (Ethernet) Modules
• 2 x SAS RAID Connectivity Modules
• 1 x DVD
• 3 x HS22 blade servers – each with 2 x Intel E5620 Xeon Processors, 24GB RAM, SAS Connectivity Card, ESXi 4.1 USB Memory Key

Total U.S. List Price (as of 7/26/2010): $30,768.00 

Yes, I also know that HP has an offering (the BladeSystem C3000) that could compete to this design – however IBM is the only blade server vendor that offers dedicated local disk storage. HP’s design takes up a blade server slot. Perhaps I’ll write up something on this in the future. 

THIS SECTION ADDED ON 7/27/2010 – I’ve added this section to cover a couple of pieces that I left off in the original post.

Power
A really valuable feature of the IBM BladeCenter S is the ability to run on 110 v or 220 v.  Use of 110 v is ideal for remote or small offices.  The BladeCenter S power supplies are auto-sensing so you can use the same power supplies.  There are a few power categories to choose from with the BladeCenter S:

• Redundant AC Power Source – in this policy, the power limit is set to equal the capacity of N power modules.  According to IBM’s Implementing the IBM
  BladeCenter S Chassis Redbook, this policy is the most conservative approach and is recommended when all four power modules are installed. When the chassis is correctly wired with dual AC power sources, one AC power source can fail without affecting your blade server operation. 
• Redundant Power Module Policy – in this policy, the power limit equals the capacity of one less than the number of power modules installed (more than one power module must be present). One power module can fail without affecting blade server operation.  If a single power module fails, all the blade servers that are powered on will continue to operate at normal performance levels.
• No Redundancy – all power modules are used, there is no redundancy and if you lose a power supply and the power demands exceed the capacity of the available power modules, the chassis will power down… Not recommended.

Office Enablement Kit
For those environments where a standard server rack is not ideal, IBM offers the Office Enablement Kit (part # 201886X).  This adds an additional few hundred dollars, but it gives you an 11u rack enclosure, complete with front and rear locking doors and wheels.  It also comes with an acoustic attenuation module that helps muffle the sound.  (YouTube video on this can be seen here.)  As mentioned above, the IBM BladeCenter S is only 7U tall, so the additional 4U can be used for an optional Flat Panel Monitor kit (shown in the image to the left) or perhaps additional storage or networking.  This kit really helps to finalize an “all-in-one” solution for small or remote environments.

Yes, I know it’s only 4 x 1TB drives, and I know it’s Near-Line SAS drives, however it is enough resources to help YOU create that VMware infrastructure that you need. Of course, the VMware licensing will be extra, but I just saved you a ton of money – now you can afford it… 

So – what do you think? Is this appealing, or is this just a pipe dream? Let me know your thoughts. I’m really interested in getting an idea of whether this design would really work in your world.

(Updated to include links to results)

I’ve had a few questions lately about “the best” blade server to use for virtualization – specifically VMware virtualization. While the obvious answer is “it depends”, I thought it would be an interesting approach to identify the blade servers that ranked in the top 5 in VMware’s VMmark benchmark.  Before I begin, let me explain what the VMmark testing is about.   VMmark enables equipment manufacturers, software vendors, system integrators and other organizations to:

• Measure virtual machine performance accurately and reliably
• Determine the performance of different hardware and virtualization platforms
• Make appropriate hardware decisions for your virtual infrastructure

VMware developed VMmark as a standard methodology for comparing virtualized systems. According to VMware’s VMmark website, the benchmark system in VMmark is comprised of a series of “sub-tests” that are derived from commonly used load-generation tools, as well as from benchmarks developed by the Standard Performance Evaluation Corporation (SPEC®). In parallel to VMmark, VMware is a member of the SPEC Virtualization subcommittee and is working with other SPEC members to create the next generation virtualization benchmark.

In testing the terms, a “tile” is simply a collection of virtual machines (VM’s) that are executing a set of diverse workloads designed to represent a natural work environment.   The total number of tiles that a server can handle provides a detailed measurement of that server’s consolidation capacity.  The more tiles, the better.  The faster the performance, the better.

THE RESULTS (as of 6/2/2010)…click on the link to open the details

24 Cores (4 Sockets)
HP ProLiant BL685c G6 (Six Core AMD Opteron 8435) running VMware ESX v4.0 – 29.19@20 tiles (published 7/14/2009)
HP ProLiant BL680c G5 (Six Core Intel Xeon E7458) running VMware ESX v3.5.0 Update 3 - 18.64@14 tiles (published 3/30/2009)

16 Cores (4 Sockets)
Dell PowerEdge M905 (Four Core AMD Opteron 8393 SE) running VMware ESX v4.0 – 22.90@17 tiles (published 6/19/2009)
HP ProLiant BL685 G6 (Four Core AMD Opteron 8389) running VMware ESX v4.0 – 20.87@14 tiles (published 4/24/2009)

12 Cores (2 Sockets)
Cisco UCS B250 M2 (Six Core Intel Xeon X5680) running VMware ESX v4.0 Update 1 – 35.83@26 tiles (published 4/6/2010)
Fujitsu BX922 S2 (Six Core Intel Xeon X5680) running VMware ESX v4.0 Update 1 – 32.89@24 tiles (published 4/6/2010)

8 Cores (2 Sockets)
Fujitsu BX922 S2 (Four Core Intel Xeon X5677) running VMware ESX v4.0 Update 1 – 27.99@18tiles(published 5/10/2010)
HP ProLiant BL490c G6 (Four Core Intel Xeon X5570) runningVMware ESX v4.0 – 25.27@17tiles (published 4/20/2010)

THE WINNER IS…
Cisco UCS B250 M2 running VMware ESX v4.0 Update 1 – 35.83 with 26 tiles

Cisco’s Winning Configuration
So – how did Cisco reach the top server spot?  Here’s the configuration:

server config:

• 2 x Intel Xeon X5680 Processors
• 192GB of RAM (48 x 4GB)
• 1 x Converged Network Adapter (Cisco UCS VIC M81KR)

storage config:

• EMC CX4-240
• Cisco MDS 9134
• 1173.48GB Used Disk Space
• 1024MB Array Cache
• 50 disks used on 5 enclosures/shelves (1 with 14 disk, 4 with 9 disks)
• 55 LUNs used
  *21 at 38GB (file server + mail server) over 20 x 73GB SSDs
  *5 at 38GB (file server + mail server) over 20 x 73GB SSDs
  *21 at 15GB (database) + 2 LUNs at 400GB (Standby, Webserver, Javaserver) over 16 x 450GB 15k disks
  *5 at 15GB (database)  over 16 x 450GB 15k disks
  * 1 LUN at 20GB (boot) over 5 x 300GB 15k disks
• RAID 0 for VMs, RAID 5 for VMware ESX 4.0 O/S

As you can see from the information above, the Cisco UCS B250 M2 is the clear winner above all of the blade server offerings.  As you can see, none of the Xeon 7500 blade servers have yet to be tested but when they do, I’ll be sure to let you know.

Cisco is getting some (more) recognition with their UCS blade server product, as they recently achieved the top position for “8 Core Server” on VMware’s VMmark benchmark tool.  VMmark is the industry’s first (and only credible) virtualization benchmark for x86-based computers.  According to the VMmark website, the Cisco UCS B200 blade server reached a score of 25.06 @ 17 tiles.  A “tile” is simple a collection of virtual machines (VM’s) that are executing a set of diverse workloads designed to represent a natural work environment.   The total number of tiles that a server can handle provides a detailed measurement of that server’s consolidation capacity.

Cisco’s Winning Configuration
So – how did Cisco reach the top server spot?  Here’s the configuration:

server config:

• 2 x Intel Xeon X5570 Processors
• 96GB of RAM (16 x 8GB)
• 1 x Converged Network Adapter (Cisco UCS M71KR-Q)

storage config:

• EMC CX4-240
• Cisco MDS 9130
• 1154.27GB Used Disk Space
• 1024MB Array Cache
• 41 disks used on 4 enclosures/shelves (1 with 14 disk, 3 with 9 disks)
• 37 LUNs used
  *17 at 38GB (file server + mail server) over 20 x 73GB SSDs
  *17 at 15GB (database) + 2 LUNs at 400GB (Misc) over 16 x 450GB 15k disks
  * 1 LUN at 20GB (boot) over 5 x 300GB 15k disks
• RAID 0 for VMs, RAID 5 for VMware ESX 4.0 O/S

While first place on the VMmark page (8 cores) shows Fujitsu’s RX300, it’s important to note that it was reached using Intel’s W5590 processor – a processor that is designed for “workstations” – not servers.  Second place, of server processors, currently shows HP’s BL490 with 24.54 (@ 17 tiles)

Thanks to Omar Sultan (@omarsultan) for Tweeting about this and to Harris Sussman for blogging about it.