(Updated 7/27/2010 – 11 am EST – added info on power and tower options)

 When you think about blade servers, you probably think, “they are too expensive.” When you think about doing a VMware project, you probably think, “my servers are too old” or “I can’t afford new servers.” For $8 per GB, you can have blade servers preloaded with VMware ESXi 4.1 AND 4TB of storage! Want to know how? Keep reading. 

No, I’m not smoking something. I’ve done the configuration and I can show you how to achieve a 4 TB SAN and 3 ESX hosts on blade servers with the IBM BladeCenter S. Before I can explain what I’ve done, let me give you the basics of the IBM BladeCenter S. 

Overview of the IBM BladeCenter S 

At 7U high, the chassis of the IBM BladeCenter S is the same height as the original IBM BladeCenter (now called the BladeCenter E). The chassis uses the same blade servers as the rest of the IBM blade chassis family, but the chassis holds only 6 blade servers – primarily due to the addition of locally attached storage drives. In addition, the chassis has the option to add a DVD drive for access to local media. 

In contrast, the SAS RAID Controller Module (IBM part #43W3584) allows for you to pool the storage and offer access to these arrays to each blade server that has the SAS Connectivity Card installed. Volumes that are created can be assigned to a specific blade or shared by several blade servers. The IBM SAS RAID Controller Module supports RAID levels 0, 1, 5 and 10 and each module also comes with RAID battery backup module. There are some caveats to be aware of: only SAS or NL SAS drives are supported (no SATA); the maximum volume size is currently limited to 2TB and the maximum number volumes each blade server can have is 8 (for a total of 48 volumes per chassis.) Another important thing to take note is that you must have 2 x SAS RAID Controller Modules, which sit in I/O Bays #3 and #4. This provides a redundant connection for each blade server with the SAS Connectivity Card. In fact, since I brought it up, let’s take a closer look at how the modules work in the IBM BladeCenter S. 

BladeCenter S a Look From BehindWhen you look at the back of the IBM BladeCenter S, it may look confusing, but don’t worry – it’s pretty straight forward. The chassis comes standard with 2 x 1450w power supplies standard and a single Advanced Management Module. If you are using high wattage blade servers or using the second DSM, you probably will need the 2nd set of power supplies (IBM part # 46C7438). If your budget is tight and you can’t afford to pony up the additional $599 U.S. list I recommend you take advantage of IBM’s Power Calculator prior to purchase to see if you need the 2nd set of power supplies. Following the design of the other IBM BladeCenter chassis, the IBM BladeCenter is cooled by a set of 4 redundant hot-swap blower modules. Don’t bother looking for any other fans or cooling devices, because you won’t find them. These four blowers cool the entire chassis, modules and blade servers. 

Management 
The Advanced Management Module (AMM) is the device that provides you with LOCAL keyboard, video and mouse connectivity (although only USB for keyboard and mouse) as well as an ethernet port to connect into your management network. The AMM gives you the ability to manage / monitor all of the chassis’ thermals as well as remotely control the blade servers and the I/O modules. In all honesty, the AMM is feature rich, so if you want to take a peek at what it can do, take at look at this IBM BladeCenter S Advanced Management Module simulator. Unlike the other IBM blade chassis, there is not an option for a redundant AMM, however in the event of a failure your blade servers, I/O modules, fans and blowers will continue to function without penalty. 

I/O Architecture 

The biggest technical confusion I see from engineers and customers alike is around the I/O layout of the blade chassis. The IBM BladeCenter S is a bit dissimilar to the other IBM chassis in the BladeCenter family so let me explain how it works. There are 4 I/O bays in the IBM BladeCenter S. The 1st I/O bay maps to the NICs that come on the motherboard of each blade server. If you are familiar with rack-mount servers, you know they typically have 2 x 1Gb Ethernet ports. The IBM Blade servers are no different – they also have 2 x 1Gb Ethernet NICs. In order for them to be “lit up” you need to have a module in bay 1 that can allow the signal from the blade server to extend out of the chassis. To simplify things – think of having a power outlet in the wall at home and connecting an extension cord to it so you can turn on a light that is a few feet away. The same rudimentary concept is how it works in the blade infrastructure. The only difference is, with the IBM BladeCenter S, both NIC ports 0 and 1 go to the I/O Module Bay #1. This means if that module has an issue, then those 2 NICs located on the motherboard of each blade server will be dead. There is no redundancy with the onboard NICs in the IBM BladeCenter S (unlike the other IBM BladeCenter chassis.) Why did IBM design it this way? Well, the original target market for the IBM BladeCenter S was small businesses and remote offices. When you look at those environments – how many have redundant NICs for the rack / tower server NICs? Odds are none. With that in mind, IBM designed the BladeCenter S to only have a single I/O module for both onboard NICs. Never fear, though. After a few months, IBM revised the design to allow for I/O Module #2 to provide an additional 2 NICs, using the 2/4 port Ethernet adapter (IBM part # 44W4479) on each blade server. The card is designed to provide 4 Ethernet Ports, however with the BladeCenter S, only 2 ports are connected. Therefore with network modules in I/O Module Bay 1 and 2 you can get 4 NICs. Add this to the 2 x SAS storage cards that we mentioned in the sections above and you “should” have adequate architecture to provide a VMware environment. 

The $32,000 Design 

Now that I’ve spent a few moments telling you what the IBM BladeCenter S is all about, perhaps you understand the potential. So how did I get to the $32,000 design that enables you to have 4 TB and 3 ESX hosts? I won’t devulge in the actual bill of materials, but here’s what I came up with: 

• 1 x IBM BladeCenter S chassis
• 1 x Disk Storage Module
• 4 x 1 TB Near Line Storage Disk Drives
• 1 set of 1450W Power Supplies
• 2 x Server Connectivity (Ethernet) Modules
• 2 x SAS RAID Connectivity Modules
• 1 x DVD
• 3 x HS22 blade servers – each with 2 x Intel E5620 Xeon Processors, 24GB RAM, SAS Connectivity Card, ESXi 4.1 USB Memory Key

Total U.S. List Price (as of 7/26/2010): $30,768.00 

Yes, I also know that HP has an offering (the BladeSystem C3000) that could compete to this design – however IBM is the only blade server vendor that offers dedicated local disk storage. HP’s design takes up a blade server slot. Perhaps I’ll write up something on this in the future. 

THIS SECTION ADDED ON 7/27/2010 – I’ve added this section to cover a couple of pieces that I left off in the original post.

Power
A really valuable feature of the IBM BladeCenter S is the ability to run on 110 v or 220 v.  Use of 110 v is ideal for remote or small offices.  The BladeCenter S power supplies are auto-sensing so you can use the same power supplies.  There are a few power categories to choose from with the BladeCenter S:

• Redundant AC Power Source – in this policy, the power limit is set to equal the capacity of N power modules.  According to IBM’s Implementing the IBM
  BladeCenter S Chassis Redbook, this policy is the most conservative approach and is recommended when all four power modules are installed. When the chassis is correctly wired with dual AC power sources, one AC power source can fail without affecting your blade server operation. 
• Redundant Power Module Policy – in this policy, the power limit equals the capacity of one less than the number of power modules installed (more than one power module must be present). One power module can fail without affecting blade server operation.  If a single power module fails, all the blade servers that are powered on will continue to operate at normal performance levels.
• No Redundancy – all power modules are used, there is no redundancy and if you lose a power supply and the power demands exceed the capacity of the available power modules, the chassis will power down… Not recommended.

Office Enablement Kit
For those environments where a standard server rack is not ideal, IBM offers the Office Enablement Kit (part # 201886X).  This adds an additional few hundred dollars, but it gives you an 11u rack enclosure, complete with front and rear locking doors and wheels.  It also comes with an acoustic attenuation module that helps muffle the sound.  (YouTube video on this can be seen here.)  As mentioned above, the IBM BladeCenter S is only 7U tall, so the additional 4U can be used for an optional Flat Panel Monitor kit (shown in the image to the left) or perhaps additional storage or networking.  This kit really helps to finalize an “all-in-one” solution for small or remote environments.

Yes, I know it’s only 4 x 1TB drives, and I know it’s Near-Line SAS drives, however it is enough resources to help YOU create that VMware infrastructure that you need. Of course, the VMware licensing will be extra, but I just saved you a ton of money – now you can afford it… 

So – what do you think? Is this appealing, or is this just a pipe dream? Let me know your thoughts. I’m really interested in getting an idea of whether this design would really work in your world.

They make it look so complicated in the movies.  Detailed covert operations with the intent to hack into a casino’s mainframe preceeded by weeks of staged planned rehearsals, but I’m here to tell you it’s much easier than that.  

This is my story of how I had 20 seconds of complete access to The Venetian Casino’s data center, and lived to tell about it.

Now, before I go on, I must apologize to all of the hopeful future visitors of The Venetian Hotel and Casino in Las Vegas.  I have a feeling that security will get tightened down following this story.  So, let me set the stage.  I was asked to attend the HP Technology Forum in Las Vegas to blog about the event and cover it via Twitter.  Part of the agenda for the blogging attendees, and the media, was a tour of The Venetian Hotel and Casino’s Data Center.  I jumped at the opportunity.  I always enjoy customer data centers since it gives me the chance to view what “real people” are doing.  Once we landed at the Sands Hotel and Casino, I was fortunate to be part of the first group of 15 or 20 people so we hiked over to the hidden area where the data center was located.  Well, it wasn’t really hidden, just a door that said, “Employees Only.”  We rode up an elevator one or two floors and came out into a desolate barren hallway that gave the appearance of a hospital.  The hallway made a few turns and we came out into a set of office cubicles.  On the other side were the double doors leading to the data center.  No metal barred doors, no retina scans, no lasers – just two plain doors leading into the data center.  I should’ve know this was a sign of things to come.

Quick Venetian Casino Data Center Facts

* 3 Major Data Centers: Las Vegas, Singapore and China
* 500 physical servers in Las Vegas (very little virtualization)
* They use pure Ethernet (IP) for the machines in Singapore; a mix of Serial and IP in Las Vegas 

As we entered the data center, I quickly realized The Venetian was a Dell shop.  Did I mention this was a tour hosted by HP?  After inquiring about the large numbers of Dell servers, the CTO of The Venetian Casino, Steve Vollmer, made it clear that they do use a wide variety of manufacturers for their servers, however they are 100% HP Networking.   Usually when you are given a customer tour by a vendor, like HP, it’s an ALL VENDOR customer – meaning they have HP everything  but this was a “real customer data center” tour.  Kudos to Michael Thacker and the HP Public Relations team on finding a real customer with a mix of vendors’ gear.  Which leads me to how I got left behind.

I’m a server guy – networking gear is out of my expertise, so while the rest of the media and bloggers were interviewing Mr. Vollmer, I started wondering around taking pictures (no video allowed.)  I noticed that all of the physical servers (Dell, IBM and HP) were using the on-board Ethernet adapters only.  Very interesting.  It was mentioned that the casino machines are simply computers, so it makes sense that all of the communication is IP based.  As I wondered around to the last aisle of racks, I came across an HP BladeSystem!  Very cool.  I thought they were stuck on old, aging equipment, but here stood the royal crown (INSERT angelic sounds and bright light here).  Sitting right above it was a Dell M1000e Blade Chassis.  Wow – these guys are true to being a multi-vendor customer.  As I was wiping up the drool off the floor, I heard the door slam to the Data Center.  NOTHING but the whirl of server fans and A/C air blowing around.   Now my first thought was that I was the last person and I’m about to get pulled out by the Network Admin who was on the tour with us.  I turned the corner to find no one.   I was alone!

You’ve got to be kidding me.  Surely I’m not standing in a Las Vegas casino’s data center BY MYSELF – but I was.  Now, many things ran through my head – so many movies have been made about breaking into casinos and here I was in the heart of millions of electronic dollars.  I’d like to be able to conclude this story with some amazing details on how I logged onto the casino servers and transferred money to my offshore account but I can’t.  a) I’m a good person.  I don’t steal, lie or cheat and b) I didn’t have a USB key on me.  The reality is, once I realized I was alone I knew I had to get out of the data center as quickly as possible, so I looked up at the overhead camera, gave a smile and walked out the double doors to quickly catch up to the group heading back downstairs.

There’s a few lessons to be learned here:
1) Real customers use a mix of vendors for their hardware – not just one vendor
2) Always carry a USB key with you – you never know when you need it
3) Don’t waste your time breaking in to a casino – just tour their data center with a large crowd

That’s my story.  Now I just hope I don’t get any calls from the F.B.I…

Disclaimer: airfare, accommodations and some meals are being provided by HP, however the content being blogged is solely my opinion and does not in anyway express the opinions of HP.

NOTE: IDC revised their report on May 28, 2010.  This post now includes those changes.

IDC reported on May 28, 2010 that worldwide server sales for Q1 2010 factory revenues increased 4.6  4.7% year over year to $10.4 billion in the first quarter of 2010 (1Q10).  They also reported the blade server market accelerated and continued its sharp growth in the quarter with factory revenue increasing 37.1% 37.2% year over year, with shipment growth increasing by 20.8% compared to 1Q09.  According to IDC, nearly 90% of all blade revenue is driven by x86 systems, a segment in which blades now represent 18.8% of all x86 server revenue.

While the press release did not provide details of the market share for all of the top 5 blade vendors, they did provide data for the following: 

#1 market share: HP increased their market share from 52.4% in Q4 2009 to 56.2% in Q1 2010

#2 market share: IBM decreased their market share from 35.1% in Q4 2009 to 23.6% in Q1 2010.

The remaining 20.2% of market share was not mentioned, but I imagine they are split between Dell and Cisco.  In fact, based on the fact that Cisco was not even mentioned in the IDC report, I’m willing to bet a majority of that  I’m working on getting some visibility into clarification on that (if you’re with Dell or Cisco and can help, please shoot me an email.)

According to Jed Scaramella, senior research analyst in IDC’s Datacenter and Enterprise Server group,  ”"In the first quarter of 2009, we observed a lot of business in the mid-market as well as refresh activity of a more transactional nature; these factors have driven x86 rack-based revenue to just below 1Q08 value. Blade servers, which are more strategic in customer deployments, continue to accelerate in annual growth rates. The blade segment fared relatively well during the 2009 downturn and have increased revenue value by 13% from the first quarter of 2008.”

For the full IDC report covering the Q1 2010 Worldwide Server Market, please visit http://www.idc.com/getdoc.jsp?containerId=prUS22356410

new link: http://www.idc.com/getdoc.jsp?containerId=prUS22360110

Updated 5/24/2010 – I’ve received some comments about expandability and I’ve received a correction about the speed of Dell’s memory, so I’ve updated this post.  You’ll find the corrections / additions below in GREEN.

Since I’ve received a lot of comments from my post on the Dell FlexMem Bridge technology, I thought I would  do an unbiased comparison between Dell’s FlexMem Bridge technology (via the PowerEdge 11G M910 blade server) vs IBM’s MAX5 + HX5 blade server offering.  In summary both offerings provide the Intel Xeon 7500 CPU plus the ability to add “extended memory“ offering value for virtualization, databases and any other workloads that benefit from large amounts of memory.

The Contenders

IBM
IBM’s extended memory solution is a two part solution consisting of the HX5 blade server PLUS the MAX5 memory blade.

• HX5 Blade Server
  I’ve spent considerable time on previous blogs detailing the IBM HX5, so please jump over to those links to dig into the specifics, but at a high level, the HX5 is IBM’s 2 CPU blade server that offers the Intel Xeon 7500 CPU.   The HX5 is a 30mm, ”single wide” blade server therefore you can fit up to 14 in an IBM BladeCenter H blade chassis. 
• MAX5
  The MAX 5 offering from IBM can be thought of as a “memory expansion blade.”  Offering an additional 24 memory DIMM slots, the MAX5 when coupled with the HX5 blade server, provides a total of 40 memory DIMMs.    The MAX5 is a standard “single wide”, 30mm form factor so when used with a single HX5 two IBM BladeCenter H server bays are required in the chassis.

DELL
Dell’s approach to extended memory is a bit different.  Instead of relying on a memory blade, Dell starts with the M910 blade server and allows users to use 2 CPUs plus their FlexMem Bridge to access the memory DIMMs of the 3rd and 4th CPU sockets.  For details on the FlexMem Bridge, check out my previous post.

• PowerEdge 11G M910 Blade Server
  The M910 is a 4 CPU capable blade server with 32 memory DIMMs.  This blade server is a full-height server therefore you can fit 8 servers inside the Dell M1000e blade chassis.

The Face-Off

ROUND 1 – Memory Capacity
When we compare the memory DIMMs available on each, we see that Dell’s offering comes up with 32 DIMMs vs IBM’s 40 DIMMs.  However, IBM’s solution of using the HX5 blade server + the MAX 5 memory expansion has a current maximum memory size is 8Gb whereas Dell offers a max memory size of 16Gb.  While this may change in the future, as of today, Dell has the edge so I have to claim: 

Round 1 Winner:  Dell

ROUND 2 – Memory Performance
As many comments came across on my posting of the Dell FlexMem Bridge technology the other day, several people pointed out that the memory performance is something that needs to be considered when comparing technologies.  Dell’s FlexMem Bridge offering reportedly runs at a maximum memory speed of 833Mhz,  runs at a max of 1066Ghz, but is dependent upon the speed of the processor.  A processor that has a 6.4GT QPI supports memory @ 1066Ghz ; a processor that supports 5.8GT/s QPI supports memory at 978Mhz, and a processor with a QPI speed of 4.8GT runs memory at 800Mhz.  This is a component of Intel’s Xeon 7500 architecture so it should be the same regardless of the server vendor.  Looking at IBM, we see the  HX5 blade server memory runs at a maximum of 978Mhz.    However, when you attach the MAX5 to the HX5 for the additional memory slots, however, the memory runs at speed of 1066Mhz, regardless of the speed of the CPU installed.  While this appears to be black magic, it’s really the results of IBM’s proprietary eXa scaling – something that I’ll cover in detail at a later date.   Although the HX5 blade server memory, when used by itself, does not have the ability to achieve 1066Ghz, this comparison is based on the Dell PowerEdge 11G M910 vs the IBM HX5+MAX5.  With that in mind, the ability to run the expanded memory at 1066Mhz gives IBM the edge in this round.

Round 2 Winner:  IBM

ROUND 3 – Server Density
This one is pretty straight forward.  IBM’s HX5 + MAX5 offering takes up 2 server bays, so in the IBM BladeCenter H, you can only fit 7 systems.  You can only fit 4 BladeCenter H chassis in a 42u rack, therefore you can fit a max of 28  IBM HX5 + MAX5 systems into a rack.

The Dell PowerEdge 11G M910 blade server is a full height server, so you can fit 8 servers into the Dell M1000e chassis.  4 Dell chassis will fit in a 42u rack, so you can get 32 Dell M910′s into a rack.

Round 3 Winner:  Dell

(NEW) ROUND 4 – Expandability
It was mentioned several times in the comments that expandability should have been reviewed as well.  When we look at Dell’s design, we see there two expansion options: run the Dell PowerEdge 11G M910 blade with 2 processors and the FlexMem Bridge, or run them with 4 processors and remove the FlexMem Bridge.

The modular design of the IBM eX5 architecture allows for a user to add memory (MAX5), add processors (2nd HX5) or both (2 x HX5 + 2 x MAX5).  This provide users with a lot of flexibility to choose a design that meets their workload.

Choosing a winner for this round is tough, as there a different ways to look at this:

Maximum CPUs in a server: TIE – both IBM and Dell can scale to 4 CPUs. 
Maximum CPU density in a 42u rack:  Dell wins with 32 x 4 CPU servers vs IBM’s 12.
Maximum Memory in a server: IBM with 640Gb using 2 x HX5 and 2 x MAX5
Max Memory density in a 42u Rack: Dell wins with 16Tb

Round 4 Winner:  TIE
Summary
While the fight was close, with a 2 to 1 win, it is clear the overall winner is Dell.  For this comparison, I tried to keep it focused on the memory aspect of the offerings. 

On a final note, at the time of this writing, the IBM MAX 5 memory expansion has not been released for general availability, while Dell is shipping their M910 blade server. 

There may be other advantages relative to processors that were not considered for this comparison, however I welcome any thoughts or comments you have. 

UPDATED 4/14/2010 - IBM announced today their newest blade server using the POWER7 processor.  The BladeCenter PS700, PS701 and PS702 servers are IBM’s latest addition to the blade server family, behind last month’s announcement of the BladeCenter HX5 server, based on the Nehalem EX processor.  The POWER7 processor-based PS700, PS701 and PS702 blades support AIX, IBM i, and Linux operating systems.  (For Windows operations systems, stick with the HS22 or the HX5.)  For those of you not familiar with the POWER processor, the POWER7 processor is a 64-bit, 4 core with 256KB L2 cache per core and 4MB L3 cache per core.  Today’s announcement reflects IBM’s new naming schema as well.  Instead of being labled “JS” blades like in the past, the new POWER family blade servers will be titled “PS” – for Power Systems.  Finally – a naming schema that makes sense.  (Will someone explain what IBM’s “LS” blades stand for??)  Included in today’s announcement are the PS700, PS701 and PS702 blade.  Let’s review each.

IBM BladeCenter PS700
The PS700 blade server is a single socket, single wide 4-core 3.0GHz POWER7
processor-based server that has the following:

• 8 DDR3 memory slots (available memory sizes are 4GB, 1066Mhz or 8GB, 800Mhz)
• 2 onboard 1Gb Ethernet ports
• integrated SAS controller supporting RAID levels 0,1 or 10
• 2 onboard disk drives (SAS or Solid State Drives)
• one PCIe CIOv expansion card slot
• one PCIe CFFh expansion card slot

The PS700 is supported in the BladeCenter E, H, HT and S chassis.  (Note, support in the BladeCenter E requires an Advanced Management Module and a minimum of two 2000 watt power supplies.)

IBM BladeCenter PS701
The PS701 blade server is a single socket, single wide 8-core 3.0GHz POWER7
processor-based server that has the following:

• 16 DDR3 memory slots (available memory sizes are 4GB, 1066Mhz or 8GB, 800Mhz)
• 2 onboard 1Gb Ethernet ports
• integrated SAS controller supporting RAID levels 0,1 or 10
• 2 1 onboard disk drive (SAS or Solid State Drives)
• one PCIe CIOv expansion card slot
• one PCIe CFFh expansion card slot

The PS701 is supported in the BladeCenter H, HT and S chassis only. 

IBM BladeCenter PS702
The PS702 blade server is a dual socket, double-wide 16-core (via 2 x 8-core CPUs) 3.0GHz POWER7 processor-based server that has the following:

• 32 DDR3 memory slots (available memory sizes are 4GB, 1066Mhz or 8GB, 800Mhz)
• 4 onboard 1Gb Ethernet ports
• integrated SAS controller supporting RAID levels 0,1 or 10
• 2 onboard disk drives (SAS or Solid State Drives)
• 2 PCIe CIOv expansion card slots
• 2 PCIe CFFh expansion card slots

The PS702 is supported in the BladeCenter H, HT and S chassis only. 

For more technical details on the PS blade servers, please visit IBM’s redbook page at: http://www.redbooks.ibm.com/redpieces/abstracts/redp4655.html?Open

Cisco recently announced their first blade offering with the Intel Xeon 7500 processor, known as the ”Cisco UCS B440-M1 High-Performance Blade Server.”  This new blade is a full-width blade that offers 2 – 4 Xeon 7500 processors and 32 memory slots, for up to 256GB RAM, as well as 4 hot-swap drive bays.  Since the server is a full-width blade, it will have the capability to handle 2 dual-port mezzanine cards for up to 40 Gbps I/O per blade. 

Each Cisco UCS 5108 Blade Server Chassis can house up to four B440 M1 servers (maximum 160 per Unified Computing System). 

How Does It Compare to the Competition?
Since I like to talk about all of the major blade server vendors, I thought I’d take a look at how the new Cisco B440 M1 compares to IBM and Dell.  (HP has not yet announced their Intel Xeon 7500 offering.)

Processor Offering
Both Cisco and Dell offer models with 2 – 4 Xeon 7500 CPUs as standard.  They each have variations on speeds – Dell has 9 processor speed offerings; Cisco hasn’t released their speeds and IBM’s BladeCenter HX5 blade server will have 5 processor speed offerings initially.  With all 3 vendors’ blades, however, IBM’s blade server is the only one that is designed to scale from 2 CPUs to 4 CPUs by connecting 2 x HX5 blade servers.  Along with this comes their “FlexNode” technology that enables users to have the 4 processor blade system to split back into 2 x 2 processor systems at specific points during the day.  Although not announced, and purely my speculation, IBM’s design also leads to a possible future capability of connecting 4 x 2 processor HX5′s for an 8-way design.  Since each of the vendors offer up to 4 x Xeon 7500′s, I’m going to give the advantage in this category to IBM.  WINNER: IBM

Memory Capacity
Both IBM and Cisco are offering 32 DIMM slots with their blade solutions, however they are not certifying the use of 16GB DIMMs – only 4GB and 8GB DIMMs, therefore their offering only scales to 256GB of RAM.  Dell claims to offers 512GB DIMM capacity on their the PowerEdge 11G M910 blade server, however that is using 16GB DIMMs.  REalistically, I think the M910 would only be used with 8GB DIMMs, so Dell’s design would equal IBM and Cisco’s.  I’m not sure who has the money to buy 16GB DIMMs, but if they do – WINNER: Dell (or a TIE)

Server Density
As previously mentioned, Cisco’s B440-M1 blade server is a “full-width” blade so 4 will fit into a 6U high UCS5100 chassis.  Theoretically, you could fit 7 x UCS5100 blade chassis into a rack, which would equal a total of 28 x B440-M1′s per 42U rack.Overall, Cisco’s new offering is a nice addition to their existing blade portfolio.  While IBM has some interesting innovation in CPU scalability and Dell appears to have the overall advantage from a server density, Cisco leads the management front. 

Dell’s PowerEdge 11G M910 blade server is a “full-height” blade, so 8 will fit into a 10u high M1000e chassis.  This means that 4 x M1000e chassis would fit into a 42u rack, so 32 x Dell PowerEdge M910 blade servers should fit into a 42u rack.

IBM’s BladeCenter HX5 blade server is a single slot blade server, however to make it a 4 processor blade, it would take up 2 server slots.  The BladeCenter H has 14 server slots, so that makes the IBM solution capable of holding 7 x 4 processor HX5 blade servers per chassis.  Since the chassis is a 9u high chassis, you can only fit 4 into a 42u rack, therefore you would be able to fit a total of 28 IBM HX5 (4 processor) servers into a 42u rack.
WINNER: Dell

Management
The final category I’ll look at is the management.  Both Dell and IBM have management controllers built into their chassis, so management of a lot of chassis as described above in the maximum server / rack scenarios could add some additional burden.  Cisco’s design, however, allows for the management to be performed through the UCS 6100 Fabric Interconnect modules.  In fact, up to 40 chassis could be managed by 1 pair of 6100′s.  There are additional features this design offers, but for the sake of this discussion, I’m calling WINNER: Cisco.

Cisco’s UCS B440 M1 is expected to ship in the June time frame.  Pricing is not yet available.  For more information, please visit Cisco’s UCS web site at http://www.cisco.com/en/US/products/ps10921/index.html.

(Updated 4/22/2010 at 2:44 p.m.)
IBM officially announced the HX5 on Tuesday, so I’m going to take the liberty to dig a little deeper in providing details on the blade server. I previously provided a high-level overview of the blade server on this post, so now I want to get a little more technical, courtesy of IBM.  It is my understanding that the “general availability” of this server will be in the mid-June time frame, however that is subject to change without notice.

Block Diagram
Below is the details of the actual block diagram of the HX5.  There’s no secrets here, as they’re using the Intel Xeon 6500 and 7500 chipsets that I blogged about previously.

As previously mentioned, the value that the IBM HX5 blade server brings is scalability.  A user has the ability to buy a single blade server with 2 CPUs and 16 DIMMs, then expand it to 40 DIMMs with a 24 DIMM MAX 5 memory blade.  OR, in the near future, a user could combine 2 x HX5 servers to make a 4 CPU server with 32 DIMMs, or add a MAX5 memory DIMM to each server and have a 4 CPU server with 80 DIMMs. 

The diagrams below provide a more technical view of the the HX5 + MAX5 configs. Note, the “sideplanes” referenced below are actualy the “scale connector“.  As a reminder, this connector will physically connect 2 HX5 servers on the tops of the servers, allowing the internal communications to extend to each others nodes.  The easiest way to think of this is like a Lego .  It will allow a HX5 or a MAX5 to be connected together.  There will be a 2 connector, a 3 connector and a 4 connector offering. 

 (Updated) Since the original posting, IBM released the “eX5 Porfolio Technical Overview: IBM System x3850 X5 and IBM BladeCenter HX5” so I encourage you to go download it and give it a good read.  David’s Redbook team always does a great job answering all the questions you might have about an IBM server inside those documents. 

If there’s something about the IBM BladeCenter HX5 you want to know about, let me know in the comments below and I’ll see what I can do.

Thanks for reading!

InfoWorld.com posted on 3/22/2010 the results of a blade server shoot-out between Dell, HP, IBM and Super Micro. I’ll save you some time and help summarize the results of Dell, HP and IBM.

The Contenders
Dell, HP and IBM each provided blade servers with the Intel Xeon X5670 2.93GHz CPUs and at least 24GB of RAM in each blade.

The Tests
InfoWorld designed a custom suite VMware tests as well as several real-world performance metric tests. The VMware tests were composed of:

• a single large-scale custom LAMP application
• a load-balancer running Nginx
• four Apache Web servers
• two MySQL servers

InfoWorld designed the VMware workloads to mimic a real-world Web app usage model that included a weighted mix of static and dynamic content, randomized database updates, inserts, and deletes with the load generated at specific concurrency levels, starting at 50 concurrent connections and ramping up to 200.  InfoWorld’s started off with the VMware tests first on one blade server, then across two blades. Each blade being tested were running VMware ESX 4 and controlled by a dedicated vCenter instance.

The other real-world tests included serveral tests of common single-threaded tasks run simultaneously at levels that met and eclipsed the logical CPU count on each blade, running all the way up to an 8x oversubscription of physical cores. These tests included:

• LAME MP3 conversions of 155MB WAV files
• MP4-to-FLV video conversions of 155MB video files
• gzip and bzip2 compression tests
• MD5 sum tests

The ResultsDell
Dell did very well, coming in at 2nd in overall scoring.  The blades used in this test were Dell PowerEdge M610 units, each with two 2.93GHz Intel Westmere X5670 CPUs, 24GB of DDR3 RAM, and two Intel 10G interfaces to two Dell PowerConnect 8024 10G switches in the I/O slots on the back of the chassis

Some key points made in the article about Dell:

• Dell does not offer a lot of “blade options.”  There are several models available, but they are the same type of blades with different CPUs.  Dell does not currently offer any storage blades or virtualization-centric blades.
• Dell’s 10Gb design does not offer any virtualized network I/O. The 10G pipe to each blade is just that, a raw 10G interface.  No virtual NICs.
• The new CMC (chassis management controller) is a highly functional and attractive management tool offering new tasks like pusing actions to multiple blades at once such as BIOS updates and RAID controller firmware updates.
• Dell has implemented more efficient dynamic power and cooling features in the M1000e chassis. Such features include the ability to shut down power supplies when the power isn’t needed, or ramping the fans up and down depending on load and the location of that load.

According to the article, “Dell offers lots of punch in the M1000e and has really brushed up the embedded management tools. As the lowest-priced solution…the M1000e has the best price/performance ratio and is a great value.”

HP
Coming in at 1st place, HP continues to shine in blade leadership.  HP’s testing equipment consisted of a c7000 nine BL460c blades, each running two 2.93GHz Intel Xeon X5670 (Westmere-EP) CPUs and 96GB of RAM as well as embedded 10G NICs with a dual 1G mezzanine card.  As an important note, HP was the only server vendor with 10G NICs on the motherboard.  Some key points made in the article about HP:

•  With the 10G NICs standard on the newest blade server models, InfoWorld says ”it’s clear that HP sees 10G as the rule now, not the exception.”
• HP’s embedded Onboard Administrator offers detailed information on all chassis components from end to end.  For example, HP’s management console can provide exact temperatures of every chassis or blade component.
• HP’s console can not offer  global BIOS and firmware updates (unlike Dell’s CMC) or the ability to powering up or down more than one blade at a time.
• HP offers “multichassis management” – the ability to daisy-chain several chassis together and log into any of them from the same screen as well as manage them.  This appears to be a unique feature to HP.
• The HP c7000 chassis also has power controlling features like dynamic power saving options that will automatically turn off power supplies when the system energy requirements are low or increasing the fan airflow to only those blades that need it.

InfoWorld’s final thoughts on HP: “the HP c7000 isn’t perfect, but it is a strong mix of reasonable price and high performance, and it easily has the most options among the blade system we reviewed.”

IBM
Finally, IBM’s came in at 3rd place, missing a tie with Dell by a small fraction.  Surprisingly, I was unable to find the details on what the configuration was for IBM’s testing.  Not sure if I’m just missing it, or if InfoWorld left out the information, but I know IBM’s blade server had the same Intel Xeon X5670 CPUs as Dell and HP used.   Some of the points that InfoWorld mentioned about IBM’s BladeCenter H offering:

• IBM’s pricing is higher.
• IBM’s chassis only holds 14 servers whereas HP can hold 32 servers (using BL2x220c servers) and Dell holds 16 servers.
• IBM’s chassis doesn’t offer a heads-up display (like HP and Dell.)
• IBM had the only redundant internal power and I/O connectors on each blade.  It is important to note the lack of redundant power and I/O connectors is why HP and Dell’s densities are higher.  If you want redundant connections on each blade with HP and Dell, you’ll need to use their “full-height” servers, which decrease HP and Dell’s overall capacity to 8.
• IBM’s Management Module is lacking graphical features – there’s no graphical representation of the chassis or any images.  From personal experience, IBM’s management module looks like it’s stuck in the ’90s – very text based.
• The IBM BladeCenter H lacks dynamic power and cooling capabilities.  Instead of using smaller independent regional fans for cooling, IBM uses two blowers.  Because of this, the ability to reduce cooling in specific areas, like Dell and HP offer are lacking.

InfoWorld summarizes the IBM results saying, “ if you don’t mind losing two blade slots per chassis but need some extra redundancy, then the IBM BladeCenter H might be just the ticket.”

Overall, each vendor has their own pro’s and con’s.  InfoWorld does a great job summarizing the benefits of each offering below.  Please make sure to visit the InfoWorld article and read all of the details of their blade server shoot-out.

IBM BladeCenter H

Cisco UCS 5108

Server Density and Server Offerings: winner in this category is IBM. IBM’s BladeCenter E and BladeCenter H chassis offer up to 14 blade servers with servers using Intel, AMD and Power PC processors. In comparison, Cisco’s 5808 chassis offers up to 8 server slots and currently offers servers with Intel Xeon processors. As an honorable mention Cisco does offer a “full width” blade (Cisco UCS B250 server) that provides up to 384Gb of RAM in a single blade server across 48 memory slots offering up the ability to get to higher memory at a lower price point.

 Management / Scalability: winner in this category is Cisco.
This is where Cisco is changing the blade server game. The traditional blade server infrastructure calls for each blade chassis to have its own dedicated management module to gain access to the chassis’ environmentals and to remote control the blade servers. As you grow your blade chassis environment, you begin to manage multiple servers. Beyond the ease of managing , the management software that the Cisco 6100 series offers provides users with the ability to manage server service profiles that consists of things like MAC Addresses, NIC Firmware, BIOS Firmware, WWN Addresses, HBA Firmware (just to name a few.)

Cisco UCS 6100 Series Fabric Interconnect

With Cisco’s UCS 6100 Series Fabric Interconnects, you are able to manage up to 40 blade chassis with a single pair of redundant UCS 6140XP (consisting of 40 ports.)

If you are familiar with the Cisco Nexus 5000 product, then understanding the role of the Cisco UCS 6100 Fabric Interconnect should be easy. The UCS 6100 Series Fabric Interconnect do for the Cisco UCS servers what Nexus does for other servers: unifies the fabric. HOWEVER, it’s important to note the UCS 6100 Series Fabric Interconnect is NOT a Cisco Nexus 5000. The UCS 6100 Series Fabric Interconnect is only compatible with the UCS servers.

Cisco UCS I/O Connectivity Diagram (UCS 5108 Chassis with 2 x 6120 Fabric Interconnects)

If you have other servers, with CNAs, then you’ll need to use the Cisco Nexus 5000.

The diagram on the right shows a single connection from the FEX to the UCS 6120XP, however the FEX has 4 uplinks, so if you want (need) more throughput, you can have it. This design provides each half-wide Cisco B200 server with the ability to have 2

CNA ports with redundant pathways. If you are satisified with using a single FEX connection per chassis, then you have the ability to scale up to 20 x blade chassis with a Cisco UCS 6120 Fabric Interconnect, or 40 chassis with the Cisco UCS 6140 Fabric Interconnect. As hinted in the previous section, the management software for the all connected UCS chassis resides in the redundant Cisco UCS 6100 Series Fabric Interconnects. This design offers a highly scaleable infrastructure that enables you to scale simply by dropping in a chassis and connecting the FEX to the 6100 switch. (Kind of like Lego blocks.)

On the flip side, while this architecture is simple, it’s also limited. There is currently no way to add additional I/O to an individual server. You get 2 x CNA ports per Cisco B200 server or 4 x CNA ports per Cisco B250 server.

As previously mentioned, IBM has a strategy that is VERY similar to the Cisco UCS strategy using the Cisco Nexus 5000 product line with pass-thru modules. IBM’s solution consists of:

• IBM BladeCenter H Chassis
• 10Gb Pass-Thru Module
• CNA’s on the blade servers

Even though IBM and Cisco designed the Cisco Nexus 4001i  switch that integrates into the IBM BladeCenter H chassis, using a 10Gb pass-thru module ”may” be the best option to get true DataCenter Ethernet (or Converged Enhanced Ethernet) from the server to the Nexus switch – especially for users looking for the lowest cost. The performance for the IBM solution should equal the Cisco UCS design, since it’s just passing the signal through, however the connectivity is going to be more with the IBM solution. Passing signals through means NO cable

BladeCenter H Diagram with Nexus 5010 (using 10Gb Passthru Modules)

consolidation – for every server you’re going to need a connection to the Nexus 5000. For a fully populated IBM BladeCenter H chassis, you’ll need 14 connections to the Cisco Nexus 5000. If you are using the Cisco 5010 (20 ports) you’ll eat up all but 6 ports. Add a 2nd IBM BladeCenter chassis and you’re buying more Cisco Nexus switches. Not quite the scaleable design that the Cisco UCS offers.

IBM also offers a 10Gb Ethernet Switch Option from BNT (Blade Networks) that will work with converged switches like the Nexus 5000, but at this time that upgrade is not available. Once it does become available, it would reduce the connectivity requirements down to a single cable, but, adding a switch between the blade chassis and the Nexus switch could bring additional management complications. Let me know your thoughts on this.

IBM’s BladeCenter H (BCH) does offer something that Cisco doesn’t – additional I/O expansion. Since this solution uses two of the high speed bays in the BCH, bays 1, 2, 3 & 4 remain available. Bays 1 & 2 are mapped to the onboard NICs on each server, and bays 3&4 are mapped to the 1st expansion card on each server. This means that 2 additional NICs and 2 additional HBAs (or NICs) could be added in conjunction with the 2 CNAs on each server. Based on this, IBM potentially offers more I/O scalability.

And the Winner Is…

It depends. I love the concept of the Cisco UCS platform. Servers are seen as processors and memory – building blocks that are centrally managed. Easy to scale, easy to size. However, is it for the average datacenter who only needs 5 servers with high I/O? Probably not. I see the Cisco UCS as a great platform for datacenters with more than 14 servers needing high I/O bandwidth (like a virtualization server or database server.) If your datacenter doesn’t need that type of scalability, then perhaps going with IBM’s BladeCenter solution is the choice for you. Going the IBM route gives you flexibility to choose from multiple processor types and gives you the ability to scale into a unified solution in the future. While ideal for scalability, the IBM solution is currently more complex and potentially more expensive than the Cisco UCS solution.

Let me know what you think. I welcome any comments.

Intel officially announced today the Xeon 5600 processor, code named “Westmere.” Cisco, HP and IBM also announced their blade servers that have the new processor. The Intel Xeon 5600 offers:

• 32nm process technology with 50% more threads and cache
• Improved energy efficiency with support for 1.35V low power memory

There will be 4 core and 6 core offerings. This processor also provide the option of HyperThreading, so you could have up to 8 threads and 12 threads per processor, or 16 and 24 in a dual CPU system. This will be a huge advantage to applications that like multiple threads, like virtualization. Here’s a look at what each vendor has come out with:

Cisco
The B200 M2 provides Cisco users with the current Xeon 5600 processors. It looks like Cisco will be offering a choice of the following Xeon 5600 processors: Intel Xeon X5670, X5650, E5640, E5620, L5640, or E5506. Because Cisco’s model is a “built-to-order” design, I can’t really provide any part numbers, but knowing what speeds they have should help.

HP
HP is starting off with the Intel Xeon 5600 by bumping their existing G6 models to include the Xeon 5600 processor. The look, feel, and options of the blade servers will remain the same – the only difference will be the new processor. According to HP, “the HP ProLiant G6 platform, based on Intel Xeon 5600 processors, includes the HP ProLiant BL280c, BL2x220c, BL460c and BL490c server blades and HP ProLiant WS460c G6 workstation blade for organizations requiring high density and performance in a compact form factor. The latest HP ProLiant G6 platforms will be available worldwide on March 29.” It appears that HP’s waiting until March 29 to provide details on their Westmere blade offerings, so don’t go looking for part numbers or pricing on their website.

IBM
IBM is continuing to stay ahead of the game with details about their product offerings. They’ve refreshed their HS22 and HS22v blade servers:

HS22
7870ECU – Express HS22, 2x Xeon 4C X5560 95W 2.80GHz/1333MHz/8MB L2, 4x2GB, O/Bay 2.5in SAS, SR MR10ie

7870G4U – HS22, Xeon 4C E5640 80W 2.66GHz/1066MHz/12MB, 3x2GB, O/Bay 2.5in SAS

7870GCU – HS22, Xeon 4C E5640 80W 2.66GHz/1066MHz/12MB, 3x2GB, O/Bay 2.5in SAS, Broadcom 10Gb Gen2 2-port

7870H2U -HS22, Xeon 6C X5650 95W 2.66GHz/1333MHz/12MB, 3x2GB, O/Bay 2.5in SAS

7870H4U – HS22, Xeon 6C X5670 95W 2.93GHz/1333MHz/12MB, 3x2GB, O/Bay 2.5in SAS

7870H5U – HS22, Xeon 4C X5667 95W 3.06GHz/1333MHz/12MB, 3x2GB, O/Bay 2.5in SAS

7870HAU – HS22, Xeon 6C X5650 95W 2.66GHz/1333MHz/12MB, 3x2GB, O/Bay 2.5in SAS, Emulex Virtual Fabric Adapter

7870N2U – HS22, Xeon 6C L5640 60W 2.26GHz/1333MHz/12MB, 3x2GB, O/Bay 2.5in SAS

7870EGU – Express HS22, 2x Xeon 4C E5630 80W 2.53GHz/1066MHz/12MB, 6x2GB, O/Bay 2.5in SAS

HS22V
7871G2U - HS22V, Xeon 4C E5620 80W 2.40GHz/1066MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871G4U - HS22V, Xeon 4C E5640 80W 2.66GHz/1066MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871GDU - HS22V, Xeon 4C E5640 80W 2.66GHz/1066MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871H4U - HS22V, Xeon 6C X5670 95W 2.93GHz/1333MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871H5U - HS22V, Xeon 4C X5667 95W 3.06GHz/1333MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871HAU - HS22V, Xeon 6C X5650 95W 2.66GHz/1333MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871N2U - HS22V, Xeon 6C L5640 60W 2.26GHz/1333MHz/12MB, 3x2GB, O/Bay 1.8in SAS

7871EGU - Express HS22V, 2x Xeon 4C E5640 80W 2.66GHz/1066MHz/12MB, 6x2GB, O/Bay 1.8in SAS

7871EHU - Express HS22V, 2x Xeon 6C X5660 95W 2.80GHz/1333MHz/12MB, 6x4GB, O/Bay 1.8in SAS

I could not find any information on what Dell will be offering, from a blade server perspective, so if you have information (that is not confidential) feel free send it my way.