384GB RAM in a Single Blade Server? How Cisco Is Making it Happen (UPDATED 1-22-10)

UPDATED 1/22/2010 with new pictures 
Cisco UCS B250 M1 Extended Memory Blade Server
Cisco UCS B250 M1 Extended Memory Blade Server

 Cisco’s UCS server line is already getting lots of press, but one of the biggest interests is their upcoming Cisco UCS B250 M1 Blade Server.  This server is a full-width server occupying two of the 8 server slots available in a single Cisco UCS 5108 blade chassis.  The server can hold up to 2 x Intel Xeon 5500 Series processors, 2 x dual-port mezzanine cards, but the magic is in the memory – it has 48 memory slots.  

This means it can hold 384GB of RAM using 8GB DIMMS.  This is huge for the virtualization marketplace, as everyone knows that virtual machines LOVE memory.  No other vendor in the marketplace is able to provide a blade server (or any 2 socket Intel Xeon 5500 server for that matter) that can achieve 384GB of RAM. 

 

So what’s Cisco’s secret?  First, let’s look at what Intel’s Xeon 5500 architecture looks like.

 
 

intel ram

 

As you can see above, each Intel Xeon 5500 CPU has its own memory controller, which in turn has 3 memory channels.  Intel’s design limitation is 3 memory DIMMs (DDR3 RDIMM) per channel, so the most a traditional server can have is 18 memory slots or 144GB RAM with 8GB DDR3 RDIMM. 

With the UCS B-250 M1 blade server, Cisco adds an additional 15 memory slots per CPU, or 30 slots per server for a total of 48 memory slots which leads to 384GB RAM with 8GB DDR3 RDIMM. 

 

b250-ram

How do they do it?  Simple – they put in 5 more memory DIMM slots then they present all 24 memory DIMMs across all 3 channels to an ASIC that sits between the memory controller and the memory channels.  The ASIC presents the 24 memory DIMMs as 1 x 32GB DIMM to the memory controller.  For each 8 memory DIMMs, there’s an ASIC.  3 x ASICs per CPU that represents 192GB RAM (or 384GB in a dual CPU config.) 

It’s quite an ingenious approach, but don’t get caught up in thinking about 384GB of RAM – think about 48 memory slots.  In the picture below I’ve grouped off the 8 DIMMs with each ASIC in a green square (click to enlarge.)

Cisco UCS B250 ASICS Grouped with 8 Memory DIMMs

Cisco UCS B250 ASICS Grouped with 8 Memory DIMMs

With that many slots, you can get to 192GB of RAM using 4GB DDR3 RDIMMs– which currently cost about 1/5th of the 8GB DIMMs.  That’s the real value in this server.

Cisco has published a white paper on this patented technology at http://www.cisco.com/en/US/prod/collateral/ps10265/ps10280/ps10300/white_paper_c11-525300.html so if you want to get more details, I encourage you to check it out.

UNVEILED: First Blade Server Based on Intel Nehalem EX

The first blade server with the upcoming Intel Nehalem EX processor has finally been unveiled.  While it is known that IBM will be releasing a 2 or 4 socket blade server with the Nehalem EX, no other vendor has revealed plans up until now.  SGI recently announced they will be offering the Nehelem EX on their Altix® UV platform. 

Touted as a “The World’s Fastest Supercomputer”, the UV line features the fifth generation of the SGI NUMAlink interconnect, which offers up a whopping 15 GB/sec transfer rate, as well as direct access up to 16 TB of shared memory. The system will have the ability to be configured with up to 2048 Nehalem-EX cores (via 256 processors, or 128 blades) in a single federation with a single global address space.

According to the SGI website, the UV will come in two flavors:

SGI Altix UV 1000

Altix UV 1000  – designed for maximum scalability, this system ships as a fully integrated cabinet-level solution with up to 256 sockets (2,048 cores) and 16TB of shared memory in four racks.

Altix UV 100 (not pictured) – same design as the UV 1000, but designed for the mid-range market;  based on an industry-standard 19″ rackmount 3U form factor. Altix UV 100 scales to 96 sockets (768 cores) and 6TB of shared memory in two racks.

SGI has given quite a bit of techinical information about these servers in this whitepaper, including details about the Nehalem EX architecture that I haven’t even seen from Intel.  SGI has also published several customer testimonials, including one from the University of Tennessee – so check it out here.

Hopefully, this is just the first of many announcements to come around the Intel Nehalem EX processor.

(UPDATED) Blade Servers with SD Slots for Virtualization

(updated 1/13/2010 – see bottom of blog for updates)

Eric Gray at www.vcritical.com blogged today about the benefits of using a flash based device, like an SD card, for loading VMware ESXi, so I thought I would take a few minutes to touch on the topic.

As Eric mentions, probably the biggest benefit of using VMware ESXi on an embedded device is that you don’t need local drives, which lowers the power and cooling of your blade server.  While he mentions HP in his blog, both HP and Dell offer SD slots in their blade servers – so let’s take a look:

HP
HP currently offers these SD slots in their BL460 G6 and BL490 G6 blade servers.  As you can see from the picture on the left (thanks again to Eric at vCritical.com) HP allows for you to access the SD slot from the top of the blade server.  This makes it fairly convenient to access, although once the image is installed on the SD card, it’s probably not ever coming out.  HP’s QuickSpecs for the BL460 G6 state offer up an “HP 4GB SD Flash Media” that has a current list price of $70, however I have been unable to find any documentation that says you MUST use this SD card, so if you want to try and use it with your own personal SD card first, good luck.  It is important to note that HP does not currently offer VMware ESXi, or any other virtualization vendor’s software, pre-installed on an SD card, unlike Dell.

Dell
Dell has been offering SD slots on select servers for quite a while.  In fact, I can remember seeing it at VMworld 2008.  Everyone else was showing “embedded hypervisors” on USB keys while Dell was using an SD card.  I don’t know that I have a personal preference of USB vs SD, but the point is that Dell was ahead of the game on this one.

Dell currently only offers their SD slot on their M805 and M905 blade servers.  These are full-height servers, which could be considered good candidates for a virtualization server due to its redundant connectivity, high memory offering and high I/O (but that’s for another blog post.)

Dell chose to place the SD slots on the bottom rear of their blade servers.  I’m not sure I agree with the placement, because if you needed to access the card, for whatever reason, you have to pull the server completely out of the chassis to service.  It’s a small thing, but it adds time and complexity to the serviceability of the server.  

An advantage that Dell has over HP is they offer to have VMware ESXi 4 PRE-LOADED on the SD key upon delivery.  Per the Dell website, an SD card with ESXi 4 (basic, not Standard or Enterprise) is available for $99.  It’s listed as “VMware ESXi v4.0 with VI4, 4CPU, Embedded, Trial, No Subsc, SD,NoMedia“.  Yes, it’s considered a “trial” and it’s the basic version with no bells or whistles, however it is pre-loaded which equals time savings.  There are additional options to upgrade the ESXi to either Standard or Enterprise as well (for additional cost, of course.)

It is important to note that this discussion was only about SD slots.  All of the blade server vendors, including IBM, have incorporated USB slots internally to their blade servers, so whereas a specific server may not have an SD slot, there is still the ability to load the hypervisor onto an USB key (where supported.)

1/13/2010 UPDATE –SD slots are also available on the BL 280G6 and BL 685 G6.

There is also an HP Advisory discouraging use of an internal USB key for embedded virtualization.  Check it out at:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01957637&lang=en&cc=us&taskId=101&prodSeriesId=3948609&prodTypeId=3709945

Interesting HP Server Facts (from IDC)

As you can see from my blog title, I try to focus on “all things blade servers”, however I came across this bit of information that I thought would be fun to blog.  An upfront warning – this is an HP biased blog post, so sorry for those of you who are Cisco, Dell or IBM fans.

Market research firm, IDC released a quarterly update to their Worldwide Quarterly Server Tracker, citing market share figures for the 3rd calendar quarter of 2009 (3Q09).  From this report, there are a few fun HP server facts (thanks to HP for passing these facts along to me:)

HP is the #1 vendor in worldwide server shipments for the 30th consecutive quarter (7.5 years). HP shipped more than 1 out of every 3 servers worldwide and captured 36.5 percent total unit shipment share.

According to IDC:

  • HP shipped over 161,000 more servers than #2 Dell.
  • HP shipped 2.6 times as many servers as #3 IBM
  • 9.0 times as #4 Fujitsu
  • 12.9 times as many as #5 Sun.
  • HP ended up in a statistical tie with IBM for #1 in total server revenue market share with 30.9 percent.  This includes all server (UNIX and x86 revenues.)

HP leads the blade server market, with a 50.7 percent revenue share, and a 47.7 percent unit share.

I blogged about this in early December (see this link for details),but it’s no surprise that HP is leading the pack in blade sales.  Their field sales team is actively promoting blades for nearly every server opportunity and they continue to make innovative additions to their blades (like 10Gb NICs standard on G6 blades.)   HP Integrity blades claimed the #1 position in revenue share for the RISC+EPIC blade segment with a 53.2 percent share gaining 1.8 points year over year.

For the 53rd consecutive quarter, more than 13 years, HP ProLiant is the x86 server market share leader in both factory revenue and units, shipping more than 1 out of every 3 servers in this market with a 36.9 percent unit share.

HP’s x86 revenue share was 14.6 points higher than its nearest competitor; Dell. HP’s x86 revenue share was 19.2 percentage points higher than IBM.

 For the 3 major operating environments UNIX®, Windows and Linux combined (representing 99.3 percent of all servers shipped worldwide), HP is number 1 worldwide in server unit shipment and revenue market share.

HP holds a 36.5 percent unit market share worldwide, which is 2.6 times more than IBM’s unit market share and 12.9 times the unit share of Sun.

HP holds a 35.4 percent revenue market share worldwide which is 2.2 times the revenue share of Dell and 4.0 times the revenue share of Sun.

FINAL NOTE:  All of the following market share figures are for the 3rd quarter (unless otherwise noted) and represent worldwide results as reported by the IDC Worldwide Quarterly Server Tracker for Q309, December 2009.

IBM BladeCenter Rumours

I recently heard some rumours about IBM’s BladeCenter products that I thought I would share – but FIRST let me be clear:  this is purely speculation, I have no definitive information from IBM so this may be false info, but my source is pretty credible, so…

4 Socket Nehalem EX Blade
I posted a few weeks ago my speculation IBM’s announcement that they WILL have a 4 socket blade based on the upcoming Intel Nehalem EX processor (https://bladesmadesimple.com/2009/09/ibm-announces-4-socket-intel-blade-server/) – so today I got a bit of an update on this server.

Rumour 1:  It appears IBM may call it the HS43 (not HS42 like I first thought.) I’m not sure why IBM would skip the “HS42” nomenclature, but I guess it doesn’t really matter.  This is rumoured to be released in March 2010.

Rumour 2:  It seems that I was right in that the 4 socket offering will be a double-wide server, however it appears IBM is working with Intel to provide a 2 socket Intel Nehalem EX blade as the foundation of the HS43.   This means that you could start with a 2 socket blade, then “snap-on” a second to make it a 4 socket offering – but wait, there’s more…  It seems that IBM is going to enable these blade servers to grow to up to 8 sockets via snapping on 4 x 2 socket servers together.  If my earlier speculations (https://bladesmadesimple.com/2009/09/ibm-announces-4-socket-intel-blade-server/) are accurate and each 2 socket blade module has 12 DIMMs, this means you could have an 8 socket, 64 cores, 96 DIMM, 1.5TB of RAM (using 16GB per DIMM slot) all in a single BladeCenter chassis.  This, of course, would take up 4 blade server slots.  Now the obvious question around this bit of news is WHY would anyone do this?  The current BladeCenter H only holds 14 servers so you would only be able to get 3 of these monster servers into a chassis.  Feel free to offer up some comments on what you think about this.

Rumour 3: IBM’s BladeCenter S chassis currently uses local drives that are 3.5″.  The industry is obviously moving to smaller 2.5″ drives, so it’s only natural that the BladeCenter S drive cage will need to be updated to provide 2.5″ drives.  Rumour is that this is coming in April 2010 and it will offer up to 24 x 2.5″ SAS or SATA drives.  

Rumour 4:  What’s missing from the BladeCenter S right now that HP currently offers?  A tape drive.  Rumour has it that IBM will be adding a “TS Family” tape drive offering to the BladeCenter S in upcoming months.  This makes total sense and is well-needed.  Customers buying the BladeCenter S are typically smaller offices or branch offices, so using a local backup device is a critical component to insuring data protection.  I’m not sure if this will be in the form of taking up a blade slot (like HP’s model) or it will be a replacement for one of the 2 drive cages.  I would imagine it will be the latter since the BladeCenter S architecture allows for all servers to connect to the drive cages, but we’ll see.

That’s all I have.  I’ll continue to keep you updated as I hear rumours or news.

The Hit Movie, AVATAR Processed on HP Blade Servers

Since the hit movie AVATAR surpassed the $1 Billion Revenue mark this weekend I thought it would be interesting to post some information about how the movie was put together – especially since the hardware behind the magic was the HP BL2x220c.

According to an article from information-management.com, AVATAR was put together at a visual effects production house called Weta Digital located in Miramar, New Zealand.  Weta’s datacenter sits in a 10,000 square foot facility however the film’s computing core ran on 2,176 HP BL 2x220c Blade Servers.  This added up to over 40,000 processors and 104 terabytes of RAM.  (Check out my post on the HP BL 2x220c blade server for details on this 2 in 1 server design by HP.)

The HP blades read and wrote data against 3 petabytes of fast fiber channel disk network area storage from BluArc and NetApp.  According to the article, all of the gear was  connected by multiple 10-gigabit network links. “We need to stack the gear closely to get the bandwidth we need for our visual effects, and, because the data flows are so great, the storage has to be local,” says Paul Gunn, Weta’s data center systems administrator.  

The article also highlights the fact that the datacenter uses water cooled racks to keep the racks and storage cooled.  Suprisingly, the water cooled design, along with a cool local climate, allows Weta to run their datacenter for less cost than running air conditioning (all they pay for is the cost of running water.)  In fact, they recently won an energy excellence award for building a smaller footprint that came with 40% lower cooling cost.

Summary of Hardware Used for AVATAR:

  • 34 racks – each with 4 HP BladeSystem Chassis, 32 servers (16 BL2x220c)
  • over 40,000 processors
  • 104 TB RAM

Since I don’t want to re-write the excellent article from information-management.com, I encourage you to click here to read the full article.

Happy New Year!

Happy New Year to all of my readers. As we enter a new decade, I wanted to give everyone who takes the time to read a few stats on how I’ve done since my inaugural posting on September 23, 2009. First a bit of a background. My main website is now located at BladesMadeSimple.com, however a few months prior to that I had a blog on WordPress.com at http://kevinbladeguy.wordpress.com/.  Even though I have my own site, I have kept the WordPress.com site up as a mirror site primarily since Google has the site indexed and I get a lot of traffic from Google.  SO – how’d I do?  Well, here’s the breakdown:

On http://kevinbladeguy.wordpress.com, I received 4,588 page views since Sept 23, 2009 with my article on “Cisco UCS vs IBM BladeCenter H” receiving 399 page views.

On http://BladesMadeSimple.com, I received 2,041 page views which started up on November 1, 2009 with my article on Cisco UCS vs IBM BladeCenter H receiving 238 page views.

Combined, that is 6,629 page views since September 23, 2009!  As I’m still a virgin blogger, I’m not sure if that’s a good stat for a website devoted to talking about blade servers, but I’m happy with it.  I hope that you will stay with my as I continue my voyage on keeping you informed on blade servers.

Happy New Year!!

Great Dell Blade Server Page – "Dell TechCenter Wiki"

Looking through my previous posts it would appear that I’m against Dell’s blade server.  That’s not the case.  The reality is that it is hard to find good technical info on Dell’s product – that is until recently.  A search on SwagBucks.com led me to a little-known Dell website, called “Dell TechCenter Wiki“. 

The site, located at http://www.delltechcenter.com/, is a Dell sponsored wiki where you can find technical info on servers, storage, and even virtualization.  The wiki also provides recorded demos, white papers and even weekly chats with Dell experts to get your hard-to-answer questions answered.  Last week’s chat offered  an overview of new features available with the Dell Chassis Management Controller (CMC) for remote monitoring and access of system component information and status of Dell PowerEdge™ M1000e modular blade enclosures. 

As you can tell, this information can be extremely valueable to an IT professional with Dell blade servers, so I encourage you to check it out and let me know what you think.

immigration reform news
best buy mastercard
monster high wiki
debt to income ratio calculator
microsoft powerpoint templates

Cisco Wants IBM’s Blade Servers??

In an unusual move Tuesday, Cisco CEO, John Chambers, commented that Cisco is still open to a blade server “partnership” with IBM.  “I still firmly believe that it’s in IBM’s best interests to work with us. That door will always be open,” Chambers told the audience at the Cisco’s financial analyst conference yesterday at Cisco’s HQ in San Jose. 

John Chambers and other executives spent much of the day talking with financial analysts about Cisco’s goal to become the preeminent IT and communications vendor because of the growing importance of virtualization, collaboration and video, a move demonstrated by their recent partnership announcement with EMC and VMware.  According to reports, analysts at the event said they think Chambers is sincere about his willingness to work with IBM. The two companies have much in common, such as their enterprise customer base, and Cisco’s products could fit into IBM’s offerings, said Mark Sue of RBC Capital Markets.

So – is this just a move for Cisco to tighten their relationship with IBM in the hopes of growing to an entity that can defeat HP and their BladeSystem sales, or has Cisco decided that the server market is best left to manufacturers who have been selling servers for 20+ years?  What are your thoughts?  Please feel free to leave some comments and let me know.

IDC Q3 2009 Report: Blade Servers are Growing, HP Leads in Shares

IDC reported on Wednesday that blade server sales for Q3 2009 returned to quarterly revenue growth with factory revenues increasing 1.2% year over year.  However there was a 14.0% year-over-year shipment decline.  Overall blade servers accounted for $1.4 billion in Q3 2009 which represented 13.6% of the overall server revenue.  Of the top 5 OEM blade manufacturers, IBM experienced the strongest blade growth gaining 6.0 points of market share.  However, overall market share for Q3 2009 still belongs to HP with 50.7%, with IBM following up with 29.4% and Dell in 3rd place with a lowly 8.9% revenue share.Q3_2009_Blades According to Jed Scaramella, senior research analyst in IDC's Datacenter and Enterprise Server group,  "Customers are leveraging blade technologies to optimize their environments in response to the pressure of the economic downturn and tighter budgets. Blade technologies provide IT organizations the capability to simplify their IT while improving asset utilization, IT flexibility, and energy efficiency.  For the second consecutive quarter, the blade segment increased in revenue on a quarter-to-quarter basis, while simultaneously increasing their average sales value (ASV). This was driven by next generation processors (Intel Nehalem) and a greater amount of memory, which customers are utilizing for more virtualization deployments. IDC sees virtualization and blades are closely associated technologies that drive dynamic IT for the future datacenter."