384GB RAM in a Single Blade Server? How Cisco Is Making it Happen (UPDATED 1-22-10)
Cisco’s UCS server line is already getting lots of press, but one of the biggest interests is their upcoming Cisco UCS B250 M1 Blade Server. This server is a full-width server occupying two of the 8 server slots available in a single Cisco UCS 5108 blade chassis. The server can hold up to 2 x Intel Xeon 5500 Series processors, 2 x dual-port mezzanine cards, but the magic is in the memory – it has 48 memory slots.
This means it can hold 384GB of RAM using 8GB DIMMS. This is huge for the virtualization marketplace, as everyone knows that virtual machines LOVE memory. No other vendor in the marketplace is able to provide a blade server (or any 2 socket Intel Xeon 5500 server for that matter) that can achieve 384GB of RAM.
So what’s Cisco’s secret? First, let’s look at what Intel’s Xeon 5500 architecture looks like.
As you can see above, each Intel Xeon 5500 CPU has its own memory controller, which in turn has 3 memory channels. Intel’s design limitation is 3 memory DIMMs (DDR3 RDIMM) per channel, so the most a traditional server can have is 18 memory slots or 144GB RAM with 8GB DDR3 RDIMM.
With the UCS B-250 M1 blade server, Cisco adds an additional 15 memory slots per CPU, or 30 slots per server for a total of 48 memory slots which leads to 384GB RAM with 8GB DDR3 RDIMM.
How do they do it? Simple – they put in 5 more memory DIMM slots then they present all 24 memory DIMMs across all 3 channels to an ASIC that sits between the memory controller and the memory channels. The ASIC presents the 24 memory DIMMs as 1 x 32GB DIMM to the memory controller. For each 8 memory DIMMs, there’s an ASIC. 3 x ASICs per CPU that represents 192GB RAM (or 384GB in a dual CPU config.)
It’s quite an ingenious approach, but don’t get caught up in thinking about 384GB of RAM – think about 48 memory slots. In the picture below I’ve grouped off the 8 DIMMs with each ASIC in a green square (click to enlarge.)
With that many slots, you can get to 192GB of RAM using 4GB DDR3 RDIMMs– which currently cost about 1/5th of the 8GB DIMMs. That’s the real value in this server.
Cisco has published a white paper on this patented technology at http://www.cisco.com/en/US/prod/collateral/ps10265/ps10280/ps10300/white_paper_c11-525300.html so if you want to get more details, I encourage you to check it out.