Moving to POWER7 from SPARC

Last weekend I was reading a very interesting thread about moving to POWER7 from SPARC. I found it very interesting hence thought to share it with you.


There is an attention-grabbing discussion going on under LinkedIn. The subject is –

“Moving to POWER7 from SPARC”

I think this is quite exciting topic or rather very strategic decision to make so I decided to have a brief summary note on it.

The discussion starts with question –

“My employer is considering seriously moving to POWER7 from SPARC as we retire EOSL (End of Service Life) hardware. Has anyone considered such a move or made the move from POWER to SPARC? “

Very difficult to answer isn’t it? Yes, it is!


I’m just trying to summarize the comments came from singular experts.


- There is a very valid point I agree upon - I would suggest not changing everything unless there is a very compelling reason. Point to ponder that if there is a particular need to run AIX, you can just integrate an AIX servers or cluster into your data centre without changing everything else which will be less overhead and sensible decision. Though this is not a technical point however first thought in my mind if my management ask me to do so.


- Datacenter power usage, cooling and space point of view - If one particular is struggling with power, cooling and space issue @ datacenter then consider mixing T- and M-series SPARC systems. T-series are great as power savers, they work best with highly parallel applications. M-series work great on single-threaded apps. The T-series are very efficient on power. However they do not do well on single threaded applications. For those you need to stick with M series.


- Another point - cost factor point of view - Solaris has x86 (Intel/AMD architecture), which is an option we don't have with AIX. For raw processing power and large memory footprints, Solaris 10 on Nehalem Intel is very motivating. You don't get all the RAS features of SPARC hardware, but if you have a load balanced applications or edge layer you can move there it can be a great fit. Also Solaris Supports x64 CPUs for excellent price/performance.


- The very vital point - The recent SPARC T3 servers aka “Rainbow Falls” that were announced last week at Oracle OpenWorld, that POWER7 isn't as desirable of a platform. Considering that a SPARC T3-4 can perform as well and in many benchmarks better than a 4 socket POWER7 box, but at considerably lower TCO, I don't see the point in the IBM/AIX/POWER route.

The SPARC T3 processor has the following specifications:

 - 16 Cores x 8 Threads = 128 Threads Running at 1.65Ghz
- 2 x Execution Units per Core with 4 x Threads Each
- 1 x Floating Unit and 1 x Crypto Unit per Core
- 6MB of Shared L2 Cache
- 2 x DDR3 Memory Controllers
- 6 x Coherency Links for Glue-Less SMP up to 4 Sockets
- 2 x 10GbE NIU Controllers
- 2 x PCI-E 2.0 Controllers -> 8GB/s Bi-Directional I/O Each

Not done yet! Here is more details of various flavors of T3.

- Licensing factor - IBM will charge you for licenses left and right for each feature, especially on the virtualization front (LPARs, MPARs, and WPARs,), not to mention all the external components (HMCs, etc.). As where you can use Oracle VM Server for SPARC (LDoms) for free with the server and only pay for a RTU and support for S10 or S11 once for the whole machine (you can have hundreds or thousands of guests for no additional charge)! And don't forget that Solaris Containers are free and available on x86 and SPARC. Plus they can be used in LDoms(T-Series) and Dynamic System Domains (M-Series) for free! Oracle core licensing factor on SPARC T3 is 0.25

- AIX has no equivalent technology to ZFS.


 - Solaris can scale to >64 CPUs to solve extremely large problems.


- Point to be noted - AIX on POWER is a good platform, I don't want to badmouth AIX. But here are the two biggest issues with the platform:


1. Costs
2. Finding enough AIX folks to support you! [It doesn’t mean that out there lot of resources available for supporting Solaris, I mean "GOOD RESOURCES"…]


- Punch line - SPARC is not dead, actually is more alive than ever and Solaris is the most advanced OS in the market: why change ?


- Virtualization point of view – AIX got LiveMotion! But not Sun BUT---BUT there are ways. Lets discuss it in details.


You can migrate your LDoms in a cold or warm migration, look out for LDoms 1.3 Admin guide in the chapter "Migrating Logical Domains", in particular the section on "Migrating an Active Domain" on page 129. (Oracle VM Server for SPARC 2.0). With the warm migration, the LDom is suspended and moved within seconds->minutes depending on the size and network bandwidth. As for rebooting service domains, if you're on something like a T5240 or T5440, you can use external boot storage (SAN/JBOD/iSCSI) to keep the second service domain up and have enough PCI-E slots for redundancy. By splitting your redundancy (network and storage) between them, your LDom guests will continue running with IPMP and MPXIO. So no down-time. FYI, even if you reboot the Primary domain without a secondary service domain, the guests will continue to run and wait for the Primary to return, so all is not lost. The only way you'd lose all of your LDoms is if you lose power or reset the SC. You can also use Solaris Cluster to automate the migration or fail-over of LDom guests.


The bad thing about the IBM Power VM setup is the following:


1. High-Over Head! The VIO overhead can easily consume over 40%-50% of your resources! As where on a T-series with LDoms, it's one core for the Primary domain and less than 10% overhead for the networking and storage I/O virtualization. The CPU threads are partitioned at the hypervisor and CPU level. RAM is virtualized at the hypervisor and MMU level. And on the M-series with Dynamic Domains you have 0% overhead because the CPU, Memory, and I/O are electrically partitioned on the centerplane. Even Solaris Containers have extremely low overhead, usually less than 5%. So I can get more out of my Solaris servers than you can in the IBM PowerVM world.


2. If there is a fault on a CPU or memory module, you can take out multiple LPAR/WPARs. As where on M-series with Dynamic Domains, faults like that would only affect the Domain the CMU module was on. You can even do a mirrored Memory configuration on the M-Series to be fully fault tolerant. Not to mention that the Dynamic Domains are electrically isolated, something you don't get anywhere else.


3. Costs, you have to get licenses to enable PowerVM features and they add up quickly. As where LDoms and Dynamic Domains are free with the hardware.


NOTE: In AIX WPAR (equivalent to Sun Container) can be migrated on the fly BUT on Solaris Containers cannot be migrated on the fly BUT not to worry it is something Sun-Oracle is working on and we'll probably see down the road. As of now for live migrating something around, I would use an LDom for now!


Not biased about AIX POWER architecture as this is a very vital decision to make and at the same time very difficult situation to conclude on but taking help of above points what I can think of – I’ll stick to SPARC architecture!


Your comments on this will be very much appreciated.