Compute Cycles

Free Grill v1.5 and Grill Recipe Challenge announced!

We’re pleased to announce the release of Grill version 1.5 for Opscode Chef! And it’s FREE!

We’ve added a few features that came in useful during our last Utility Supercomputing run code-named Naga, including:

• New customizable reports
• Backporting Chef alert system to CycleServer core
• Greatly improved performance for large data sets
• Hosts are now removed after a certain amount of inactivity

But we have a dirty secret: we don’t have a publicly available cookbook to install it.

We know, we know. We wrote a chef report visualization tool that can handle several thousand nodes all converging at once but we don’t have an installer cookbook.

Well, we do actually have an installer, but it’s tightly-coupled to CycleCloud and we just haven’t had time to make a rock-solid generally-useful cookbook.

So we’d like to announce the Grill Recipe Challenge. Write an awesome Grill cookbook and we’ll send you a CycleComputing Mug and T-shirt. Here’s the simple rules:

1. Submit your cookbook to grill-recipe@cyclecomputing.com
2. Release your cookbook under the Apache license. 
3. We’ll accept entries until 11:59 PM PST on June 12th 2012. 
4. We’ll pick the best one and send the winner a sweet Cycle mug and T-shirt 
5. (our decisions are final! ;)

So if you need to explore Chef converge data, download Grill today, kick the tires, write an awesome cookbook, and win fabulous prizes... :D

CycleCloud Achieves Ludicrous Speed! (Utility Supercomputing with 50,000-cores)

By now, we've shown that our software is capable of spinning up cloud computing environments that run  at massive scale and produce real scientific results.  After some of our previous efforts, we realized we were onto something with the CycleCloud and Utility Supercomputing concept. However, even we underestimated the scales researchers would want to use and the scope of the research that this would impact. 

Among the requests were some from a leader in computational chemistry research, Schrodinger. In collaboration with Nimbus Discovery, they needed to virtually screen 21 million molecule conformations, more than ever before, against one possible cancer target using their leading docking application, Glide. And they wanted to do it using a higher accuracy mode early-on in the process, which wasn’t possible before because it is so compute intensive!

This is exactly what we did with our latest 50,000 core utility supercomputer that CycleCloud provisioned on Amazon Web Services, code-named Naga.  And Schrodinger/Nimbus got useful results they wouldn't have seen without utility supercomputing. We will describe how we accomplished this below, and in future articles and future blog posts.

From a scale perspective, the most revolutionary concept implemented for Naga was scaling out all the components of an HPC environment. In our previous megaclusters, we performed a great deal of optimization to scale one scheduler to handle as many cores (slots) as possible. With Naga, it was clear that we needed to scale out multiple aspects of the system so we could use capacity wherever we found it.

We ran Naga across each of the 7 regions that AWS currently supports, scaling-out all supporting systems of a cluster (scheduling, software configuration, etc.) so we could use idle capacity wherever we found it. However, the magic really happened when we layered CycleServer on top of Naga and allowed our revolutionary new job submission algorithm to intelligently dole out work to each region based on real-time measurements from that region. Using this architecture, we had built ourselves a secured, automated 50,000 core supercomputer in under two hours using AWS infrastructure.

The amazing thing about this cluster was that it was launched via CycleCloud by the enduser and the jobs were submitted to the cluster through CycleServer by the enduser as well. All of the automation work that went into the Naga megacluster means that Cycle engineers largely got to sit back and monitor the progress of the workload as it progressed.

Here are the cold hard stats:

Resource	Count
Total Instances	6742
- c1.xlarge instances	5,435
- c1.medium instances	779
- cc1.4xlarge instances	297
- cc2.8xlarge instance	231
cores	51,132
RAM	58.78-TB
AWS Regions	All (7: us-east, us-west1, us-west2, eu-west, sa-east, ap-northeast, ap-southeast )

Here is a map of the compute resources harnessed during the Naga run:

Here is an analysis of the staggering amount of compute power that Naga provided:

Metric	Count
Compute Hours of Work	109,927 hours
Compute Days of Work	4,580 days
Compute Years of Work	12.55 years
Ligand Count	~21 million ligands

 

THE REAL IMPORTANT NUMBERS!

The most important thing about this run are the following two numbers:
Time to run the impossible science: ~3 hours
Cost at peak: $4828.85/hour!

You read that right, 20+ million dollars of infrastructure was used for $4,828 per hour. Wow!

As always, we used a large suite of CycleServer visualizations to monitor the run.  These include CycleCloud compute provisioning, Condor job scheduling visualizations, OS-level performance statistics, CycleServer-Grill to monitor OpsCode Chef, and others that we’ll be talking about in the coming weeks.  Here is a sampling of some of the dashboards we were monitoring in real time:

As you can see, Cyclecloud was able to launch an outrageous number of machines in AWS, our Chef environment converged all of them into useful workers, CycleServer was able to dole out jobs to Condor schedulers, the Condor schedulers were able to quickly fill the compute resources with jobs, and we were able to monitor the overall utilization of all 50,000 cores working in unison.  

 

THE RESULTS

 

As part of this run, Schrodinger, in collaboration with Nimbus Discovery, has already identified compounds they would not have seen if it weren't for CycleCloud Utility Supercomputing! Now they're acquiring the compounds, and will be testing them in the lab. 

And all of this was possible by under $5000 an hour in AWS fees. We’re talking about a supercomputer that would cost tens of millions of dollars to purchase, power, cool, and maintain. If that isn’t a transformational capability, we don’t know what is! 

 

There is no industry that won’t be impacted by this technology, from insurance to life science, manufacturing to computer-aided product design, aerospace to energy. So, if you have a big problem that you’d like to solve in a few hours instead of a few years, give us a call. If you don’t currently have a big problem, sit back and think about what big problems face your organization that could really move it forward, but you might never ask because it would take too much computation. It’s time to think about research differently. We’d love to introduce you to Utility Supercomputing using CycleCloud.

Astounding Science: the CycleCloud BigScience Winners

 

“If we did all the things we are capable of, we would literally astound ourselves.”
  -- Thomas A. Edison

It is with this quote in mind that we get to do something wonderful. Today, we will honor the winner of the CycleCloud BigScience Challenge 2011.

 

What is the CycleCloud BigScience Challenge?

Last October, we defined "utility supercomputing" and challenged researchers to break out of a common habit: limiting the questions they asked to the size of their local compute environment.  Instead we asked them to propose big questions whose answers can move humanity forward. 

We offered the winner use of “utility supercomputing”, providing resources at a scale of the Top 500 supercomputing list, to run their BigScience for a few hours, then turn it off. With Cycle offering $10,000 in time, and Amazon adding another $2,500, together the Winner will have an equivalent of 10 hours on a 30,000-core CycleCloud cluster.

We announced the Finalists at Supercomputing 2011, and had a group of industry luminaries agree to be judges, including Kevin Davies of Bio IT World, Matt Wood of AWS, and Peter Shenkin of Schrodinger. Many thanks to all of you for your help.

 

The Finalists

And then we saw the Finalists' presentations. The experience was inspiring. The finalists all sought to tackle BigScience problems, which only utility supercomputing could help. And it was awesome!

So it is with great pleasure I would like to recognize the Finalists that presented:

• Alan Aspuru-Guzik & Johannes Hachmann, Harvard Clean Energy Project
• Jesus Izaguirre, University of Notre Dame
• Victor Ruotti, Morgridge Institute for Research
• Martin Steinegger, TU Munich ROSTLAB

Next up, we created an Honorable Mention for Alan Aspuru-Guzik & Johannes Hachmann of the Harvard Clean Energy Project, for great research into creating more efficient energy from organic photovoltaic cells. We will give Alan and Johannes the idle compute capacity on Cycle's R&D clusters.

 

And the CycleCloud BigScience Winner is...

Victor Ruotti, a computational biologist at the Morgridge Institute for Research. He and his team will use utility supercomputing to help create an index of RNA alignments that affect the differentiation of human stem cells into specific kinds of cells. The research itself is amazing, and the ability for it to be put into practice quickly in wet labs should have a big impact in the use of stem cells to treat disease.

In closing, we hold high hopes for all of this research, and are excited to report back to you on the results. And equally importantly, we hope that you start to think about how utility supercomputing, whether at 30 or 30,000 cores, that can help ignite your work, make your research more agile, develop products and treatments faster, and enable "all the things that we are capable of". As Edison wisely pointed out, it will literally be astounding.

Jason Stowe
CEO, Cycle Computing

CycleCloud BigScience Challenge Finalists = +5 for humanity

So we announced the finalists of the CycleCloud BigScience Challenge 2011 at Supercomputing 2011 in Seattle last night. Finalists were selected based on their proposal’s long-term benefit to humanity, originality, creativity and suitability to run on CycleCloud clusters.

We're excited to announce that in addition to the $10,000 Grand Prize, and $500 for Finalists, the awesome folks over at Amazon Web Services are adding their own $2500 for the Grand Prize, and $1000 per Finalist, bringing our totals to $12500 for the BigScience award and $1500 per Finalist. With $12500 we will be able to do about 10 hours on a 30000 core environment, or 30 hours on 10000 cores.

The finalists will be judged by me, and a panel of industry luminaries, including Kevin Davies, editor-in-chief, Bio-IT World, Matt Wood, technology evangelist for Amazon Web Services, and Peter S. Shenkin, vice president, Schrödinger. Thanks to Kevin, Matt, and Peter for helping judge the winner. Picking Finalists from all the entrants we got was hard enough. We're going to have our hands full picking a winner from all this great research!

So without further ado, below are the Finalists for the inaugural CycleCloud BigScience Challenge:

Alan Aspuru-Guzik, Harvard Clean Energy Project, professor in department of chemistry and chemical biology and Johannes Hachmann, postdoctoral fellow: Hachmann and Aspuru-Guzik wish to conduct computational screening and design of novel materials for the next generation of organic photovoltaics (OPVs). The goal is to facilitate creating the next generation of photovoltaic cells.

Jesus Izaguirre, University of Notre Dame, associate professor of computer science and engineering and concurrent associate professor of applied and computational mathematics and statistics: Izaguirre intends to explore mutations in proinsulin case misfolding and analyze the ability to stimulate the folding pathways of these mutations to provide mechanic insight into the events of onset of diabetes. He also plans to examine the dominant states in the folding pathways to enable structure-based drug design and the production of new therapies to combat this disease.

Soumya Ray, Harvard Medical School, assistant professor of neurology: Ray’s team has identified a mutation that represents the majority of Parkinson’s disease patients. They seek to utilize the additional computational power to explore the dynamics of the protein and how it interacts with inhibitors to understand how drugs interact with the mutation, benefitting a large number of researcher and other drug discovery programs around the world.

Victor Ruotti, Morgridge Institute for Research, computational biologist: Ruotti aspires to collect genetic information, specifically RNA alignments, from different types of cells to build an RNA-based indexing system for stem cells. Once these alignments are identified, analysis based on this knowledgebase will provide a better understanding of the overarching signaling mechanisms used by stems cells to support generation of personalized, cell-based therapies for a variety of diseases.

Martin Steinegger, TU Munich ROSTLAB, bioinformatics researcher: Steinegger’s team’s goal is to provide access to every possible mutation in the gene sequence that will ever be observed in humans. To achieve, they have started a new project called SNAP-Map, which strives to calculate every possible single-nucleotide polymorphism (SNP) in human proteins to make this technology and data available worldwide. With this data available, researchers will have the ability to access the effect of mutations in individuals and advance the efforts towards individual medicine based on understanding human diversity and variation.

We had many other wonderful submissions, including the runner-up from Purdue's Gerhard Klimeck, of the NanoHUB project. If you're interested in finding out more about the Challenge, and you're at Supercomputing, visit us at SC11 BOOTH 443.

Thanks to all the entrants, and we'll keep you posted on the winner!

Jason Stowe
CEO, Cycle Computing

CycleCloud BigScience Challenge Finalists Announced at SC11 Amazon Booth #6202 on Monday at 8PM

We got a great response to the CycleCloud BigScience Challenge 2011, and wanted to thank all of the entrants. The entries included calculations to fight Alzheimer's, fight Parkinson's, help understand stem cell differentiation, and improve photovoltaics to make greener energy. It's going to be hard to pick Finalists!

We are also excited that the gracious folks at Amazon Web Services are adding to the prize money that Cycle is providing. Amazon's contribution will help us get even more BigScience done in shorter periods of time using utility supercomputing.

We'll announce the Finalists, the final Judges, and more details about the additional prizes, at 8pm Monday, November 14th, at SC11 at the Amazon Booth #6202. Additional information will be available at Cycle Computing's Booth #443 throughout the show and on the BigScience Challenge Website.

Thanks again to the entrants. We're very excited about these scientists using utility supercomputing to perform research that will benefit humanity. We'll see you at SC11!

Mad Scientist could win CycleCloud BigScience Challenge…

Just kidding, he's just a potential finalist! ;)

As some of you may know, Cycle wants to help scientists answer big research questions that might help humanity by donating compute time using our utility supercomputing softare. But in the overwhelming response we've gotten to the CycleCloud BigScience Challenge we announced last week, we repeatedly get the question, "What kind of research benefits humanity?" And the answer isn't Dr. Evil researching "sharks with frickin' laser beams"!

Let's highlight a couple of the entries already received that might move us forward: There is the researcher doing quantum mechanics simulations for materials science to improve solar panel efficiency that might help "electrify 2.5 Billion people" with greener energy. Or the computational biologist that wants to use meta-genomics analysis to create a knowledgebase indexing system for stem cells and their derivatives, helping us "speed development of personalized cell-based therapies". Very exciting!

Maybe you analyze public government data to provide clarity. Or you research science that might help in the race to treat Alzheimer's, Cancer and Diabetes. Or you're simulating ways to more efficiently distribute food in places that need it. There's plenty of utility supercomputing applications ahead of us that could benefit humanity, and now's your chance to start.

Remember entries are due November 7th. So come join us. There's just four questions between you and the equivalent of 8 hours on a 30000 core cluster. So submit early & submit often, and let's change the speed that BigScience gets done!

Jason Stowe
CEO, Cycle Computing

CycleCloud BigScience Challenge 2011

I’m planning on offering the equivalent of 8 hours on a ~30,000-core cluster ($10,000 in free CycleCloud time) to help researchers answer questions that will help humanity. But before we get there, let’s talk about why:

Recently, Cycle got significant press coverage for using CycleCloud to create a ~30,000 core cluster on Amazon EC2 for a Top 5 Pharma to run research. It hit Ars Technica, then Slashdot, then Engadget (which had a great depiction of Nekomata, btw), then Wired's CloudLine ... unreal. (Update: Now Forbes and Wired too!)

In reading all the comments and questions, like "Would this run high-res Crysis?" or "How much capacity does Amazon have?", a thought went through my mind:

I became concerned. Concerned because these shouldn't be the questions we're asking.

I worried that in all this glitter, we would miss what is truly gold: that this type of computing can speed up scientific research and solve problems we’d traditionally never dream of tackling. So I'm writing this to introduce a new concept to you, and implore you to think about how to move the human race forward through science and research.

To start, let's answer a question implied in many comments: Why is this important? For years cloud computing has been about paying for what you use, and accessing the compute power you need, when you needed it.

The problem is, today, researchers are in the long-term habit of sizing their questions to the compute cluster they have, rather than the other way around. This isn’t the way we should work. We should provision compute at the scale the questions need. We're talking about taking questions that require a million hours of computation, and answering them in a day. Securely. At reasonable cost.

I call this “utility supercomputing”, that is, provisioning resources at a scale of the Top 500 supercomputing list, for researchers to use for a few hours, then turn off. For example, take the 5 years of back data that you never were able to re-analyze with a new, innovative algorithm because it would take 2 years on your internal compute cluster, and instead get results back in 9 hours. For $12000.  Wow...

Scratch the surface of this idea, and you'll see a world of research the way I see it. No more waiting. No more R&D folks task-switching for days or weeks while compute is run. Only answers at the speed of thought, at the speed of invention, at the scale of the question.

But unlike other innovations, this is not a tool for the have's and not the have-not’s. This tool is equal opportunity and an extreme competitive advantage. You pay for what you use with no up-front investment. We've already enabled individual researchers to rent computing equivalent to all of the Top 5 Pharma's HPC clusters combined, for a couple thousand dollars an hour. And all this without the researcher needing to know IT, or scripting, or APIs.

Now, on behalf of the Cycle team, I'd like to answer an additional question we’ve been asked repeatedly over the past several weeks: Will you go bigger?

Absolutely. We will. There is no more thinking that having a single, fixed-size internal cluster is the right idea. You might buy internal compute capacity for day-to-day usage. But the answer isn't there. Not anymore.

This is truly exciting. And not exciting simply because technologically Cycle can now create 30000 core clusters in minutes, leveraging robust automation, performance analytics, and moving data securely across continents to answer un-askable questions.

It's exciting because someone is going to take these clusters and cure cancer, or Alzheimer's, or my personal affliction, Type-1 diabetes. And hopefully cure them faster because they have better tools.

So there's no more running. We aim to misbehave. We aim to wreck the status quo of HPC clusters and computational science. We will enable those crazy questions from the misfit geniuses, the ones so big that you would never even ask them, you might even joke about how silly they are, and we will help answer them. Efficiently. Cheaply.

It is with this in mind that I'd like to ask you, what's your question? What is the analysis or simulation that could change your field? What is the work that would bring you, your university, your research organization, your company, or all of humanity to the next level?

Because these are the questions we should be asking.

And to put our money where our mouth is, we're going big or going home by offering the equivalent of 8 hours on a ~30000-core cluster (specifically $10000 of CycleCloud time), for free, along with free support from Cyclers, to help a single researcher with an un-askable question. It's the CycleCloud BigScience Challenge 2011.

If you have one of these questions please fill out the form, with brief entries, describing your career, the work you do, why you believe in it, and how you want to use 10000 or more cores to push us forward. Proposals are due Nov 7th, 2011, and we'll announce the Finalists at Booth #443 at Supercomputing 2011.

We want the runts, the misfits, the crazy ideas that are normally too big to ask, but might, just might, help humanity.

Jason Stowe
CEO, Cycle Computing

New CycleCloud HPC Cluster Is a Triple Threat: 30000 cores, $1279/Hour, & Grill monitoring GUI for Chef

Update: Wow, we've gotten tremendous feedback from this run on Arstechnica, Wired, and others, and man has it been a busy few days. We did have many people ask a quesiton that we wanted to clarify:
Q: How long would the run-time take in-house vs. in-cyclecloud?
A: The clients indicated the workload would never have happened in-house because it would have used everything they had for week(s). The in-cyclecloud run time was 7-8 hours.

=========================

In more ways than one, the Nekomata cluster is three times as impressive as our last public mega-cluster. A few months ago, we released details of the Tanuki cluster, a 10,000-core behemoth launched within AWS with the click of a button. Since then, we have been launching large clusters regularly for a variety of industries. We kept our eye open for a workload large enough to push us to the next level of scale. It didn’t take very long.

We have now launched a cluster 3 times the size of Tanuki, or 30,000 cores, which cost $1279/hour to operate for a Top 5 Pharma. It performed genuine scientific work -- in this case molecular modeling -- and a ton of it. The complexity of this environment did not necessarily scale linearly with the cores.

In fact, we had to implement a triad of features within CycleCloud to make it a reality:
1) MultiRegion support: To achieve the mind boggling core count of this cluster, we launched in three distinct AWS regions simultaneously, including Europe.

2) Massive Spot instance support: This was a requirement given the potential savings at this scale by going through the spot market. Besides, our scheduling environment and the workload had no issues with the possibility of early termination and rescheduling.

3) Massive CycleServer monitoring & Grill GUI app for Chef monitoring: There is no way that any mere human could keep track of all of the moving parts on a cluster of this scale. At Cycle, we’ve always been fans of extreme IT automation, but we needed to take this to the next level in order to monitor and manage every instance, volume, daemon, job, and so on in order for Nekomata to be an efficient 30,000 core tool instead of a big shiny on-demand paperweight. Our new Grill monitoring GUI app for CycleServer helped show what's cooking with our Opscode Chef environment that helped with cloud infrastructure automation for all the instances for this cluster.

The Stats

Before we step through these enhancements one by one, let’s take a moment to sit back and contemplate the sheer scale of this compute environment.

Resource	Count
c1.xlarge instances	3,809
cores	30,472
RAM	26.7-TB
AWS Regions	3    ( us-east, us-west, eu-west )

 

• Where Tanuki certainly encompassed a significant number of physical racks of compute hardware in the AWS datacenter, this scale meant that our footprint was a significant number of aisles. Certainly, the power, cooling, and floorspace that we occupied during this run were larger than what one would find at many moderately sized datacenters.
• Our system placed individual staged requests to AWS that eclipsed the size of entire clusters that we considered to be large at the time (such as Oni). These requests were made relentlessly to Amazon and were fulfilled with impressive efficiency.
• The amount of RAM available on the compute nodes was roughly 30TB, which means that it could hold the entirety of the current raw Wikipedia database 5 times over in memory (citation needed -- get it?)

The workload that was run for the client in this case was a software package that consisted of  molecular dynamics approaches applied to molecular structure and functioning that were run against millions of compound targets. The expected run time on the internal cluster environment was expected to be at least a week worth of run time consuming the entire cluster.

To manage the high throughput workload of the jobs, we used the condor scheduler for load distribution and CycleServer to monitor progress of workload and job scheduling. The table below provides a synopsis of the total work that was done in this massive scale analysis.

Metric	Count
Compute Hours of Work	95,078 hours
Compute Days of Work	3,961 days
Compute Years of Work	10.9 years
Number of condor jobs	154,116
Avg. Run Time per job	37 minutes

New Multi-Region Support

Because of the scale of this run, we opted to go across multiple regions to spread out the spot instance requests and increase our probability of getting the core counts we required. Drilling down a little deeper into the numbers shown in the table above, here is what the breakout across the three regions looked like. We were very pleased that provision times were very consistent across the regions.

Region	Server Count	Core Count	RAM
US-East	2,825	22,600	19.8-TB
US-West	484	3,872	3.4-TB
EU-West	500	4,000	3.5-TB

Massive Spot Instance Support

Nekomata also marks our first use of Spot Instances. The Spot Market provides a huge discount over traditional on-demand instances which means we can do more science for less money. We’ve added support for Spot Instance bidding for CycleCloud execute hosts. Nekomata made use of over 3700 c1.xlarge Spot Instances at an average cost of 0.286 USD / instance / hour (0.036 USD / core / hour). Compare that to the 0.68 USD / instance / hour for the same On Demand instance. That’s 57% savings!

Massive Telemetry and Monitoring: CycleServer & Grill

We used Nekomata to give a workout to some of the new tools we’ve been working on that provide unprecedented views into large-scale computing environments. We’re extremely excited about this, as this is the first time that we are leveraging the plugin architecture of CycleServer to enable both the collection and visualization of diverse types of data. One such tool is a new offering that we have named Grill for CycleServer.

Cycle has professed its love for Opscode’s Chef project in the past on our blog, and we continue to use it to provision even our largest clusters--Nekomata is no different. When you have hundreds of nodes converging each minute, you need a way to see precisely which converges are failing so that you can resolve the issue(s).

Using the Handler feature of open-source Chef, we wrote a Ruby gem that reports converge information to our flagship telemetry engine, CycleServer. Grill shows the result of every converge, as you can see in the screenshot below.

For Nekomata, we built a Grill view that shows a timeline of all converges and a window showing drill-down to individual hosts and their converge history. Important information such as converge duration, number of resources updated and an exception stack trace on failure are all just a click away. On the right side of the display is a stacked histogram of converges per minute with successful in green and failed in red. We have many more features planned for our upcoming release of the chef dashboards.

Along with the chef dashboards, we have also extended our telemetry engine and visualization dashboards to include  heat maps to succinctly highlight servers that are running host as well as including the data captured by ganglia to see resource utilization trends over time as illustrated with a point-in-time screenshot below from the Nekomata run.

 

We’ll be releasing more information in the coming weeks about these tools, so please stay tuned.

Next time around...

While all parties involved are absolutely thrilled at the achievement of the Nekomata cluster, we would be remiss if we did not share a couple of lessons learned during such a large run. You can be sure that when you run at massive scale, you are bound to run into some unexpected gotchas. As a result, it’s critical that you always execute a post mortem after such an event to identify what went well, what didn’t go well and how to do it better next time.

In our case, one of the gotchas included such things as running out of file descriptors on the license server. In hindsight, we should have anticipated this would be an issue, but we didn’t find that in our pre-launch testing, because we didn’t test at full scale ( another lesson learned....keep reading ). We were able to quickly recover from this bump and keep moving along with the workload with minimal impact. The license server was able to keep up very nicely with this workload once we increased the number of file descriptors.

Another bump that we hit were EBS volume and byte limits on EBS-backed volumes as outlined in their documentation here: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?Concepts_BootFromEBS.html. Once again, once identified, necessary limits were raised and we kept moving right a long.

The Future?

What’s next from CycleComputing’s large scale clusters? Well, we already have our next use-case identified and will be turning up the scale a bit more with the next run. While we are excited about the ability to scale clusters to larger and larger core counts and forging new frontiers in cloud computing, we are most excited about the possibilities we are creating by enabling scientists to spin up such large clusters with minimal effort to answer questions that they would never have considered asking a year ago because they didn’t have access to resources that could answer those questions. Today, we have enabled them to have access to massive computing scales in minutes which in turn gives them the ability and opportunity to ask these sorts of complex questions, not once or twice, but multiple times with multiple different perspectives and multiple different variables leading to improvements in their discovery research efforts. We understand that ultimately, it’s not about core counts or TB of RAM or PB of data. Rather, it’s about how we are helping to transform how science is done.

If you’re interested in talking about high volumes of calculations of compute with Cycle Computing, or using CycleCloud, CycleServer, and Grill, then please contact us.

Fast and Cheap, pick two: Real data for Multi-threaded S3 Transfers

Gentleman start your uploads! They're free now but how fast can we do them? Lately we’ve been working with clients solving big scientific problems with Big Data (Next Generation Sequencing analysis is one example) so we’ve been working hard to transfer large files into and out of the cloud as efficiently as possible. We’re optimizing two costs here: money and time.

Lucky for us, Amazon Web Services continues to drive down the costs of data transfer. We were excited to see that all data transfer into AWS will be free as of July 1st! They’re also reducing the cost to transfer data out of AWS. Less money, more science, yes!

We still need to optimize for time, however. The scalability of the Elastic Compute Cloud (EC2) means we can throw as many cores at a scientific problem as we can afford in a very short time. But what if our input or result data is so large that the time to transfer it far outweighs the time to analyze it?

Our previous work has shown that file transfers often do not fill the pipe to capacity, and are often limited by disk I/O and other factors. Therefore, we can speed transfers by using multiple threads to fill the pipe.

 

As shown above, this work involved moving data directly to a file system using rsync. But since that time, we’ve begun to rely upon the Simple Storage Service (S3) as both a staging area and long-term storage solution for input and result data. S3’s availability and scalability are far superior than even striped Elastic Block Store volumes running on an instance. Besides, using S3 over an instance serving an EBS file system means we have a smaller support footprint. So how can we move data quickly using multiple streams into and out of S3?

Multi-stream Uploads

S3 supports multiple stream data transfers differently depending on the direction of the transfer. If you move data into S3, you can make use of the multi-part upload feature. This enables you to break up a large file into chunks and upload them separately. When you’ve uploaded each chunk, you issue another call to S3 to stitch the chunks together. So we can take a large file, chunk it up, and transfer the chunks simultaneously over many threads to fill our upload pipe.

Multi-stream Downloads

When downloading S3 objects, you can make use of the ranged GET feature of S3. The Hyper Text Transfer Protocol (HTTP) defines a Range header you can add to GET requests. This feature allows you to specify a byte range and only transfer that part of the resource. Because S3 supports the Range header, we can create a list of chunks - specified by byte ranges - and download them simultaneously. After all chunks are downloaded, we can concatenate them and checksum the resulting file to verify integrity.

Test Environment and Results

We tested a new transfer tool by repeatedly uploading and downloading a 2.2 GB NGS file using different numbers of threads. Tests were run on an m1.large EC2 instance. The time to upload the entire file and the time to download it from S3 (including time to reassemble and verify the checksum) were recorded and averaged. As you can see in this first graph, the total bandwidth used in both directions increased with an increasing number of threads.

By using many threads, the time to transfer decreased dramatically. This graph shows the percent increase by number of threads. In the best case, we were able to decrease the time to upload the file to S3 by nearly 2.5 times!

Multi-stream Challenges

Even with a multi-part transfer tool, you still run into the following issues:

• You need to track the transfers and retry any that fail
• You can be too efficient with your transfers and overload your data pipe
• Some Internet proxies drop the Range header from requests silently, resulting in many copies of the entire S3 object to be downloaded
• The time to concatenate downloaded chunks can outweigh some of the transfer gains
• Many S3 tools assume the etag header of the object contains the MD5 checksum of that object (this is not the case with mutipart uploaded objects!) causing integrity checks to fail
  
  

Coming soon: A JetS3t app update

We’ll be addressing these challenges in the coming weeks. Unfortunately, the most commonly used S3 transfer tools have limited support for parallelization. We are big fans of s3cmd, but it doesn’t support multiple threads, and it lacks support for multipart transfers. JetS3t is another option. It’s Synchronize application has support for multiple threads, but it’s focused on moving lots of smaller files in parallel rather than one big file in chunks. Because the JetS3t library has support for multi-threaded transfers, and because it’s open source, we used it as the basis for our own multipart data transfer tool. It’s this tool that generated the results you see above.

We’ll be releasing an application to do this automatically in the next week or so, to enable anyone to easily do parallel uploads.

Why Baking Your Cluster AMI Limits the Menu: DevOps for HPC clusters

You may have read our last blog post about Tanuki, the 10000-core HPC supercomputer we built to predict protein-protein interactions. We’re back to tell you a little bit about how we provisioned the 1250 c1.xlarge instances that made up Tanuki. In fact, it’s the same technology that builds all of our CycleCloud instances whether you select a single-node stand-alone Grid Engine cluster or a super kaiju Condor cluster like Tanuki. But before we get into how we do things today, lets talk about where we’ve been and what we’ve learned.

 

Pre-Built Custom Images: Basic Cloud Cluster “Hello World”

It seems every one's first foray into building HPC clusters in a public virtual cloud (like Amazon’s EC2) involves baking a specialized image (AMI) complete with all the tools required to handle the workload. The most basic architecture includes software for pulling work from a queue or central storage location (e.g. Amazon’s SQS or S3), running the work, and pushing the results back. If you’re feeling especially clever, you may even use a scheduler like Condor, SGE, or Torque.

This first cluster comes up fast, but like all first attempts, probably has some bugs in it. Maybe you need to fix libraries to support your application, add an encrypted file system, or tweak your scheduler configuration. Whatever the case, at some point you’ll need to make changes to it. If you’ve got just one cluster with a handful of nodes, making these changes manually can be done but it’s a pain. Alternatively, you can make your changes, bake new images and restart your cluster with the new images. This is a slow development cycle. It’s painful, but it can be made less so with automated build and test scripts.

With a little time and some perseverance you can get yourself to a point where you’ve got a couple of clusters up and they are working hard for you. Maybe you’ve got a few different image types to handle different types of workloads with different scheduler configurations for each cluster. Maybe you’ve even created images for different operating systems. But then there’s a critical OS security update, or you realize you need to change encryption keys on all your nodes. What do you do?

Figure 1: Explosion of AMI versions

Baked AMIs create problems for everything but the most trivial of use cases: How long will it take you to make new versions of your images and replace the old with the new? How do you keep track of what image versions are affected? Wouldn’t it be easier if we could just change all these instances on the fly?

Absolutely! And you can make that happen with automated configuration management.

 

Base Images Configured with Chef

Having experienced and learned from many of the nuances of building AMIs, configuration management and scaling clusters over the years, we have chosen to use Chef, the open-source project from Opscode, to configure our CycleCloud cluster images. We maintain a small number of base images and layer cluster and role-specific software and configuration on top of the base images using Chef cookbooks. We find the ability to describe the configuration of an instance with a simple Ruby domain specific language to be invaluable. Why?

• Cookbooks are easy to write, read and understand
• Cookbooks are version-controlled in our git repository
• Cookbooks can be modified and tested without baking a new image

Chef allows us to make changes to running clusters. We can patch operating systems, change scheduler configuration, grow file systems and more just by changing a few lines of code and waiting for the scheduled Chef client run.

Figure 2: The DevOps update procedure is much more efficient

So the “get coffee” part is cool! But baked AMIs, for anything but trivial usage, wastes your time, users time, and $$$. DevOps configuration methodologies are a key part of good cloud management.

 

Scaling Chef

In a typical datacenter environment, the life cycle of a machine can range from weeks to years.  In general, bringing new hardware online is staggered, allowing a configuration management system such as Chef to breathe. It tends to only do the heavy lifting when performing the initial configuration of a brand new machine.  In the world of cloud computing, the life cycle tends to be from minutes to weeks.  In addition, the launches tend to be stacked right on top of each other, meaning the demands on the Chef infrastructure are immediate and requests come one after another in rapid succession causing immediate and significant load for Chef.

When we started regularly launching clusters that contained hundreds of machines (thousands of cores) instead of tens of machines (hundreds of cores) - our 4000-core cluster Oni is a good example - it became immediately apparent that we needed to work hard to scale our Chef infrastructure.  It was obvious to us that we had to be able to handle the spike of activity as bare bones EC2 instances came online as naked as the day they were born. Once these systems are converged within Chef, the configuration management load is significantly less, both because there are fewer changes that need to be made and because we decrease our polling interval as a result.

We couldn’t publish a blog post without including some technical details, so here’s how we initially scaled our Chef servers to handle the Tanuki load. We took inspiration from one of the only blog posts we could find on the subject: Joshua SS Miller’s Chef Tuning Part 1. The following tips are for CentOS 5.5.

First, start with a larger EC2 instance, one with quite a few cores to handle the additional application server processes. Then, turn off your chef server and edit /etc/sysconfig/chef-server. Change the OPTIONS line to start as many processes as you wish. In this example, we are only starting two.

# Configuration file for the chef-server service

#CONFIG=/etc/chef/server.rb
#PIDFILE=/var/run/chef/server.pid
#LOCKFILE=/var/lock/subsys/chef-server
#LOGFILE=/var/log/chef/server.log
PORT=4005
#ENVIRONMENT=production
#ADAPTER=thin
#CHILDPIDFILES=/var/run/chef/server.%s.pid
#SERVER_USER=chef
#SERVER_GROUP=chef
# Any additional chef-server options.
OPTIONS="-c 2"

Now start chef-server back up and verify that you have the right number of merb worker processes.

-bash-3.2# ps -ef | grep merb
root      7285     1  0 May11 ?        00:00:02 merb : chef-server (api) : spawner (ports 4005)

chef      7287  7285  0 May11 ?        00:11:46 merb : chef-server (api) : worker (port 4005) 

chef      7288  7285  0 May11 ?        00:11:38 merb : chef-server (api) : worker (port 4006) 

root     12859 12830  0 09:39 pts/4    00:00:00 grep merb

You will now proxy chef-server using Apache. Install apache, then create a new configuration file in /etc/httpd/conf.d called chef_server.conf. Add these lines:

Listen 4000

ServerName devchef.cyclecomputing.com
DocumentRoot /usr/lib64/ruby/gems/1.8/gems/chef-server-api-0.9.12/public/

<Proxy balancer://chef_server>
    BalancerMember http://127.0.0.1:4005
    BalancerMember http://127.0.0.1:4006
</Proxy>

LogLevel info
ErrorLog /var/log/httpd/chef_server-error.log
CustomLog /var/log/httpd/chef_server-access.log combined

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://chef_server%{REQUEST_URI} [P,QSA,L]

You will need to add more BalancerMember entries if you set /etc/sysconfig/chef-server to start more than two processes. You may also have to modify the DocumentRoot to point to the location and version of your chef-server-api gem.

Reload apache with /etc/init.d/httpd reload and test with a knife command.

 

What’s Next

In general, we were universally pleased with the performance of CycleCloud as we began to launch these “mega scale” clusters within cloud environments.  However, there are still improvements to be made.  More specifically, we are always looking to maximize the amount of useful compute work that can be done for the compute time that our clients have paid for.  We want to see clusters of arbitrary scale launch as quickly as possible, and we want the configuration of the cluster to proceed quickly as well.  Stay tuned for our architectural solutions to these challenges. Rest assured that Cycle will not stop driving scalability by generational leaps while making the process of launching a cluster easier and quicker than ever.

As always, if you have questions, or comments, or also love big clusters, please contact us.