Nati Shalom's Blog

The Elephant in the Cloud -- Putting Hadoop on any Cloud

In one of my earlier posts I discussed  in general terms why it makes a lot of sense to put  Big Data on the cloud.

The first step that we took in this regard with Cloudify was to make NoSQL databases, such as Cassandra and MongoDB, run on any the cloud through Cloudify recipes. Uri Cohen gave an excellent talk on this during the Cassandra Summit where he provided a insights into this work.

In the past couple of weeks we've been working on the next phase of this project: Putting Hadoop on the cloud.

As there are various solutions that aim to do something similar, I thought that a good start would be to first outline where we fit in this ecosystem.

Cloudifying Hadoop -- What Does That Actually Mean?

gaji bekerja di singapura

angka keluar togel singapura

keputusan perlawanan persahabatan malaysia vs singapura

kode nomor telepon negara singapura

lowongan kerja di singapura untuk orang indonesia

arti lambang negara singapura

nomor togel singapura online

daftar keluaran togel hari ini

daftar keluaran togel singapura hari ini

singapura layout bangalore price

alamat sekolah indonesia singapura

Yet another Hadoop distribution?

There are multiple distributions of the Hadoop framework today. All come with strong management and sets of tools and do a fairly good job. With that in mind, it was clear to us that our goal wouldn't be to come up with yet another Hadoop distribution but rather to integrate with the ones that are already out there. We picked IBM BigInsights and the Cloudera distribution as the first targets for the project.

When we realized that this was the path we wanted to pursue, the main question that immediately came up was:

What value can we add on top of IBM BigInsights and the Cloudera distribution?

Big Data systems tend to be complex to manage and operate. BigInsights and Cloudera provide good tools to make the process of configuring and deploying Hadoop significantly simpler. That's good for the Hadoop part of the story. Big Data systems and applications often include other services such as relational databases, other NoSQL databases such as Cassandra or MongoDB, stream processing such as Twitter-Storm and GigaSpaces XAP, web front ends such as Tomcat, Play framework and Node.js. Each framework comes with its own management, installation, configuration, and scaling solutions, as shown in the diagram below:

  

Managing each component of your Big Data system seperately is an operational nightmare. That complexity grows exponentially as the system gets bigger, and in Big Data that's just to be expected. 

We realized that one of the areas in which we can reduce this operational complexity is through Consistent Management. By Consistent Management, I'm referring specifically to consistent deployment, configuration, and management across the stack. Consistent management applies not only to the deployment phase, but also to post-deployment, including fail-over, scaling, and upgrades. In addition, Big Data systems tend to consume a lot of infrastructure resources that can easily pile up to thousands of nodes. We realized that we can optimize the infrastructure cost for running the Big Data system through Cloud Enablement and Cloud Portability. Cloud Portabiltiy enables you to choose the right cloud for the job. For example, you could choose a bare-metal cloud for I/O intensive workloads or a virtualized/public cloud for more sporadic workloads. Below is a more detailed description on how we implemeted these two properties:

1. Consistent Management 

With consistent management, we wanted to make the experience of managing each of the tiers and services in the Big Data System consistent throughout the entire stack. This is where Cloudify plugs in.

Cloudify already plugs into a variety of web containers, databases, and through the integration with Chef also into hundreds of services available through the Chef Cookbook.

The process for achieving consistent management for BigInsights and Cloudera Hadoop distribution basically maps to the creation of a Cloudify recipe. The purpose of the recipe in this specific case was to map all the capabilities that come with the IBM BigInsights and Cloudera distribution in a way that would be later consistent with other services in the stack. 

For this, we needed to come up with the following:

• Deployment automation -- A Cloudify recipe enables us to automate the installation, configuration, and deployment of a given service. In the context of a Hadoop distribution, which comes with its own scripts for automating these phases, the process of creating a Cloudify recipe basically meant mapping the specific Hadoop distribution scripts into the Cloudify convention, as well as using the Cloudify discovery, global context, and other services to dynamically inject configuration values that are driven on-the-fly through the deployment process. In the case of BigInsights, this basically mapped into a simple recipe that would launch the machines, set up the SSH environment, and update the BigInsights configuration file with the relevant information. We then called BigInsights to install the NameNode, DataNode, Management, Hive and other services that come with the BigInsights distribution. 
• Automation of post-deployment operations -- In the Hadoop scenario, post-deployment operations mapped into the abilities to add a node, rebalance, and test the environment. To automate these processes, we used Cloudify custom commands. Custom commands enable us to give an alias to those maintenance scripts, and then it enables the users to call those scripts using a simple naming convention without knowing the physical location of Hadoop deployment. 
• SLA-based monitoring and auto-scaling -- SLA-based monitoring enables measure how the Hadoop distribution behaves after it's been deployed, and also to map specific actions when a given SLA is breached. For example, you can monitor a situation where a node fails and then spawns a new machine to take over that fail-node. You can also use the monitors to trigger any of the actions defined in the custom commands, which basically means that you can automate not only a fail-over process but also the ability to scale, by adding more capacity. To do this with Cloudify, use Custom Metrics to monitor the specific metrics that are of interest. One great power of custom metrics IMO is that you're not bound to metrics that are provided by the Hadoop distribution, and you can easily generate compound metrics that can be generated out of correlating metrics from multiple sources, such as such as network statistics or a cloud management system. You can then use the Cloudify Scaling Rules to attach rules to each of those metrics and trigger a fail-over or scaling event.

2. Cloud Enablement and Portability 

The Cloud is basically a more efficient way to run IT. Today, there are growing numbers of cloud offerings and types of clouds, each providign different SLAs and pricing models. There are the public clouds such as Amazon, Rackspace, and also HP, IBM and Dell who's coming out with their own cloud offering. Microsoft and Google are also coming out with new offerings that will undoubtedly change the dynamic in this space. There are also the bare-metal clouds, the private cloud, the OpenSource cloud, etc., and obviously the VMware cloud, which are also an important part of the mix.

Enabling your Big Data systems for all these environments means you can leverage all the agility, efficiency, and flexibiltiy of the cloud, while also allowing you to choose the right cloud for the job.

Given the rapid dynamic of the cloud world, you also want to keep your options open so that you can leverage any future developments and services as they become available.

Controlling Your Cloud Infrastructure is a Critical Part of Controlling Your Big Data Costs

There are various ways to control Big Data costs through controlling  infrastructure. Here are a few examples:

• Bare-Metal -- Reduce the number of machine instances for high I/O workloads.
• Private-Cloud --Optimize performance for your specific workload through specialized hardware (Inifiband, dedicated network, specialized hard drives, etc.) and also have better control over our data secrurity.
• Hybrid-Cloud -- Offload some of the work into the public cloud and optimize costs through more elastic computing.

Take Zynga for example. Zynga recently launched their own private cloud offering, zCloud, and is now using a hybrid cloud strategy in which they move some of the heavy workload from the Amazon cloud into their private cloud environment. With this approach Zynga was able to increase utilization by 3x, which means that they will need 1/3 of the servers that they would need from Amazon for the same workload, as noted by CTO Allan Leinwand on Zynga’s engineering blog:

For social games specifically, zCloud offers 3x the efficiency of standard public cloud infrastructure. For example, where our games in the public cloud would require three physical servers, zCloud only uses one. We worked on provisioning and automation tools to make zCloud even faster and easier to set up. We’ve optimized storage operations and networking throughput for social gaming. Systems that took us days to set up instead took minutes. zCloud became a sports car that’s finely tuned for games.

How Does Cloudify Add Cloud Portability to Your Hadoop Distribution?

Cloudify recipes uses a feature called machine templates. Machine templates are an abstraction of the underlying compute resource. The mapping of the machine template into the particular cloud infrastructure is done through the Cloud Driver.

Cloudify comes with a large and continuously growing list of Cloud Drivers  available for HP, Rackspace, Amazon, Azure, and CloudStack, as well as for completely non-virtualized environments, which is basically just a bunch of bare-metal machines with an IP and network. The Cloud Driver also plugs in with JClouds and as such can plug into any cloud that is supported through the JClouds framework. 

With all this, running your Hadoop deployment on any cloud becomes just a matter of a simple configuration of the target cloud end-point.

Current Status:

We've made the entire project available on  GitHub. These days, we're working closely with our IBM partners to harden and optimize our BigInsights integration, and will be coming out with more updates shortly. Obviously, this is meant to be an ongoing process, so I'd really appreciate any feedback or comments or contributions.

In the next post we will outline more specifics about the work with IBM BigInsights. 

Final Notes

Adding consistent management and cloud portability to your Big Data system allows you to reduce the operational and infrastructure cost of running Big Data systems. It's clear to me that this is just the begining for what we can achieve through this work. The flexibilty that we can add through the integration of Cloudify with IBM BigInsights and the Cloudera distribution will allow us to easily plug into other services and cloud infrastructures. It will also allow us to deploy and manage even the most complex systems through a single-command... So stay tuned.

References

• Big Data in the cloud with Cloudify
• The Rise of Bare metal Clouds
• Lessons from Zynga & Sony on moving from Amazon AWS
• Putting BigData/ Cassandra on the Cloud (Online session from the Cassandra Summit 2012)
• Cloudify Hadoop Recipies peoject on GitHub

 

 

Lessons from the Heroku/Amazon Outage

Earlier this week we experienced the third significant AWS outage in the past 14 months, as reported by Justin Lee in his report Heroku, Pinterest Among Sites Knocked Offline in Amazon Data Center Outage on theWhirr magazine:

An Amazon data center in Ashburn, Virginia suffered a power outage at 9:45 p.m. PDT on Thursday, causing some websites using AWS cloud technology to go offline. High-profile websites like Heroku, Pinterest, Quora and HootSuite saw downtime, as well as many smaller sites.

In this post I'd like to briefly address the lessons from this experience, and more importantly, focus on the lessons from this sort of failure and their effect on public PaaS offerings, such as Heroku.

Lesson from the Heroku Outage: Choose the Right PaaS for the Job

One of the promises of PaaS is productivity. PaaS providers like Heroku claim to increase productivity by abstracting the user from the details of the underlying infrastructure, and even go so far as to claim that PaaS makes application operations redundant.

Lesson 1: Choose the Right PaaS for the Job

The main lesson from this outage is that relying on the PaaS provider to carry all your operations isn't always a safe bet. When we move to PaaS we still need to understand how they run their disaster recovery, high-availability, and scaling procedures. Heroku-like PaaS also forces you to a lowest common denominator approach to dealing with continuous availability and scalability. In reality, however, there are many tradeoffs between scalability, performance, and high availability. The best fit between those tradeoffs tends to be application specific, so compromising on a lowest common denominator could be less productive and more costly at the end of the day.

Which brings me to the last point in this section -- PaaS was meant to provide a higher productivity for running our apps on the cloud by abstracting the details of how we run our application (the operation) from the application developer. The black-box approach of many of the public PaaS offerings, such as Heroku, is often an extreme measure in this regard. There is often a close coupling between what the application does and the way we run it. A new class of Open PaaS platforms such as Cloudify, CloudFoundry, and OpenShift offer a different open source alternative that gives you more control of the underlying PaaS platform. Cloudify takes it even further, providing an open recipe model that integrates with the likes of Chef, enabling you to easily customize and control your operations without affecting developer productivity.

Lesson 2: Database Availability Must Address Datacenter Failure

The other area that Heroku, and to be honest, most other PaaS offering don't adequately address is database high-availability, which is obviously a tough area. Specificaly, in the event of data center failure or availability zone failure, as in the present case. To deal with database availability, it is necessary to ensure real-time synchronization of the database across sites. The example at the bottom of this post refers to a specific way this can be done, between two mySql instances running on Amazon and Rackspace with Cloudify.

General Lessons:

Lesson 3: Coping with Failure, Avoiding a Single Point of Failure

The general lesson from this and previous failures is actually not new. To be fair, this lesson is not specific to AWS or to any cloud service. Failures are inevitable, and often happen when and where we least expect them to. Instead of trying to prevent failure from happening we should design our systems to cope with failure. 
The method of dealing with failures is also not that new -- use redundancy, don't rely on a single point failure (including a data center or even a data center provider). Automate the fail-over process, etc...

Haven't Learned from Past Lessons?

The question that comes out of this experience IMO is not necessarily how to deal with failures (those lessons are as old as the mainframe or even older), but rather -- why are we failing to implement the lessons? Assuming that the people running these systems are among the best in the industry makes this question even more intresting. Here is my take on it:

• We give up responsibility when we move to the cloud: When we move our operations to the cloud, we often assume that we're out-sourcing our data center operation completely, including our disaster recovery procedures. The truth is that when we move to the cloud we're only outsourcing the infrastructure, not our operations, and the responsibility of how to use this infrastructure remain ours. 
• Complexity: The current DR processes and tools were designed for a pre-cloud world, and do not work well in a dynamic environment, such as the cloud. Many of the tools that are provided by the cloud vendors (Amazon in this specific case) are still fairly complex to use.

Implementing Past Lessons in a Cloud World

The first point is is easy -- we need to assume full responsibility for our applications' disaster recovery procedures, in the cloud world just as if we were running our own data center. The hard part in the cloud world is that we often have less visibility, control, and knowledge of the infrastructure, which affects our ability to protect our applications -- and each sub-component of our application -- from failure. On the other hand, the cloud enables us to spawn new instances easily on various data center locations, a.k.a Availability Zones. 

And so, most failures can be addressed by moving from the failed system to a completely different system regardless of the root cause of the failure. Therefore, the first lesson is that in the cloud world it is easier to implement disaster recovery plans, by moving our application traffic to a completely different redundant system in a snap, rather than trying to protect every component of our application from a failure. If we're willing to tolerate a short window of downtime, we can even use an on-demand backup site rather than pay the consistent cost and overhead of maintaining a hot backup site.

Which brings me to the next point: What do we need to build such a solution?

A consistent redundant environment that is ready to take over in case of failure needs to include the following elements:

• Workload Migration: Specifically, the ability to clone your application environment and configuration in a consistent way accorss sites, and on demand.
• Data Synchronization: The ability to maintain a real-time copy of the data between two sites. 
• Network Connectivity: Enabling the flow of network traffic between two sites.

Which leads to the second challenge: Complexity. Here, I would use an example of a simple web-app and show how we can easily create two sites on demand. I would even go so far as to set this environment on two separate clouds to show how we can ensure an even higher degree of redundancy by running our application across two different cloud providers.

A Step by Step Example: Fail-Over from AWS to Rackspace

In this example, we picked Amazon and Rackspace as the two target sites. The same solution would also work between two availability zones in Amazon or data centers. We've also tried the same example with a combination of HP Cloud Services and a flavor of a private cloud.

The example demonstrates a very simple web application with global load-balancer (Rackspace), and a Web application (Pet Clinic) based on Tomcat as the Web front end and MySQL as the database.

On both ends we used GigaSpaces XAP Transactional WAN replication as a replication channel between the two instances of MySQL and Cloudify to handle the workload migration between the sites.

The Goal: Fail-over with no change to the target application

The goals that we set for ourselves were seamlessness and no change to the target application or database. We achieved this by plugging the replication service into the existing instances of MySQL. The replicating service listened to the MySQL events and replicated every change to its peer MySQL instance. Cloudify enabled us to clone the same application in both Amazon and Racksapce while maintaining a consistent configuration setup as well as consistent scaling and fail-over SLAs. Cloudify does this by abstracting all the information through portable recipe definitions. Cloudify wraps the application instances with its management and control services based on the definition provided in the recipe. This enabled us to clone the environment as well as add elastic scaling without changing the target application (Pet Clinic in this case).

You can read the full details, including code references on Github, in Dotan Horvits' blog post: AWS Outage Thoughts on Disaster Recovery Policies.

Join GigaSpaces for our upcoming events to learn more about this topic:

• Transactional Cross-Site Data Replication webinar this Wednesday June 20th (9:00 AM PST) that will teach you how to achieve consistent data replication across sites in just 10 minutes, enabling disastery recovery, cloud bursting, and more. Register here.
• The DevOps PaaS Infusion Meetup this Tuesday June 19th at 6:30 PM - NetFlix and OpsCode will be joining GigaSpaces at this meetup in NY where one of the highlights will be a panel on lesson learning from the recent AWS outage.  Register here.

 

References:

• Heroku, Pinterest Among Sites Knocked Offline in Amazon Data Center Outage
• AWS Outage Thoughts on Disaster Recovery Policies
• Mapping the Cloud Stack
• Making Cloud Bursting a Practical Reality

Making Hadoop Run Faster

One of the challenges in processing data is that the speed at which we can input data is quite often much faster than the speed at which we can process it. This problem becomes even more pronounced in the context of Big Data, where the volume of data keeps on growing, along with a corresponding need for more insights, and thus the need for more complex processing also increases.

Batch Processing to the Rescue

Hadoop was designed to deal with this challenge in the following ways:

1. Use a distributed file system: This enables us to spread the load and grow our system as needed.

2. Optimize for write speed: To enable fast writes the Hadoop architecture was designed so that writes are first logged, and then processed. This enables fairly fast write speeds.

3. Use batch processing (Map/Reduce) to balance the speed for the data feeds with the processing speed.

Batch Processing Challenges

The challenge with batch-processing is that it assumes that the feeds come in bursts. If our data feeds come in on a continuous basis, the entire assumption and architecture behind batch processing starts to break down.

If we increase the batch window, the result is higher latency between the time the data comes in until the time we actually get it into our reports and insights. Moreover, the number of hours is finite -- in many systems the batch window is done on a daily basis. Often, the assumption is that most of the processing can be done during off-peak hours. But as the volume gets bigger, the time it takes to process the data gets longer, until it reaches the limit of the hours in a day and then we face dealing with a continuously growing backlog. In addition, if we experience a failure during the processing we might not have enough time to re-process.

Speed Things Up Through Stream-Based Processing

The concept of stream-based processing is fairly simple. Instead of logging the data first and then processing it, we can process it as it comes in. 

A good analogy to explain the difference is a manufacturing pipeline. Think about a car manufacturing pipeline: Compare the process of first putting all the parts together and then assembling them piece by piece, versus a process in which you package each unit at the manufacturer and only send the pre-packaged parts to the assembly line. Which method is faster?

Data processing is just like any pipeline. Putting stream-based processing at the front is analogous to pre-packaging our parts before  they get to the assembly line, which is in our case is the Hadoop batch processing system.

As in manufacturing, even if we pre-package the parts at the manufacturer we still need an assembly line to put all the parts together. In the same way, stream-based processing is not meant to replace our Hadoop system, but rather to reduce the amount of work that the system needs to deal with, and to make the work that does go into the Hadoop process easier, and thus faster, to process.

In-memory stream processing can make a good stream processing system, as Curt Monash’s points out on his research traditional databases will eventually end up in RAM. An example of how this can work in the context of real-time analytics for Big Data is provided in this case study, where we demonstrate the processing of Twitter feeds using stream-based processing that then feeds a Big Data database for the serving providing the historical agregated view as described in the diagram below.

Faster Processing the Google Way: Using Stream-Based Processing Instead of Map/Reduce

Due to a lack of alternatives at the time, in many Big Data systems today Map/Reduce is used in areas where it wasn't a very good fit in the first place.  A good example is using Map/Reduce for maintaining a global search index. With Map/Reduce, we basically rebuild the index, where it would actually make more sense to update it with changes as they come in.

Google moved large part of its index processing from Map/Reduce into a more real-time processing model, as noted in this recent post:

So, how does Google manage to make its search results increasingly real-time? By displacing GMR in favor of an incremental processing engine called Percolator. By dealing only with new, modified, or deleted documents and using secondary indices to efficiently catalog and query the resulting output, Google was able to dramatically decrease the time to value. As the authors of the Percolator paper write, ”[C]onverting the indexing system to an incremental system … reduced the average document processing latency by a factor of 100.” This means that new content on the Web could be indexed 100 times faster than possible using the MapReduce system!

..Some datasets simply never stop growing ..it is why trigger-based processing is now available in HBase, and it is a primary reason that Twitter Storm is gaining momentum for real-time processing of stream data.

Final Notes

We can make our Hadoop system run faster by pre-processing some of the work before it gets into our Hadoop system. We can also move the types of workload for which batch processing isn't a good fit out of the Hadoop Map/Reduce system and use Stream Processing, as Google did.

Interestingly enough, I recently found out that Twitter Storm came up with an option to integrate an in-memory data store into Storm through the Trident-State project. The combination of the two makes lots of sense and something were currently looking at right now so stay tuned.

 

References

• Build Your Own Twitter Like Real Time Analytics - A Step By Step Guide (Blog Post)
• Build Your Own Twitter Like Real Time Analytics - A Step By Step Guide (Online Tutorial)
• Real Time analytics for Big Data: Facebook's New Realtime Analytics System
• Beyond hadoop: fast queries from big data
• Traditional databases will eventually wind up in RAM
• Why the days are numbered for Hadoop as we know it

Related articles

Elephants and Eyeballs: Real-Time Big Data in the Real World

Cassandra on ACID

GigaOm: 'Hadoop's days are numbered...' Are they?

Hadoop Officially a Big Deal for Big Data

Woz is Worried About “Everything Going to the Cloud” — the Real Issue is Giving Up Control

 

In a recent article, Steve Wozniak, who co-founded Apple with the late Steve Jobs, predicted "horrible problems" in the coming years as cloud-based computing takes hold. 

 

"I really worry about everything going to the cloud,".. "I think it's going to be horrendous. I think there are going to be a lot of horrible problems in the  next five years. ...."...with the cloud, you don’t own anything. You already signed it away.”

 

When I first read the title I thought, Wozniak sounds like Larry Ellison two years ago, when he pitched the Cloud is hype, before he made a 180-degree turn to acknowledge Oracle wished to be a cloud vendor too. 

 

Reading it more carefully, I realized the title is instead just misleading. Wozniak actually touches on something that I hear more often, as the cloud hype cycle is moves from a Peak of Inflated Expectations into through the Trough of Disillusionment.

 

Wozniak echos an important lesson, that IMO, is major part of the reason many of the companies that moved to cloud have experienced lots of outages during the past months. I addressed several of these aspects in in a recent blog post: Lessons from the Heroku/Amazon Outage.

 

When we move our operations to the cloud, we often assume that we're out-sourcing our data center operation completely, including our disaster recovery procedures. The truth is that when we move to the cloud we're only outsourcing the infrastructure, not our operations, and the responsibility of how to use this infrastructure remain ours. 

 

Choosing better tradeoffs between producivity and control

 

For companies today, the main reason we chose to move to the cloud in the first place was to gain better agility and productivity. But in starting this cloud journey, we found that we had to give up some measure of control to achieve the agility and productivity.

 

The good news is that as the industry mature there are more choices that provides better tradeoffs between producivity and control:

• Open source cloud such as OpenStack and CloudStack 
• Private cloud offering 
• DevOps and automation tools such as Chef and Puppet
• OpenSource PaaS such as Cloudify, OpenShift and CloudFoundry
• DevOps and PaaS combined such Cloudify

As businesses look at cloud strategy today, there isn't a need to give up control over productivity. With technologies like Cloudify, businesses can get the best out of both worlds.

 

References

• Apple Co-Founder Steve Wozniak Distrusts the Cloud: Is He Right?
• Mapping the Cloud/PaaS Stack
• Why Larry Don't Get It
• Lessons from Zynga & Sony on moving from Amazon AWS
• Public vs Private clouds (Again!)- it's not about the cost

Putting DevOps and PaaS together with Cloudify

 

Putting DevOps and PaaS together with Cloudify

DevOps and PaaS represent two different paradigms for delivering applications to the cloud.

• DevOps - DevOps takes an automation approach - with DevOps we basically script the process of installation, configuration and deployment of the application stack. 
• angka ikut togel singapura hari ini


• data keluaran togel 2012
perancangan keluarga di

youtube bola sepak malaysia vs singapura 2012

tukaran mata wang malaysia ke singapura
prediksi angka togel singapura besok
• yamaha store plaza singapura
sistem pemerintahan negara singapura

times bookstore singapore plaza

rumah peranginan persekutuan di singapura
ramalan mimpi togel singapura

I couldn t help but notice the problem you had back there with Megan. He shook the bag of coke that he held in his hand creating small puffs of white clouds in the air as it spilled out onto the cement floor, You re crazy!
backpacker ke malaysia dan

kurs terhadap rupiah
PaaS - PaaS takes an abstraction approach by abstracting the details of the cloud infrastructure from the developers. 

DevOps tends to cater to operations guys that want to maintain the flexibility and control over the infrastructure and PaaS caters more to developers who just want to write code and deliver it without worrying too much about the details of the underlying infrastructure.

 

For a while DevOps and PaaS had been considered two competing paradigms as noted in Tom Mornini’s (Engine Yard) post DevOps is DOA:

"DevOps essentially requires everyone involved in the development, deployment, and management of software to change the way they work, to cross-train for multiple functions, and to collaborate with people they’ve all too often viewed as adversaries. It creates new processes and methodologies and, in many cases, uses new tools to get the job done. In addition, it often requires a small army of consultants to implement and manage the transition. That is why the “promise” and “potential” of DevOps has yet to be realized by more than a handful of real-world companies.

...The appeal of the DevOps model fades even further when you consider the fact that there is a better option! You can achieve the intended benefits of DevOps without any of the turmoil by taking advantage of Platform-as-a-Service offerings, or PaaS."

Putting DevOps and PaaS together

The truth is that it's hard to choose between the two paradigms. DevOps provides a high degree of control and flexibility but this flexibility comes at the cost of complexity and reduced productivity and often requires a culture change as Tom Mornini pointed out. PaaS on the other hand, provides an extremely simple deployment environment but it also forces you to fit into the PaaS environment, including the language of choice, middleware stack and cloud environment etc.

The best of both worlds- The control of DevOps and Productivity of PaaS

Do we really need to choose between the two? We can use DevOps as the foundation of the PaaS infrastructure. In this way, the operations guys can choose their stack of choice, package their specific blueprint, pick their cloud of choice and package all that as a pre-baked environment to the developers who in turn can be relieved of all of the details as they would be with any other PaaS. In this way we get the control of DevOps and productivity of PaaS.

At this point I'll refer to Cloudify as an implementation reference of that concept to illustrate my point in this regard.

Putting DevOps and PaaS together with Cloudify

Cloudify is an Open PaaS Stack that brings DevOps and PaaS together. Here is how:

The control of DevOps 

1. Application Blueprint - Cloudify allows you to add your own application blueprint into the platform through recipes. Recipes are at the heart of Cloudify architecture. A recipe in Cloudify is a Groovy-based DSL. The recipe is where you define your application stack, including the installation, configuration, monitoring, scaling and fail-over and tell Cloudify what, how and where to deploy your app.
2. Integration with DevOps tools such as Chef - Cloudify allows you to call Chef cookbooks from within a Cloudify recipe. This gives Cloudify users a way to leverage the wide variety of existing Chef cookbooks and community and decorate it with a Cloudify recipe if needed, to add scaling, fail-over automation etc.
3. Operational Task Automation - Custom Commands  allow you to attach common operation tasks such as update-app, such as in the case of a web app, run-snapshot in the case of a database and more, as part of a service recipe, as well as automate the post-deployment operational procedures of an application alongside its lifecycle.

The productivity of PaaS

Developers can run one of the existing recipes as they would with any other PaaS without going through any of the steps above. Unlike other Platforms-as-a-Service, they can also customize the existing recipes or pick custom recipes that cover their specific environment and application blueprint. All this without losing that same productivity. Developers can also get better visibility into the underlying infrastructure, and have a better experience when troubleshooting. They can also use Cloudify as an automation tool as part of their continuous deployment environment.

Final words

Adrian Cockcroft a highly regarded Cloud Architect at Netflix provides an excellent insight on how DevOps and PaaS work at Netflix in his post Ops, DevOps and PaaS (NoOps) at Netflix:

“We have a small number of DevOps engineers embedded in the development organization who are building and extending automation for our PaaS, and we have hundreds of developers using NoOps to get their code and datastores deployed in our PaaS.”

An interesting lesson from the Netflix experience is that an important part of running an online service is the ability to resolve any friction for developing and managing such a service. Moving a large part of the operational responsibility to the developers makes it even more important that the development platform (PaaS) come with operations tools that are pre-baked into the developer environment.  At the same time, that same environment needs to also cater to a dedicated (but a significantly smaller) operational team.  This makes the idea of putting DevOps and PaaS together even more critical to reach the business agility and operational efficiency. Netflix built their own PaaS environment to address such a need. Cloudify provides a similar solution for those who want to build their own Netflix environment but without the hassle involved with doing it themselves.

References

• PaaS, Does It Really Mean No-Ops?
• Ops, DevOps and PaaS (NoOps) at Netflix
• Mapping the Cloud/PaaS Stack
• GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker
• DevOps is DOA
• Productivity vs. Control tradeoffs in PaaS
• DevOps and PaaS - Friend or Foe?
• PaaS shouldn’t be built in Silos

 

What should Marissa Mayer do with Yahoo!

Last week the internet exploded as the news of the new appointment of Marissa Mayer as the new CEO of Yahoo sparked.

 

Most of the discussion was centered around the speculation of why Marissa left Google - clearly  the market leader in the search and advertising business compared with Yahoo! a company in deep trouble for a while, who seems to have lost its way and changed three previous CEOs in the past few years.

 

I was surprised though to see how little was said on what Marissa's plans are to save Yahoo! 

In this post, I wanted to try and explore this, as well as add my my own personal thoughts about what Marissa could do to regain a leading position in the market.

 

 

What Yahoo! shouldn't be doing

Until now, the strategy at Yahoo! has centered around advertising, which was based on its core search engine, homepage and email. 

Obviously, this strategy ought to come to its end, as both of these products are growing increasingly stale.

 

In addition to this, it would be almost impossible for Yahoo! to regain any kind of leadership position in that space as  Jason Del Rey and Michael Learmonth  argue in their article Predicting The Strange Future Relationship of Yahoo!, Google, and Facebook:

The economics of content aren't good enough, and Yahoo doesn't have the firepower to pony up, say, the $100 million YouTube is spending on original channels or the hundreds of millions more Netflix is spending to acquire rights to TV and movies.

Perhaps, a hint of Marissa future plans - build products instead of chasing search

Clearly Marissa didn't join Yahoo! to continue her predecesor's strategy i.e. focus on content. So the question is, what could be the next viable option that Marissa should pursue. A hint in that regard is given in the Jason Del and Michael Learmoth article:

Ms. Mayer represented what Yahoo could become, a company that builds products instead of chasing search -- as it did for nearly a decade -- and social, which is now sapping the company of its once-formidable engagement with users.

The real question is what could that mean?

The elephant in the room

Most people are not aware of the fact that the main company behind the Hadoop project is Yahoo!.

In fact, Yahoo! is probably the largest deployment of Hadoop, and still the major contributor to the project.

Yahoo runs Hadoop on 42,000 servers--that's 1,200 racks--in four data centers. Its largest Hadoop cluster is 4,000 nodes, but that will increase to 10,000 with the release of Apache Hadoop 2.0, expected...

Yahoo uses Hadoop for its own internal analysis of information captured from user interactions. It stores 140 petabytes in Hadoop. Since Hadoop keeps all data sets in triplicate, over 400 petabytes of storage are needed to sustain its systems.

Having said that, Yahoo! has done a fairly bad job on monetizing the project, and has let other companies such as Cloudera, Amazon, IBM, EMC to build services and solution around the project.

Now here's the real elephant in the room: During a recent interview with InformationWeek  Scott Burke, senior VP of advertising and data platforms provided a very interesting insight on some of the assets that Yahoo developed around Hadoop:

• Yahoo! can advise advertisers on the demographic that responds best to their campaigns so that they can adjust their ads as necessary to appeal to the widest audience. According to Burke, this can lead to improved results, sometimes even doubling expected response rates.
• Yahoo's data indicates not only who is viewing advertising, but also how much time they're spending on it, what their click-through rate is, and what they do on the Yahoo site after viewing an ad. This is all valuable feedback to advertisers, as it lets advertisers make decisions early in an ad campaign, and helps them plan their future advertising strategies.
• Yahoo has also developed a set of open source code called Cocktails, which gives advertisers the tools to extract information in Yahoo's Hadoop systems. Cocktails are written in JavaScript so they can run in browser windows; Manhattan is a server-side cocktail and Mojito is client-side. 

What Yahoo should become? - The Web Analytics Company

 Marissa already indicated its intent to move from the search business and into products but the real question is what that means.

This is where I wanted to add my own personal thoughts.

 

Yahoo! - The new Web Analytics company

IMO Yahoo should build on its strong Hadoop assets and become the leading Web Analytics Company.

 

What do I mean by that?

I mean that instead of providing content directly, it will give other companies services to manage and analyse their content using the tools and an experience gained within Yahoo. 

Yahoo should also use this position as its ticket into the new cloud world, leveraging its existing data center operation and assets that it developed over the years in managing large scale data services.

Here are few  concrete examples:

• Hadoop as a Service - similar to Amazon Elastic Map Reduce (EMR) but significantly simpler and richer with built-in analytics and reporting tools.
• Analytics PaaS - similar to GAE but focused in the area of Analytics where it would give users the ability to build apps that specialize in data analytics, rather than serving web apps or mobile apps.

Become the Market Place for Web Analytics

There are many services that collect data over the web including social media, and try to extract their own insights out of it.

 

By collecting this data and organizing it for other companies - Yahoo could give its users an easier path to building useful insights (rather than starting with collecting data).  This would then provide a simpler way to build their own insights on-top of that data, rather than building it from scratch. It could also become the marketplace for such insights, where they would offer other analytics companies tools to provide specialized insights for companies consuming this data.

 

Win the developers mindshare

The web and social world is driven by developers - Yahoo can and needs to win the heart and mind of the developer community.

This is specifically true as Google and Facebook are starting to lose some of that mindshare. Given that it already has good assets in the open source community, such as the Hadoop project, Yahoo can leverage this to create even more traction within this community. With the right focus, this shouldn't be too difficult.

Anyway these were my own personal thoughts on this interesting excerise.

 

Good luck Marissa! 

 

References

• 3 Challenges Yahoo’s new CEO Marissa Mayer shares with IT managers
• The real reason Marissa Mayer left Google: She had to
• Google's Marissa Mayer: 3 Leadership Traits She'll Bring to Yahoo
• Predicting The Strange Future Relationship of Yahoo!, Google, and Facebook
• Yahoo Has Seen Its Future; In One Word, It's Products
• How Yahoo Spawned Hadoop, the Future of Big Data
• Yahoo And Hadoop: In It For The Long Term

Google Now and the Future Shock

I was following the Google announcment last week as i was at a Dell Future Cloud meeting just few blocks away from the Google event. We all started throwing idea's that was mainly inspired by the new Google Now and Google Glasses announcement. A world where were going to be allways connected, monitored and managed. We even went further to describe that world.  A world where all this new devices would be planted in our body and we wouldn't even need to carry external devices like Glasses to do all that.

When we finished this session, on my way back home something felt wrong. Google Now sounds cool and scary at the same time. Is this the world that i want to be part of? more importantly is this the world that i'd like to see my kids growing to?

As i was going through this philosofical thoughts i looked around and came across an interesting book from 1972 by Alvin Toffler  titled the  Future Shock.

Alvin looks at the pace of innovation and the effect of that innovation on human being's. Alvin  book looks at whether were really improving our productivity and quality of life with all this innovation. Here some of the interesting thoughts from his book that i thought we should all think about as when we decide to adopt all this "coo" technologies.

While a human being's capacity to adjust physically, psychologically, and socially to this torrent of change is finite and quite limited, the pace of change is increasing and expanding into more and more areas of individuals' lives. Moreover, no one is asking for these profound and endless changes; they stem more from the economic impulses of the marketplace than from any kind of consumer demand, and perhaps we should be asking to what extent this flood of innovations actually enhances our lives, and personal convenience associated with all these innovations and technological improvements are worth the social, economic, and political change that follows in its wake.

To put it simple - were becomming salves of our own inventions, unlike the old generation slaves were doing it out of free will and even willing to pay for it and get excited about it.

Isn't there a better way? Food for thoughts..

 

Lessons from Amazon RDS on Bringing Existing Apps to the Cloud

Its a common believe that Cloud is good for green field apps. There are many reasons for this, in particular the fact that the cloud forces a different kind of thinking on how to run apps. Native cloud apps were designed to scale elastically, they were designed with complete automation in mind, and so forth. Most of the existing apps (a.k.a brown field apps) were written in a pre-cloud world and therefore don't support these attributes. Adding support for these attributes could carry a significant investment. In some cases, this investment could be so big that it would make more sense to go through a complete re-write. 

In this post I want to challenge this common belief. Over the past few years I have found that many stateful applications running on the cloud don't support all those attributes, elasticity in particular. One of the better-known examples of this is MySQL and its Amazon cloud offering, RDS, which I'll use throughout this post to illustrate my point. 

Amazon RDS as an example for migrating a brown-field applications

MySQL was written in a pre-cloud world and therefore fits into the definition of a brown-field app. As with many brown-field apps, it wasn't  designed to be elastic or to scale out, and yet it is one of the more common and popular services on the cloud. To me, this means that there are probably other attributes that matter even more when we consider our choice of application in the cloud. Amazon RDS is the cloud-enabled version of MySQL. It can serve as a good example to find what those other attributes could be.

The Amazon RDS Approach for Cloud Enablement of MySQL

Amazon's approach for cloud enablement of MySQL includes the followings elements:

1. Pre-tuned MySQL -- A pre-configured version of MySQL that was tuned to run well on the AWS instances.
2. Automation -- Automate the installation, deployment of MySQL database.
3. Automate a scale-up/process.
4. Automate the high availability configuration

What Can We Learn from the RDS Experience?

There are couple of lessons that we can learn from this Amazon RDS experience:

1. There is value in migrating existing applications to the cloud. Or in other words, the cloud shouldn't be considered only for green field applications.
2. Automation of the installation and deployment processes is the "low hanging fruit" for migrating application to the cloud.  
3. Elasticity -- quite often when we think of elasticity we think of scaling out, i.e. -- adding more capacity by launching more machines. As MySQL wasn't designed for this model, the RDS approach to elasticity was to automate the move from small to a bigger machine. Apparently this model had proven to be useful even if it involves downtime and is limited to scale-up.
4. Baby steps approach: When Amazon first released RDS it only included basic automation and it was only somewhere down the road htat it kept enhancing the service with more automation. This baby steps approach was a key to speed up the migration process of MySQL.

RDS vs SimpleDB

When AWS was first introduced it started with SimpleDB as its cloud database service. SimpleDB fits into the "green field" application approach to cloud. The approach with SimpleDB was to trade features and functionality for simplicity, elasticity and scalability. The popularity of RDS over SimpleDB is another interesting lesson -- trading features for scalability is definitely a simpler approach to get things running on the cloud. However, there is a considerable number of users and applications that would prefer to trade some degree of scalability for features. This also teaches us that the "green field" approach isn't necessarily the right approach. Migrating existing services has lots of value, especially if they are already in wide use.

Applying These Lessons to Existing Applications

The main take away for migrating your existing app to the cloud can be summed up as follows:

1. Use the baby steps approach
2. Start with simple automation and monitoring of the installation and deployment process
3. Add automation of post-deployment processes such as fail-over and scaling as a secondary step
4. Use a pragmatic approach to elasticity. There are areas of your app where the simplest way to apply elasticity would be simple automation of the move from a small to a bigger machine, often the stateful part of your application. There are other areas of your application, such as the web-tier, where using full elasticity can be done by spawning new instances and automating the discovery and configuration for how these instances group together, as in the case of a web tier.

DevOps & PaaS Can Be a Great Tool to Implement These Steps

There are many ways we could implement the baby steps approach, especially if we look at it as a one-time operation. If we need to move more than one application, we can easily find ourselves repeating the same effort over and over again for each application. DevOps tools such as Chef provide a good generic tool for scripting your installation and configuration processes. New open PaaS  platforms, such as CloudFoundry, Cloudify, and OpenShift address the entire application deployment life cycle, including the automation of scaling, fail-over and monitoring. An integration between DevOps and PaaS, such as the integration between Cloudify and Chef,is even better-suited tothis challenge, as it gives you the best of both worlds. 

Final Words

According to a recent HP Survey with 940 IT responders, a majority of businesses are planning to move their mission-critical apps to the cloud over the next two to five years. A recent survey by Cisco of 1300 responders shows that today only 5 percent of the responders are able to migrate meaningful parts of their application to the cloud. According to this survey, what makes migration to the cloud even more difficult is the lack of information about the process.

Many new cloud users are lost in a sea of hype-driven desire to move to cloud computing, without many proven best practices and metrics.

I believe that the lessons from Amazon RDS could serve as a good reference on what that process could be. If you are considering moving more than one application into the cloud, you should consider the use of DevOps & PaaS as the means to deliver those steps.

References:

• Cloud On-Boarding: The True Story
• Companies nearing mission critical apps in the cloud
• Cisco Global Cloud Networking Survey
• This just in: Cloud computing is hard and takes a long time
• Cloudify and Chef integration 

OpenStack is Coming to Israel

I'm very excited to announce our first OpenStack Israel event on Wednesday the 30th of May in Petach Tikva in collaboration with the IGT Cloud and Rackspace. Since Avner Algom and myself started to work on the event a few months ago I was positively surprised to see the amount of local innovation and work that is already taking place here at Israel in this space. As I write this post, it looks like we’re getting close to 200 registrations, and all of this has been achieved for the most part by word of mouth, a clear indication of the growing interest here.

We’ve got a nice lineup of presenters covering different parts of the stack:

• Chmouel Boudjnah (Rackspace):  Compute, Storage 
• Arturo Suárez  (StackOps): How to Create an OpenStack Powered Service Provider in 30 Days
• Roy Amir, (Alcatel Lucent): OpenStack Quantum - Network as a Service
• Zohar Alon (Dom9): Overview the Security Controls Built into OpenStack
• Samuel Bercovici (Radware): Cloud Load Balancing and Application Delivery Control (ADC) Services in OpenStack
• Uri Cohen (GigaSpaces), Camuel Gilyadov (ZeroVM): PaaS and Big Data on Openstack
• Alex Glikson (IBM),  OpenStack-Based Cloud Hosting Infrastructure

Shlomo Swidler will be leading two panel discussions, during the first discussion we’ll review the shift toward Open Source in the cloud and the effect of that move on the cloud industry. In the second panel we’ll try to predict some future directions and trends we expect for OpenStack.

I would also like to take this opportunity and welcome our distinguished guests from Rackspace, IBM and StackOps:

• Mac Devine, Director and CTO, Portfolio and SO Offering Management, Global Technology Services, IBM Distinguished Engineer
• Chmouel Boudjnah Principal Engineer at Rackspace Hosting
• Arturo Suárez Co-founder of StackOps Technologies

During the event we reserved a slot for lightning talks -  If you have an interesting story you'll have the chance to share it during the event.

Anyway this event is going to be a kickoff for a series of meetups around OpenStack that is planned throughout the year. We'd appreciate any help in this regard, so if you have suggestion for a meetup this is your chance.

I would like to finish up with special thanks to Marc Coollier and Luaren Sell from Rackspace who's been very instrumental and supportive throughout this process. We wouldn't have been able to pull this off without their close help.

 

Making Cloud Bursting a Practical Reality

What is Cloud Bursting?

A good definition for cloud bursting can be found here:

Cloud bursting is an application deployment model in which an application runs on a private cloud or data center and bursts to a public cloud when the demand for computing capacity spikes. The advantage of such a hybrid cloud deployment is that an organization only pays for extra compute resources only when they are needed.

The Business Impact of Cloud Bursting

Neal Sample the former CTO - X.commerce at eBay gave an interesting talk last year on the economic benefit of Cloud Bursting. Neal pointed out eBay traffic statistics and showed some real numbers of the business impact of bursting peak load activities using on-demand cloud resources as presented in the diagram below.

The diagram shows real traffic statistics on eBay site (the brown lines).

The line at the top shows the associated costs for provisioning all the resources for peak loads. The blue lines show what the cost could be if they’d provision for the average load and use on-demand resources for the peak loads.

Based on this case study, cloud bursting can yield a 40% reduction in overall costs.

The challenges with cloud bursting

While the economic model of cloud bursting can be very compelling as Neal articulated in his eBay study, practically speaking, it remains an interesting theoretical model for the most part due to the technical challenges involved in implementing this model.

Let me explain...

To make cloud bursting a practical reality you need to address the following:

1. Network connectivity between the private and public site
2. Consistent environment setup
3. Data/Security model

What's changed?

The availability of new OpenStack public clouds from both HP Cloud Services and Rackspace makes it possible to create private and public clouds using the same underlying cloud infrastructure which supports the same API.

With this it makes a lot of sense to connect both ends of OpenStack clouds into one virtual cloud.

We can also see Amazon starting to partner with private Cloud solutions that are based on the Amazon API through their partnerships with Eucalyptus, as well as Citrix’s recent announcement with Cloud Stack.

That basically solves one of the more complex challenges of the lack of consistent data-center models between private and public clouds.

While this is a great start it still doesn't address all three challenges that I outlined above.

Making cloud bursting a practical reality

1. Network connectivity

To address the network connectivity issue, we can use one of these two approaches:

• Virtual Private Cloud - With this approach we create a VPN tunnel between the private and public network making the network look like one big IP network. A good reference on the various solutions in this category can be found here.
• Load balancing web traffic (with a federated deployment) - in this approach the private and public site remain isolated from each other from a network perspective. What we’d need to do is use a global load balancer to route traffic between the private and public sites so the only connectivity is at the global router.

 

        

2. Consistent environment

In order to burst our private cloud activities into an on-demand public site, we need to be able to fork the exact set of services and application setup that we have on our private cloud, and push it to the cloud. To this end, we need to use automation and configuration tools that work in pretty much the same way in both environments. Tools like Chef do a great job with automating the installation and configuration parts, this partnered with Cloudify recipes enables us to leverage Chef recipes for setting up the installation of the individual services and Cloudify for setting up a complete deployment of any application stack on demand on both private and public cloud.

3. Data/Security

So obviously, bursting processes is one thing, but bursting data is a completely different ball game. Bursting data is one of the hardest challenges with cloud bursting today. There are basically two options when dealing with data in a cloud bursting scenario:

1. The primary site approach - Use the private cloud as the primary data site, and then point all the burst activity to that site.
2. Federated site approach - This approach is similar to the way Content Distribution Networks (CDN) work today. With this approach we maintain a replica of the data available at each site and keep their replicas in sync.

The primary site approach is usually the simpler of the two, as it doesn't involve data movement.  That said, it may open latency and security challenges, as it works on the assumption that every public site needs to have complete network access to the data at the private site, and at a latency that is reasonable. There are still many cases where these assumptions don’t apply. For these cases, a federated deployment approach is the only option.

With the federated deployment approach, we need to keep the data replicated between the private and public site…but how do we do that?

1. First we need to create a separate private data cloud and public compute cloud.  When we’re dealing with vast amounts of information, we simply can't burst data with the load, since depending on the amount of data involved, this process alone can take hours. A more practical approach would be to create a separate private data cloud in which each data cloud is physically located close to the location of the public compute cloud data center. In this way, we can have better control over the network latency when we start to burst loads, just like a CDN, only in this case we apply it to transactional information and not just static files.

2. Then, we launch the data cloud on demand. Even though, in most cases it’s not practical to burst data with the load, it is still a requirement to be able to launch new data clouds without any downtime or impact on the existing data clouds. The challenge of doing that over transactional data is huge, as we not only need to replicate the data to the new site but do so while the data keeps on changing as the new site is launched.

Example: Bursting Workload & Data between AWS and Rackspace/OpenStack

The following example demonstrate a simple web application with Global load-balancer (Rackspace load balancer), and Web Application (Petclinic) based on Tomcat as the Web front end and MySQL as the database.

On both ends we used GigaSpaces XAP Transactional WAN replication as a replication channel between the two instances of MySQL and Cloudify to handle the workload migration between the sites.

You can read the full detailes including code references of this examaple on Dotan Hortvitz post Bursting into the Clouds – Experimenting with Cloud Bursting 

Final notes

Cloud bursting comes with compelling economic benefits that will enable us to get the best out of both the private and public cloud worlds. In reality, the challenges involved with creating a true cloud bursting environment has kept it in the realm of a theoretical model for the most part until now.

The availability of OpenStack (and to a lesser degree EC2) in both private and public environments can simplify the process of building cloud bursting environments, as now we can have the same underlying data center infrastructure for both environments. 

To run real transactional workloads over a cloud bursting environment, we need to address other challenges such as consistent configuration and setup of our application stack, and the hardest part is managing consistent data across sites.

Bare-Metal PaaS

The Rise of Bare-Metal Clouds

Cloud and Virtualization are not mandatory, and the number of cloud providers that supports bare-metal clouds is growing, as David Linthicum pointed out in his article Going native: The move to bare-metal cloud services

It's a fact that virtualization is not a requirement when creating cloud computing services, but it is helpful to those who manage the service. Indeed, Google is able to provide a multitenant cloud computing platform without virtualization; there are other examples as well.

Internap, SoftLayer, Rackspace, Liquid Web, and New Servers (also known as BareMetalCloud.com) also provide access to the bare metal. You can count on more providers to join the fray as cloud computing users continue to demand that their managed hosting environments work like their native environments.

Performance Matters

The main reason for the rise of bare-metal clouds is that virtualization often comes with a cost. There is a  large class of applications where the virtulization overhead is unacceptable, as Carl Brooks, a cloud analyst at Tier 1, a division of 451 Research, points out here:

By having dedicated servers that are not virtualized, and therefore do not have a hypervisor layer, users can experience an uptick in speed, Brooks says, in some cases as much as 10% depending on the application. This can be an attractive option for high-performance compute needs, advanced Web 2.0 developers, or applications that require a large amount of database resources. Basically any situation "where the performance matters the most" 

Come to think of it, the main thing that brought us to virtualization in the first place was the ability to create new servers on demand, as opposed to it taking days or weeks with dedicated servers. Now, with the avliability of bare-metal clouds, it is possible to get the same level of flexibility without the virtualization overhead, and with more control and flexibiltiy on the specific HW configuration and setup. This makes the choice of bare-metal clouds much more attractive than in the past. 

Bare-Metal PaaS

Now that bare-metal clouds have become more popular, it only makes sense to have bare-metal PaaS to support them. Bare-metal PaaS provides the abilty to manage an elastic application workload without relaying on virtualization, providing the option to support high-performance workloads with the simplicity of PaaS.

This is particularly interesting for Big Data and any data intensive application that is heavy on I/O. Quite often, running these types of applications on top of virtualization isn't even considered a viable option. With bare-metal PaaS we can easily serve I/O intensive workload without compromising on performance or latency.

A Bare-Metal PaaS Example with Cloudify

Cloudify is an open PaaS stack. With Cloudify, we look at cloud infrastructure as just a bunch of compute resources with an operating system, IP address, SSH port. We use a generic Cloud-Driver that enables us to abstract the way those machines can be allocated and created on each different clouds infrastructure. In that context we treat a virtual machine as just generic container of an operating system. Cloudify manages applications at the process level. This means it manages the application workload by provisioning the application processes across machines.

With that, it only made sense to look at bare-metal clouds as any other virtualized cloud. So we ended up writing a bare-metal cloud driver, which is referred to as the Bring Your Own Node (BYON) cloud driver.

A BYON cloud is set simply by specifying a list or range of IP addresses as our cloud pool. Cloudify then uses this pool to provision the application and its services. Below is a simple example that shows what this configuration looks like.

templates ([
	SMALL_LINUX : template{

rumus togel singapura 2 angka
nilai tukar dolar rupiah
masjid hajjah fatimah singapura
Ted hadn t been able to track him down 69 Lietha Wards to find out if the rumors were true. Your place is with me, by my side as my wife.
asatizah
vivanews indonesia vs singapura
nomor togel jitu singapura
piscina infinita hotel singapura
kode negara xl
cat size comparison
komunitas orang indonesia di singapura
majulah singapura song download
majulah singapura guitar chords
persahabatan malaysia vs singapura
majulah singapura music sheet
plaza starhub customer service
cara bekerja di singapura
plaza singapura starhub number
singapura cat breeders southern california
perkhidmatan teksi johor bahru ke singapura
crystal jade steamboat buffet plaza
	custom ([
		"nodesList" : ([
				([
				"id" : "vm1",
				"ip" : "10.10.10.1"
				]),
				([
				"id" : "vm2",
				"ip" : "10.10.10.2"
				]),
				([
				"id" : "vm3",
				"ip" : "10.10.10.3"
				])
			])
		])
	}
])

A list of more advanced configuration options is avliable here.

Keeping Available the Choice Between Bare-Metal and Virtualized Cloud 

In reality, there are cases where the use of virtualized cloud instances makes sense, and as already discussed here, there are also cases where using a bare-metal cloud would be the right option. Ideally it would be best to abstract the application from those choices in order to easily move between the two environments, or even combine them. For example, imagine the case of a typical web application. It is very likely that we would want to run our web traffic workload on virtualized instances and our database on bare-metal instances.

In the case of Cloudify -- keeping the details of the specific cloud target at the CloudDriver enables us to achieve just that. We can define our application as a recipe, which is kept unchanged between the cloud choices. The details of whether to use a bare-metal or virtualized cloud is left to the deployment stage where all that is needed is to point Cloudify to the target cloud driver, and that's it.

References

• Going native: The move to bare-metal cloud services
• New bare metal cloud offerings emerging
• How much overhead does x86/x64 virtualization have?
• Amazon EC2 versus Bare Metal and KVM? The inside story on what you thought you knew about EC2

 

Related articles

• Mapping the Cloud/PaaS Stack (natishalom.typepad.com)
• GigaSpaces Cloudify & VMware CloudFoundry the new PaaS Jailbreaker (natishalom.typepad.com)
• PaaS is the New Middleware (redmonk.com)
• Big Data In the Cloud Using Cloudify (natishalom.typepad.com)

Mapping the Cloud/PaaS Stack

Cloud is evolving at an incredible pace, changing almost every aspect of our industry. It’s only natural then, that with this change also comes a continuous evolution of how we classify and categorize the various components of the cloud stack.

In earlier days, it was enough to draw the lines between IaaS, PaaS, and SaaS, and while this categorization is still very handy, we are beginning to see the lines between these categories start to blur. On the one hand, we see more IaaS providers starting to move up the stack, and on the other hand, we are seeing new categories such as DevOps gaining popularity and bridging the gaps between the IaaS and the PaaS layers.

 

I was recently asked to draw a map of how I see the current cloud stack. This ended up being an interesting exercise. To make things simpler I decided to focus on the IaaS/PaaS piece, both because this is my area of expertise, and also because IMO this is where many interesting developments are taking place. I ended up with the diagram below, which I thought would be worth sharing in order to get feedback and different perspectives.

It’s important to note, that this diagram is not based on any scientific scoring of different products in the market. I'll leave that part to the analysts. The specific products I chose to use are only there to illustrate and support my analysis of the components in the stack and where they map to within each category.

What's Driving the Cloud Stack?

The two main (and often conflicting) driving forces behind the evolution of the cloud stack are Productivity and Control. As cloud started to emerge it was productivity that took center stage, often by giving up almost completely on the level of control. As the industry started to mature we started to see an evolution where we could have various degrees of tradeoffs between these two driving forces. A good indicator of this, is the rise of DevOps and Private Clouds. We could also see the new open source movement as yet another major step in that direction - the most notable one is OpenStack, and the recent Citrix announcement about the moving of Cloud Stack into the Apache Foundation.

We could also see similar movement happening on the PaaS layer, where VMware launched Cloud Foundry as both a self-service PaaS for development productivity, and as an open source project to provide users with more control. Cloudify takes similar yet more explicit moves in that direction by bringing DevOps and PaaS together, in addition to making it open source.

 

Mapping the PaaS Stack

Geva Perry wrote a good summary mapping the current Java PaaS players in his post Listing the Java PaaS Public and Private Clouds . Geva chose to classify the different players based on their private/public cloud position and language support (Java in this particular case). Obviously the correlation behind public/private cloud and productivity/control is fairly close, so I can see where he was coming from. While this makes a lot of sense, I believe that it’s not accurate enough as there are still solutions that work on both private and public clouds and are not limited to just Java - Cloudify included. So is there a better way to classify the PaaS providers?

Here is my suggestion:

I would classify the PaaS providers based on those who focus on delivering PaaS through a self-service portal I’ll refer to them as PaaS service providers, where they assume full control over the underlying infrastructure and own the entire operation of your application - this is where I would put GAE, Heroku etc. Next will be  those who gives you the tools to build your own PaaS on private or public clouds, I’ll refer to them as PaaS stack providers - this is where Cloudify and I believe, Cloud Foundry and Stackato also fit in.

The second sub-category of classification would be on the level of openness and control i.e. open source, language support, cloud environment support, DevOps support and such.

In any case, we can easily get into a complex matrix of features/mapping for even more fine-grained classification but this far beyond the scope of this post which is merely drawing a rough map rather than providing a complete competitive analysis.

How would you categorize the Cloud/PaaS Stack?

As I noted earlier, this is just my own personal view on the subject. I hope that this will spark an interesting debate in which we may either come out smarter, or possibly even more confused. :)

Cloudily explained

plaza starhub contact number

singapura cats sale price

majulah singapura song free download

the times singapura-kuala lumpur mp3

amore fitness plaza singapura

surat kiriman rasmi format singapura

togel online singapura hongkong

prediksi togel singapura 2d

pengalaman mencari kerja di singapura

tabel nomor togel singapura

dolar terhadap rupiah

prediksi indotogel singapura hari ini

hasil pengeluaran togel singapura hari ini

harga tukar dollar

perbandingan pendidikan di
surat kiriman rasmi format singapura

singapura kittens for sale australia

sejarah perpisahan singapura dan malaysia

lagu kebangsaan singapura adalah

A simple whiteboard explanation on how Cloudify works

Big Data In the Cloud Using Cloudify

Edd Dumbill wrote an interesting article on O’Reilly Radar covering the current solutions for running Big Data in the Cloud

Big data and cloud technology go hand-in-hand. Big data needs clusters of servers for processing, which clouds can readily provide.

Edd touched briefly on the role of PaaS for delivering Big Data applications in the cloud

what is the average price of a singapura cat

plaza cinema seats

ramalan jitu togel singapura

rumus togel singapura 2 angka

hotel sands sky park em singapura

tres joli singapura cattery

telepon murah simpati ke

indotogel result

waktu sholat subuh singapura

You don t know her, she doesn t give up. He managed to guide her back lying down on the bed and was glad that she didn t hesitate when he urged her back in next to him.

terkini malaysia vs singapura

plaza orchard map
golden village plaza singapura contact number

singapura kittens for sale prices

kerja kosong di singapura

pengeluaran togel singapura thn 2012

small cat breeds

prediksi togel singapura hari ini 2011
paket liburan ke singapura desember 2011
resepi mee hoon goreng singapura

She set upon her task, scrubbing each item on the washboard before rinsing it in the river and wringing it by hand.

lowongan kerja di singapura februari 2011

berita harian singapura rgcx

smiggles singapore plaza singapura

nilai tukar rupiah ke dollar singapura

urut zakar di singapura

majulah singapura download mp3

singapura cat breeder price

golden village cinema plaza singapura

prediksi togel singapura rabu 27 juni 2012

tiket bola sepak malaysia lawan singapura
cari tiket pesawat murah ke singapura

sejarah bahasa indonesia

plaza singapura starhub customer service

angka ikut togel singapura hari ini

perancangan keluarga di singapura

prediksi togel singapura rabu 6 juni 2012

Why do you not give it a go? he suggested mildly, rekindling her ire. No, I guess it s not, but like my father always said, It s a nice place to visit, but I wouldn t want to live there .

tukaran wang malaysia ke singapura

prediksi togel singapura besok

Beyond IaaS, several cloud services provide application layer support for big data work. Sometimes referred to as managed solutions, or platform as a service (PaaS), these services remove the need to configure or scale things such as databases or MapReduce, reducing your workload and maintenance burden. Additionally, PaaS providers can realize great efficiencies by hosting at the application level, and pass those savings on to the customer.

Even though Edd’s article covers all the different forms of running Big Data on private and public clouds, the article focuses mainly on the public cloud offering from Amazon, Microsoft and Google.

In this post, I wanted to cover more specifically how I see the evolution of cloud application platforms (PaaS) to support Big Data. I’ll refer specifically to Cloudify which was designed primarily to support Big Data applications. 

 

 

Big Data in the cloud using Cloudify

Background

Most of the PaaS solutions out there started by focusing on simple web application deployments on Ruby, Java and Node.js. Unlike other PaaS solutions, when we designed Cloudify we picked Big Data as the primary target for Cloudify, and started by supporting popular NoSQL clusters such as Cassandra and MongoDB, as well as providing the equivalent of Amazon RDS by providing recipes for MySQL. Our goal was to make Big Data deployments a first class citizen within Cloudify. To this end,when you download Cloudify you'll notice that ALL of our examples comes with pre-integrated Big Data deployments.

There are couple of reason that brought us to make that decision: 

• Managing large data clusters is a core expertise at GigaSpaces

Most people know GigaSpaces for our In-Memory Data Grid solution known as XAP (eXtreme Application Platform). Over the past 10 years, as our customer deployments grew substantially, we realized that developing strong automation and cluster management is as critical as handling data consistency, performance, and latency in of our data-grid product. In a large cluster, if something breaks it’s going to literally be impossible to handle that failure through manual procedures.  For that reason, we developed lots of IP around automation of data cluster deployment which resulted in a unique self-managed data cluster.

• Cloudify is a natural evolution of GigaSpaces Data Cluster

When we built Cloudify it made a lot of sense to take the IP that we developed for managing GigaSpaces cluster and simply generalize it so it would fit with any other framework. In this way, we could leverage the years of experience  as well as development in this area, and gain a significant head-start.

• Big Data applications are complex

Big Data applications tend to be fairly complex, which makes them an ideal candidate for the sort of automation and management that Cloudify can offer.

• Big Data applications have a lot in common with XAP applications

Both need automation of data, failover and recovery, both fit into large cluster deployments, and both share similar partitioning and other clustering architecture. 

What makes Big Data platform different than any other application

Most of the existing orchestration systems were designed to handle stateless processes. Moving data is a completely different ballgame as you need to think of:

• Primary and Backup dependency 
• Availability - moving data without losing it.
• Moving processes to the data rather than the other way around.
• Data replication within and across sites 

Automating any of these processes through general orchestration tooling like Chef or Rightscale can become a fairly involved and complex process, with lots of pitfalls with handling edge scenarios for example the handshake process that is often involved when automating a data node failure, including a split-brain scenario.

In Cloudify we were able to curve out lots of that logic from the user, for example Cloudify will automatically ensure that primaries and backups won’t run on the same node or data center in case of disaster recovery. You don't need to do anything but tag your cluster nodes with a zone-tag. 

Managing Big Data applications != Managing Big Data storage

Managing data clusters is one thing. Being able to process the data is yet another challenge that we need to think about when we’re dealing with application platforms as I noted in one of my earlier post.

The main challenge is that quite often the management of the data processing logic is built on completely different scaling, availability and monitoring tools than the one used for managing our Big Data deployment. It turns out, that this silo thinking leads to a whole set of complexities starting from the inconsistency in having multiple managers, each determined in a different way when there is a failure or scaling event, and that quite often end up conflicting with one another. Having lots of moving parts is yet another challenge that makes the entire deployment pretty much a complete mess.

Built-in support for In Memory Stream based processing

In an earlier post, 5 Big Data predictions for 2012, Edd pointed out that streaming data processing is going to be one of the main trends for Big Data. 

Over the next few years, we'll see the adoption of scalable frameworks and platforms for handling streaming, or near real-time analysis and processing. In the same way that Hadoop has been borne out of large-scale web applications, these platforms will be driven by the needs of large-scale location-aware mobile, social and sensor use.

Being also part of XAP, Cloudify already comes with built-in support for streaming Big Data processing. This means that building your own Facebook or Twitter-like real-time analytics can be as a simple as writing only small scripts that handle the analytics counters. All the rest, i.e. scalability, availability, automation, cloud portability, management and monitoring, is covered by Cloudify as noted in this and this use case. 

Examples for Big Data applications running on Cloudify 

In the list below, I tried to put together a couple of references and examples that will make it easier for you to get started. The first reference points to a simpler scenario that will allow you to use Cloudify to deploy your Big Data database as a service. The other three references are full-application stack deployments that include the data-processing and web-tier of applications managed together with the Big Data database. 

Running a Big Data 'Database as a Service'

Cloudify comes with built-in recipes for Cassandra and MongoDB, as well as Solr (popular search engine), which makes it easy to deploy these database clusters on your local machine, data center or private/public cloud through a single command. In this way, you can use Cloudify to automate the database. 

Spring Travel application with Cassandra

Demonstrating a deployment of a JAVA-based commerce application with Cassandra as the database

The example includes recipes that provision the Cassandra database, create a schema, load the data, then spawn a Tomcat container, automatically injecting the reference to Cassandra, that then shows the custom management and monitoring of the application - all through a single command. A recent videocast showing how the travel application works on HP OpenStack cloud is available here.

Pet Clinic example with MongoDB

The Pet Clinic example does pretty much the same thing only using a sharded MongoDB cluster.

Twitter Real Time analytics example for Big Data 

The Twitter example  shows how you can attach real time stream-based processing for handling real live Twitter feeds, and how you can manage both the stream processing cluster and the Big Data cluster using Cloudify and run it on any cloud. The entire source code for this example is provided on Github.

Give it a try

To try out any of the examples you'll need to download Cloudify (latest) or (stable) build. Cloudify comes with all first three examples as part of the distribution under the recipes or examples directory. To try out these examples simply follow the steps from the Cloudify Quick Start Guide.  

 References:

• Big Data in the Cloud
• 5 Big Data predictions for 2012
• Big Data Application Platform
• Analytics for Big Data – Venturing with the Twitter Use Case
• Cloudify Download  (latest), (stable) 

Lessons from Zynga & Sony on moving from Amazon AWS

 

Earlier this month Zynga announced its move from Amazon AWS to its own private Z-Cloud. Sony also started to move increasing parts of its workload from Amazon to Rackspace OpenStack.

There isn't so much in common between these different use cases, except for the fact that they may indicate the beginning of a trend (I’ll get back to that toward the end) where companies start to take more control over their cloud infrastructure.

So what really brought Zynga and Sony to make such a move?

Zynga moves from Amazon to their own private cloud

Tanya had never spoken to her like that before, Linda yes, but not Tanya. Elsa who was involved in her textbook, looked up to see what the rising voices were all about their conversation no registering on her, What is with you two?

golden village plaza singapura showtimes

prediksi toto togel singapura hari ini

perlawanan bola sepak malaysia lawan

gaji bekerja di singapura

perubatan islam di singapura

majulah singapura song.mp3

berita harian singapura rgcx

lowongan kerja di singapura 2011

plaza singapura contact us

The truth was he did actually have some muscle, and she did enjoy touching him.

singapura ou cingapura onde fica

kimage salon singapore- plaza singapura

prediksi togel singapura hari ini 2011

liburan ke singapura lewat batam

dasar negara dan konstitusi singapura
Fortunately for her the small desk lamp gave her some perception of the room.
sejarah singapura keluar dari malaysia

indotogel singapura-togel hongkong-togel macau

musee platinum tokyo singapore plaza singapura

singapura cattery north carolina

sistem pemerintahan di negara

what is the average price of a cat

rumus togel jitu singapura

singapura breeders northern california

musee singapore plaza singapura

plaza singapura starhub contact number

dolar singapura ke rupiah

nomor togel jitu

Especially since that night was your first date, and he only had eyes for you. He sat next to Megan, keeping a safe distance from her.
perlini bags plaza singapura

liburan murah ke singapura 2011

cara kerja di singapura

resep tumis buncis ala singapura

indonesia di mata orang singapura

Zynga ran their gaming services on Amazon for a while. It was noted that the cost of running these games on Amazon cost Zynga $63M annually. This cost and the continuous increase of their workload forced Zynga’s management to look for ways to control their cloud operational costs. Zynga realized that at that scale, they are just better off building their own cloud infrastructure. By doing so, they could also optimize the cloud infrastructure for their own needs and reduce the operational cost margins substantially as a result.

According to Zynga their private cloud operation is reported to increase utilization by 3x, which means that they will need 1/3 of the servers that they would need from Amazon for the same workload, as noted by CTO Allan Leinwand on Zynga’s engineering blog:

For social games specifically, zCloud offers 3x the efficiency of standard public cloud infrastructure. For example, where our games in the public cloud would require three physical servers, zCloud only uses one. We worked on provisioning and automation tools to make zCloud even faster and easier to set up. We’ve optimized storage operations and networking throughput for social gaming. Systems that took us days to set up instead took minutes. zCloud became a sports car that’s finely tuned for games.

What's interesting in this case, is the cost analysis. Quite often when we measure the cost of running machines, we don't measure what would be the cost of running a particular workload, which is a combination of many factors, not just the cost of servers. 

The other thing that is interesting with Zynga’s move is that they didn't make the move just to cut costs. The move was part of a more strategic direction to create a gaming platform for additional games. Apparently when your service becomes a platform, controlling your infrastructure becomes more strategic than just the cost factor. It’s a big differentiator that can put Zynga in a completely different ball game than its competitors; it will also enable them to have better control over the dependency on Facebook and build their own independent ecosystem.

According to Dave Wehner, CFO at Zynga, the company will lower its cost of revenue over the next 18 to 24 months as third-party hosting costs decrease. Wehner said that Zynga plans to “roll off the majority of those third-party hosting arrangements.”
Zynga’s capital expenditures in the fourth quarter were $50 million, down from $63 million in the third quarter. Most of that spending is focused on zCloud. For 2011, capital investments were $238 million.

The building of its own infrastructure will help the bottom line. Zynga can depreciate its gear and lower quarterly expenses. “We believe this investment will have a short payback period and enable us to expand gross margins in the long term,” said Wehner

The important thing to note is that Zynga’s move wasn't because of Amazon’s failure as it first reads. It’s more of a natural evolution and maturity cycle of the company.

Sony move from Amazon to Rackspace/OpenStack

Sony’s gaming arm faced a number of security breaches last year, as reported by NetworkWorld, which compromised the personal identity information of millions of players within Sony’s gaming network. This event forced Sony to look for ways to have better control of their infrastructure exposure. Their approach was to move some of their workload to Rackspaces’ OpenStack.

By splitting their operations between Amazon and Racksapce they are able to have better control of a particular cloud failure. They are also better positioned to control their cloud infrastructure costs by reducing their dependency on a particular cloud provider, thus being in a stronger position to negotiate their cloud operational costs.

My take:

We’re often taught that infrastructure is a commodity that we shouldn't care much about, and should essentially outsource everything. It turns out, that as the cost of infrastructure get bigger, and as our needs become more unique, controlling the infrastructure becomes a critical part of our business. Controlling your infrastructure could mean having your own private cloud as is the case with Zynga, or minimizing your dependency on a particular cloud provider as is the case with Sony.  The good news is there are enough free OSS tools out there such as Chef, Puppet, and Cloudify that can help to reduce the complexity that is often involved with such a move.

References:

• Zynga's play platform a big test for its cloud computing prowess
• New problem for Amazon: Sony defects to OpenStack
• Public vs Private cloud - its not about the cost.
• Five miss conceptions on cloud portability
• Making cloud portability a practical reality
• Cloudify community site

 

Top Myths of Moving Mission-Critical Applications to the Cloud

Top Myths of Moving Mission-Critical Applications to the Cloud

Special Guest Post By Nati Shalom, at CloudNewsDaily.com

Research conducted by HP found that the majority of businesses in the EMEA region are planning to move their mission-critical apps to the cloud. Of the 940 respondents, 80 per cent revealed plans to move mission-critical apps at some point over the next two to five years. This is a fairly dramatic shift in how most companies and enterprises in particular plans to embrace the Cloud.

This would be a good time for a reality check, to look at some of the predominant myths about cloud adoption, and what we should change as we start to embrace mission critical applications in the cloud.

 

Myth 1: The cloud is a silver bullet solution

We have all borne witness to failures with Amazon and other major cloud providers over the past few years.  Because of this, we need to realize that in the cloud, failure is prone to happen even more often than with traditional enterprise IT. This instability is built into cloud economics – to move mission-critical application into the cloud we need to cope with these failures, rather than simply try to prevent them from happening. Mission-critical apps are often built to survive failure, but the fail-over processes assumes manual intervention and static configuration.

Reality Check: To move mission-critical applications to the cloud we need to completely automate the fail-over and recovery processes.

 

Myth 2: The cloud cuts costs

Many people think that cloud economics starts to pay dividends immediately when you move to an on-demand usage model, paying only for what you use. Cloud can actually be fairly expensive when hosting environments in use are not elastic. I was surprised to see how many startups and SaaS organizations still run their applications in the cloud just as they would in any static hosting environment. While the pay-per-use model has a lot of promise for cost-savings if our applications aren’t designed for elasticity the cost of running the application in the cloud may end up costing you more. Most of the mission-critical applications have not been designed for elasticity and on-demand usage.

Reality Check: To benefit from the overall cloud economics, with mission-critical application we need to provide more flexibility in the way we employ elasticity. For example, for applications that were not designed for scale-out we could automate the move from small instances to large instances – this is basically an elastic scale-up model. We can also embrace hybrid models where most of our resources are statically provisioned and we can employ elasticity only to part of our resources that are easy to scale – such as the web tier.

 

Myth 3: The cloud is simple

When cloud came out it was touted as a much simpler model for running IT. The common example given was usually the time it takes to get a new machine into our data center. In traditional enterprises this could take weeks, where in cloud getting a new machine can be done in a matter of minutes.  However, as IT becomes more efficient, this is no longer the case.

If we compare the process of running applications within our IT with the way we run applications in the cloud, we will see that the time it takes to bring in a new machine is only small part of the actual process of running and managing the application. Quite often this process is actually much more complex in the cloud. This is mostly due to fact that cloud is more dynamic in nature, and there is a lack of tools that can provide the level of control that is expected when dealing with mission-critical applications.

Additionally, most of the existing mission-critical applications come with the requirement for a high degree of management and control. However, a lot of the management and monitoring tools involve a high degree of manual configuration just to get the required monitoring capabilities in place, and manual processes to get a new version or feature of the application into production.

Reality Check: To move mission-critical application into the cloud all these processes needs to be completely automated from the development stage up to the production.

 

Myth 4: The cloud is inherently scalable

While cloud can scale to 100’s of machines via an API, the truth is that most applications were not designed to take advantage of this approach.  So while cloud itself can be fairly scalable the application running on the cloud doesn’t become anymore scalable by simply running in that environment, unless they were designed for scaling from the onset.

Most of the existing mission-critical applications were not designed for cloud scaling. People believe to fully benefit from cloud scaling often involves a complete re-write of the application, and most enterprises just aren’t ready to make such as major shift.

Reality Check: To move mission critical application to the cloud we need to do this in baby steps, starting with only automation and repackaging of the apps, without any code changes.  Later steps would involve changing of modules within the application, as needed.

 

Myth 5: Cloud Bursting

Cloud bursting is an economical model that enables you to combine statically provisioned resources for the part of your application that needs 24/7 utilization and use on-demand resources on the cloud only in cases where you’re running out of resources. In this model, you can design your cost at the optimum level between fixed and on-demand cost.

Many cloud providers offer network connectivity (a.k.a VPC) between private and public cloud resources to enable cloud-bursting and hybrid clouds. Having said that, many applications (mission-critical application in particular) tend to be data sensitive.

Reality Check: To implement cloud-bursting not enough to have network connectivity, you also need to be able to move data not just processes. Content delivery networks (CDN) now make it possible to bring static data closer to the processes. For mission critical applications the same is needed for transactional and analytics data – without it cloud-bursting may remain an interesting theoretical model.

 

Final Thoughts

As the industry becomes more mature and ready to move mission critical apps to the cloud, we need to manage our expectations from the cloud infrastructure.  It’s important to realize that while cloud holds a lot of promise for cost savings, scaling and more, how we run our application and business in the cloud remains our sole responsibility. In this post I tried to outline more specifically the top misconceptions we need to make clear in order to start moving mission-critical applications into the cloud.

I also believe that Hybrid cloud is the way to go – not just for legacy applications but also for new applications. For this to happen we need to think of how do we make cloud-portability a practical reality, as well as how do we practically migrate mission critical applications that were not built for a dynamic environment into the cloud without forcing a complete re-write. Based on the experience with many of the DevOps automation tools over the past years I believe that we can gain a lot by starting with automation of deployment, fail-over and scaling processes as well as automation of the management and monitoring of our applications.

With Cloudify we took many of those lessons and packaged them into a product that makes the process of moving mission-critical applications into the any cloud vastly simpler. Our experience on that regard has proven to be tremendously successful as we proved to be able to take real life telco application that wasn’t written to cloud and migrate it into the cloud in matter of 4 weeks without any code change.

 

References

• Companies nearing mission critical apps in the cloud
• Making cloud portability a practical reality
• Cloudify community site

GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker

I was reading Krishnan Subramanian's post, Two Events That “Clouded” Our Thinking In 2011. The thing that caught my attention was Krishnan's comments on why PaaS is a superior alternative to DevOps: 

The shift in the thinking about the enterprise cloud consumption also poured water into the “DevOps” concept advocated by vendors and pundits with their foot in the IaaS world. When organizations embrace PaaS instead of infrastructure services, we don’t need the DevOps marriage and the associated cultural change (believe me, this cultural change is giving sleepless nights to many IT managers and some consultants are even making money helping organizations realize this cultural change). With PaaS, organizations can keep the existing distinction between the Ops and Dev teams without worrying about the cultural change. In fact, with cloud computing, the role of the Ops is not going away but it stays in the background offering an interface which developers can manage themselves.

Krishnan represents one of the common attitudes and subjects of debate between two main paradigms for developing and managing applications on the cloud:

• PaaS -- PaaS takes a developer, application-driven approach. A PaaS platform provides generic application containers to run your code. The PaaS platforms deals with all the operational aspects needed to run your code such as deployment, scaling, fail-over, etc.
• DevOps -- DevOps takes a more operations-driven approach. With DevOps, you get tools to automate your operational environment through scripts and recipes, and keep full visability and control over the underlying infrastructure.

The Difference Between PaaS and DevOps

Both PaaS and DevOps aim toward the same goal -- reducing the complexity of managing and deploying applications on the cloud. But they take a fairly different approach to deliver on that promise.

Christoper Knee summarized it in his blog post DevOps and PaaS -- Friend or Foe? as the difference between Developers and SysAdmin:

• Developers may ask: "if I have a self-service portal for deploying applications (aka PaaS), do I need SysAdmins at all?"
• SysAdmins may ask: "isn't PaaS just a monstrous black box that prevents me from provisioning the specific services we need to deploy real-world apps?" ... The typical SysAdmin thinks that they can get to 75% of PaaS functionality with DevOps tools like Chef without giving up any systems architecture flexibility.

In one of my earlier post on the subject, Productivity vs. Control tradeoffs in PaaS, I tried to outline the main limitation of most of the current "blackbox PaaS" implementations:

I thought that Carlos Ble's post Goodbye Google App Engine (GAE) is a good example that illustrates why the initial perception behind GAE as a simple platform that provides extreme productivity can be completely wrong.

...developing on GAE introduced such design complexity that working around it pushes us 5 months behind schedule.

Part of the reason that brought Carlos through that experience IMO is that in the course of trying to make GAE extremely productive the owner made the platform too opinionated, to the point where you lose all the potential productivity gains by trying to adopt their model. In addition, with a platform like GAE you have very little freedom to leverage existing frameworks such as your own database, or messaging system, or any other third-party service that can in itself be a huge contributor to productivity.

Instead, you're completely dependent on the platform provider's stack and pace of development, and that in itself can work against agility and productivity in yet another dimension. In this specific example, Carlos couldn’t use a specific version of a Python library that would have made his productivity higher, and instead had to work around issues that were already solved elsewhere. This is a good example how the lack of flexibility leads to poorer productivity even in the case of simple applications.

Putting DevOps & PaaS togatehr 

It looks like more people in the the industry have come to recognize that rather than looking at DevOps and PaaS as two competing paradigms, it might be best to combine the two, as Christoper Knee pointed out in his post:

 

What if you could get a PaaS that wasn't a black box, enabling developers to deploy apps easily while still giving SysAdmins the ability to provision any services they needed (a la Cloud Foundry)?

I also came to that realization myself, as I pointed out in my  2012 predictions: 

In 2012, we’ll see many of the DevOps tools, such as Chef and Puppet, integrated into application platforms, making it easier to deploy complex applications onto the cloud. In the same way, we’re going to see more Application Platforms adopting the automation and recipe model from the DevOps world into the application platform. The latter have the potential to transform the opinionated PaaS offerings as we know them today, with Heroku and GAE leading that trend, into a more open PaaS offering that better fits the way users develop apps today, and provide more freedom to choose your own stack, cloud, and application blueprint.

What Makes a Cloudify and CloudFoundary a PaaS Jailbreaker?

The CIO Maginze article Cloud computing disrupts the vendor landscape defines a new class of PaaS platforms that I think represent the definition for a DevOps PaaS:

A PaaS that allows developers to use whatever tool they want to build their cloud applications and the platform tackles the deployment, scaling and management of these apps in the cloud data center.

VMware CloudFoundry is one of the more notable references in that category. Quoting Christoper Knee:

CloudFoundry runs anywhere, incuding on your laptop. CloudFoundry's service container concept is particularly strong, kind of an appliance on steroids.

These ideas were the founding concept behind Cloudify, i.e. putting DevPops & PaaS togather in a single framework. As with CloudFoundry, Cloudify enables you to break away from the "blackbox PaaS". However, even though CloudFoundry is significantly more open than most other PaaS alternatives, at the core it is still based on the "my way or the highway" approach (aka opininated architecture), which forces you to fit into a specific blueprint that is mandated by the platform. Cloudify, on other hand, pushes the envelope even further by adopting the concpet of recipes that was first introduced by DevOps frameworks such as Chef and Puppet. It introduces  more application-driven recipes through the introduction of Domain Specific Language that extends upon the groovy language.

The Cloudify recipes give you the full power to plug in any application stack on any cloud (including a non-virtualized environment) and manage them in similar way to the way you would manage them in your own datacenter or machines. You can also call Chef and Puppet from within the recipes. All this, without hacking the framework itself. As with other similar DSLs, the Cloudify DSL was designed to express even the most complex application management tasks, such as <recovery from a data center failure> in a single line and avoid all the verbose script and API calls that are often involved when you work at the infrastructure level.

All this makes Cloudify an even more open alternative that fits a large variety of the current enterprise application stacks, including:

1. JEE applications
2. Big Data applications
3. Multi-tier applications
4. Native applications (C++,..)
5. .Net, Ruby, Python, PHP applications
6. Multi-site applications
7. Low-latency applications (that can't run on VMs)

It also makes Cloudify more open for special customization of the existing stack, like in the case of:

1. An application that needs a certain version of MySQL (not the one that comes with the framework)
2. An application that needs to run on Redhat (not Ubuntu). or even more interesting -- a case where there are mutiple applications, each needing a different OS served at the same time.

Cloudify also provides a more advanced level of control geared for mission-critical apps, including:

1. Monitoring the application stack and topology
2. Adding custom application metrics
3. Adding custom SLAs

It can also work on wide variety of cloud environments, including Microsoft Azure and non-virtual enviroments.

One of the great powers of the recipe is that it is a great collaboration tool. Once you develop a recipe, it is very easy to share it wtih different groups -- whether internal groups like developement, QA, and operations, where the recipie provides a programmatic definition of thier environment, or it can be shared between the product team and the pro-services team, where your sales and pro-services can easily install and update product versions in a consistent way, as well as reproduce customer scenarios and share them with the support team. Recipes are also a great tool for collaboration through a wider community network, where users can collaborate by sharing common recipes and best practices over the web.

Quick Introduction to Cloudify Recipes

Below is a short snippet that shows a simple application recipe, of a typical java-based web application based on JBoss as the web container and MySQL as a database. The application recipe describes the services that comprise the application, and thier dependencies. The details on how to run MySQL and JBoss is provided in a seperate recipe for each of the individual services. A more detailed description of how a service recipe would look like can be seen here. 

application {
       name="simple app"

       service {
              name = "mysql-service"
      }
       service {

              name = "jboss-service"
              dependsOn = ["mysql-service"]
       }
}

To get a taste, you can try one of the available recepes such as Cassandra, MongoDB, Tomcat, JBoss, Solar, etc., just as a simple way to try out these products on your own desktop or on any of the supported clouds, without the hassle that is often involved in doing so and without even direct relation to Cloudify per-se.

 

References

• Productivity vs. Control tradeoffs in PaaS
• DevOps and PaaS - Friend or Foe?
• PaaS shouldn’t be built in Silos
• Two Events That “Clouded” Our Thinking In 2011
• Cloud computing disrupts the vendor landscape
• Cloudify docs

Realtime Analytics for Big Data: A Facebook Case Study

via www.youtube.com

2012 Cloud, PaaS, NoSQL Predictions

2011 is coming to its end and now is a good time to start planning for 2012. I thought that a good start would be too look at my 2011 predictions and if my previous (and first) attempt to predict someting in that turbulent environment held any water...so, here is a quick recap of 2011.

Recap of 2011

Private vs. Public Cloud - As I noted in my recent post Public vs. Private Clouds I felt that during 2011 the debate around public vs. public cloud would become less interesting, as most of the industry has started to accept the fact that there is a need for both environments, and the important issue would become how to make them work well together. The most interesting development in that regard was Rackspace’s recent announcement about their plan to support OpenStack based private clouds, which shows that even public cloud providers have fully embraced this idea.

OpenStack is evolving from a movement into a viable reality - the momentum around OpenStack has gone through ups and downs throughout the year as happens with every new technology. However looking back, it appears that 2011 was a fairly successful year for OpenStack with its first public cloud available already in the market.  Dell and HP have started to offer the OpenStack based cloud to their customers, as has Citrix.  Rackspace announced their plan to provide official support including for those who want to build their own OpenStack environment…that's quite big considering the short timeframe from when the technology was first introduced…still there is a long way to go but the future looks promising - check out this survey in that regard.

PaaS adoption has been happening at a slower pace than expected, despite the fact that the trend remains consistent.  For PaaS startups 2011 was a fairly significant year with the acquisition of Heroku by Salesforce.  Amazon Redhat and VMware joined the PaaS arena; Amazon with Elastic Beanstalk, Redhat with their OpenShift initiative, VMware with CloudFoundry, adding to its previous acquisition of SpringSource vFabric. This was a fairly significant year for us at GigaSpaces as we launched a new product in this same domain that aims to completely change the way PaaS is being taught today (stay tuned…).

Google App Engine have no doubt been the disappointment of the year by literally killing GAE as we knew it (amongst many other things) with their new pricing model. 

Big Data has gone real time.  Facebook made a big announcement on how they moved their batch-oriented analytics system to real time analytics (See my previous posts on this subject here and here).  Twitter announced a launch of a new Real Time Analytics dashboard; while both join Google and Yahoo who have already started to make this shift.  Google has also been transforming their web analytics framework into real time. As I noted in my 2011 predictions, the entire debate around NoSQL and SQL didn't make sense, and indeed we’ve seen quite a few announcements both from Cassandra and Couchbase on their support for SQL-like query support.

In Memory Data Grids have also taken a similar approach where, with GigaSpaces, we’ve launched our JPA support, other Data Grid implementations such as Infinispan and Gemfire seems to be heading in that same direction each adding different levels of SQL support. The interesting development in this regard is that we were able to prove that you could actually mix and match Document/Schemaless APIs with SQL APIs and have the flexibility to choose the right language for the job (See online demo Same Data Any API).

All in all I think that I came fairly close - don't you think…?

Ok that gives me enough confidence to try the same thing for 2012. 

rumus togel jitu singapura

berita bola indonesia vs

download lagu majulah singapura

togel singapura prediksi besok

pemain bola sepak malaysia lawan singapura

siaran langsung perlawanan bola sepak malaysia vs singapura 2012

jalan kayu prata cafe plaza singapura

hukuman mati di singapura

golden village plaza singapura contact number

bagaimana mencari kerja di

video konser lady gaga di singapura

perencanaan transportasi di singapura
melayu murtad di singapura

2012 predictions

Cloud

iCloud everywhere - IMO the biggest shift in Cloud is the fact that it’s going to become pretty much invisible to many of the end users as new mobile devices, operating systems and applications start to be designed with cloud support in mind. Apple iCloud and DropBox mark the beginning of this trend. Using cloud for collaboration and synchronization is definitely a killer app for many of the consumer based apps. I expect that in 2012 we’re going to continue to see a big push of many SaaS-based offerings in that space toward rich client support that uses the cloud as a backend and leverages the power of the new generation of advanced mobile devices. The difference is that those clients won’t be just another frontend for the same web UI, but something that will run almost entirely on the mobile device and will use more generic cloud services for synchronization and collaboration. This will create the need for more generic cloud services such as database as a service and other middleware services that can interact directly from mobile applications.

Moving from Amazon-centric clouds to Cloud Mashups – In 2011 we started to see new kinds of clouds starting to pop up. Literally every hardware vendor (IBM, Dell, HP,..), telco (ATT, Verizon, KT), and software provider (Oracle, Microsoft) are either developing or already offer something in this space. Each one tries to maintain a unique position to compete with Amazon either through SLAs, locality, security, or being more open through the support of OpenStack. In 2012, this movement is going to become even stronger as many of the players that have been making the investment during 2011 will come out full speed ahead in 2012.

Microsoft finally gets it with Azure - Microsoft has been around for a while with Azure with somewhat marginal success mostly around its .NET user base, an approach that is too narrow a play when it comes to cloud.  Their cloud strategy is coming into focus with the offering of a more ubiquitous cloud supporting technologies that were previously unheard of on a MSFT cloud platform - such as Java, PHP and it wouldn't be too far to assume that they will be supporting Linux applications in the cloud as well. 

Cost-driven Application Management - One of the things that is still fairly hard to measure in the cloud is cost, and more specifically how each component of our application and architecture contribute to cost.  This is specifically true during current market conditions which are going to put even more pressure on cost savings. Cost-driven application design patterns will start to emerge, and will become an integral part of any design for cloud applications just as scalability and performance are today. A new form of Cost Driven Application Management (CDA) will start to emerge to provide better insight on how our application behaves from a cost analysis perspective - Newvem is a new startup in that space that already launched their private beta.

Mission Critical Apps move into the Cloud - As the industry matures there is no reason why we should draw the line for cloud adoption at simple apps. The challenge will be mostly around performance, latency, and ensuring continuous availability. A new class of middleware and application platforms that are designed specifically for cloud environments will become more popular to help in that transition. On the other hand, Java and JEE specifically will finally become more cloud ready as I noted in an earlier post - Java and the Center Stage.

Network Gets into Cloud API Stack - While compute and storage have become virtualized to fit into the cloud, we haven't seen much advancement on the network layer. Many of the networking providers are now launching APIs to enable better control over the cloud network. Alcatel recently announced an interesting cloud proposition in this domain specifically targeting telcos.  The idea is to use the network as a vehicle for making distributed data centers look like one big cloud, making it possible to better leverage existing assets and offer SLA driven compute resources based on latency, location etc. Other cloud providers are also starting to open their network APIs starting from the Load Balancer down to the core switch. This opens up a new set of opportunities for integrating these network APIs with the upper layer of the application stack.

More OpenStack Clouds - 2011 was the just the beginning of that trend, 2012 will see more public and private cloud providers offering support for OpenStack APIs with RackSpace, Dell, and HP already making public announcements in this area. The interesting question in this regard would be how Citrix will play out their CloudStack acquisition with its OpenStack strategy. 

 

PaaS

DevOps and PaaS Converge into App DevOps PaaS - One of the topics that drew a lot of my personal interest last year was the DevOps movement. For odd reasons, most of that movement was driven by Ops and less by Devs.  In 2012, we’ll see many of the DevOps tools such as Chef and Puppet integrated into application platforms making it easier to deploy complex applications onto the cloud. In the same way, we’re going to see more Application Platforms adopting the automation and recipe model from the DevOps world into the application platform. The latter have the potential to transform the opinionated PaaS offerings as we know them today, with Heroku and GAE leading that trend, into a more open PaaS offering that better fits into the way users develop apps today and giving more freedom to choose your own stack, cloud, and application blueprint.

Beyond Google App Engine, Heroku - Heroku established itself as a one of the early PaaS providers in the market and is now expanding their offering to Java. CloudFoundry, DotCloud and others are slightly different but still follow the same path.  In 2012 you should expect more choices for completely different PaaS platforms starting with JEE PaaS offerings from Redhat, IBM, and Oracle, to private PaaS offerings which essentially are frameworks to build your own PaaS, DevOps PaaS offerings (see note above), as well as vertical PaaS for specific industries. Magento announced their plan to provide PaaS for eComm and it wouldn’t be crazy to assume that others will follow that same path. 

BigData

Not only Hadoop Centric - During 2010 and to a lesser degree 2011 Big Data discussions were pretty much centered around Hadoop.  NoSQL solutions such as Cassandra and Mongo are gaining fast adoption mainly due to the operational and development complexity that comes with Hadoop. That movement is going to continue at an even greater pace, as Hadoop gets fragmented between many vendors and frameworks such as EMC, MapR, Cloudera, Yahoo, IBM each claiming to own their own Hadoop distro. With new funding in the hands of many of the NoSQL startups I’d expect to see more complete solution stacks targeted at Big Data.

In Memory Data-Grid and NoSQL Become Integrated - During the early days of the NoSQL movement it wasn't clear how the two technologies fit together. As I noted in my previous post here, it actually makes more sense to integrate the two technologies in a context of real time analytics for Big Data or real time data processing for Big Data.  Indeed during 2011 I started to see more case studies showing the use of the two technologies as with Facebook and Twitter. MemBase is also a good example for that approach with their announcment earlier this year about their integration of Memcached and CouchDB together into a single product. At GigaSpaces we added built-in integration for Cassandra and MongoDB, as noted here, and plan to invest more in that direction during 2012.

A New Class of Big Data Application Platforms will Address the Development and Operational Complexity of Big Data Applications - As Big Data application become more mainstream we start to hit the next level of complexity, development, and operational complexity. Clearly plugging NoSQL into your architecture may address your scalability requirements but at the same time it’s going to make your development and management experience more complex.  Not because the products themselves are complex, but mostly it’s because it is less obvious how to build and design the application around these new technologies. As in previous years, the goal of application platforms is to ease that task by putting together an integrated stack that makes it easier to develop Big Data applications as I noted in my post on Big Data Application Platforms.

References

• The Future of Cloud Computing: Industry Predictions for 2012 | Cloud Computing Journal
• NoSQL Consolidation: CouchOne and Membase Merge to Form Couchbase
• 10 Predictions About Cloud Computing
• 2011 Cloud, PaaS, NoSQL Predictions
• Public vs Private clouds (Again!)- it's not about the cost
• Introducing Magento Go and the Magento Go Developer Platform. The Next Evolution in eCommerce
• Real Time Analytics for Big Data: An Alternative Approach
• Big Data Application Platform
• Java and the Center Stage.
• Same Data Any API

Making Cloud Portability a Practical Reality

In one of my previous posts Five Misconceptions On Cloud Portability I argued that:

The term "cloud portability" is often considered a synonym for "Cloud API portability," which implies a series of misconceptions. If we break away from dogma, we can find that what we really looking for in cloud portability is Application portability between clouds which can be a vastly simpler requirement, as we can achieve application portability without settling on a common Cloud API. ..

The following presentation shows how I could use the ideas from this post and provide a practical cloud portability solution today using Cloudify and JClouds. 

 

 

via www.youtube.com

Public vs Private clouds (Again!)- it's not about the cost

I was reading Geva Perry's post, 10 Predictions about Cloud Computing  ..

 

As always I enjoyed reading Geva's throughts and for the most part comletely agree with him. The thing that caught my attention is Geva's point on private vs. public cloud. Geva re-iterated why the economical benefit of public cloud would turn private cloud into a small niche. Quoting Geva:

"Internal clouds will be niche. In the long-run, Internal Clouds (clouds operated in a company's own data centers, aka "private clouds") don't make sense. The economies of scale, specialization (an aspect of economies of scale, really) and outsourcing  benefits of public clouds are so overwhelming that it will not make sense for any one company to operate its own data centers."

Last week I had an interesting discussion with Yoav Abrahami from Wix which made me rethink this entire argument. Wix runs their own managed data center on one of the hosters. They were able to build a pretty agile environment, which enables them to update their software a few times a day and in this way continually grow their business in this model. If they want extra capacity it takes them a couple of hours to get the hardware that they need. They pay a monthly fee and they don't need to commit to three years in advance for thier resources.  The example of Wix is interesting as it doesn't fall exactly into the public cloud definition, yet they don't carry large part of a cost associated with  the operation of a data center which makes the economical argument somewhat less relevant.

 

This brings me to the first point:

What makes a cloud  private or public?

For most people, public cloud is synonymous with Amazon or Rackspace and private cloud is everything else.  Which basically means that if you don't pay by the hour and can spawn x1000 of machines by a call of an API your not considered public cloud - is that really the right place to draw the line?

I beg to differ - the main thing that brought us to cloud in the first palce is agility at a reasonble cost.  Agility in that context is our abiilty to launch new products, new features, and meet a growing demand continuously. Previously, it was the IT who stood in the way of the business and  Cloud (public & private) emerged as a significantly better way to run IT. Wix is one good example of what some would consider a private cloud, but at the same time they were able to serve the agility and cost requirements of  thier business.  The thought that the only way to meet our business agility and cost would be through public cloud is wrong and there are many examples like Wix that prove otherwise. This brings me to the next point...

Its not about cost but ownership, and privacy.

The point in my previous argument is that the cost of running your own managed data center today can be fairly low and even lower than running on some of the known public clouds because you can optimize for a specific use case and workload. BeaTune (a referece from my previous post) is another good example in this regard. They claim that since they switched to HQ8 they were able to cut their infrastructure costs in half. Here are the exact numbers:

Amazon: “two servers $380/month. Add roughly $220/month for backup snapshots and EBS for the (pretty large) database and your total yearly costs are $,7200”

EQ8: “two 24GB RAM, i7-920 Quad-Core, 2x1.5TB HDD (Software-RAID 1) servers for $3,530/year at a hosting company that you can actually call

Source: Goodbye Amazon EC2 See you later cloud

The explanation that I have for that, is that with public clouds you're limited in the type of compromises that you can make, and therefore carry a fairly large cost just to be able to live to the promise of meeting huge peak loads, a high degree of failures, and more. If you design for a specific work load you could come up with different structure that fits a potentially more relaxed set of assumptions that could fit your business better and can result in a lower cost and even better performance and SLA. The other part of the explanation is that there is more than one mean to gain from the economy of scale even in the case of private clouds such as leasing the resources of our infrastrcture and using hosted data centers. A good analogy to that is leasing versus renting a car. If we run a fleet we don't have to buy the cars of the entire fleet, instead, we could lease those cars from a third party provider which uses the economy of scale to manage the cost and risk through his entire customer base.  By leasing the car we still maintain pretty much the same degree of control as we would if we would own the car ourselves, but we have a more economimcal model to do so. 

This brings me to the main point of this post. The important difference between private and public cloud is less about the cost but more to do with ownership and flexibility. I'll use another transportation analogy to illustrate my point.  The cost of using public transportation is significantly lower than owning our own private car. If I only use the cost argument I could easily argue that owning a private car doesn't make any economical sense, still most of us wouldn't give up our private car, not because it is less expensive, but because we want the flexibility, and to control when and where we want to go, and are willing to pay extra to accommodate these needs.  Pubic cloud in this example is analogous to public transportation; the fact that you can share the same infrastructure with many users enables the sharing of costs with all the users, and thus the cost per user is significantly reduced, while at the same time it means that you agree to compromise on some sort of a least common denominator which is also an attribute of that cost benefit.

If we agree on this observation, then its clear that niether private nor public clouds would ever go away but instead we'll be moving away from the old way of running data centers into a more modern data center, a hybrid data ceneter that can serve the need for control and cost. In both cases, controlling the cooling, real estate, and electiricity doesn't makes any sense, and indeed all that will be outsourced completely if that's not already the case. The main thing that will make private cloud different is who controls where and how our data is maintained, where and how our services will run, what type of infrastrcture (compute, network, storage) we choose to run our business, and most importantly how much we're willing to share that environment with other users to reduce the cost. Even with that, it's hard to draw  a clear line between private and public cloud, as it may very well be the case that what's right in a particular case is a mashup between the two. For example, imagine the case where we will keep our data in a private cloud but run the compute nodes on a public cloud, is that a private, public cloud...who cares?

Final note

Many use the term private cloud to point to the old (and pretty clueless) way of running a data center. I argued that private cloud, as the name suggests, represents a business need to maintain high degree of privacy and control of its data and resources by ensuring that they are not shared with others. The key is the sharing aspect. The resources may very well be hosted somwhere else but the key is that they remain private. These needs will never cease to exist in the same way that we have public and private transportation, or the option to lease and rent a car, and will continue to have.  The old way of running data centers must change and is changing to meet the cost and agility demand of the business, and is adopting many of the features of public cloud. With that the cost margin differences between private and public clouds becomes less signficant and for some scenarios has proven to be even cheaper (assuming that you can design for a particular work load and use case) in a private cloud sceanrio. All this makes the cost argument even less significant a factor for choosing between the two models.

Public cloud providers would also need to adapt to meet the demand of private cloud. Rackspace recently announced a new offering in that space around OpenStack.

Rackspace's announcement is another sign that any player that wants to compete in the IaaS market needs to offer some way to have a private and public cloud. Any offering with only half that is going to be left wanting.

In addition to that, Rackspace is conneting their hosting business which some could view as another form of private cloud with their public cloud offering which will give thier customers an even greater degree of flexability. Amazon already provides for a while now the Virtual Private Cloud (VPC) service which enables them to offer resources from the public cloud as just another node in the customer's private cloud. Amazon VPC doesn't provide the full degree of privacy and isolation of their network and resources, but is definitely another form of connecting private with public cloud offerings, and an indication of how Amazon has adapted to this demand. There are still many questions to be answered and challenges ahead, namely how to combine the two models well both technically and echonomically, but the main point is that the entire debate around private and public cloud, who's going to win should become moot. It's now time to focus our energy on how we combine the two.

References

• 10 Predictions about Cloud Computing 
• Goodbye Amazon EC2 See you later cloud
• Is Rackspace Ready to Support Private Clouds?
• Amazon VPC

Powered by Qumana