Capacity planning is not a black art. But when it comes to sites doing a scheduled flash sale or a person with a large number of social media followers (over 1mil) it can get ugly fast! I witnessed this today for a site that was opening up new memberships at a specific time. Their site runs on Magento and that is about all I know since they are not a client. To not help matters they were also using Cloudflare which just showed their own error message.
So how can this be prevented?
Magento and most LAMP applications like WordPress are known for not scaling very well. You have to leverage caching and CDNs along with using the correct config files such as my.cnf (MySQL) to optimize performance. Using default configurations will NOT do you any favors!
If using Amazon Web Services (AWS) you should be leveraging:
RDS Cluster (MySQL)
Correct EC2 Sizing
Elastic Load Balancers (ELB)
Getting back to the example that inspired to me write this. During the first hour the complaints on their Instagram were quick and heavy. Users even taking pictures of the Cloudflare error page and posted those in frustration. Needless to say it is not the best PR after building up to an event and getting more interest than you saw coming. After two hours they gave up and posted a page to postpone their event. With correct capacity planning and configuration of Magento this could have gone much better!
I was reading an article on BBC about Kelly Sutton of CultofLess.com and it got me to thinking about the potential for a new sub-culture in the US and rest of the world. Basically it is a movement (if you can call it that yet) of mainly 20-somethings relinquishing the vast majority of their posessions and living with as little as possible as long as they have their laptops and other digital goodies. Depending on friends and family to provide a place to sleep in some instances. This may yet be another symptom of steps toward what many call Technological Singularity in the sense that a new subset of people following this digital living and minimalism of possessions are becoming more popular. Now there are Hacker Spaces, imagine a day soon where there will be “Hacker Flop Houses” or “Hacker Hostels”. Even work could be transformed for this subset of people in terms of things already occurring. With the ability to telecommute or work on Human Intelligence Tasks (HITs) that services like Amazon’s Mechanical Turk, Crowd Flower, or others could allow people to live an almost digital hobo existence bedding down in hacker flop houses and moving on to another place whenever they choose. With no physical address these people could also become hired digital guns if they chose to follow a less reputable path. Using public Internet access to do their deeds and be gone to another town by the time anyone notices. Almost sounds like something from a William Gibson book but the groundwork for this sort of thing is already in place in many instances. Roving gangs of black hats working to do the deeds of the highest bidder. Sounds pretty scary to me. Although that is just an example of what might happen. Most people as in the original article I read are honest working people who are following a route less taken, which I can understand and support. It would also be a very effective way to save money if you lack the overhead of a place to live.
I have been reading recently about the “Lean Startup” concept, which is funny since I have been working with and been part of more than a few without knowing there is a nifty catch phrase for them. So now I am all up on the lingo and hash tag #leanstartup I can speak on it. Today is the perfect time to take a gamble and dive into a lean startup. Being that there are so many free tools and services you can use along with very inexpensive solutions to infrastructure that were not available even a few years ago. By using the power of Open Source software, free online services, and cloud computing one can build a scalable and very cost effective lean startup.
Let’s first look at your new companies hosting and servers. being that you are lean you may most likely be working from home. So let’s look at hosting and services you will need in order to host some of the Open Source applications will will talk about in a moment. In my experience the basic things you need to get going in terms of setting up shop are the basics: