Capacity planning is not a black art. But when it comes to sites doing a scheduled flash sale or a person with a large number of social media followers (over 1mil) it can get ugly fast! I witnessed this today for a site that was opening up new memberships at a specific time. Their site runs on Magento and that is about all I know since they are not a client. To not help matters they were also using Cloudflare which just showed their own error message.
So how can this be prevented?
Magento and most LAMP applications like WordPress are known for not scaling very well. You have to leverage caching and CDNs along with using the correct config files such as my.cnf (MySQL) to optimize performance. Using default configurations will NOT do you any favors!
If using Amazon Web Services (AWS) you should be leveraging:
RDS Cluster (MySQL)
Correct EC2 Sizing
Elastic Load Balancers (ELB)
Getting back to the example that inspired to me write this. During the first hour the complaints on their Instagram were quick and heavy. Users even taking pictures of the Cloudflare error page and posted those in frustration. Needless to say it is not the best PR after building up to an event and getting more interest than you saw coming. After two hours they gave up and posted a page to postpone their event. With correct capacity planning and configuration of Magento this could have gone much better!
I have been reading recently about the “Lean Startup” concept, which is funny since I have been working with and been part of more than a few without knowing there is a nifty catch phrase for them. So now I am all up on the lingo and hash tag #leanstartup I can speak on it. Today is the perfect time to take a gamble and dive into a lean startup. Being that there are so many free tools and services you can use along with very inexpensive solutions to infrastructure that were not available even a few years ago. By using the power of Open Source software, free online services, and cloud computing one can build a scalable and very cost effective lean startup.
Let’s first look at your new companies hosting and servers. being that you are lean you may most likely be working from home. So let’s look at hosting and services you will need in order to host some of the Open Source applications will will talk about in a moment. In my experience the basic things you need to get going in terms of setting up shop are the basics:
I have been working on some new projects since leaving my last job about 6 months ago. One is to build an entire infrastructure that is highly available and redundant. With Amazon Web Services this is a snap and almost makes my job obsolete. With EC2, S3, EBS, and CloudFront you can build a scalable solution with dependable backups with ease. My goal now is to also use AWS to create an intranet for the company. I have not found much on this topic so I ma taking the time to document it here. My ideas so far:
Use Fedora Directory Server as main LDAP solution. This can be used with EBS (Elastic Block Storage) with striped volumes to store the LDAP data. Along with having redundant multi-master replication geographically depending on where the EC2 instance is set.
Subversion server using EBS for storage. Using EC2 for something along the lines of SVN is a good use since it speeds up performance. Along with having the backup ability of EBS and EBS Snapshots