1. AWS is slow!
2. AWS is expensive!

I’ve covered IO performance on EC2 in-depth before and have compared the IO benchmarks, favorably, against numbers from Digg and Media Temple’s systems engineers. The notion that AWS is too slow for your application is, largely, not supported by the numbers and comparisons. The second point I often make with regards to performance on AWS is that Amazon uses this to run large portions of their own infrastructure. Trust me, if it’s good enough for the largest online retailer in the world, it’s good enough for you.

The second point is a bit harder to defend sometimes. Amazon’s AWS can be cheaper than running your own hardware and vice versus. If you run huge amounts of servers AWS can be a few hundred thousand more by comparison on raw numbers that compare cost of your own hardware to cost of AWS. The problem with this vanilla comparison is it forgets one extremely important cost for startups – opportunity cost.

I have a few rhetorical questions as to why people are not using AWS.

• How many people does it take to maintain your own DC? People have to wrangle hardware, travel around to various DCs, RMA hardware, etc. If they weren’t doing those things, or you didn’t need those people, what could you be doing with those resources if they weren’t wiring your DC?
• How much time, money, effort, and overhead is it going to take to create multiple data centers? Have you negotiated bandwidth contracts before? Do they have power from multiple providers? Do they have power and bandwidth failover? Amazon has amazing economies of scales and has spent thousands of man hours (years?) preparing for power/bandwidth failover, floods/natural disasters, etc.
• Managing multiple data centers requires a small army of highly trained network operations people. Have you built DC failover before? Have you implemented load balancing across multiple DCs? It took me about 30 minutes to set up an Elastic Load Balancer that spread traffic across three Availability Zones (Amazon’s term for DCs).
• Have you thought about building your own automation and self-service APIs for the DC you want to build? Fabric/Chef/Puppet/Capistrano combined with AWS’s automation API is an extremely potent combination for automating large clusters. For instance, we use Fabric and Boto to automate the creation of all nodes in our cluster. I can run a command in Fabric that creates an API server out of thin air, bootstraps it, and puts it into our ELB. This takes about five or so minutes.
• Have you ever set up a DC in Europe? What about Asia? Would you even know where to start? I can spin up a server in Europe in a matter of seconds. How much might you spend on flying your network operations folks to and fro all of these DCs you plan on building?

These are just a few of the nooks and crannies that people often forget when comparing running their own data centers that I think are extremely important. The two biggest costs, in my opinion, that people forget are opportunity cost and cost of creating automation systems.

To expound a bit on opportunity cost, I’d like to quote the ever-thoughtful Joi Ito.

“If you want to increase the pace of innovation, you need to lower the cost of failure.” — Joi Ito

I can fire up an entire DC for SimpleGeo with a 20-30 node cluster with a few commands, totally automated, run load/consumption/system tests against it, find flaws in my system, and iterate in a matter of hours at a cost of a few hundred dollars.

The simple fact is, SimpleGeo wouldn’t be anywhere near as robust, indeed it might not even exist, as it is without leveraging the cloud.

So here is a small list of things you’re going to have to prepare yourself to do if you want to hire, encourage, and foster a team of top engineers.

• Don’t skimp on making them comfortable. You’re paying this person over $100,000 a year so why cheap out on their hardware, desk, chair, snacks, etc.? What we do at SimpleGeo is we tell them they have about a $3000 budget to purchase an Apple MacBook Pro of their choice and whatever monitor setup they desire. We make sure there’s plenty of water in the water cooler, soda in the fridge, nice chairs, whiteboards, etc. in the office as well.
• You need to be prepared to fire a great engineer. There are going to be great engineers on your team who, for one reason or another, are not going to work out. They’re causing more harm than good – act quickly and decisively when you identify this. Yes, I just told you to fire great engineers.
• Do not allow language and other technology zealotry to take hold amongst your engineers. Truly great engineers will run benchmarks, be pragmatic in their technology choices, and abide by bake-offs. I hate Java with a passion, but if it will pull five times more jobs a second off of RabbitMQ than Python then we’re using Java for our queue processors.
• Allow ample time for proper development and, if that’s not possible due to arbitrary deadlines, allow developers to circle back after the fact to clean up. Great developers loath writing shitty code; at least give them the opportunity to revisit quickly/poorly written code. They’ve got a reputation to uphold you know!
• Institute and promote proper development cycles and practices. Great engineers will naturally do this on their own and, when they do, get out of their way or figure out how to help with the process.
• Hire or promote a great engineer to manage them. Great engineers, for lots of reasons, tend not to trust non-engineers to ensure their voices are heard within the company. If you think about it, engineering is one of the only disciplines where people who were not trained in the vocation are allowed to manage a trained team. You wouldn’t put a marketing person in charge of accounting would you?
• This is a personal preference of mine, but I don’t think you should hire or have any dedicated product/project managers.
• Your engineers should, for the most part, drive new feature development. You’ve just hired a small army of the best technologists on the planet. Nobody in your company knows technology trends better. This is where it becomes crucial that you’re adequately communicating your vision for the product and company. If your engineers buy into that vision they’ll dream up and program things you never thought possible. If they don’t they’ll tell you they don’t and why; listen to them carefully as your business’s success might depend on it.
• Within reason, give them flexibility in where, when, and how they work. It’s true that face-to-face time and pair programming lead to better code, but sometimes an engineer just wants to walk to his favorite cafe and work with IM turned off. Let them.
• The phrase is “Work hard. Play hard.” not “Work hard and then work hard some more.” This applies generally to the entire team, but especially so with highly cognitive vocations such as accounting, engineering, etc. Give your team ample vacation time and encourage them to use it. I’ve gone so far as to enforce reasonable hours at the office. Your engineers simply aren’t producing great code if you’re forcing them to work 12 hours a day, 7 days a week. Some engineers will resist this, if they do, read Alex Payne’s great post on not being a hero.
• Promote a culture amongst engineers where their first response to a technical question is, “I don’t know.” A good way to tell a good engineer from a great engineer after you’ve hired them is a willingness to quickly admit when they don’t know something.

The above is just a small list of things I’m actively doing in an attempt to maintain a team solely built of great engineers that are producing great products at a very rapid rate. The usual caveat being that your mileage may vary when you attempt to do this yourself.

First, and foremost, I haven’t heard of anyone saying MySQL or PostgreSQL on comparable hardware is faster than NoSQL options. The best I’ve heard is that MS SQL setups on SSD drives with lots of RAM could do 6,100 result sets a second. I guess, based on these posts, I’d like to ask a few questions to the people who honestly think RDBMSs can compete with NoSQL solutions at large scale.

• Do you honestly think that the PhDs at Google, Amazon, Twitter, Digg, and Facebook created Cassandra, BigTable, Dynamo, etc. when they could have just used a RDBMS instead?
• Has anyone ran RDBMS benchmarks with highly heterogeneous datasets with lots of varying indexes on them? At Digg we had probably a hundred or so tables, each table had varying indexes (a char here, an integer there, a date+time here). Disk IO becomes a serious problem when indexes for different tables are stored on different parts of disks and you have concurrent reads/writes. I know that people have found ways around this, such as 37Signals systems guy putting 15 x 15k RPM drives on his DB server. Assuming $500 a disk (15k disks range from $300 to $800 on Newegg) that’s $7,500 just for disks.
• Anyone out there running an EC2 large instance with a RDBMS on it that’s doing 1,800 reads/second? I’ve got a Cassandra node that was getting hammered with a load of 6 serving that much traffic without falling over, which I think is pretty decent when you consider each node could easily do that and adding more nodes to handle more load is trivial.
• How much are you spending on those MS SQL servers with SSD drives that serve up 6,100 results a second? MS SQL is $5,999 per processor. Windows Server 2008 is another $1029. Decent 128GB SSDs appear to cost around $450 each. You see where I’m going with this. Nobody is arguing you can’t get RDBMSs to scale up to a few thousand reads/writes a second if you can afford to spend $50,000 or $100,000 per server. The problem is that very few startups can spend that much money on a single server.
• How much time are your DBAs spending administering your RDBMSs? How much time are they in the data centers? How much do those data centers cost? How much do DBAs cost a year? Let’s say you have 10 monster DB servers and 1 DBA; you’re looking at about $500,000 in database costs.
• How easy is it to add a new server to your cluster? If we identify a hot spot in our Cassandra cluster, we can have a new node bootstrapped into our cluster in about five minutes. And I mean it’s in production taking writes and serving reads.
• Does your RDBMS automatically rebalance the entire cluster when a new node is bootstrapped into it?
• I’m running a 50 node cluster, which spans three data centers, on Amazon’s EC2 service for about $10,000 a month. Furthermore, this is an operational expense as opposed to a capital expense, which is a bit nicer on the books. In order to scale a RDBMS to 6,000 reads/second I’d need to spend on the order of five months of operation of my 50 node cluster.
• Has anyone ran benchmarks with MySQL or PostgreSQL in an environment that sees 35,000 requests a second? IO contention becomes a huge issue when your stack needs to serve that many requests simultaneously. I know of one company that’s managing to scale portions of their PostgreSQL servers by purchasing $250,000 servers. This would cover my 50 node EC2 cluster for two years.

I guess what I’m saying is that my decision to use NoSQL, and I’m guessing others’ decisions to do so, has less to do with the fact that we can’t squeeze a few thousand writes a second out of MySQL and more to do with management and cost overhead. NoSQL solutions allow us to serve absurd amounts of data for a really, really low price. I’m happy to put my $/write, $/read, and $/GB numbers for my NoSQL setup against anyone’s RDBMS numbers.

We’re not nearly as dumb as everyone thinks we are; I promise.

1. How on earth did you manage to build a team like this in such a short period of time?
2. Will you please leave some engineers for the rest of us?

The answer to #2 is easy; no. The answer to #1 is somewhat funny; I’m a better recruiter than I am a coder or architect. No, it’s true. Ask Jay if he was more sorry to see his lead architect or his top recruiter leave Digg. My guess is he’s more upset his top recruiter left (I recruited about 40% of engineering and over 10% of the company by the time I’d left). The simple fact is that I would have made more money as a recruiter for Digg than as their lead architect.

But, how do I do it? Good question. I honestly don’t know, but I am going to share some insights that might help you land your own rock star. Here’s an arbitrarily ranked list of do’s and don’t’s for finding, recruiting, and hiring great engineers.

• Make sure your ego is in check. My only rule in hiring is to hire people that are smarter than me. Top engineers like this will be smarter than you and most likely command a higher salary than you. You need to be fully prepared to pay handsomely for someone who will likely make you look and feel like a fool on a daily basis.
• If you take nothing else away from this post, please remember this: amazing engineers are not perusing job boards for their next job. Do you honestly think Alex Payne or Ian Eure actively seek out employment?
• Great engineers generally seek out two things when looking for new employment: interesting problems and awesome people. If your startup is “like Twitter plus blah”, you’re not likely going to be able to recruit top engineers.
• Get involved in local meetups, bleeding edge protocols, open source, etc. I’ve found, and recruited, two of the best engineers I know via meetups (Arin Sarkissian) and open source projects (Chris Goffinet).
• Do not, under any circumstances, send a recruiter after an engineer you covet. Send your most senior technologist after them. Pitch them on your team. Pitch them on your problem sets. Tell them about your strict adherence to TDD and Agile. My point on this is you need to send in someone who can speak engineering and sell that aspect of your company.
• Pay them what they’re worth. If you don’t, someone else will. You aren’t going to lure these guys in with dreams of IPO or M&A riches. They’re smart enough to know that 1% of your company won’t lead to “fuck you money.” So don’t bicker over paying them an extra $10,000.

I’ve got a lot more ideas on how to manage and foster such a team once it’s been built, but those will have to wait for another blog post.

1. It didn’t support replication.
2. There wasn’t any inherent support for partitioning. This means we’d have to use Memcached’esque hashing to partition our data, which might work for many, but won’t work for our use cases.
3. It was licensed under the AGPL.

The first two precluded me from using it merely on technical reasons, which aren’t huge hurdles because I was looking for a foundation to build on and would have gladly built that into Neo4j and released it back. The AGPL, however, was a deal breaker.

For those who don’t know, the AGPL is an incredibly viral license that says that not only do you have to redistribute your code changes, as the GPL states, but you must also publicly release any code that connects to an AGPL piece of software over a network. I fear the day that an HTTP server is built using AGPL. Think about it.

I’m extremely opposed to viral software. Since when is the term “viral” a good thing? Ultimately, though, my biggest complaint is the blatant hypocrisy of companies using AGPL. They’re basically stating that they’re “open” and “free”, when in reality they’re outsourcing free labor while retaining the right to charge for a proprietary license. The hypocrisy comes from companies trumpeting this so-called freedom when, in reality, they’re forcing certain behavior on their users. Reminds me of pious people saying, “You can live your life however you want! As long as you live it according to our rules.”

You want real freedom in software? Then release your code into the public domain, use CC0, or a BSD/Apache/MIT license. That’s true freedom. Anyone who tells you otherwise is either lying or delusional.

So do I have an issue with proprietary software? No, because Microsoft isn’t trying to play me for a fool. They’re up front saying, “Hey, it’s $199 to install Windows.” I can respect that. They’ve spent a lot of time building a product. They think it’s worth something. Consumers think it’s worth something. Simple economics come into effect after that.

What about services like SimpleGeo and AWS? No, because it’s not that you’re buying software. You’re buying a service. The crucial differentiator here is if your data is free. You’re not paying for AWS software or SimpleGeo software, you’re paying for service, support, and knowledge. What you pay SimpleGeo for is our knowledge in scaling, managing large server clusters, our data agreements with our various partners, and ease-of-use. It’s the same reason you go to the restaurant for a dish you could make at home.

What about Twitter or Digg? No, because those are communities. The notion that the software that runs Digg or Twitter is what makes them so special is ludicrous. The reason people keep going back to those sites is to participate with like-minded people in an environment they find fun and interesting. It’s like going to the bar.

So, in conclusion, if you’re going to release software then grow a pair and truly set your code free. If not, that’s fine too. Just don’t release your code using some zealot’s license and pretend your code is truly free when it’s not.

The one point they brought up as a lessoned learned was that they didn’t focus on learning and failing fast until it was too late, which reminded me of a quote from Joi Ito I recently heard at a conference.

“Want to increase innovation? Lower the cost of failure.” — Joi Ito

This is absolutely crucial. As a startup you need to fail and fail often. Not a lot of people know that SimpleGeo was, in fact, born of failure. Matt and I originally started the company as a location-based gaming company. Due to market realities (investors rarely invest in gaming companies in the seed stage), personnel realities (neither Matt nor I had built games), and other factors we had to have what I call our “come to Jesus” moment. Since then we’ve focused on small, quick iterations and quickly scrapping or moving on from our failures (or in many cases simply adjusting our assumptions).

Easy to say, but how are we doing it?

1. We use a loose approach to Agile development with two week sprints. This means that, worst case scenario, we’ve wasted two weeks of effort in order to find out if a tweak, product, etc. is going to fail. I can’t emphasize how important this is. If you push your team 6 months and release a giant product only to find out users hate it or don’t use it, you’re screwed. 6 months is a lot of opportunity cost for a startup.
2. We actively engage our user feedback. If users don’t like something we abandon it. If users are demanding something we move it up the priority list. For instance, we thought for sure that people wanted location-based search first and foremost after storage. Turns out, users actually wanted better tools for accessing and managing the data they’re sending us. So we’ve focused on churning out more SDKs and tools for managing data.
3. We use Amazon EC2. Specifically, for testing and prototyping, we use spot instances. It’s an extremely cost efficient way to destroy servers during testing.
4. We start small. Our current product offering is probably only 10% of what users expect from us long-term as far as pure number of features go. That being said, we picked the initial use-case and, I think, nailed it. The reality is 60% or more of the use cases for our users is simply storing millions of points and running radius queries on those points quickly. It’s also the foundation that all of our other products will be built on. So we’ve spent time focusing on building and iterating on that use-case. As a result, we’re multi-homed in three datacenters (actively serving data from all three), redundant across all three (meaning your data is stored in every datacenter), searches are extremely fast (under 50ms in some cases if you’re on EC2), and all points are backed up to S3 in YOUR account (it’s your data after all).

We’re not perfect by any means, but I think Matt and I realized early on that we need to be most efficient at recognizing and owning our failures and deficiencies. If you focus on minimally viable products, small iterations, and prioritizing user feedback you, too, can fail fast and often.

“Failure is useful.” — Larry Page

The code above is a simple class that will implement all of the option parsing and such to run a Nagios plugin from the command line. From there you need to implement a plugin. Below is the Gearman plugin that I created. It runs a simple job I created that sums numbers and returns the results. If all goes well then the job should run and the correct sum of the randomly selected numbers should be returned.

Once you’ve got all of that working it’s pretty simple to then add the following snippet of code to your setup.py file.

      entry_points = {
          'console_scripts': [
              'check_gearmand = path.to.nagios.plugins:check_gearmand',
          ]
      },

Dear Engineers, It’s not a “distributed” system if you have a master/slave setup. That’s simply a “redundant” system. #

Not long after this I got a few inquiries along with some great input from the ever-wise Jan Lehnardt from the CouchDB project along with Christopher Brown:

RT @joestump: “…It’s not a ‘distributed’ system if you have a master/slave setup. That’s simply ‘redundant’” — Same: distributed != sharding #

And it’s not “distributed” if it’s all hub-n-spokes.  Might as well have a central server with some extra storage. #

Basically, Jan and I are making the argument that redundancy or sharding/partitioning doesn’t really add up to a truly distributed system. Well, not in the sense of what I think of when I think of distributed. Going back to my old friend, the CAP Theorem, I think we can see why.

Redundancy could be argued to be the A in CAP; always available. I’d even argue that partitioning, in the sense that most people think of (e.g. partitioning 300m users across 100 MySQL servers), is the A in CAP. The P for partitioning in CAP is that a system is highly tolerant to network partitioning. Because of the master/slave setup of Katta, it really only implements the A in CAP.

Of course, CAP says that you can only have two in any distributed system. I’d make the argument that no system is truly distributed unless it fully implements two of the three properties in CAP. I’d further argue that if you had a highly available (A), highly consistent system (C) even it wouldn’t be distributed (lacking the ability to be highly tolerant to network partitioning).

The problem that I have with Katta’s definition of distributed is that I can’t truly distribute Katta in the raw sense of the word. For instance, I can’t spread Katta across three data centers and not worry about a data center going down. If I lose access to the master then Katta is worthless to me.

To have a truly distributed system I’d argue you need the following characteristics in your software:

• There can be no master (hub).
• There must be multiple copies of data spread across multiple nodes in multiple data centers.
• Software and systems must be tolerant of data center failures.

My point is, I think the word “distributed” is applied too freely to systems that are, more appropriately, called “redundant” or “fault tolerant”.

UPDATE: Seems I confused some people about my master/slave argument. You can build distributed systems that have master/slave pieces in the puzzle. Large distributed systems are rarely one cohesive piece of software, rather they tend to be a series of systems glued together. What I am saying is that a simple master/slave system is not distributed. Through a series of bolts, glue, add-ons, and duct tape you can make a master/slave system sufficiently distributed, but don’t confuse the totality of that system with the base master/slave system.

According to TripIt and Dopplr I spent 142 days traveling this year. I don’t have complete numbers, but I’m guessing I logged over 80,000 miles this year on various airlines. As in the tradition of last year, I think it’s only appropriate that I create a list of my year in cities.

• Koh Samui, Thailand
• Koh Phangan, Thailand
• Bangkok, Thailand
• Seattle, Washington
• Miami, Florida
• Austin, Texas
• Sebastopol, California
• Ypsiltanti, Michigan
• Ann Arbor, Michigan
• East Jordan, Michigan
• San Francisco, California
• Prague, Czech Republic
• Phoenix, Arizona
• Boulder, Colorado
• Amsterdam, The Netherlands
• Roskilde, Denmark
• Oslo, Norway
• Bergen, Norway
• Askvoll, Norway
• Dublin, Ireland
• Cork, Ireland
• London, United Kingdom
• New York City, New York
• San Diego, California
• Minneapolis, Minnesota
• Ashland, Wisconsin

We’ve been ironing out kinks in our production environment for the last few weeks and one of the things that worried me was if these assertions were true. So, I set out to run a fairly comprehensive test of Disk IO and throughput. I ran hdparm -t, bonnie++, and iozone against ephemeral drives in various configurations along with EBS volumes in various configurations.

For all of my tests I tested the regular ephemeral drives as they were installed (”Normal”), LVM in a JBOD setup (”LVM”), the two ephemeral drives in a software RAID0 setup (”RAID0″), a single 100GB EBS volume (”EBS”), and two 100GB EBS volumes in a software RAID0 setup (”EBS RAID0″). All of the tests were ran on large instances (m1.large). All tests were ran using the XFS file system.

hdparm -t

While hdparm -t isn’t the most comprehensive test in the world, I think it’s a decent gut check for simple throughput. To give a little context, I remember Digg’s production servers, depending on setup, ranging from 180MB/sec. to 340MB/sec. I’m guessing if you upgraded to the XL instance type and did a RAID0 across the four drives it has you’d see even better numbers with the RAID0 ephemeral drives.

What I also found pretty interesting about these numbers is that the EBS volumes stacked up “okay” against the ephemeral drives and that the EBS volumes in a RAID0 didn’t gain us a ton of throughput. Considering that the EBS volumes run over the network, which I assume is gigabit ethernet, 94MB/sec. is pretty much saturating that network connection and, to say the least, impressive given the circumstances.

For most applications, I’d guess that EBS throughput is just fine. The raw throughput only becomes a serious requirement when you’re moving around lots of large files. Most applications move around lots of small files, which I’d likely use S3 for anyways. If your application needs to move around lots of large files I’d consider using a RAID0 ephemeral drive setup with redundancy (e.g. MogileFS spreading files across many nodes in various data centers).

bonnie++

Disregarding the Input/Output Block performance of the ephemeral RAID0 setup here, it’s extremely interesting to note that EBS IO performance is better than the ephemeral drives and that EBS in a RAID0 was better in almost every metric as the ephemeral drives.

That all being said, RAID0 ephemeral drives are the clear winner here. I do wonder, however, if you could set up a RAID0 EBS array that had, say, four or six or eight volumes that’d be faster than the RAID0 setup.

If your application is IO bound then I’d probably recommend using EBS volumes if you can afford it. Otherwise, I’d use RAID0. Again, the trick with the ephemeral drives is to ensure your data is replicated across multiple nodes. Of course, this is cloud computing we’re talking about, so you should be doing that anyways.

Here’s the CPU numbers of the various configurations. One thing to note here is that EBS, LVM, and software RAID all come with CPU costs. Somewhat interesting to note is that the EBS has substantially less CPU usage in all areas except Input/Output Per Char.

If your application is both CPU and IO bound then I’d probably recommend upgrading your instance to an XL.

The last bonnie++ results are the random seeks per second and, wow, was I surprised. A single EBS runs pretty much dead even with the LVM JBOD and the EBS RAID0 is on par with the RAID0 ephemeral drives.

To say I was surprised by these numbers would be an understatement. The lesson here is that, if your application does lots of random seeking, you’ll want to use either EBS RAID0 volumes or RAID0 ephemeral drives.

iozone

Before running these tests I’d never even heard of this application, but it seemed to be used by quite a few folks so I thought I’d give it a shot.

Again, some interesting numbers from EBS volumes. What I found pretty interesting here is that the EBS RAID0 setups actually ended up being slower in a few metrics than a single EBS volume. No idea why that may be.

The other thing to note is that the single EBS volume outperformed the ephemeral RAID0 setup in a few different metrics, most notably being random writes.

Conclusions

I think the overall conclusion here is that disk IO and throughput on EC2 is pretty darn good. I have a few other conclusions as well.

• If you can replicate your data across multiple nodes then the RAID0 ephemeral drives are the clear winners.
• If you are just looking to store lots of small files and serve them up, then definitely use S3 with CloudFront.
• You’d very likely get even more impressive numbers using the XL instances with RAID0 striped across four ephemeral drives.
• Another potential disk setup would be to put different datasets on different ephemeral drives. For instance, put one MySQL database on one ephemeral drive and your other one on another.
• If your setups are IO bound and you’re looking for lots of redundancy, then EBS volumes are likely the way to go. If you don’t need the super redundancy on a single box then use RAID0 on ephemeral drives.