HOWTO: Maintain a Rock Star Culture

In a recent post I described a few bullet points on how I’ve managed to go about recruiting great engineers. That’s all and good, but what happens when you manage to land a few of these people? A common retort to my recruiting efforts at SimpleGeo has been, “How are you going to get them to work together without killing each other?” It’s a common misperception that all top engineers are also prima donnas who are impossible to work with. I’d argue that the truly talented engineers are far from prima donnas.

So here is a small list of things you’re going to have to prepare yourself to do if you want to hire, encourage, and foster a team of top engineers.

  • Don’t skimp on making them comfortable. You’re paying this person over $100,000 a year so why cheap out on their hardware, desk, chair, snacks, etc.? What we do at SimpleGeo is we tell them they have about a $3000 budget to purchase an Apple MacBook Pro of their choice and whatever monitor setup they desire. We make sure there’s plenty of water in the water cooler, soda in the fridge, nice chairs, whiteboards, etc. in the office as well.
  • You need to be prepared to fire a great engineer. There are going to be great engineers on your team who, for one reason or another, are not going to work out. They’re causing more harm than good – act quickly and decisively when you identify this. Yes, I just told you to fire great engineers.
  • Do not allow language and other technology zealotry to take hold amongst your engineers. Truly great engineers will run benchmarks, be pragmatic in their technology choices, and abide by bake-offs. I hate Java with a passion, but if it will pull five times more jobs a second off of RabbitMQ than Python then we’re using Java for our queue processors.
  • Allow ample time for proper development and, if that’s not possible due to arbitrary deadlines, allow developers to circle back after the fact to clean up. Great developers loath writing shitty code; at least give them the opportunity to revisit quickly/poorly written code. They’ve got a reputation to uphold you know!
  • Institute and promote proper development cycles and practices. Great engineers will naturally do this on their own and, when they do, get out of their way or figure out how to help with the process.
  • Hire or promote a great engineer to manage them. Great engineers, for lots of reasons, tend not to trust non-engineers to ensure their voices are heard within the company. If you think about it, engineering is one of the only disciplines where people who were not trained in the vocation are allowed to manage a trained team. You wouldn’t put a marketing person in charge of accounting would you?
  • This is a personal preference of mine, but I don’t think you should hire or have any dedicated product/project managers.
  • Your engineers should, for the most part, drive new feature development. You’ve just hired a small army of the best technologists on the planet. Nobody in your company knows technology trends better. This is where it becomes crucial that you’re adequately communicating your vision for the product and company. If your engineers buy into that vision they’ll dream up and program things you never thought possible. If they don’t they’ll tell you they don’t and why; listen to them carefully as your business’s success might depend on it.
  • Within reason, give them flexibility in where, when, and how they work. It’s true that face-to-face time and pair programming lead to better code, but sometimes an engineer just wants to walk to his favorite cafe and work with IM turned off. Let them.
  • The phrase is “Work hard. Play hard.” not “Work hard and then work hard some more.” This applies generally to the entire team, but especially so with highly cognitive vocations such as accounting, engineering, etc. Give your team ample vacation time and encourage them to use it. I’ve gone so far as to enforce reasonable hours at the office. Your engineers simply aren’t producing great code if you’re forcing them to work 12 hours a day, 7 days a week. Some engineers will resist this, if they do, read Alex Payne’s great post on not being a hero.
  • Promote a culture amongst engineers where their first response to a technical question is, “I don’t know.” A good way to tell a good engineer from a great engineer after you’ve hired them is a willingness to quickly admit when they don’t know something.

The above is just a small list of things I’m actively doing in an attempt to maintain a team solely built of great engineers that are producing great products at a very rapid rate. The usual caveat being that your mileage may vary when you attempt to do this yourself.

NoSQL vs. RDBMS: Let the flames begin!

I’ve been getting solidly flamed recently, as have my former coworkers at Digg, my friends at Twitter, etc. about our adoption and promotion of various NoSQL storage systems. It seems that some DBAs are very, very upset that us internet kids are considering abandoning SQL’s ship. I’m not here to throw out a bunch of insane numbers, benchmarks, or flame back, but I did want to point out why SimpleGeo and others are jumping onto the NoSQL bandwagon.

First, and foremost, I haven’t heard of anyone saying MySQL or PostgreSQL on comparable hardware is faster than NoSQL options. The best I’ve heard is that MS SQL setups on SSD drives with lots of RAM could do 6,100 result sets a second. I guess, based on these posts, I’d like to ask a few questions to the people who honestly think RDBMSs can compete with NoSQL solutions at large scale.

  • Do you honestly think that the PhDs at Google, Amazon, Twitter, Digg, and Facebook created Cassandra, BigTable, Dynamo, etc. when they could have just used a RDBMS instead?
  • Has anyone ran RDBMS benchmarks with highly heterogeneous datasets with lots of varying indexes on them? At Digg we had probably a hundred or so tables, each table had varying indexes (a char here, an integer there, a date+time here). Disk IO becomes a serious problem when indexes for different tables are stored on different parts of disks and you have concurrent reads/writes. I know that people have found ways around this, such as 37Signals systems guy putting 15 x 15k RPM drives on his DB server. Assuming $500 a disk (15k disks range from $300 to $800 on Newegg) that’s $7,500 just for disks.
  • Anyone out there running an EC2 large instance with a RDBMS on it that’s doing 1,800 reads/second? I’ve got a Cassandra node that was getting hammered with a load of 6 serving that much traffic without falling over, which I think is pretty decent when you consider each node could easily do that and adding more nodes to handle more load is trivial.
  • How much are you spending on those MS SQL servers with SSD drives that serve up 6,100 results a second? MS SQL is $5,999 per processor. Windows Server 2008 is another $1029. Decent 128GB SSDs appear to cost around $450 each. You see where I’m going with this. Nobody is arguing you can’t get RDBMSs to scale up to a few thousand reads/writes a second if you can afford to spend $50,000 or $100,000 per server. The problem is that very few startups can spend that much money on a single server.
  • How much time are your DBAs spending administering your RDBMSs? How much time are they in the data centers? How much do those data centers cost? How much do DBAs cost a year? Let’s say you have 10 monster DB servers and 1 DBA; you’re looking at about $500,000 in database costs.
  • How easy is it to add a new server to your cluster? If we identify a hot spot in our Cassandra cluster, we can have a new node bootstrapped into our cluster in about five minutes. And I mean it’s in production taking writes and serving reads.
  • Does your RDBMS automatically rebalance the entire cluster when a new node is bootstrapped into it?
  • I’m running a 50 node cluster, which spans three data centers, on Amazon’s EC2 service for about $10,000 a month. Furthermore, this is an operational expense as opposed to a capital expense, which is a bit nicer on the books. In order to scale a RDBMS to 6,000 reads/second I’d need to spend on the order of five months of operation of my 50 node cluster.
  • Has anyone ran benchmarks with MySQL or PostgreSQL in an environment that sees 35,000 requests a second? IO contention becomes a huge issue when your stack needs to serve that many requests simultaneously. I know of one company that’s managing to scale portions of their PostgreSQL servers by purchasing $250,000 servers. This would cover my 50 node EC2 cluster for two years.

I guess what I’m saying is that my decision to use NoSQL, and I’m guessing others’ decisions to do so, has less to do with the fact that we can’t squeeze a few thousand writes a second out of MySQL and more to do with management and cost overhead. NoSQL solutions allow us to serve absurd amounts of data for a really, really low price. I’m happy to put my $/write, $/read, and $/GB numbers for my NoSQL setup against anyone’s RDBMS numbers.

We’re not nearly as dumb as everyone thinks we are; I promise.

HOWTO: Recruit Rock Stars

Yesterday, SimpleGeo announced the hiring of five new employees. Four will be engineers and one is an engineer moonlighting as a developer advocate for us. The feedback I’ve received about the team we’ve managed to put together in such a short period of time usually involves two statements:

  1. How on earth did you manage to build a team like this in such a short period of time?
  2. Will you please leave some engineers for the rest of us?

The answer to #2 is easy; no. The answer to #1 is somewhat funny; I’m a better recruiter than I am a coder or architect. No, it’s true. Ask Jay if he was more sorry to see his lead architect or his top recruiter leave Digg. My guess is he’s more upset his top recruiter left (I recruited about 40% of engineering and over 10% of the company by the time I’d left). The simple fact is that I would have made more money as a recruiter for Digg than as their lead architect.

But, how do I do it? Good question. I honestly don’t know, but I am going to share some insights that might help you land your own rock star. Here’s an arbitrarily ranked list of do’s and don’t’s for finding, recruiting, and hiring great engineers.

  • Make sure your ego is in check. My only rule in hiring is to hire people that are smarter than me. Top engineers like this will be smarter than you and most likely command a higher salary than you. You need to be fully prepared to pay handsomely for someone who will likely make you look and feel like a fool on a daily basis.
  • If you take nothing else away from this post, please remember this: amazing engineers are not perusing job boards for their next job. Do you honestly think Alex Payne or Ian Eure actively seek out employment?
  • Great engineers generally seek out two things when looking for new employment: interesting problems and awesome people. If your startup is “like Twitter plus blah”, you’re not likely going to be able to recruit top engineers.
  • Get involved in local meetups, bleeding edge protocols, open source, etc. I’ve found, and recruited, two of the best engineers I know via meetups (Arin Sarkissian) and open source projects (Chris Goffinet).
  • Do not, under any circumstances, send a recruiter after an engineer you covet. Send your most senior technologist after them. Pitch them on your team. Pitch them on your problem sets. Tell them about your strict adherence to TDD and Agile. My point on this is you need to send in someone who can speak engineering and sell that aspect of your company.
  • Pay them what they’re worth. If you don’t, someone else will. You aren’t going to lure these guys in with dreams of IPO or M&A riches. They’re smart enough to know that 1% of your company won’t lead to “fuck you money.” So don’t bicker over paying them an extra $10,000.

I’ve got a lot more ideas on how to manage and foster such a team once it’s been built, but those will have to wait for another blog post.