Author Archives: joestump

My patent-pending 3 question technical interview

Posted on by

I’ve done a lot of interviewing, hiring, and firing in my career. It’s inevitable as you start to build your own companies, become a team leader, or move into management. I fully subscribe to the notion that you should hire slowly and fire quickly. That being said, triaging candidates quickly is also important if you don’t want to spend half your time vetting candidates. Adding to the complexity of hiring an individual is that it can be extremely difficult to get to the heart of someone’s technical abilities without pairing with them.

In general my interview process is as follows:

  1. Stalk the candidate on the internet via Twitter, LinkedIn, and GitHub. I look at Twitter to see what they talk about, what interests them, etc. I look at LinkedIn for obvious reasons. On GitHub I pay special attention to their commit frequency, what projects they’ve forked, and considerable attention to any repositories that they’ve created and put out there.
  2. Meet with them for a coffee or have a short 30 minute phone conversation with them. This is when I’ll put them through my patent-pending 3 question technical interview. Due to the resources required for the next two steps, it’s imperative that I triage candidates quickly and effectively.
  3. Bring the candidate in for a full day of pair programming across the stack. Spend half of the day fixing a frontend ticket and the second half of the day working on a backend ticket. I do this regardless of skill or specialties in an effort to evaluate problem solving skills, how candidates work others, and how candidates approach problems that they know little about.
  4. A final, and optional, step is a 30 to 90 day contract. I consider this step to be not optional if the candidate lacks an established track record, wasn’t personally referred to me, or is an early hire at a startup when each hire has an enormous impact on company culture.

I actually have a mixed bag of questions that I’ve slowly added to over time. When talking with a candidate I attempt to match up questions with their experience and generally never go beyond three of them. Before asking them any of these questions I ask the candidate to tell me, on a scale of 1 to 10, where their expertise sits on each question’s topic (e.g. “On a scale of 1 to 10 how capable are you with the Unix command line?”).

So, without further ado…

There is a multi-level directory tree, say a local git checkout, that has a number of files all with the extension .tmp. How would you delete all files with the .tmp extension in a single command regardless of where they sit in the directory tree?

I’ve had candidates who will tell me they’re an 11 on a 10 point scale answer this question with rm -r *.tmp. In fact, I got into an argument with an existing Digg employee about that answer and how patently wrong it is. Overall, I’d say my pass rate on this question is below 20% Pretty poor, especially considering I’ve resorted to accepting any answer that mentions using the find command.

In PHP, what is the difference between isset() and empty()?

Every language has its edge cases and gotchas. This just happens to be one of my favorites from PHP because, quite frankly, PHP is a fucking moron when it comes to types. What I like about these edge case gotcha questions about languages is it really allows the language wonks and those who are true blue veterans of a language geek out and really shine. I’ve found that the pass rate on this, even with fairly seasoned PHP developers, was hit or miss. Maybe 50%

What is the -i option on grep used for?

I probably use grep a 100 times a day when I’m doing full-time development. Particularly when I’ve put my DevOps hat on. When I was in college I made myself read one man page a day on all of the major Gnu/Unix command line tools. Want to know what the flags on wc are for? Or how to scp a directory? I know this off the top of my head and I expect seasoned developers to as well. If you don’t know what the -i option on grep means, it likely means you’ve not committed yourself to ninja level status on the Unix command line. The pass rate on this question is also abysmal. Maybe 20-30%

What is the difference between an abstract class and an interface?

This is more of a theory question as it kind of spans languages. Anyone who has coded in PHP, C++, Java, etc. should have a pretty good handle on the fact that these two are different beasts. What I like about this question is that anyone who really, truly knows how to implement really effective OOP code will absolutely know the subtle differences between the two and how to most effectively use them. The pass rate on this has also been abysmal. At most 10% of developers have been able to clearly articulate the difference between the two.

What is the difference between an INNER and OUTER JOIN in SQL?

I love this question because it washes out every idiot who learned how to use the Rails or Django ORM and now thinks they’re some wizard at SQL. The most common answer to this question, I shit you not, is, “I don’t know. I just let the ORM handle that.” As my basketball coaches always would say, “It’s about the fundamentals!” Pass rate on this is also below 10% Bonus related question: What field of mathematics are RDBMSs based on?

What is your favorite OOP design pattern and why?

Another language theory question. I like this question because it doesn’t seem like a pass or fail question at first, but you’d be shocked how many developers have no idea what a design pattern is, much less their formal names, how to implement them, or where to use them most effectively. This is another question where true language polyglots will really shine. A true star candidate will find a way to talk about patterns that are more esoteric, such as the chain of responsibility pattern. While I’ll accept things like the factory or singleton patterns, I’m looking for more substance and look at those answers as a net neutral.

These days I don’t waste my time unless the developer can answer all three to my satisfaction. There is enough latitude in the questions that they work equally well for systems engineers, application engineers, and backend engineers. Any failures from someone purporting to be an architect should be seen as enormous red flags.

To raise or not to raise

Posted on by

If you follow startups at all, you’ll likely notice that there are precisely two camps when it comes to raising venture funding of any sort: those who trumpet each new round as if the company won the Super Bowl and those who believe bootstrapping is the only legitimate way to build a business. There are, of course, merits to both camps’ arguments, but, as with all things, reality tends to live somewhere in the middle. There are dozens of reasons why raising money could be a good idea and dozens of reasons why raising money might not be a good idea.

Whether or not to raise funding is an extremely complicated question with dozens of angles, numerous points of data, and, in the end, gut instinct and your emotions. Your company is a very personal topic. Don’t forget that, because your emotions will effect your business decisions.

I look at capital, whether it comes from an angel, a VC, top money lenders in Singapore or a traditional bank loan, as an accelerant. It’s like gas. Sometimes your fire is burning nice and bright. You’re warm and have plenty of heat so why pour more gas on the fire? Sometimes, however, you’d like to start another fire, or maybe you’d like to burn down a house, or, possibly, you’d like your little glowing embers to be a fire more quickly than is otherwise naturally possible. These are all great reasons to reach for the can of gas.

There are, literally, hundreds of reasons why you might consider raising money. The following is a small list of reasons that I think are in support of raising funding:

  • You’d like to create a product which requires you to build hardware, open storefronts, or otherwise produce something physical. Manufacturing is a capital intensive adventure. I doubt someone could easily bootstrap the Tesla or the Nest.
  • You’d like to focus all of your resources on getting the product to a new level. Sometimes your company’s bottom line isn’t growing quickly enough for your product’s needs. Do you think Steve Jobs would have hesitated to raise capital for Apple if they needed it to ship the iPhone? Investors call this an “inflection point” and love to invest in companies at these stages. Could you wait a year and save your pennies? Sure, but you could give up couple of points on a convertible note and be in a position to get there in a few months instead. And, in this industry, a year is a long time.
  • You’ve built a successful business with the best technology like blockchain IoT and now that it’s going good you have no interest in selling, and would like to extract some liquidity. We’ve all seen this, but nobody ever talks about it. Some company raises $10’s of millions and people wonder why a company making that much money would need it. They don’t, but their founders are now multimillionaires and free to get back to building that 50-year business without wondering when the big payday will come.
  • Not everyone is in a position, financially, to pursue their dreams. There are a litany of reasons why this might be, but you’d have to be a gigantic asshole to pooh-pooh someone for raising capital for this reason. Kids, family, mortgages, salary requirements, and time constraints are all legitimate reasons to say, “I either hunker down at this giant corporation for the rest of my life or I raise some capital, mitigate my family’s risk, and chase my dream.”
  • It can, at times, make sense to raise capital simply to gain access. Many large corporations have investment arms that keep an eye out for promising young companies and products that could help the mother ship out. Raising capital from Verizon or Salesforce is a great way to gain access to lucrative partnerships, which, of course, you’ll parlay into partnerships with other companies.

That being said, there are legitimate pitfalls to raising capital. Luckily, there is a solid book, Venture Deals, that very clearly and plainly lays out your options, how such deals are structured, and what to look out for. If you are considering seeking capital, or even if you are not, this is a must-read book.

Venture Deals raises a number of concerns, though not directly, about raising capital for founders. For every reason to raise money there is likely a good reason to not raise money. Specifically, I’ll talk about reasons related to what investors call a “priced round”, rather than convertible notes, which I find to be spectacular vehicles for founders seeking to take investment. A few reasons you might want to second-guess your decision to raise capital:

  • Raising capital is an enormously distracting process. Your company, product, and sanity will absolutely take a hit throughout the process. If you’re in the middle of a major product push, or negotiating a big contract, do not think for a second you should be out looking for money – you’ve got more important work at the moment.
  • There is such a thing as raising too much capital. If you raise too much money in an early round, you’re likely going to get severely pinched in future rounds. It’s not fun to hit your Series B and realize that you only own 7% of your own company (Yes, I know founders who’ve been in that position. Yes, they were at companies you’ve heard of.).
  • There is such a thing as too high of a valuation. I’m going to tell you a dirty little secret – investors think valuations are bullshit too. The real issue, though, is raising at a valuation that takes other options off the table. Setting the bar too high in your Series A or Seed can take early exits off the table.
  • When you set up two classes of stock and structure your company the way VCs usually require them to be configured, you give up all control of your company. Yes, you’re on the board and, yes, you’re CEO, but that hasn’t stopped Sequoia from (proudly, no less) firing 45% of their founding CEOs within 18 months. Remember, you’re on a vesting schedule too, so you’ll lose all of the equity that’s unvested.
  • Investors can be a huge distraction. It takes time to keep them up-to-date, coordinate conversations, extract input and insights. Additionally, 99% of the investors you’ll work with haven’t built a product like yours (if they’ve even built one), worked at a company like yours, interacted with users like yours, etc. They’ll suggest things to you that are completely insane. Stay the course – nobody, including your investors, know your business, product, and customers like you do.
  • Investors play portfolio theory. If you’re one of the ones they think are going to strike it big, you’ll be showered with praise and attention. If you’re not, you’re in for a rough reality – you’re now competing for their attention with the dozens of other losers in their portfolio.

I don’t have any hard and fast rules when it comes to raising capital. I think every business, product, and team are unique with their own unique set of challenges. That being said, I do have a set of guidelines I like to follow when raising early stage capital, which are as follows:

  • Don’t raise money before you have a working prototype. If at all possible, wait until you’ve launched a private beta. Every line of code you write, every beta signup, will lead to better terms with any investor. The downside, of course, is that you might prove to yourself and potential investors that your idea is dumb.
  • Only raise money from people with their own money in play. With the advent of the “super angel”, we’re now seeing many angels who are not, in reality, investing their own money. Investors who invest their own money tend to be a lot more attentive.
  • Look for investors with a low “deal flow”. Do you honestly think the guy funding 20 startups a year is going to have a whole hell of a lot of time to help you out? I’d shoot for people that do 3-5 deals a year.
  • Whenever possible, take money from the former founder. I jokingly refer to being a founder as being a member of the Fraternal Order of Founders. You simply don’t know what it’s like until you’ve been there and tried it. Having an investor with the “founder context” is going to be hugely important when you hit rough waters.
  • Seek out investors who have skillsets that compliment your own. If you’re a coder and your cofounder is a designer, look for a guy who used to be VP of Business Development somewhere or find a former COO/CEO. Those skills are not the bullshit most makers would have you believe. The tricky part is finding someone who treats those trades as craft and science rather than pure hustle or vapor.

In the end, do what you feel is best for your product and business. No book or blog post (including this one) can replace your insight into your own business.

Year in Review

Posted on by

2011 was a tumultuous year for just about everyone in the world, particularly Osama bin Laden. I’m no different than the average person on the planet and can be included in the masses thankful that an extremely stressful year has come to a close. 2012 is already shaping up to be an exciting time for myself and Diana, but more on that next year. The following is a highlight reel of my 2011.

  1. January started out with me working at SimpleGeo in San Francisco, Diana in Denver, and SimpleGeo’s team still split between Boulder and San Francisco. All three of these facts would be upended by the end of the year.
  2. Also in January, I went out to the UK to work with Holiday Extras for the second time. I made three more trips to Hythe, UK and have been extremely impressed with the team. It’s been a pleasure to watch passionate people reinvent many aspects of their technology stack, team, and processes in such a short period of time.
  3. The beginning of the year brought more tattoo work. I’ve been slowly, but surely, working on finishing the rest of my left arm. I don’t think I planned on getting two full sleeves when I started out, but I strangely find that it’s completing me in some weird way.
  4. I went to Grand Rapids, MI to talk to a local user group there about scaling, managing engineers, and general geekery. Thanks to Atomic Object for having me out!
  5. In early February my dear friend Josh “Dorkboy” Lynn came out to San Francisco for a trip. We drove down HWY 1, went to a basketball game, and ate a few hundred burritos.
  6. February also brought the closing of the Boulder office. A few people moved to San Francisco to join us and a few people moved on to other opportunities. The lesson learned is that startups are far too small to sustain two offices unless absolutely necessary.
  7. Finally, in February, Attachments.me, a side project of mine that ended up becoming a company with two amazing founders, received $500k in seed funding from the Foundry Group. I continue to be impressed with the hustle and professionalism of Ben and Jesse.
  8. In March I went to the entirety of SXSW. Diana joined me for the last day of Interactive and all of the Music portion. I nearly didn’t get out of Austin at the end due to craziness at the airport and was thoroughly burnt out on the entire event. It’s too big, too corporate, and far too hectic for my taste. I doubt I’ll be spending as much time there this year.
  9. In late March, SimpleGeo launched SimpleGeo Storage, which was built entirely on top of a distributed graph database system that our team developed internally. This patent pending technology is some of the most impressive technical work I’ve seen happen in a startup and I’m proud as hell with the team and their work. In particular, it was a pleasure to watch Mike Malone challenge himself and grow into an absolutely spectacular engineer.
  10. In April I went out to Denver a few times to see Diana showcase her artwork for her senior thesis. In May I once again traveled to Denver to watch Diana graduate from Metro State. We then packed up her apartment and moved her out to SF so that we could live together. This March 2nd will be two years. I can’t imagine someone filling a void in my life as completely as Diana has.
  11. Also in May was a trip to Amsterdam to speak at a location related conference. I managed to visit Amsterdam three times this last year and would happily visit it three hundred more times. I’ve never felt so at home in a foreign country as I do walking the canals of Amsterdam.
  12. May and June also brought the first serious interest from potential acquirers at SimpleGeo. We got pretty far down the road with one potential acquirer before walking away. This was my first foray into the dreaded M&A process and it set the tone for all future discussions with acquirers.
  13. In June Diana and I went to Vegas for the Future of Web Apps conference. The Carsonified crew are, in my opinion, some of the very best in the business. I gave a talk titled Starting Your Startup, which appeared to be well received.
  14. In June I started following the Four Hour Body’s Slow Carb diet. I ended up losing 35 pounds over the next five months and now weigh under 200 pounds for the first time since high school.
  15. July brought a trip to Europe with Diana. We went to Paris, Amsterdam, Rome and the UK. It pretty much rained the entire time we were in Europe and we both got sick. We tried to make the best of it and managed to take in some great sites in Rome and Paris. We had a great time hanging out with friends in Amsterdam. I think Diana might share my feelings on Amsterdam after the trip.
  16. In September conversations to sell SimpleGeo started to pick up steam. We talked with quite a few companies about the possibility and, at the end of October, ended up being acquired by Urban Airship. As part of the acquisition, I moved on from the company.
  17. After leaving SimpleGeo, I dedicated myself full-time to Sprint.ly, a product management tool that I’ve been wanting to build for almost my entire career. We launched as an invite-only beta on the 3rd of December.
  18. At the beginning of November Diana and I bought an RV with the hopes of driving around the Southwest all winter. We made it as far as San Diego before Sprint.ly and a new puppy caused us to rethink the trip. Our current plan is to head up to Seattle for a couple of months.
  19. Puppy? Yes, we got a puppy. He’s a Lhasapoo and his name is Sheldon.

Another year, another ridiculous amount of travel. I flew over 110,000 miles to two different continents and visited 17 different cities throughout the year. Needless to say, jet lag is my copilot. My year in cities follows.

  • San Francisco, CA
  • Denver, CO
  • Portland, OR
  • San Diego, CA
  • Chicago, IL
  • Grand Rapids, MI
  • Ann Arbor, MI
  • East Jordan, MI
  • Amsterdam, The Netherlands
  • Rome, Italy
  • Hythe, UK
  • London, UK
  • Paris, France
  • New York, New York
  • Austin, TX
  • Las Vegas, NV
  • Orlando, FL
  • Seattle, WA

Looking back while moving forward

Posted on by

A little over the last two years of my live has been devoted to building, funding, and growing SimpleGeo. The experience was, without a doubt, life changing in many ways. It was a crash course in a bunch of disciplines that I knew little to nothing about. Managing clients, iterating on your product, sales, business development, raising money from investors, etc. were all wide open holes in need to my attention and input. It goes without saying that this was not an easy endeavor.

Along with being constantly challenged and learning new things, I was fortunate enough to work with some of the brightest engineers in Silicon Valley. While at SimpleGeo I saw them build a patent-pending distributed graph database built on top of theory that had not been implemented yet in the real world. At scale no less. The operations team built an infrastructure that was scarily automated and resistant to failure. So much so that the SimpleGeo engineers get regular calls to talk about how we did it all on top of AWS.

On October 31st, we announced that we’d been acquired by Urban Airship. I’m extremely excited to see what the combined teams cook up in the coming months. Having locationally aware push notifications is going to allow businesses to engage with their customers in ways they’ve never dreamed. Additionally, I know SimpleGeo’s world class engineering team will be able to help the new company build features at scale that the competition won’t be able to match. The best is truly yet to come from this company and I’m sure Scott Kveton will be a great shepherd moving forward.

As for me, I’ve decided to move on post-acquisition. I need to step away from the echo chamber and spend time focusing on what is important to me in general; not just professionally. To that end, my lovely lady and I have bought an RV and plan on touring around the Southwest this winter. I’d like to visit as many incubators and coworking places as possible. So if you’re in the Southwest and want me to swing by and say hello, please drop me a line on Twitter or via email.

Onward and upwards.

Dear Yahoo!, hire me as your next CEO

Posted on by

News is going around that Yahoo! is looking for a new CEO. I have no idea if this is true or not, but if it is, I would like to announce that I’m ready, willing, and insane enough to go long and go big with Yahoo! as your new CEO. Yahoo! showed glimmers of hope when it bought Flickr and Delicious. It’s been a bastion of some of the most impressive technology of the last 15 years. I believe it can be great again.

Sounds great, how the hell am I going to do it? I’m going to take the $20bn in market cap and build an empire of product and design talent that will be beyond reproach. Then I will give them the support and freedom to do what they do best: innovate.

  1. I’d buy Instagram and put them in charge of both Instagram and Flickr. They would have 100% autonomy over the entire “Yahoo! Photo” division.
  2. I’d buy Soft Facade and run them as an internal design and branding agency for all of our products.
  3. I’d figure out a way to wrestle The Barbarian Group into the fold and put them in charge of all PR and marketing initiatives.
  4. I would buy Twitter and Square in order to bring Jack Dorsey on full-time to run a new division called “Yahoo! Mobile”. He would have 100% autonomy over the entire mobile strategy.
  5. I’d buy Path and With for the sole reason of bringing Dave and his team on to lead the new “Yahoo! Social” division.
  6. I’d buy the NYT (for a mere $1.5bn!) and recruit John Gruber to be Editor in Chief of the “Yahoo! News” division.

Just think of what we could accomplish if we just let amazing people do what they do best.

HOWTO: Spend your investors’ money

Posted on by

I’ve invested in two startups and advise, officially and unofficially, a dozen or so other startups. Recently, a company that I’m involved with, attachments.me, raised $500,000 from Foundry Group. Since their raise, the two cofounders, Ben and Jesse, have been on a tear adding features, solidifying the infrastructure, and ramping things up to a public beta.

Attachments.me is a unique consumer service in that a single user could have gigabytes of data to crawl across multiple accounts. As a result of this unique challenge, Ben has been spending a great deal of time working out how the underlying infrastructure is going to scale. This, of course, involves spinning up a decent amount of servers on AWS. In doing so, Ben was extremely worried about keeping costs down. I had to laugh as the numbers he was worrying about was less than 1% of the total amount raised or, as Chris Lea said, “Your investors didn’t give you the money so you could look at the large balance in your bank account.”

But, it’s a good question, and I get asked it often. How should you spend all that money your investors just gave you? How should you spend that 15% employee option pool? So I set up a Google Form and asked a dozen or so of my favorite investors what they thought. I got six responses from four seed stage investors and two Series A investors. Here’s what they had to say…

  • If you took total monthly burn and divided it by total number of employees, how much would you expect the per-employee burn be? The average response was $12,000 per employee with the majority saying $15,000 was expected. This means if you have 10 employees you should be comfortable with a total monthly burn of between $120,000 and $150,000 per month.
  • What percent of a given round of funding do you expect will be spent on personnel? The average response was 73% of the total round with the majority saying 80% This would indicate that my friends at attachments.me should feel comfortable spending $400,000 on building out the team.
  • What percent of a given round of funding do you expect will be spent on servers and infrastructure? The average response was 18% with the vast majority saying they expect a company to spend 15% of their total raise on servers and infrastructure. If you’re burning $150,000 a month, you should try to keep your AWS bills below $22,500 per month.
  • How much should rent be, roughly, per employee per month? The average response was $666 with the vast majority of investors saying $500. So a team of 10 shouldn’t be spending more than $5000/mo. on rent.
  • How much equity, on average, should early engineers get? Two investors recommended less than 0.5%, which seems extremely low for your first couple of engineers. One said 1.5% to 3%, which I think is fair for your first engineer, but on the high end for your third and fourth engineers. The other three said 0.5% to 1.5%, which seems to be the universal standard when I talk about this topic with other founders in Silicon Valley.
  • How much equity, on average, should an early executive hire get? The consensus, with four investors agreeing, was between 2% and 5% The other two investors thought 1% to 2% was appropriate. My personal recommendation, pre-Series A, would be 1% for a Director, 2-3% for VPs, and 6-9% for CEOs. This, of course, depends greatly on salary and other benefits offered. What I tell people is that I have two dials: salary and equity. Dial one up and the other gets dialed down.
  • How much, if any, of a premium would you expect there to be on burn for
    SaaS and PaaS companies?     One investor said there should be no premium, one said 10%, one said 30%, and three said 20% The 20% number resonates with me as that’s about the premium SimpleGeo has spent on our per-employee burn. In other words, if an investor expects you to spend $15,000 per employee per month, they most likely will be okay with a platform company spending $18,000 per employee per month in total burn. The thinking here is that SaaS/PaaS companies require more infrastructure, better/higher quality infrastructure, more bandwidth, and more senior/seasoned engineers.

These are, of course, rules of thumb, but it should give you a good feeling of where you and your company stands. Your investors put money into your company under the expectation that it’s going to be spent so you shouldn’t feel bad about spending that money.

UPDATE: A lot of people have questioned the $12,000 per month, per employee number. Keep in mind that’s 100% of all burn for the entire company and not just their salary. This includes server costs, travel, rent, office supplies, etc. On average, an engineer in Silicon Valley will have a base salary of $100,000 a year. Add roughly 20% for benefits (healthcare, vacation, payroll taxes, etc.), $3k every two years for hardware, rent ($500/mo.), etc. and you’re at $10,800 per month just to pay for them to walk in the door. I doubt it’s too hard to imagine spending $1,200 per employee on travel, servers, office supplies, etc.

The Cloud is not a Silver Bullet

Posted on by

There has been much brouhaha over the recent EBS outage. I say EBS to be specific as this, despite the sensationalistic headlines, was not an AWS or EC2 outage. RDS was also affected as it is built on top of the EBS product. As has been reported just about everywhere, the outage affected many large websites that are built on top of AWS. Much of the banter was mainly around how the “AWS outage” had “brought down” these sites, which couldn’t be further from the truth. What brought down these services was poor architectural choices combined with a lack of understanding of the unique characteristics of cloud infrastructure.

Our friends at Twilio and SmugMug, along with SimpleGeo, were able to happily weather the storm without significant issues. Why were we able to survive the storm while others were not? It’s simple; anyone who’s spent a lot of time talking with the fine folks at AWS or researching the EBS product would know that it has a number of drawbacks:

  • EBS volumes are not anywhere near as fault tolerant as S3. S3 has 11 9’s of durability. EBS volumes are no more or less durable than a simple RAID1.
  • EBS volumes increase network IO to and from your instance to some degree and can degrade or interfere with other network services on a given instance.
  • EBS volumes lack the IO characteristics that many IO-bound services require. I’ve heard insane stories of people doing RAID0 across dozens of EBS volumes to get the IO performance they’re looking for. At SimpleGeo, we RAID0 the ephemeral drives and split data across multiple ephemeral RAID0 volumes to get the IO we’re looking for.

This doesn’t mean that you should avoid EBS or that EBS is a bad product. Quite the contrary. I think EBS is a pretty fucking cool product when you consider the features and simplicity of it. What it does mean is that you need to know it’s weaknesses in order to properly build a solution on top of it.

I think the largest surprise of the EBS outage, and one that no doubt will be reviewed in depth by the engineers at AWS, was that an EBS problem in one AZ was able to degrade services in other AZs. My general understanding is that the issue arose when EBS in one AZ got confused about possible network outages and started replicating degraded EBS volumes around the downed AZ. In a very degenerate case, when EBS can’t find any peers in their home AZ, they attempted to reach across other AZs to replicate data to other peers. This led to EBS being totally unavailable in the originally problematic AZ and degrading network-based services in other AZs as this degenerate case was triggered.

All this being said, what is so shocking about this banter is that startups around the globe were essentially blaming a hard drive manufacturer for taking down their sites. I don’t believe I’ve ever heard of a startup blaming NetApp or Seagate for an outage in their hosted environments. People building on the cloud shouldn’t get a pass for poor architectural decisions that put too much emphasis on, essentially, network attached RAID1 storage saving their asses in an outage.

Which brings me to my main point: the cloud is not a silver bullet. S3, EC2, EBS, ELB, et al have their strengths and weaknesses, just like any piece of hardware you’d buy from a traditionally enterprise vendor. Additionally, the cloud has wholly unique architectural characteristics. What I mean by this is that the tools AWS provides you need to be assembled and leveraged in unique ways that differ, sometimes greatly, to how you’d build a system if you were building out your own datacenter.

The #1 characteristic that people fail over and over to recognize is that the cloud is entirely ephemeral. Everything from EBS, to EC2, to EBS volumes could vaporize in an instant. If you are not building your system with this in mind, you’ve already lost. When you’re dealing with an environment like this, which by the way most large hosted services deal with, your architecture needs to exhibit a few key characteristics:

  • Everything needs to be automated. Spinning up new instances, expanding your clusters, backups, restoring from backups, metrics, monitoring, configurations, deployments, etc. should all be automated.
  • You must build share-nothing services that span AZs at a minimum. Preferably your services should span regions as well, which is technically more difficult to implement, but will increase your availability by an order of magnitude.
  • An avoidance of relying on ACID services. It’s not that you can’t run MySQL, PostgreSQL, etc. on the cloud, but the ephemeral and distributed nature of the cloud make this a much more difficult feature to sustain.
  • Data must be replicated across multiple types of storage. If you run MySQL on top of RDS, you should be replicating to slaves on EBS, RDS multi-AZ slaves, ephemeral drives, etc. Additionally, snapshots and backups should span regions. This allows entire components to disappear and you to either continue to operate or restore quickly even if a major AWS service is down.
  • Application-level replication strategies. To truly go multi-region, or to span across cloud services, you’ll very likely have to build replication strategies into your application rather than relying those inherent in your storage systems.

Many people point out that Amazon’s SLA for it’s AWS products is “only” 99.9% What they fail to recognize is those are per-AZ numbers. You can compound your 9’s in a positive manner by spanning multiple AZs. This means that the simple act of spanning two AZs takes you from 99.9% uptime on EC2 to 99.9999% If you went to three AZs, with an appropriate share-nothing architecture, you’d be looking at a theoretical 99.9999999% uptime. That’s 0.03 seconds of downtime a year.

This all being said, I think it is fair to give Amazon a hard time for a couple of things. Personally, I think the majority of the backlash this outage caused was driven by a few factors:

  1. Amazon and the AWS team need to more clearly communicate, and reinforce that communication, about the fallibility of each of their services. For too long, Amazon has let the myth run rampant that the cloud is a magic silver bullet and it’s time to pull back on that messaging. I don’t think Amazon has done this with malice, rather they’re a victim of their own success. The reality is that people are building fantastic systems with outrageously impressive redundancy on top of AWS, but it’s not being clearly articulated that these systems are due to a combination of great tools and great architecture – not magic cloud pixy dust.
  2. Amazon’s community outreach and education needs to be much stronger than it is. They should have a small army of AWS specialists preaching proper cloud architecture at every hackathon, developer day, and conference.
  3. A lot more effort needs to go into documenting proper cloud architecture. The cloud has changed the game. There are new tools and, as a result, new ways of building systems in the cloud. Case studies, diagrams, approved tools, etc. should all be highlighted, documented, and preached about accordingly.

So what have we learned? The cloud isn’t a silver bullet. You still have to build proper redundancy into your systems and applications. And, most importantly, you, not Amazon, is ultimately responsible for your system’s uptime.

Netflix is hiring social data/integration nerds

Posted on by

I’ve previously used my blog to talk about a lot of things in technology, rant about politics, and ramble on about meaningless trivia. Today I’m trying something new; talking about one of the many amazing job opportunities I’ve recently heard about in Silicon Valley. I’ve built up a somewhat random reputation for helping to connect companies with technical talent. I love helping people, I love nerds, and I’m enthusiastic about helping companies succeed, so it comes pretty naturally to me.

Which brings me to a rad opportunity I recently found out about …

I’m an avid fan of Netflix and I use the service daily on my Apple TV. I’ve met a few people who work there and recently ran into Mike Hart from Netflix and he mentioned that he’s working on forming a team to focus entirely on building out Netflix’s social strategy. He said that the company views this as a big opportunity. You can’t help but hear Mike out on these kinds of initiatives. After all, he was the guy who created, incubated, and led the Netflix API team. You may recall that this small team now powers about 200 retail devices on the market and is the 5th largest API in the world with 20 billion calls a month.

What’s not to like about Netflix? They have a CEO that Fortune named Business Person of the Year. They spent $1m on the Netflix Prize to improve their algorithms. They’re the fastest growing public company in Silicon Valley. They have a cultural thesis that is widely emulated around the world. Hell, they’re even licensing original TV shows now!

So what, exactly, would you be doing if you’re lucky enough to land this gig? Well, there are two opportunities open: a senior-level Senior Facebook Integration Engineer, and a Lead Test Engineer for Social Systems. The team you’d be joining would be small, maybe four to six people in total, and you’d be, basically, figuring out ways to integrate Netflix into the social services you love and use every day. Long term you’ll be helping to figure out how the social graph can be leveraged to to help you, and your friends, discover great TV and movie content. Your daily hacking session would include writing Java, wrangling servers in AWS’s cloud, and munging data using Apache’s Cassandra project. You test engineers can expect lots of automation and debugging. The best part is that those pesky UI/UX tests are handled by a separate team! You’d be focused solely on continuous integration, testing, and triaging the underlying social infrastructure fabric at Netflix.

If working for an amazing leader, at a company with an outstanding culture, working on complex problems with millions of users and tons of data wasn’t enticing enough, I’ve been assured by Mike that you’ll be highly compensated. How highly? Well, Netflix, being the awesome company they are, give you a total compensation number and then allow you the flexibility to choose how you want that portioned out (e.g. If you want more options, they’ll lower the salary or if you want better healthcare/401(k)/benefits, they can move around options and salary). Total compensation? Let’s just say I’ve confirmed it will definitely be at top of market (In fact, Mike caught me off guard with their compensation; it’s seriously the best I’ve seen in Silicon Valley).

Engineers interested in learning more about this opportunity should contact Mike Hart for more information. If you know of anyone interested, please pass this along. Additionally, if you tweet out the link to this blog post, I’ll be randomly selecting two people to receive 6 month streaming subscriptions from Netflix for free.

Disclaimer: I have not, in any way, been compensated to write this post. If, in the future, people do compensate me for writing up job opportunities, I’ll be sure to let you all know. Mike gave me a few talking points and made sure this post wouldn’t anger Netflix, but otherwise the words and details outlined are my own.

Year in Review

Posted on by

This year I spent a lot of time iterating on SimpleGeo’s products, working with partners, listening to customers, recruiting employees, and trying to establish an open culture at the company. Being a founder of a startup is an extremely stressful job. I tell the people at SimpleGeo that I hope they all found a company someday so they’ll know how awesome it is and how terrible it is to be a founder.

The rest of my year was filled with travel, good friends, good food, time with Diana, and the usual insanity. A short recap follows.

  1. I started the year with a trip to Singapore the Global Leaders 2010 summit. It was a pretty surreal experience debating the relevance and consequences of metadata with the former CIO of Google and the Chief Privacy Officers of Microsoft and Oracle. Extremely enlightening debates.
  2. I tried to spend more time blogging. Most of it was about startups, engineering culture, and entrepreneurship: Fail fast and often, HOWTO: Recruit Rock Stars, HOWTO: Maintain a Rock Star Culture, and Why I’ll never own another server.
  3. In February I met a girl on OkCupid of all places. She’s improved my life greatly and I’m happy she continues to put up with me.
  4. I also went to Atlanta in February for PyCon. Mike Malone and I gave a talk on using OAuth, OpenID, and other open web technologies in Python and Django. If you’re into OAuth and use Python, check out my python-oauth2 library.
  5. March brought my 30th birthday and SXSW. I spoke on a panel with a bunch of really smart people about how location and location-aware applications are going to change the face of social networking, mobile, and the internet.
  6. I went out to Dublin for Funconf in April, despite it being canceled by a volcano, and ended up in Amsterdam for Queen’s Day, which is basically the Dutch version of St. Patrick’s Day.
  7. SimpleGeo raised $8.2m in June of this year to ramp up operations and bring simple tools for location-aware applications to market.
  8. Diana and I went to South Africa for the TECH4AFRICA conference. The trip was intense overall. We met a ton of great people, hung out in Johannesburg, saw Soweto, went to Cape Town, and visited a game reserve.
  9. The Madikwe Game Reserve was one of the highlights of the trip to South Africa. Waking up to monkeys on your porch and zebras wrestling behind your cabin is a surreal experience.
  10. At the beginning of September I moved back to San Francisco. I’ve given up attempting to move anywhere else at this point. San Francisco is home now and that’s fine by me.
  11. September also brought my first two angel investments, which sounds really weird to be saying. I placed small bets on StyleSeat and Kiip. They’re both ramping up operations and doing well. Excited to see what Melody and Brian cook up in 2011.
  12. I went back to Dublin to attend the postponed Funconf and also went to London to speak at Future of Web Apps. I’ll be speaking at the Las Vegas Future of Web Apps this summer as well.
  13. We hired one of my mentors and good friends, Jay Adelson, to be CEO of SimpleGeo in November.
  14. In November I made good on my promise to get a geek tattoo and to get an old school Americana tattoo by getting a single tattoo that professes my love for the Internets.
  15. November also brought a happy ending for a startup I’d advised from nearly inception. ngmoco:) sold to DeNA for $400m. Huge kudos to the team over there. Incredible watching them grow from 9 employees to over 100 in two short years.
  16. A project that I started over a year ago, gained steam through 2010, gained a team, and was released into private beta in early December by Jesse and Ben. attachments.me aims to give you advanced tools for mining the unstructured data locked up in your email inbox.
  17. Just in time for 2010, I started work on finishing my left sleeve. It’s going to be Mechagodzilla vs. Godzilla fighting over a fictitious skyline comprised of buildings from cities I’ve lived in.

Despite thinking there was no way I could top last year’s travel schedule, I somehow managed to set a record for miles traveled, sights seen, and continents stepped on. I traveled less days this year with “only” 137 days on the road, but managed to travel 124,392 miles (200,189km), visit 10 countries, 4 continents, and 32 cities over the course of 29 trips. My year in cities is as follows.

  • Boulder, CO
  • San Francisco, CA
  • Seattle, WA
  • Austin, TX
  • Ann Arbor, MI
  • Lansing, MI
  • Detroit, MI
  • Singapore, Singapore
  • Atalanta, GA
  • Johannesburg, South Africa
  • Cape Town, South Africa
  • London, UK
  • Dublin, Ireland
  • Amsterdam, Netherlands
  • Kansas City, MO
  • Toronto, Canada
  • Salt Lake City, UT
  • Glenwood Springs, CO

The World Famous Chicken Zigafoos Recipe

Posted on by

About a decade ago my mom got a recipe from a friend called “chicken zigafoos.” These absurdly named dinner pastries have been a big hit in the family since then. So what, exactly, is a chicken zigafoo? Well, it’s kind of like a fat kid chicken pot pie. With gravy. To really get the full effect of the chicken zigafoo I highly recommend pairing it with homemade chicken gravy and my mashed potatoes recipe. What follows is a step-by-step manual for making this delicious dinner.

Chicken Zigafoos

Ingredients

  • 2 healthy sized chicken breasts without skin.
  • 1 red onion.
  • Some celery.
  • 2 chicken bullion cubes.
  • Salt, pepper, garlic powder.
  • 2 8oz packages of cream cheese.
  • 3 packages of croissant rolls. In the US, Pillsbury has apparently decided Americans are too stupid to pronounce or spell croissant so they go by, I shit you not, “crescent” rolls.
  • 1 stick of melted butter.
  • 3 tablespoons of milk.

Instructions

  • Cut up about 1/4 cup of celery and red inion (or shallots or white onion or whatever) and put it into a glass casserole dish with the chicken breasts.
  • Add the chicken bullion cubes and seasoning as you wish.
  • Add two cups of water to the casserole dish. Be sure to save this for your chicken gravy!
  • Bake in the oven for an hour at 350F (177C).
  • Take the chicken out and shred it.
  • Strain the celery and onion out of the chicken stock into a sauce pan. Add the celery and onion to a bowl with the shredded chicken.
  • Add the melted butter, milk, and cream cheese to the bowl and mix thoroughly into a cream cheese and chicken paste of sorts.
  • Roll out the croissant/crescent rolls out flat. You’ll notice they’re notched into four squares that are notched into two triangles each. Use two triangles (one square) for each zigafoo. Pinch the notching of the two triangles together so you get four rough squares. Add a large dollop (1/4 cup or so) of zigafoo innards to the middle of the square and then pinch up the four corners over the top of the dollop.
  • Bake on a cookie sheet per the instructions on the croissant roll packaging (usually 375F for 13-15 minutes).
  • Place onto a large serving plate for serving.

Mashed Potatoes

Ingredients

  • 4 large russet potatoes.
  • A cup of sour cream.
  • 6 tablespoons of butter.
  • 2 chicken bullion cubes.

Instructions

  • Peel the potatoes. Leave a little skin on if you like.
  • Cut the potatoes into small squares (1” x 1” or smaller).
  • Throw the cut up potatoes into a pot and add water until the potatoes become buoyant in the pot.
  • Add the chicken bullion cubes to the water and heat the pot until the water starts to boil. Boil the potatoes for a minimum of 10 minutes. Test with a fork to ensure potatoes are soft enough to easily mash.
  • Strain water out of potatoes.
  • Add butter and sour cream to pot and pour the boiled potato cubes on top. Season as preferred with salt and pepper.
  • Mix and mash to your preferred texture.
  • Dump the mashed potatoes into a bowl for serving.

Chicken Gravy

Ingredients

  • Chicken stock leftover from the chicken zigafoos.
  • 2 tablespoons of corn starch.
  • 1/2 cup of water.

Instructions

  • Add the chicken stock to a sauce pan and bring to a boil.
  • Mix the corn starch with the water until the corn starch is thoroughly dissolved into the water with no clumps.
  • Slowly add the corn start/water mixture to the boiling chicken stock one teaspoon or so at a time. Mix thoroughly into the chicken stock before adding more.
  • Once done pouring into the stock, boil the gravy for about a minute.
  • Pour into a gravy dish for serving.

That’s about it. I highly recommend putting a zigafoo or two on your plate, a healthy helping of mashed potatoes next to that, and then cover the whole thing in chicken gravy. You’ll thank me later.