Programmer for Hire

[UPDATE: To all those who got here by clicking a link entitled "Why I Will Never Feel Threatened by Programmers in India", that title has been retired as an unfortunate misfire on my part: much of the conversation was lead astray as though I meant to single out India in particular, and it landed for some as having racist connotations.

It is now titled "Why I Will Never Feel Threatened by Cheap Overseas Programming", which much more aptly captures the meat and message of this essay.  And now without further ado...]

I got a call from a friend of a friend the other night. It was a fellow with whom I’d talked 11 months ago about a project he and his partner were looking to start. We established then that I wasn’t the guy for him, that I was likely too expensive for their big-dreams, small-means budget.

Fast forward to present day: their project is still not launched, it’s still not right. They’ve paid for something between 600-700 hours of development with a firm in India, and they should have launched 6 weeks ago.

Sure they’re only being charged $14/hour for that work, but I think the Indian firm is, as the saying goes, making up for it in volume.  And that’s to say nothing of the time our heroes have spent proctoring the whole process: in his words, they’ve got to be constantly super explicit in their instructions, or it comes back wrong–then then have to spend more hours getting it fixed (this is but one instance of why I am suspect of the hourly model in my industry).

“I’m wondering if you’re available–my partner says we just need an American programmer to get in their and clean up a few things to get us out the door, we figure it would take the right person 10 hours, instead of 50 or more with these guys.”

Outsourcing programming came in to vogue as a cost-effective strategy, what, 10 years ago?

For a number of years in my industry there’s been a sense of not-necessarily impending doom, but something akin to that due to the rise in outsourcing programming work abroad, most prominently to India.  Hoards of smart people, as the alarmist vision goes, willing and able to do highly talented technical work for a fraction of the going rate for US programmers.  The labor force here in the states can’t possibly compete, and what savvy, responsible company could possibly pass up the opportunity of such cost-effective globalization?

The thing is: I have yet to see a project done overseas at that sort of hourly rate that has actually gone well.  Version 1 of MonsterMarketplace was first done overseas–took 4 months, billed for gobs of hours including a lot of testing and QA, and crashed predictably under the strain of campaign traffic.  (By contrast, my complete rewrite of the front end was done in 4 weeks–at peak times it produced about 4% server CPU utilization, never crashed, and had correspondingly faster page loads.)   This was my first inkling that the “outsourcing problem” wasn’t all it was cracked up to be.

My second close look at the outsource phenomenon was the code base for Zowzee.  As I discussed in the tale of building and launching SpotlightDenver.com, rebuilding it made more sense than trying to extend the $12/hour piece of work (to this day I’ve been too polite to ask how many of those $12 hours it took–I do know that my work was vastly cheaper, faster, and smoother).

And now this.  That makes three out of three instances in which outsourcing abroad turned not that great, if not outright regrettable¹.  This is a small sample, to be sure, but still serves to bolster my confidence as a US-based programmer.

Reclaiming our Competitiveness

So how is it that we can compete with programming talent overseas, with figures like $12/hour stacked against $125/hour?  My experience leads me to believe that US-based programmers can be the sound and “right” choice for development projects, and we don’t even need to resort to protectionist arguments for supporting our local economy & talent.

I think we can observe a few generalizations which, while they may not give due credit to the exceptional firms overseas, serve to counter the equally general notions that hiring offshore talent is an economical no-brainer.

Consider the idea that the rock-bottom hourly rates creates a certain cognitive dissonance in decision makers.  If I tell you that your job is going to take 500 hours, but don’t worry, it’s only $14/hour, you might think it a god send that such a rate is even possible–you might even think it not worthwhile it to consider US based options at $50+ per hour.  Sure, the quote from India might be a little inflated, but it couldn’t be that inflated… right?  Turns out it was in my 3 examples.

But even if multiplying hours times rate gives you a real bargain, there’s a very good chance the hours term will balloon.  If they get something wrong, they can fix it with 20 more hours of work.  But don’t worry, it’s only $14/hour.  Again, impaired judgment based on what a great deal you’re getting makes this palatable and thus probable: you wouldn’t tolerate it if you were paying a substantial hourly, and besides, you’re already in 500 hours, so what’s a few more?  I think it’s safe to presume that overseas firms understand this reasoning and mindset very well, and at very least appear to exploit it accordingly.

Our heroes’ need to be super explicit in order to get what they want reveals another major advantage that local programmers have: nobody wants to have to babysit their programmer with constant direction and correction.  Communicate the overall vision of what you’re trying to create for your customers, and any programmer worth their salt will bring their A-game to solve it from that shared understanding.  The net result is a project that is completed faster, racks up far fewer billable hours, and saves you headaches and time.

I’ve noticed a number of other reasons to think twice about off-shoring:

• real-time communication made inconvenient and response times made long by the time zone difference,
• a reduced sense of accountability, commitment and partnership inherent in the long distance relationship,
• and text like “Link will be sent to your mail for to update your Password.” sprinkled throughout public facing parts of the website, which just doesn’t give your customers the best impression of you and your business.

These all come with the turf but seem seldom considered: the allure of cheap labor seems to trump these potential problems, or they are unimagined all together.

The good news is that each of these reasons constitute a means by which US programmers can justify their value as a worthwhile and economically sound alternative to outsourcing.

There’s fruit to all of this on more idyllic grounds, too: just as the threat of overseas outsourcing would tend to discourage mastery of programming here in the US (what’s the point when you’re told that someone else is going to eat your lunch?), the notion that the fight is more fair than typically envisioned may as well encourage it.

I believe when the myth of cheap, abundant programming abroad is shattered, there is a lot more reason for the next generation of talent to take on and excel at a craft that creates real value.

[UPDATE: There has been a lot of comments and conversation on this topic here and elsewhere.  See the response to many of the consistently raised issues in the follow up post, Cheap Overseas Programming Revisited.]

View the discussion on Hacker News or Slashdot or Reddit.

Notes:

This is project that was a long time in coming, and in a short time in doing.

I first became acquainted with the Zowzee team back in January when I went in to pitch on their then new project, DealNation.  Frustrated with the progress and process of working with a firm in India, they wanted to know if, in addition to building DealNation, I could expand their Zowzee application (built specially as a deal-of-the-day site) to be more of a general purpose marketplace of deals: let deals run for longer than a day, enable visitors to peruse and purchase from a collection of deals, and so on.

Tempting as it would be to pick up the extra bit of work, after poking around the code I had no interest: it was buggy, carelessly architected, and loaded with code duplication (meaning if you wanted to change something, you better be sure you found and updated every instance of it, or face new bugs and inconsistencies).  I couldn’t responsibly quote prices or time frames for any non-trivial work, to me it would all be subject to pitfalls and gotchas.

I said to Paul with earnest sympathy and apology (because I know it sucks to get the kind of news I had to give) that it would need to rebuilt if I were to make any substantial changes on it (I mean, who wants to hear that 3 months after you launch?).  I scoped it out and wrote it up, and estimated a solid two weeks to do everything and add in the big new features.  “But at that point” I told him, “you’ll have a super solid foundation from which point I’ll be able to make it do whatever you want, easily and without concern of introducing obscure bugs.”

Paul was open to it (a real testament, I think, to how it was going with the firm in India) but it was not in the budget at the time, thus Zowzee would trudge along in its current state for a while longer.

This month, with me now in house as CTO of DealNation, we found gap enough in my usual duties to direct our attention back to the Zowzee redesign: our sights were now to reincarnate it as SpotlightDenver.  Paul took the initiative to mock up some fab wire frames that really gave a clear vision, hired some design help to give me some pretty pixels to work with, and we were off.

Centered around the concept of a marketplace of deals, I built an e-commerce application nearly from scratch that was perfectly tailored to the needs in just over two weeks amidst usual DealNation duties.  The end result is www.spotlightdenver.com, launched Monday, November 21th.

My team tells me that the whole thing went smoother for them than launching some of the deals on Zowzee, let alone the whole Zowzee site.  And the best part for my team is me delivering on my original promise of the rebuild: we’ve now got a super solid foundation that is easy to extend upon, and since launch I’ve been taking requests and building out new features that make everyone’s life easier and better.

The whole experience has reaffirmed for me that yes: though generally unpleasant news upfront, sometimes you gotta rebuild.  The good news is the other side can be quite gorgeous, workable, and profitable.

I like to create games.

Games that motivate.  Games that create focus, pressure, supreme concentration.  Done right they can be a fantastic hack for programmer productivity.

A few weeks ago I set out to build a working prototype of the DealNation members zone in a single day.  To make it interesting, I let everyone in on the game.   At 9:45am I setup my day as a collection of mini-milestones in building the system from start to finish, each with a target completion time, and put the progress chart + time line on the office whiteboard for all to see.

I had declared my fate for the day, and effectively given my word to “this is what will get done with every passing hour today”.  Folks were curious to get that kind of high-resolution look into how I work, and I promised they could view the work in progress as I pushed each mini-milestone to our dev server.

To make it more real and pressing, I openly advertised my progress throughout the day.  When a milestone was completed I would rise from my desk, write the actual completion time on the whiteboard, and settle back into my trance-like concentration at the computer once again.  No chit-chat, I was on a mission and everyone knew it.  Green marker meant I was on time, red meant I was behind.

At the end of the day, this is what everyone in my office got to see:

Yep, I gave it a zany, over-the-top name. Whatever, it's my game.

There are a lot of payoffs to this structure.  First, this day was a ton of fun for me.  It was like being in the zone of pure productive concentration for a hour, 7 times over.  I had zero problem blocking out distraction, and the mini-race to the next milestone keyed me in with renewed focus each time (it’s WAY fun to write the next actual time on the board, and striving to use the green marker keeps it interesting).

The final show off was the grand prize.  My team was delighted to see all that got accomplished in just one day, and I think their sense of appreciation was heightened by being vicariously in on the process along the way. This is about as close I know to the experience of performing for an audience as a programmer, and it creates a certain buzz which theatrical and musical performers will understand.

I’ve been invited to join the team of DealNation as their CTO.

This is a very good fit in a lot of ways, most of all because it has me doing my magic in a way that lends rocket fuel to a big venture that’s already got a lot going for it.

In retrospect there are 3 things I would say I did very right to precipitate this opportunity:

1. I set the bar for my performance very high from the very beginning.  The first impression I made on the DealNation team was an act of remarkable ability, and I, with confidence bordering on arrogance, stated plainly then that that’s just how it goes with me.  I’ve been delivering on that tall promise ever since.
2. I gave them a taste of what it’s like to have me in house.  When I offered the free day of me working with them side by side on my laptop, they got to experience the luxury of immediate communication to bounce around ideas, and the velocity at which development can get done when you sidestep the spec’ing/bidding/approving cycle of contract work¹
3. I shared with them a single-sheet write up of the value of me being their CTO. Drawing heavily from the coursework of Ramit Sethi’s Earn1K, I took the time to really articulate the how and why I could be of most value to DealNation in the CTO capacity.  Then I unabashedly shared as much.  Two months later, when the time was right, they came around to taking me up on that offer.

The path that we’ve taken together since January has us both clear that we’re a fantastic fit, and thus I’ve accepted the position and begin on July 11th.  Exciting times ahead, I can hardly wait to weave more web dev magic into everything they (soon to be we) are doing.

Notes:

“Dude, that was a total baller move.” my friend Eliav reminisces of it, months later.

This refers to the manner in which I first met the good people behind a young company here in Denver called DealNation back in January.  As a close collaborator to the project, Eliav got me invited to pitch myself on developing their system.

Things moved fast this particular week in January.  While hosting and hanging out with Eliav on Tuesday night, he loaded onto my browser a few mock up images of the system.  During that same evening I got a call inviting me to come to the office and pitch on the project.  We would meet the next day, at noon.

Fast forward to 9am on Wednesday.  I’m all ready to go, and have about 3 hours to kill before I meet with some strangers to convince them that yes, I have web development superpowers.

An idea hit me.  I checked my browser’s history from the night before looking for the mock up images.  Bingo.  Instead of pacing about for 3 hours, psyching myself out for what is in essence a job interview, I would spend the time building a working prototype of the system as a real-as-can-be demonstration of what 3 hours of John Larson time are worth.

It was an inspired 180 minutes.  I fired up Photoshop, sliced up the mockups, designed a suitable database, whipped up the 3-step front end user experience, and even created an administrative back end for configuring facets of the system and viewing results of user visits.  A quick deployment to a dev URL and I was out the door for my noon meeting.

Like any such meeting I was met by the usual platitudes of greeting and new acquaintance.  “Hi John”, “Good to meet you”, “Thanks for being here”, “How’s it going?”

I was waiting for that last one.

“Good”, I said, “I’ve had a fun morning.  Now I’ll be honest to say that Eliav shared the mock ups you guys are working on with me last night, so I’ve had a little advance info on what you guys are up to.  I had time to kill before this meeting since 9, so I cooked up something I think you’ll find interesting.  May I show you on a computer?”

As I punched in the dev URL in the address bar, I continued to say “So I built your system.  I mean not all of it, and it’s not the air-tight, bullet proof masterpiece that you could expect with a more thorough effort, but enough to give you a good sense of what I’m capable of in a few solid hours.”

I let them play around on the front end, and had them log in with their own username and password in the back end and see the several controls and reports available in this hurried piece of web work.

I went on to narrate: “Now I understand you guys are used to paying twelve bucks an hour with India, but perhaps you will find me worth the roughly $125 I charge¹–you get a bit more bang for your buck, I reckon.”

With them clear that I was the real deal, the rest of the meeting was a fun, free flowing exchange of ideas and everyone getting to know each other.  Again, this was Wednesday.  Later that afternoon I had a proposal with firm price tag in their inbox.  Thursday I had the thumbs up for the project, and on Friday I dropped by to pick up my project inception check.

The powerful first impression set by those 3 hours of work set the tone for our entire working relationship, namely “you can and should expect miracles of me that will defy what you’ve been trained to expect in software development”.  By the gesture I deliberately set the bar for myself very high, and to our mutual delight: they get to enjoy ultra-high performance out of me on a regular basis, and I get to delight in the opportunity and rewards of providing it.

Notes:

Just for show and tell, here’s a tidy example of what web application development prowess can achieve to bring more life and vigor to a project:

http://www.beingawesomeisawesomefest.com/

Being Awesome is Awesome Fest is something I created as part of coaching a leadership program with the folks at Landmark Education.  Using the fantastic design work of Anne Richardson, I took about 2 days to create a website to power the project: user profiles, project submissions, status updates, email communications.

I stood in the office of my clients having just wrapped up a successful quick visit, and gave a strategic, pensive pause.

“So… you guys get a free day.” I said plainly, as though what I meant was perfectly clear and well established.

It wasn’t of course, and immediately aroused the earnest curiosity that I was aiming for.  “What’s that mean?” asked one of my clients, as if right on cue.

“A free day.  Meaning I’ll come into the office at 9am prompt, laptop in hand, and be at your disposal to bang out whatever new features, tweaks, enhancements, or anything else you’d like to add to your system.  My super powers are yours to wield for a full day, no charge.  I totally dig working with you guys, and this is my way of saying thanks for the opportunity.”

(As an aside, let me point out that our working relationship had thus far been 100% based on me assigning dollar amounts to chunks of work (projects, features, enhancements, etc).  Hourly availability was never on the table, as I don’t really believe in that model for the kind of work that I do.  So to have me in the office to perform whatever work I could in a solid 8 hour day was quite the novelty, and a rather valuable one at that.)

“It’s a chance for you to get whatever enhancements or tweaks made to have you really love the system without worrying about the cost.” I went on to explain.  “Having me in house will enable me to quickly identify how you’re using things and what’ll make them even better for you.”  Eyes lit up at the prospect, they were just about to go live with the system for the first time, and having me after a few days of really using things was in their estimation perfect and fortuitous timing.  We picked a day during the following week.

“It’s good form to buy me lunch, during that day.” I said with a smile as I walked out, “Keeps me at my desk working on stuff for longer.  Oh, and for best results I also recommend having a spare monitor available… I can get more done when rockin’ the dual screen.”

The day itself was a hit.  My second monitor as well a USB keyboard (“in case you prefer it to your laptop keyboard) were there and waiting in my conference room setup when I arrived, and the whiteboard was loaded up with a huge list of check box-adorned to-dos.  A spirit of “yay, John’s in the house!” was palpably in the air.  I was well taken care of and lunch ordered in was delicious.

I say this all not to boast, but to point out how the human element can massively work in everyone’s favor in programming.  I’m human, so the excitement that I’m in the house pumps me up, which gives me massive focus and determination to have a rockin’ day, which creates superb results.  Programming at its purpose-filled, invigorated best.

And when 5pm rolled around, what was the net result of this 8-hour generosity?  My client loves their system even more, have gotten massive value out of the programming I did, and appreciates the heck out of me for it.  I walk on air as I leave the office after having the sensation of being a hero for a day, I’m better related to what my clients are trying to accomplish which henceforth makes me a better consultant to them, I’ve got a bunch more billable work to perform to wrap up and extend upon various bits that got done, and I got free lunch.

A day in the life of a programmer well spent.

I had an enjoyable meeting with a peer of web dev wizardry the other week.  He was illustrating a development framework for web apps that he created, talking me through the many layers of the architecture.  While enjoying the discourse there was one small thing that he said that I thought profound: “no one’s a master of the entire [web technology] stack”.  He said it so matter-of-factly, no big deal, just a transition sentence to frame an illustration of the beauty of one smart abstraction or another in his framework.

No one is a master of the entire web tech stack.

(For those of you not familiar with such nerd speak, the “stack” we’re referring to is the several varied technologies, each layered upon the previous, which collectively comprise and power web applications.  Server runs upon operating system, database runs on a server, business logic uses database, and so on.)

No one is a master of the entire web stack. Conventional wisdom.  Self evident.  But of course.

I disagree. My musings in the last few months have led me to curiosity about my staunch opposition to conventional wisdom that [non-trivial] application development requires a large team, large budget, and long time lines: am I full of it?  Is there something I’m missing? Where’s the loophole?

Mastering the entire stack might be that loophole.

(A quick technical caveat for the nerdy purists in the house: I don’t pretend that I’ve mastered the literal entire stack, for example I know only a little about the L in LAMP stack¹.  But since you do things nearly identically on the so-called WAMP stack (swap in Windows for the operating system), I don’t think my craft suffers for it.  This suggests there are some shortcuts one can take while effectively maintaining mastery over the technologies that make a difference when creating web applications.  Similarly, one can get by with only crude [or even zero] knowledge of, say, hardware level logic gates and assembly language, which are also part of the rigorously defined entire stack.)

For this discourse I say the entire web stack that matters in web application development is comprised of: database, server side language, HTML, CSS, JavaScript, and enough knowledge about server software and hardware performance to not be blindsided by rookie mistakes nor dumbfounded by scalability challenges.  That’s it, and there is plenty of depth and mastery in each of these layers.  Regarding this web stack, my friend was approximately right: developers who are masters of it all are super rare².

This I think may be my loophole.  In a world where designations like “database admin”, “front end developer”, and “back end developer” comprise commonplace job titles, it’s obvious that most teams and individuals in my trade are sold on segmentation of skills along slices of the technology stack as the best way to do it.  Specialization and depth over broad, general skill sets.  Makes sense, our civilization has been going in that direction for centuries.

But does it need to be?  It’s certainly not ideal: communication overhead and inflexibility makes it easy for me to best most teams on price, time and quality, I have enough client horror stories to assure me of that.  So then is your average developer stuck in a niche, unable to branch out their mastery because it’s too much to handle?  Am I just a genius?  I don’t think so.

I think above all it’s a matter of just trying.  Overcome that little voice in your head that says “who am I to design a schema without the fancy title of Database Administrator?” and just do it³.  Play with a nice JS framework and make some cool AJAX stuff happen on the client side of things, most of the hard stuff is done for you anyway.  Learn a few CSS tricks and experiment with making things pretty instead of stark utilitarian.  I bet the average pigeon-holed developer could explore in other parts of the stack and be at least baseline proficient in all of it with just a few weeks of dabbling on the side.  It’s those Renaissance-type developers that you really want.

Notes:

I recently had a good sense to add this new boilerplate section to my standard project contracts:

3.5 RIGHT TO DEMAND SATISFACTION

Acceptance of this agreement includes the right to demand satisfaction for all described features, no more, no less.  I’m on the hook to complete all the described features to the point that you love it, and you get to demand satisfaction to that extent.  So do it.  If you’re not happy, and you haven’t exercised this clause, you’re technically in breach of this agreement.  Just sayin’.

In my years of experience I have never been burned putting myself on the hook called “my work is not done until you love it” , so I realized that I may as well get the benefit of advertising as much up front.

I don’t know that this sort of clause exists anywhere else in my industry.  Flippant language aside (which in my estimation is rad: if that sort of playfulness backed up solid performance turns you off, you’re not my client anyway), I think there’s a general fear about having to appease some fictional, nightmare client who is endlessly demanding.  In my experience, when it comes to my world of web programming, they don’t exist¹.  Oh sure, there’s always room to want more features, or a cheaper price, but that’s not what what’s on the table here.

This here is a promise to do great with all of the [meticulously outlined] features within the scope of work at hand.  Software development is generally a complex endeavor so I think the average bear understandably shies away from such a tall promise.  Fair enough.  I embrace it.  It keeps me honest and fosters a healthy pride in my craft, and if I just bring my art to it using my full facility with leading edge technologies and tricks, I don’t have to worry about falling short.

So I don’t fall short.  If my first attempt isn’t loved by my client, they tell me and I tweak accordingly.  No fuss, no muss².

That’s the second beauty of stating up front the right to demand satisfaction: it creates a conversational dynamic between my client and I that deliberately CALLS FOR that kind of feedback.  We become partners in the endeavor to create software that perfectly suits, and they have a role to play called “speak up if you don’t love it.”  When the invitation to do that is clear and on the table it is easy and fun to exercise, and moreover doesn’t get compromised by a desire to be polite.

Notes:

A fair question indeed.

I bill myself as an ultra effective development team of one, citing of course virtues like eliminated communication overhead, a shorter route from business vision to application design and implementation, nimble ability to adapt the application to changing business needs when they come up, and 100% accountability when it comes to turning out quality.

Oh, and it’s cheap to hire one person instead of six.

So naturally it has occurred to my clients on more than one occasion to ask me “what happens if you get hit by a bus?”  If I’m the guy who’s doing it all, naturally I represent  a sizable single point of failure.

Here’s my answer.

You replace me.  Though I do have  a publicly documented love affair with Classic ASP, Classic ASP developers are a dying breed, so it would be irresponsible to start any new substantial project for a client in that language. No, anything that’s built to last I would start in PHP or perhaps Ruby on Rails, languages that boast thriving communities of developer talent from which you could draw to replace poor old mangled (or dead) me.

Now then, when you replace me my successor will need to be trained on how to work on the code base I’ve laid down.  If I’m NOT dead I can guide them through to get them up to speed quickly.  I if AM dead the good news is that my code is quite beautiful and well organized, with perfectly uniform indentation and formatting, completely consistent conventions, and immaculately well factored.  Any developer who is worth their salt in the language I have laid down will be able to understand, trace, and build upon my code within one day of poring over it.  (If not, fire them.  Trust me.)

Furthermore, whatever I did manage to complete before carelessly failing to look the other way while crossing that hypothetical street will constitute a massive leg up on your project.  Remember, typical development projects tend to be heavily loaded on the front end with meetings, spec docs, planning, project management, and other such things that don’t get you closer to a usable piece of software.  Before that Greyhound sent me to a better place even a half-completed prototype done by me will have enabled you to shortcut through all of that, leaving you with a sensible solution strategy and design foundation already in existence.

So in conclusion: if I get hit by a bus, you replace me.  You’ll miss me as you move forward at not as smooth or fast a clip as before, but your project will have gotten the benefit of my precious last remaining pre-bus-hit days.

It’s kinda like getting Frank Lloyd Wright to draw up the plans for your house and then he can’t finish it.  It’s still gonna be a pretty snazzy house even if some junior architect needs to rough in details like which tiles to use in the bathroom.