Hunting Heads and Developing Back Ends

I’d like to share with you a few thoughts about tech recruiting. This is not a post about how to write resumes, read resumes, ask or answer tech interview questions. Enough is written on the subject by people who are better at all of this than me. No, it’s about that email that arrives in almost every developer’s inbox about “a full-time, permanent position with a very competitive salary” for “a very prestigious company located in mid/down/up-town” which is looking for “an alphabet soup of technologies”. Half the time it’s a phone call. These happen during the times of boom and bust in the tech sector, although with varying frequency.

Now, people who are not programmers would probably think that this is spammers or scammers who will ask for a fee. Well, there is a fee involved, but this is not scammers: every time there’s a reasonably well paying job involved. A huge number of programming positions gets filled this way.

There’s always a shortage of decent software developers, project managers, system administrators and allied tradespeople. This was true back when I was starting my career, in 1997, but it only got worse. First of all the cost of running a startup came down from hundreds of thousands of dollars to thousands, and many talented programmers with ideas, tolerance of Ramen-heavy diet, slumming, and a little risk exited the corporate workforce to build url shorteners and travel sites for hipsters. Top notch people with kids and/or love of brightly colored furniture and fridges stocked with fancy sodas were vacuumed up by Google, Microsoft, Facebook, Yahoo and other modern Xerox Parc-wannabees. The hedge funds sucked in all those who are good at implementing complicated formulas and algorithms in code that can make a Kessel run in 12 parsecs or less.

The biggies look for a healthy mix of specialists, but they can easily afford to hire jack-of-all trades hackers. Everybody else usually looks for a person who has proven experience with a certain “stack” – an alphabet soup of languages, operating systems, and servers. Sometimes stacks are ageless like COBOL, CICS, DB2 – even today typing it into brings up 155 results. Some come out of favor: LISP – only 16 results though. Somer are old standbys: Php MySQL Apache – 370 results: C# SQL Server IIS – 439 results. Some, like RoR are up and coming – 93 results.

If you are relatively young(cheaper that way) and have done some work with a fashionable stack your phone will be ringing weekly with recruiter calls in the dreariest days of a tech bubble burst. If you are a specialist who deals with a particularly gnarly piece of hot tech – it will ring a few times a week.

Now here we come to a somewhat interesting phenomenon. Most of the time the person calling is a “headhunter”. Here’s what’s happening:

Web department in company Webco Enterprises has an opening. It could be caused by a number of things, but most commonly

1) after gettig tired of incessant complaining of the VP of Web Things, the Bean Counter finally agreed to budget in Y dollars for another headcount.

2) an experienced headcount who was making Y dollars accepted an offer of Y + $30,000 + an office with a door from company iWeb 2.0 and left.

So here is VP of Web Things with a budget of Y dollars. She’s swamped with work. In the next team meeting she tells her nerds to ask all of their friends if they know somebody and offers a referral bonus. Next she puts together a typical job ad complete with technological alphabet soup and takes it to the Master of HR. The Master of HR posts it on the company intranet, on the corporate site and on,, etc. Next VPoWT goes to the same sites and looks for candidates who posted their resumes there. It’s mostly drek, and the one person who might be ok wastes a lot of her time because he has another offer for Y + $10,000 from another company.

Now more desperate, she starts using “headhunters”. They send a couple of somewhat crappy candidates, followed by one barely ok one, followed by one who’s good enough. After a little song and dance the good one accepts Y dollars per year minus – 15% that silently go to the recruiter. The next few years he’ll be getting modest raises and maybe even a bonus or two that will not break the budget: the Bean Counter approved Y dollars, and it stays in the budget like that, and the headhunter only gets paid once.

Now, here’s what was happening on the other end: headhunters start pounding searches into their computers. LinkedIn, all recruiting sites, internal databases, etc. They might not understand what those letters in the alphabet soup mean, but they sure can try to match them up with what comes back from searches. Then they start shooting in the dark – it’s easier to get people who are currently looking for jobs, but it’s the shallow end of the pool that VP of Web Things already explored somewhat. They start pinging people who are working at the moment – this is where the good heads are.

Finally someone hunts up the the right head for VPoWT’s headcount and pockets the 15% of a yearly salary of that head.

Now, here comes the interesting part: most of these headhunters who are calling are trying to line up publicly available job listings with people on LinkedIn. For instance, I recently received two emails about a rather awkwardly named position “Director of Back End Development” (second recruiter spelled it “Backend”).

I looked it up on one of the recruiting sites (I think, and it was an old and infamous pre-dot-com-boom company that chose such a unique job title for what I do for a living. It was literally the only one with that creative title. I could have applied directly, bypassing the 15% headhunter’s fee. This is the reason why headhunters rarely name the company name.

In the past, through my random and numerous connections I’ve heard stories about what the setup is like at that place, and it’s a doozy. I’m pretty sure that the person who will take that job will take a lot of development up his or her back end.

The moral of the story is this: it’s easy enough to bypass recruiters – they are just engaged in arbitrage: the hiring manager is busy, the potential headcount is not even looking. That 15% is OPM (other people’s money) to the hiring managers, but not so much to the headcounts.

Are Tables Important?

I was talking to a former co-worker about Inc Magazine’s cover story about Markus Frind and his very profitable, but godawfully ugly dating website

My co-worker (a programmer) loaded up the website. He took a quick look around and opened the source of the ratings page. Giggling like Bevis he could not believe what he saw: a gradient bar that was coded as [gasp!] an HTML table with bgcolor attributes.

It looked like this:

And was coded like that:

<table border=0 cellspacing=0 cellpadding=0 width=100%>
<tr height=5><td bgcolor=#204080><img width=1 height=5 border=0>
</td><td bgcolor=#202F70><img width=1 height=5 border=0></td>
<td bgcolor=#3F2060><img width=1 height=5 border=0></td>
<td bgcolor=#5F2050><img width=1 height=5 border=0></td>
<td bgcolor=#7F1F4F><img width=1 height=5 border=0></td>
<td bgcolor=#90103F><img width=1 height=5 border=0></td>
<td bgcolor=#B0102F><img width=1 height=5 border=0></td>
<td bgcolor=#CF0F1F><img width=1 height=5 border=0></td>
<td bgcolor=#E0000F><img width=1 height=5 border=0></td>
<td bgcolor=#F00000><img width=1 height=5 border=0></td>

He was going on and on and on about how tables are bad, and mwu-ha-ha-ha — look at this.

I was fully expecting him to take umbrage at the logo, the overall look and feel of the site, at the grotesquely skewed photo thumbnails. But no, all he was seeing is that Mr. Frind “used a table”.

I tried to tell my co-worker that despite “tables” or ugliness this website generates tens of millions of dollars of profit to its creator, that it has as much web traffic as Yahoo while being served a small handful of very powerful servers, that it was created and maintained by a single person who gets to keep most of the profits – but to no awail. The kid could not get over “tables”.

A famous hacker JWZ once was asked about his feelings about “an open source groupware system”. In a famous rant that followed he produced some of the best advice importance that I’ve ever seen:

“So I said, narrow the focus. Your “use case” should be, there’s a 22 year old college student living in the dorms. How will this software get him laid?”

While I’ve never heard of HTML tables (not the furniture kind) playing any role in getting laid, must have resulted in a mind boggling amount of action., being a technological and aestetical abomination that it is, is firmly rooted in the lower, fundamental layers of Maslow’s Hierarchy and my Web Heirarchy.

At the most basic people need oxygen, water, food, to take a dump/whiz, sleep, sex, and a predictability in environment.

On the web people need hypertext, images, search, speed, and community features. If you provide all of these for a topic that is important to people, you will be successful. Start thinking about “html tables vs divs” first, and likely you won’t get to the important stuff.

Doing it another way – saying, look, I’ll do a site just like plentyoffish but prettier and without HTML tables does not work very well: Frind’s competiors at who set out to do just that are not succesful in toppling plentyoffish.

Ugliness for the sake of ugliness is not a good thing. In the long run people want things to be pretty, like Apple products and not ugly like Microsoft products. But taste, being pretty high up in the pyramid of needs only becomes a factor after all the basic needs are met.

The Vault

A friend of mine, a contractor, recently came to me with a strange problem. He did an excellent job renovating my apartment, and since then he got used to me delousing his Windows computer and coming up with creative googled-up solutions for just about anything. This one has me stumped though.

Right now he is demolishing a location, previously occupied by a bank. It has a vault door in it that my friend needs to cart away.

He wants to sell it. I mean, the thing looks valuable – but I have no idea of who would want something like that. Movie people? A restaurant? Eric Sink (his company has a product called Vault and he must be flush after Microsoft buying a chunk of his stuff). I wonder what would happen if we did place it on eBay. Well, in fact there are a few of them there, and people don’t seem to be buying.

Happy New Year!

I am continuing this blog’s tradition of a New Year’s cards, even while thinking of closing down I might do it the Dr. Fun way – get it to 10 years and call it quits.

This was a tough fricking year. I spent it on call for application breakdowns, learning system administration, stressing out of my mind, hitting hard deadlines, missing soft ones, gaining 15 pounds, etc, etc.

I was stupid enough not to listen to my very smart friend and keeping my 401K in S&P 500 instead of in cash equivalent funds. I did have enough sense to sell it after Lehman Brothers shat the bed.

This was my first year of being 99% free of Microsoft Windows. Indeed, once you go Mac you don’t go back.

An interesting new web application that I worked on in my spare time should launch around February. I did waste a lot of my free time watching “my stories” (“The Wire”, “Mad Men”, “How It’s Made”, and other yuppie deligths), but I did not waste all of it. You’ll see.

In any case, I wish you all a very happy, healthy and prosperous New Year!

The Problem With Code Reuse and Abstraction

Currently I am neck deep in some multi-layered multi-moduled code that I inherited from some consultants. Younger developers who work with me are also upset with the convolutions, but they are convinced that their way of snapping together Lego blocks is vastly superior and much more maintainable. Just like the consultants though they believe that the more modular, abstract and generic they make their code – the better it is, even if they are building a one-off super specific application. Modularity and code reuse is their number one priority, only then followed by performance, brevity, and lack of bugs.

Development frameworks and code reuse that they promote seem to the the way things are going these days. I was reminded of this trend when I was reading this passage in Ian Bank’s “The Bridge” today:

“”Are there laws against what they did?”
“There’s no law to permit it, Orr, that’s the point. Good grief man, you can’t have people going off and doing things just because they want to, just because they think something up! You have to have a… a framework””

Don’t get me wrong, I am all for code reuse, abstractions and other such fineries. It’s just that I tend to look at certain tasks and think – there’s a piece of custom code needed here! The younger developers always look at me with great disapproval and tell me that I should use module X or Y, or how they are working on a module that will cover all problems of this kind in a generic way.

Today I came upon a perfect metaphor for such thinking. Apparently it originated from an origami forum, but it seems to me I encountered it in Fidonet signature years ago. In any case, the quote goes like this:

“An elephant consists of a trunk, ears, and a hippopotamus”.

When you are building things out of Lego blocks, you must fight the temptation to substitute a hippo for the starting point of an elephant. It looks like Lego did mess up the young minds just the way Sir Harold Kroto said when he delivered a crotch punch to the Lego sales:

“New toys (mainly Lego) have led to the extinction of Meccano and this has been a major disaster as far as the education of our young engineers and scientists is concerned. Lego is a technically trivial plaything and kids love it partly because it is so simple and partly because it is seductively coloured. However it is only a toy, whereas Meccano is a real engineering kit and it teaches one skill which I consider to be the most important that anyone can acquire: This is the sensitive touch needed to thread a nut on a bolt and tighten them with a screwdriver and spanner just enough that they stay locked, but not so tightly that the thread is stripped or they cannot be unscrewed.”

If I knew about some of the things that were lurking in the codebase I’m working with now, I’d work something like this into my hiring contract:

“In April of 2000 while on a business trip I received a near-frantic email from someone with an unusual request. It seems that she represented a fellow with a company in Seattle (no, NOT Microsoft). Turns out that last September this guy was hired, and in his contract of employment it stipulated that he wanted a desk made out of LEGO.”

Sometimes I am also reminded of other classics of early Internet literature.

I only play a doctor on TV

So, those magnum Jupiter brains at Crispin Porter + Bogusky managed to crap the bed with their Microsoft ads instead of making Microsoft cool. For some reason this made me remember these three factoids:

According to Wikipedia, PC Guy John “PC” Hodgman is a Mac user, and Justin “Mac” Long is computer illiterate.

Kazuo Kawasaki, the Japanese designer for Sarah Palin’s glasses, is grateful to the Republican vice presidential candidate for making his product famous, although he acknowledged he also likes Democratic presidential hopeful Sen. Barack Obama“. Palin’s frames without lenses cost as much as John Edward’s haircut.

I’ve read somewhere that Norm Abram hates plaid shirts. A person who worked for the show told me that his shirts are mostly from Land’s End, by the way.

I don’t know, I guess this is how my puny Pluto brain works…

Surf Naked

I had to clean out my in laws’ Windows laptop this weekend, and this was my first major contact with Windows in weeks, outside of testing IE6 quirks on Parallels. The horror that is Sharepoint is only a distant memory. Even more distant (and not as unpleasant) memory of MS Commerce Server surfaced recently.

I received this shirt as a gift from Microsoft developers, some of whom actually wrote major parts of Commerce Server and Site Server when I visited Redmond. They were very competent, knowledgeable and sociable. I remember being surprised by seeing a lot of Perl books in one office, and being told that for parsing log files even the mighty Beast of Redmond relied on Perl.

I did not get to see Lake Bill or the Microsoft museum, but I did have a Blibbet Burger at the cafeteria. My favorite piece of swag was this polo shirt which I almost completely wore out. It is about to completely disintegrate, so I decided to scan the logo from it.

CRUD Ain’t Hard

And now for a little exercise in armchair software architecture — the most despicable coder’s pastime. Dear non-coding readers: despite its name, this blog is still mostly not about programming. Just skip this post or something. Dear coders, many of you will probably disagree with me. I am not a very good or accomplished coder myself, and you probably should not be taking your advice from me. But then again, I could be right, so keep your mind open.

You might have been aware of the very popular, but uptime-challenged social networking tool called Twitter. They have one of the best problems to have: too many very active users. The site is so popular that it constantly goes down and displays and “over capacity” screen that the users have nicknamed The Fail Whale.

Rapidly writing and displaying short chunks of text with high concurrency on the web is not one of them unsolvable problems in programming. It’s not easy, but with right people and tools Twitter could be rewritten inside a month. Twitter founders should do some soul searching. Meanwhile the critical mass has already been reached, the niche for bloggers who want to SMS instead of blogging is big, and even horrible uptime can’t this service. I use it myself.

There is a lot of speculation in the blogocube about whether the reason behind the Fail Whale is the wrong choice of technology — the highly hyped and sexy Ruby on Rails and if it can “scale”. Or is it just simple incompetence?

To me Ruby on Rails falls into a class of technologies that are affected by what I call “the VRML syndrome.” Basically, if I wait long enough the hype will go away, the recruiters will stop posting job listings requiring 4 years of experience in a 4 month old technology, books as fat as my two fists will stop being published, and I will not have to learn it.

What’s the problem with Ruby on Rails? Well, it’s the same problem that slightly affects the content management system that I am currently working with (Drupal), and is the reason why I completely gave up using Microsoft web technologies which are saturated with this shit. See, software craptitechts all of a sudden decided that writing CRUD applications is too difficult for regular developers, and complicated GUI tools and frameworks need to be created to help the poor things. CRUD stands for “Create, Read, Update, Delete” and is just a funny way to say “a browser-based application chock-full’o forms”.

The default way to build these is to rather simple. You hand-code the html forms, then you write functions or classes to deal with the form input — validators and SQL queries for creating, updating and deleting. Then you write some code that will query the database and display the saved data in various ways: as pages, xml feeds, etc. None of this is difficult or non-trivial. Bad coders don’t do a good job of validation and input sanitizing resulting in the Little Bobby Tables-type situation, but these things are not very hard to learn and there are great libraries for this.

Ruby on Rails makes it very easy to create CRUD apps without hand-coding forms or writing SQL. RoR goes to great lengths to abstract out SQL, not trusting the developers to do it right. SQL is more functional than procedural, and thus a difficult thing for many programmers to grasp, but it’s not that hard. Really. SQL is located far enough levels from the machine that abstracting it out becomes a horrible thing due to the Law of Leaky Abstractions. Even when you have full control of SQL queries optimizing them is sometimes hard. When they are hidden by another layer it becomes next to impossible.

In short, RoR makes something that is easy (building CRUD apps) trivial, and something that’s hard – optimizing the database layer next to impossible.

In Drupal there are two modules, CCK and Views that allow you to create CRUD entirely through web interfaces. This is a feature that exist in just about every major CMS, it’s just that in Drupal it’s a little buggier and overcomplicated than necessary. These are fine for small websites and are really useful to amateurs. The problem arises when these are used for high traffic websites.

I think that a lot of people will agree with me that writing HTML and SQL queries using GUI tools is amateur hour. You just can’t make a good website with Microsoft Front Page. You can’t, you can’t, you can’t. But in Drupalland it’s all of a sudden fine to use Views to build queries for high traffic sites. Well, it’s not. Dealing with Views and Views Fast Search has been an ongoing nightmare for me. Hell is not even other people’s code in this case. It’s other people’s Views.

RoR, Views, CCK are one level of abstraction higher than you want to be when building a high performance application. The only way the can be an “Enterprise” tool if your enterprise is a) run by a morons that require 100 changes a day AND b) has very few users. In short, if it’s an app for the HR department of a company with 12 employees – knock yourself out. If you are building a public website for millions of people – forget about it.

Your, Deadprogrammer.

P.S. Yes, I know, you can abstract just about everything and reduce your software application to a single button labled “GENERATE MONEY”. You have to be a very smart LISP developer for that.

Oy, Again With The Moving…

Using Dreamhost is quickly turning into nightmare. It’s a cheap, full featured and generous web host, except only good for websites that do not matter (and I think that mine do). There is no upgrade path to a virtual private server (which is one step below a dedicated machine both in price and performance), their overall uptime is not something I’d trust, and their blog is just driving me nuts. At the suggestion of a friend I’m moving over to Webintellects.

As a web developer I specialize in content management systems. I have wasted many years of my career on Microsoft technologies, although my personal website was always built using open source tools. In recent years, when faced with the twin horrors of Sharepoint and MS CMS, I just could not go on any more. I just can’t imagine an entrepreneur who would willingly use this stuff to build a business. I quit my job of almost 6 years, took some time off and went on to a job that allows me to use open source tools. We’ve had quite a bit of success with Drupal, a leading open source CMS.

WordPress is a great tool for blogs, but it makes good sense for me to start using Drupal for my own sites, as well as at work. Drupal grows at an astronomical rate, improving in leaps and bounds. I have a couple of modules almost ready for contribution (once I make them a little neater and better documented). Drupal is very scalable, very well designed and has a huge following. I could not be happier with it as a developer.

In the five years that my website existed in blog format I moved 3 times. Livejournal -> Movable Type -> WordPress. Now it’s Drupal‘s turn.

I apologize in advance for any annoying symptoms of the move, like refreshing of the RSS feed where already read articles might show up as new, etc. Please bear with me.

Where’s My Flying Car Part I : KABOOM!

“Celebrating Gertsen, we clearly see three generations,
three classes acting in the Russian Revolution. First –
noblemen and landowners, Decembrists and Herzen.
Horribly distant from the people. But their work was not in wain.
Decembrists woke Herzen. Herzen began revolutionary agitation.”
V.I. Lenin

Computers have existed like for 200,000 years in Internet time, yet the innovation in computer technology seems to be a little slow. Brick and mortar slow. Let me present to you an approximate timeline:

In 1945 Dr. Vannevar Bush wrote an article As We May Think about a device called the Memex.

In 1960 Theodor Holm Nelson, inspired by Bush, coined the term “hypertext” and started on Project Xanadu, a vaporware Superinternet.

In 1968 Dr. Douglas Engelbart delivered the MOAD, demonstrating videoconferencing, email, hypertext, copy and paste, as well as some novel input devices including a mouse.

Bush, Nelson and Engelbart show a progression from a dream into reality. Bush was a pure dreamer – he never intended to actually try and build the Memex. Nelson at least tried to build Xanadu, although he failed miserably. He could not even get to the demo stage. Engelbart actually built enough stuff to make very impressive demos, although never to build actual successful products except the mouse. These guys suffered from the RAND Corporation syndrome–the common joke went that RAND stood for Reasearch And No Development.

The problem with these three was that they could not focus on individual problems. Luckily for us, next came Xerox PARC. Xerox corporation had money coming out of its wazoo, decided to invest in a world class R&D center. They used the same approach that Google is using today: spend the extra money on hiring the brightest technologists around and let them run free and wild.

Bush, Nelson and Engelbart were a lot like a character named Manilov in Gogol’s Dead Souls. Manilov was an owner of a large rundown estate. He spent his days dreaming about improving it. Wouldn’t it be nice to build a bridge over the river and on it build little merchant booths so that the peasants could buy stuff there. Of course, none of his projects ever went anywhere, and if they did, they were quickly abandoned.

PARC engineers were men of action. Each concentrated on a particular aspect, and they’ve built working models of many things that we enjoy today: personal computer with GUI interfaces, Ethernet, WYSIWYG text editor, laser printer, and even a computer animation system amongst other things. Sadly, Xerox was able to capitalize mostly on the laser printer, which actually probably paid for all of PARC’s expenses. PARC indirectly influenced Apple and Microsoft in the development of GUI OS. Also Charles Simonyi left PARC to develop Word and Excel for Microsoft, thus creating an enormous amount of wealth. Bob Metcalfe and David Boggs also left PARC, took Ethernet and turned it into 3COM. John Warnock and Charles Geschke left PARC, took PostScript and created a little company called Adobe Systems. Well, you get the picture.

To give you another analogy, the technological revolution of the 60s, 70s and 80s was like a hydrogen bomb. A hydrogen bomb is made of three bombs: a conventional explosive that ignites a fission explosive that in turn ignites a fusion explosion. Semiconductor industry created by William Shockley and the Traitorous Eight was the fuel, Bush and Company–the conventional explosion, PARC–fission, what came after–fusion. KABOOM!