I'd like to share with you a few thoughts about tech recruiting. This is not a post about how to write resumes, read resumes, ask or answer tech interview questions. Enough is written on the subject by people who are better at all of this than me. No, it's about that email that arrives in almost every developer's inbox about "a full-time, permanent position with a very competitive salary" for "a very prestigious company located in mid/down/up-town" which is looking for "an alphabet soup of technologies". Half the time it's a phone call. These happen during the times of boom and bust in the tech sector, although with varying frequency.

Now, people who are not programmers would probably think that this is spammers or scammers who will ask for a fee. Well, there is a fee involved, but this is not scammers: every time there's a reasonably well paying job involved. A huge number of programming positions gets filled this way.

There's always a shortage of decent software developers, project managers, system administrators and allied tradespeople. This was true back when I was starting my career, in 1997, but it only got worse. First of all the cost of running a startup came down from hundreds of thousands of dollars to thousands, and many talented programmers with ideas, tolerance of Ramen-heavy diet, slumming, and a little risk exited the corporate workforce to build url shorteners and travel sites for hipsters. Top notch people with kids and/or love of brightly colored furniture and fridges stocked with fancy sodas were vacuumed up by Google, Microsoft, Facebook, Yahoo and other modern Xerox Parc-wannabees. The hedge funds sucked in all those who are good at implementing complicated formulas and algorithms in code that can make a Kessel run in 12 parsecs or less.

The biggies look for a healthy mix of specialists, but they can easily afford to hire jack-of-all trades hackers. Everybody else usually looks for a person who has proven experience with a certain "stack" - an alphabet soup of languages, operating systems, and servers. Sometimes stacks are ageless like COBOL, CICS, DB2 - even today typing it into dice.com brings up 155 results. Some come out of favor: LISP - only 16 results though. Somer are old standbys: Php MySQL Apache - 370 results: C# SQL Server IIS - 439 results. Some, like RoR are up and coming - 93 results.

If you are relatively young(cheaper that way) and have done some work with a fashionable stack your phone will be ringing weekly with recruiter calls in the dreariest days of a tech bubble burst. If you are a specialist who deals with a particularly gnarly piece of hot tech - it will ring a few times a week.

Now here we come to a somewhat interesting phenomenon. Most of the time the person calling is a "headhunter". Here's what's happening:

Web department in company Webco Enterprises has an opening. It could be caused by a number of things, but most commonly

1) after gettig tired of incessant complaining of the VP of Web Things, the Bean Counter finally agreed to budget in Y dollars for another headcount.

2) an experienced headcount who was making Y dollars accepted an offer of Y + $30,000 + an office with a door from company iWeb 2.0 and left.

So here is VP of Web Things with a budget of Y dollars. She's swamped with work. In the next team meeting she tells her nerds to ask all of their friends if they know somebody and offers a referral bonus. Next she puts together a typical job ad complete with technological alphabet soup and takes it to the Master of HR. The Master of HR posts it on the company intranet, on the corporate site and on monster.com, dice.com, etc. Next VPoWT goes to the same sites and looks for candidates who posted their resumes there. It's mostly drek, and the one person who might be ok wastes a lot of her time because he has another offer for Y + $10,000 from another company.

Now more desperate, she starts using "headhunters". They send a couple of somewhat crappy candidates, followed by one barely ok one, followed by one who's good enough. After a little song and dance the good one accepts Y dollars per year minus - 15% that silently go to the recruiter. The next few years he'll be getting modest raises and maybe even a bonus or two that will not break the budget: the Bean Counter approved Y dollars, and it stays in the budget like that, and the headhunter only gets paid once.

Now, here's what was happening on the other end: headhunters start pounding searches into their computers. LinkedIn, all recruiting sites, internal databases, etc. They might not understand what those letters in the alphabet soup mean, but they sure can try to match them up with what comes back from searches. Then they start shooting in the dark - it's easier to get people who are currently looking for jobs, but it's the shallow end of the pool that VP of Web Things already explored somewhat. They start pinging people who are working at the moment - this is where the good heads are.

Finally someone hunts up the the right head for VPoWT's headcount and pockets the 15% of a yearly salary of that head.

Now, here comes the interesting part: most of these headhunters who are calling are trying to line up publicly available job listings with people on LinkedIn. For instance, I recently received two emails about a rather awkwardly named position "Director of Back End Development" (second recruiter spelled it "Backend").

I looked it up on one of the recruiting sites (I think dice.com), and it was an old and infamous pre-dot-com-boom company that chose such a unique job title for what I do for a living. It was literally the only one with that creative title. I could have applied directly, bypassing the 15% headhunter's fee. This is the reason why headhunters rarely name the company name.

In the past, through my random and numerous connections I've heard stories about what the setup is like at that place, and it's a doozy. I'm pretty sure that the person who will take that job will take a lot of development up his or her back end.

The moral of the story is this: it's easy enough to bypass recruiters - they are just engaged in arbitrage: the hiring manager is busy, the potential headcount is not even looking. That 15% is OPM (other people's money) to the hiring managers, but not so much to the headcounts.

Let me tell you about web development and fishing, my two great passions. Here's a fishing story. When I was a kid, i fished off the long piers in the Black Sea. I did not catch all that much, and I mostly thought that was because of my lack of skill and resources: I though if I could cast further, have a better fishing rod, or be able to go out on a boat, maybe I could catch more. Then I noticed that one fisherman was catching huge quantities of fish.

He had an interesting technique. Instead of using a single rod and switching from a place to a place, he's bring ten. Each one was cheap and simple bamboo rod. He'd bait them, and drop the hook in shallow water in clear water, where the sea floor was covered with concrete blocks with holes used to stabilize the sand. I tried fishing near those holes before, but never caught anything. He'd set up his ten rods, and then just wait. An interesting thing happened: after about an hour the fish started biting, and were mostly just catching themselves: all he had to do was walk from rod to a rod and take off the fish. Sometimes just a single hole would be producing, then he would take that rod and catch fish after fish from the same place.

This technique is called deadsticking: you leave the bait motionless, and thus exposed to the fish for much longer periods of time. Most fish grab the bait and run: you don't even need to set the hook, the fish catches itself. When on the boat the same technique often works. Having a number of rods fishing all the time gives you two benefits: it shows you the hot spots and exposes your hooks to more fish.

I see this again and again: a company redesigns a website, changes the core technology used to build it, spends a lot of money, and then the traffic and search rankings fall, and thus revenues fall.

I am pretty sure I know the cause of this: broken links. Any redesign of a website of just about any complexity, especially when technology changes breaks a lot of links. Search engines are like fish: they do not like things moving from a place to a place in an unnatural manner. A fisherman once told me: hey, do you think a Tautog (a kind of fish) ever seen a dead fiddler crab jump three feet up and down? Fish do like movement, jiggling the bait often entices them to bite. But the important thing is, the jiggling can't be too vigorous and take the bait out of the view! Google likes to see changing content, but if the location of the content darts around - you betcha boots you are going to see your Pagerank take a hit.

The best thing to do when faced with with less traffic from Google is not to redesign the site again, but to dead stick: fix all the broken links, keep the site stable, and better yet, bring in more rods - build more sites.

In my time I've seen a large number of websites and careers that were set back by CMS switches and redesigns.

Further reading: The Russian Tea Room Syndrome and Deadprogrammer's Hierarchy of Web Needs.

I have some advice for those in the business of building large websites with content management systems.

1) Do not implement search yourself.

Your CMS sucks at search, and so do you. I see this again and again and again. Everyone is implementing search on large websites instead of using Google. Developers are afraid of looking unprofessional. Managers are answer yes to the question "do you want advanced/faceted search" (the correct answer is no - user's don't like it and don't use it). As a result a lot of resources (both server and developer) go into implementing something that Google is awesome at. Even some very smart people, like Jeff Atwood roll their own search, and their users end up going to google.com and typing "foo site:stackoverflow.com".

Users are very happy with Google CSE, and don't mind the text ads. Those text ads - well, that's revenue that you would otherwise would not have, however small this is. If you absolutely can't do Google CSE - buy their search appliance. If you can't do that either - well, you better be using Solr.

2) Do not implement comments yourself (unless comments are what you do for a living).

It is extremely difficult to get comments right. Users absolutely abhor comments. Spammers - well, they love it. Luckily, you can just go and get DISQUS to do all the heavy lifting for you. The time saved on using DISQUS can be used on building something else, meanwhile users absolutely love leaving comments through it, while spammers hate it.

3) Physically separate your admin interface from the stuff that is going to be used by your users.

Maciej Ceglowski has some words of advice about not having your blog hacked: cache your output in flat files and hide the admin interface. The benefits of this are tremendous: cached files are fast and secure. You will need to do some fancy footwork to serve up parts that change a lot, but you can do it the same way DISQUS and Google CSE do it - through the magic of AJAX.

4) Sanity check: calculate the amount of RAM in the home computers of all of your interns. Compare that to the amount of RAM in your server farm. Who wins?

5) Use a CDN and/or caching proxy, don't be cheap. These things will save your butt when Yahoo and Digg will come a-knocking at the same time. I'm not even going to mention Memcached - you can't get big without it at all.

6) Fight WYSIWYG editors. These things are the worst. They are the Devil. They are a security hole. You never get what you see. People paste from Word. Do I need to go on?

The best middle of the road solution is something like Markdown.

Do not underestimate the user's ability to learn a few simple rules. When I worked at TV Guide there was this movie database application. Very non-technical editors were using a very scary-looking Unix-based interface at an amazing speed. When I rewrote it as a web interface, it became more "user-friendly", but they could not enter stuff as fast as before.

7) Make sure you have good backups

8) I know you won't be able to follow my advice, I know I can't either. Life is a constant compromise.