Hunting Heads and Developing Back Ends

I’d like to share with you a few thoughts about tech recruiting. This is not a post about how to write resumes, read resumes, ask or answer tech interview questions. Enough is written on the subject by people who are better at all of this than me. No, it’s about that email that arrives in almost every developer’s inbox about “a full-time, permanent position with a very competitive salary” for “a very prestigious company located in mid/down/up-town” which is looking for “an alphabet soup of technologies”. Half the time it’s a phone call. These happen during the times of boom and bust in the tech sector, although with varying frequency.

Now, people who are not programmers would probably think that this is spammers or scammers who will ask for a fee. Well, there is a fee involved, but this is not scammers: every time there’s a reasonably well paying job involved. A huge number of programming positions gets filled this way.

There’s always a shortage of decent software developers, project managers, system administrators and allied tradespeople. This was true back when I was starting my career, in 1997, but it only got worse. First of all the cost of running a startup came down from hundreds of thousands of dollars to thousands, and many talented programmers with ideas, tolerance of Ramen-heavy diet, slumming, and a little risk exited the corporate workforce to build url shorteners and travel sites for hipsters. Top notch people with kids and/or love of brightly colored furniture and fridges stocked with fancy sodas were vacuumed up by Google, Microsoft, Facebook, Yahoo and other modern Xerox Parc-wannabees. The hedge funds sucked in all those who are good at implementing complicated formulas and algorithms in code that can make a Kessel run in 12 parsecs or less.

The biggies look for a healthy mix of specialists, but they can easily afford to hire jack-of-all trades hackers. Everybody else usually looks for a person who has proven experience with a certain “stack” – an alphabet soup of languages, operating systems, and servers. Sometimes stacks are ageless like COBOL, CICS, DB2 – even today typing it into brings up 155 results. Some come out of favor: LISP – only 16 results though. Somer are old standbys: Php MySQL Apache – 370 results: C# SQL Server IIS – 439 results. Some, like RoR are up and coming – 93 results.

If you are relatively young(cheaper that way) and have done some work with a fashionable stack your phone will be ringing weekly with recruiter calls in the dreariest days of a tech bubble burst. If you are a specialist who deals with a particularly gnarly piece of hot tech – it will ring a few times a week.

Now here we come to a somewhat interesting phenomenon. Most of the time the person calling is a “headhunter”. Here’s what’s happening:

Web department in company Webco Enterprises has an opening. It could be caused by a number of things, but most commonly

1) after gettig tired of incessant complaining of the VP of Web Things, the Bean Counter finally agreed to budget in Y dollars for another headcount.

2) an experienced headcount who was making Y dollars accepted an offer of Y + $30,000 + an office with a door from company iWeb 2.0 and left.

So here is VP of Web Things with a budget of Y dollars. She’s swamped with work. In the next team meeting she tells her nerds to ask all of their friends if they know somebody and offers a referral bonus. Next she puts together a typical job ad complete with technological alphabet soup and takes it to the Master of HR. The Master of HR posts it on the company intranet, on the corporate site and on,, etc. Next VPoWT goes to the same sites and looks for candidates who posted their resumes there. It’s mostly drek, and the one person who might be ok wastes a lot of her time because he has another offer for Y + $10,000 from another company.

Now more desperate, she starts using “headhunters”. They send a couple of somewhat crappy candidates, followed by one barely ok one, followed by one who’s good enough. After a little song and dance the good one accepts Y dollars per year minus – 15% that silently go to the recruiter. The next few years he’ll be getting modest raises and maybe even a bonus or two that will not break the budget: the Bean Counter approved Y dollars, and it stays in the budget like that, and the headhunter only gets paid once.

Now, here’s what was happening on the other end: headhunters start pounding searches into their computers. LinkedIn, all recruiting sites, internal databases, etc. They might not understand what those letters in the alphabet soup mean, but they sure can try to match them up with what comes back from searches. Then they start shooting in the dark – it’s easier to get people who are currently looking for jobs, but it’s the shallow end of the pool that VP of Web Things already explored somewhat. They start pinging people who are working at the moment – this is where the good heads are.

Finally someone hunts up the the right head for VPoWT’s headcount and pockets the 15% of a yearly salary of that head.

Now, here comes the interesting part: most of these headhunters who are calling are trying to line up publicly available job listings with people on LinkedIn. For instance, I recently received two emails about a rather awkwardly named position “Director of Back End Development” (second recruiter spelled it “Backend”).

I looked it up on one of the recruiting sites (I think, and it was an old and infamous pre-dot-com-boom company that chose such a unique job title for what I do for a living. It was literally the only one with that creative title. I could have applied directly, bypassing the 15% headhunter’s fee. This is the reason why headhunters rarely name the company name.

In the past, through my random and numerous connections I’ve heard stories about what the setup is like at that place, and it’s a doozy. I’m pretty sure that the person who will take that job will take a lot of development up his or her back end.

The moral of the story is this: it’s easy enough to bypass recruiters – they are just engaged in arbitrage: the hiring manager is busy, the potential headcount is not even looking. That 15% is OPM (other people’s money) to the hiring managers, but not so much to the headcounts.

Are Tables Important?

I was talking to a former co-worker about Inc Magazine’s cover story about Markus Frind and his very profitable, but godawfully ugly dating website

My co-worker (a programmer) loaded up the website. He took a quick look around and opened the source of the ratings page. Giggling like Bevis he could not believe what he saw: a gradient bar that was coded as [gasp!] an HTML table with bgcolor attributes.

It looked like this:

And was coded like that:

<table border=0 cellspacing=0 cellpadding=0 width=100%>
<tr height=5><td bgcolor=#204080><img width=1 height=5 border=0>
</td><td bgcolor=#202F70><img width=1 height=5 border=0></td>
<td bgcolor=#3F2060><img width=1 height=5 border=0></td>
<td bgcolor=#5F2050><img width=1 height=5 border=0></td>
<td bgcolor=#7F1F4F><img width=1 height=5 border=0></td>
<td bgcolor=#90103F><img width=1 height=5 border=0></td>
<td bgcolor=#B0102F><img width=1 height=5 border=0></td>
<td bgcolor=#CF0F1F><img width=1 height=5 border=0></td>
<td bgcolor=#E0000F><img width=1 height=5 border=0></td>
<td bgcolor=#F00000><img width=1 height=5 border=0></td>

He was going on and on and on about how tables are bad, and mwu-ha-ha-ha — look at this.

I was fully expecting him to take umbrage at the logo, the overall look and feel of the site, at the grotesquely skewed photo thumbnails. But no, all he was seeing is that Mr. Frind “used a table”.

I tried to tell my co-worker that despite “tables” or ugliness this website generates tens of millions of dollars of profit to its creator, that it has as much web traffic as Yahoo while being served a small handful of very powerful servers, that it was created and maintained by a single person who gets to keep most of the profits – but to no awail. The kid could not get over “tables”.

A famous hacker JWZ once was asked about his feelings about “an open source groupware system”. In a famous rant that followed he produced some of the best advice importance that I’ve ever seen:

“So I said, narrow the focus. Your “use case” should be, there’s a 22 year old college student living in the dorms. How will this software get him laid?”

While I’ve never heard of HTML tables (not the furniture kind) playing any role in getting laid, must have resulted in a mind boggling amount of action., being a technological and aestetical abomination that it is, is firmly rooted in the lower, fundamental layers of Maslow’s Hierarchy and my Web Heirarchy.

At the most basic people need oxygen, water, food, to take a dump/whiz, sleep, sex, and a predictability in environment.

On the web people need hypertext, images, search, speed, and community features. If you provide all of these for a topic that is important to people, you will be successful. Start thinking about “html tables vs divs” first, and likely you won’t get to the important stuff.

Doing it another way – saying, look, I’ll do a site just like plentyoffish but prettier and without HTML tables does not work very well: Frind’s competiors at who set out to do just that are not succesful in toppling plentyoffish.

Ugliness for the sake of ugliness is not a good thing. In the long run people want things to be pretty, like Apple products and not ugly like Microsoft products. But taste, being pretty high up in the pyramid of needs only becomes a factor after all the basic needs are met.

8 Pieces of Architectural Advice for CMS

I have some advice for those in the business of building large websites with content management systems.

1) Do not implement search yourself.

Your CMS sucks at search, and so do you. I see this again and again and again. Everyone is implementing search on large websites instead of using Google. Developers are afraid of looking unprofessional. Managers are answer yes to the question “do you want advanced/faceted search” (the correct answer is no – user’s don’t like it and don’t use it). As a result a lot of resources (both server and developer) go into implementing something that Google is awesome at. Even some very smart people, like Jeff Atwood roll their own search, and their users end up going to and typing “foo”.

Users are very happy with Google CSE, and don’t mind the text ads. Those text ads – well, that’s revenue that you would otherwise would not have, however small this is. If you absolutely can’t do Google CSE – buy their search appliance. If you can’t do that either – well, you better be using Solr.

2) Do not implement comments yourself (unless comments are what you do for a living).

It is extremely difficult to get comments right. Users absolutely abhor comments. Spammers – well, they love it. Luckily, you can just go and get DISQUS to do all the heavy lifting for you. The time saved on using DISQUS can be used on building something else, meanwhile users absolutely love leaving comments through it, while spammers hate it.

3) Physically separate your admin interface from the stuff that is going to be used by your users.

Maciej Ceglowski has some words of advice about not having your blog hacked: cache your output in flat files and hide the admin interface. The benefits of this are tremendous: cached files are fast and secure. You will need to do some fancy footwork to serve up parts that change a lot, but you can do it the same way DISQUS and Google CSE do it – through the magic of AJAX.

4) Sanity check: calculate the amount of RAM in the home computers of all of your interns. Compare that to the amount of RAM in your server farm. Who wins?

5) Use a CDN and/or caching proxy, don’t be cheap. These things will save your butt when Yahoo and Digg will come a-knocking at the same time. I’m not even going to mention Memcached – you can’t get big without it at all.

6) Fight WYSIWYG editors. These things are the worst. They are the Devil. They are a security hole. You never get what you see. People paste from Word. Do I need to go on?

The best middle of the road solution is something like Markdown.

Do not underestimate the user’s ability to learn a few simple rules. When I worked at TV Guide there was this movie database application. Very non-technical editors were using a very scary-looking Unix-based interface at an amazing speed. When I rewrote it as a web interface, it became more “user-friendly”, but they could not enter stuff as fast as before.

7) Make sure you have good backups

8) I know you won’t be able to follow my advice, I know I can’t either. Life is a constant compromise.

The Man Who Owns the News: Inside the Secret World of Rupert Murdoch

If Rupert Murdoch isn’t making headlines, he’s busy buying the media outlets that generate the headlines. His News Corp. holdings—from the New York Post, Fox News, and most recently The Wall Street Journal, to name just a few—are vast, and his power is unrivaled. So what makes a man like this tick? Michael Wolff gives us the definitive answer in The Man Who Owns the News.

With unprecedented access to Rupert Murdoch himself, and his associates and family, Wolff chronicles the astonishing growth of Murdoch’s $70 billion media kingdom. In intimate detail, he probes the Murdoch family dynasty, from the battles that have threatened to destroy it to the reconciliations that seem to only make it stronger. Drawing upon hundreds of hours of interviews, he offers accounts of the Dow Jones takeover as well as plays for Yahoo! and Newsday as they’ve never been revealed before.

Written in the irresistible stye that only an award-winning columnist for Vanity Fair can deliver, The Man Who Owns the News offers an exclusive glimpse into a man who wields extraordinary power and influence in the media on a worldwide scale—and whose family is being groomed to carry his legacy into the future.

Hackers and Painters: Big Ideas from the Computer Age

“The computer world is like an intellectual Wild West, in which you can shoot anyone you wish with your ideas, if you’re willing to risk the consequences.” –from “Hackers & Painters: Big Ideas from the Computer Age,” by Paul Graham We are living in the computer age, in a world increasingly designed and engineered by computer programmers and software designers, by people who call themselves hackers. Who are these people, what motivates them, and why should you care? Consider these facts: Everything around us is turning into computers. Your typewriter is gone, replaced by a computer. Your phone has turned into a computer. So has your camera. Soon your TV will. Your car was not only designed on computers, but has more processing power in it than a room-sized mainframe did in 1970. Letters, encyclopedias, newspapers, and even your local store are being replaced by the Internet. “Hackers & Painters: Big Ideas from the Computer Age,” by Paul Graham, explains this world and the motivations of the people who occupy it. In clear, thoughtful prose that draws on illuminating historical examples, Graham takes readers on an unflinching exploration into what he calls “an intellectual Wild West.” The ideas discussed in this book will have a powerful and lasting impact on how we think, how we work, how we develop technology, and how we live. Topics include the importance of beauty in software design, how to make wealth, heresy and free speech, the programming language renaissance, the open-source movement, digital design, internet startups, and more. And here’s a taste of what you’ll find in “Hackers & Painters”: “In most fields the great work is done early on. The paintings made between 1430 and1500 are still unsurpassed. Shakespeare appeared just as professional theater was being born, and pushed the medium so far that every playwright since has had to live in his shadow. Albrecht Durer did the same thing with engraving, and Jane Austen with the novel. Over and over we see the same pattern. A new medium appears, and people are so excited about it that they explore most of its possibilities in the first couple generations. Hacking seems to be in this phase now. Painting was not, in Leonardo’s time, as cool as his work helped make it. How cool hacking turns out to be will depend on what we can do with this new medium.” Andy Hertzfeld, co-creator of the Macintosh computer, says about “Hackers & Painters”: “Paul Graham is a hacker, painter and a terrific writer. His lucid, humorous prose is brimming with contrarian insight and practical wisdom on writing great code at the intersection of art, science and commerce.” Paul Graham, designer of the new Arc language, was the creator of Yahoo Store, the first web-based application. In addition to his PhD in Computer Science from Harvard, Graham also studied painting at the Rhode Island School of Design and the Accademia di Belle Arti in Florence.

Inside Facebook: Life, Work and Visions of Greatness

As an early engineer, I was on the inside during Facebook’s explosive growth. In Inside Facebook, I’ll give you the scoop on the company as it became the premiere online environment for U.S. college students, including how and by whom the products were made, how you can use them best, views on what makes social networks so valuable, and where the industry is headed. You, too, can achieve startup success and attain your greatest dream; I hope to inspire you toward fulfilling your potential.

“Love the book. It captures the ethos of the place and a substantial degree of the vision and drive which is a secret to success.”
-David Kopp, Sr. Director, Community at Yahoo!

“Inside Facebook is a compelling look inside at a fascinating moment. It’s a riveting read. Karel may be an Engineer, but after reading Inside Facebook you’ll see he’s a great storyteller. I couldn’t get myself to stop reading and wanting more.”
-Ariel McNichol, CEO of mEgo.

“I love the style. It’s made for college students, like Facebook. Karel takes you into the personalities and minds behind Facebook. A must read for young entrepreneurs, and anyone into online social networking.”
-Mohammad Naqvi, UCR, creator of Facebook Notifier at

Deadprogrammer Visits Japan or Sakura in Partial Bloom Part II

Part II : Chuck Garabedian sez : “Ya gotta squeeze every penny”

My wife booked our tickets though, thriftily opting for the cheapest tickets that do not specify the airline prior to purchase. I was expecting the horrors of Aeroflot, or even worse, flying under the 5-headed eagle flag of Turkmenistan Airlines. On a crop duster or something. But we were pleasantly surprised to become proud holders of All Nippon Airways tickets. And that meant flying on a 747-400 without one of those Pokemon or Hello Kitty paint jobs, but with 4 classes of seats, on demand video monitors in all of them, nice meals and snacks, ultra clean bathrooms, and 20% more bowing.

A flight from JFK to Narita Airport takes about 14 hours non-stop. I found it to be no worse than an extra long day in a cubicle. Except instead of working I read, watched free movies (they had The Incredibles!) ate tasty meals and wondered if one could learn to smile professionally like the Japanese stewardesses. Oh, and I took a nice picture of a circular rainbow. Unfortunately I missed seeing the fish-like Sakhalin Island where my father spent a part of his childhood and my mother earned her college degree. I hope one day I’ll visit it.

Now for some photos (there will be more interesting ones in the next post). Make your own Google maps with Keyhole technology. The whole thing looks like a motherboard, doesn’t it? For some reason Japanese like to paint their towers in red and white.

In stark contrast to the heavily industrial area above, Japan is full of agricultural areas filled with rice paddies that glisten in the sun. I was gonna say like the steel of a samurai’s sword, but did not.

In Narita I came into my first contact with the wonders of Japanese wending machines. My first purchases on the Japanese soil were a $30 Hello Kitty phone card (that was way too much, a $10 one would do, besides they sell them everywhere) and a bottle of a sports drink called “Pocari Sweat”. As it turns out many Japanese products and businesses have strange, but somewhat relevant Engrish names. Overall the vending machines in Japan are way nicer that the ones in the States. I’ll write about them later, but for now I present you with the picture of a small battery of rectangular (or should I say parallelepipedal) sports drink bottles that accumulated on the windowsill of our hotel room. deal hooked us up with 5 nights in Shinagawa Prince Hotel. It’s a huge complex of a hotel located just next to a major Japan Rail station. Narita express as well as shinkansen trains stop there too. Just like ANA’s jumbo jet, the hotel seems to tailored to serve a number of different classes of customers. There’s the Executive Tower, Main Tower, New Tower and Annex Tower. We got a room in the Annex Tower which is probably like the Fiesta Deck. Still in the good old tradition of the 3rd class on a pre-war cruise ship, the room was tiny, but well designed, clean, and had very nice extras not usually found in American hotels such as yucatas, toothbrushes and razors in the “little shampoo and crap” kit and a washlet in the bathroom.

Japan is one of the last true bastions of smoking, so we could not get a non-smoking room. There was a smell of cigarettes in the air, but fortunately the windows were openable (and with a great view, including a skyscraper with a huge Canon logo) and after a short airing the room was livable. There was no hardwired Ethernet and during my stay I only figured out how to get the key to the wireless network only at the very end (you need to go to the Yahoo! cafe, fill out some paperwork, buy a drink there and ask for the access point password).

Overall it seems that Japanese businesses often treat coach class customers better than American businesses treat their business class and often first class.

We’ll Take Your Money

No, I am not talking about NYU here, even though “We’ll Take Your Money” is its unofficial motto. The official one is “Perstare et Praestare” – “To Persevere and Excel”. Yep, not PowerPoint. Excel.

Hmm, I wonder what other interesting unofficial mottos other colleges have. Brooklyn College’s official motto is “Nil Sine Magno Labore” – “Nothing Without Great Effort”. I know that firsthand. This is especially true about registering for classes. There is no popular unofficial motto, but if there was one, it would probably be pretty profane.

Cornell’s “Deus Et Humanitas” – “God and Humanity” is sometimes mistranslated as “God Went To Cornell”. And MIT’s “Mens et Manus” doesn’t mean “A man and his hand”. Well, what can expect from them, their mascot is a beaver.

Whoops, I got sidetracked. I wanted to write about charities. You see, I use Yahoo!Shopping a lot. As you spend money, they give you points. You can get some crappy merchandise for those points or you can also donate the points to a charity from There are 850,000 charities in their database. Pretty interesting. If you search, let’s say, for “goat”, you can find “American Dairy Goat Association”. Search for “butt” brings forth hilarious results, like
“Friends of the Butt-Holdsworth Memorial Library”
“Charles C. Butt Foundation”
“HE Butt Foundation”
“Hurl Butt Street School House, Inc.”
and absolutely bizzare

And they all can take your money.

Me? I am going to donate those points to my usual charity – The International Dark-Sky Association.

Some quick nerdy news

Ethernet card for my TIVO is on it’s way. I wish it was available when my TIVO’s modem got killed by a surge from a lightning storm (I had to ship it to a guy who fixed it for $50). I’ll probably install TivoWeb because Yahoo! seems to be pretty slow in rolling out online TIVO programming that was promised.

Powershot G3 is available for preorder. I can’t believe they used the same moronic silver color for it. Again. Didn’t they get enough angry letters from photographers (so many that the actually made a black color version of G2).

By the way, it’s interesting to watch the progress of design of G1, G2 and G3. One thing that I noticed is that the grip handle gets progressively bigger and more comfortable.