Semi-literate Programming

I recently finished “Coders at Work“, a series of interviews with famous programmers.

On one hand, reading a book like this is a downer: it’s very clear to me that I occupy a place that is very close to the median of the bell curve, and the skill level of programmers is a very steep non-linear curve in itself. I’ll never be as good as JWZ or Brad Fitzpatrick. But I knew that before, and I am ok with it. On the other hand, this book inspired me to read more code.

The programmers in the book disagree on many points, but they mostly agree on the importance of writing readable code and educating yourself by reading other people’s code. I make my living writing in scripting languages, and I haven’t written a line of C or C++ since college. But there’s nothing preventing me from downloading and taking a look at the source of Apache, PHP, MySQL.

It’s important for me to understand “how the sausage is made” in the PHP stack, and as it turns out, what happens between Apache PHP and MySQL in term of requests and timeouts is not as simple as one might think. I asked at StackOverflow about this, but all the diagrams that people pointed me at were of the very rudimentary type: “look, here’s a happy cow, it goes to Bovine University, look – it’s all shrink wrapped on the supermarket shelf” instead of “sausage farm/slaughterhouse/truck/factory tour, starting with cow insemenation”.

When I downloaded the source code of mod_rewrite, arguably the most useful Apache module in the world, I was amazed to find out that it’s only 5000 lines of C with comments.

The book ends with the interview of Donald Knuth, and another two major questions that the interviewer is asking everyone is – “have you read Knuth’s books and have you tried literate programming”. It was interesting to find out that most of the famous programmers use Knuth’s the same way that I do. The books sit on my bookshelf, I look at them, I sometimes try to read them, I skip most of the math. They serve as a constant reminder to me that I suck at computer science even more than I suck at programming, and luckily there are people out there who know all of this stuff who are not idiots like me.

Here’s a photo of my cubicle at TV Guide circa 2002, Knuth’s books are holding a place of honor next to the mini fridge. By the way, taking pictures of the places where you work and live is something that you should not forget to do: years from now nobody will care about those pictures of flowers, shadows, and sunsets, but

I’ve read the book about Literate Programming at the time, and was rather inspired by it. Ok, maybe I didn’t read it and more like skimmed it. I don’t think I understood what real literate programming is.

The way I understand it, Literate Programming is a way to write programs as a narrative that is readable to computers and humans. My father, in his former career a site supervisor (a type of a contractor) is very fond of giving very detailed instructions to me, the same way he used to give instructions to construction workers. His instructions usually are exaustive algorithms, with error handling. I think that his instructions, expressed as a flow of conciousness, would work not only on me and construction workers, but on computers as well, and are similar to what Donald Knuth has in mind. All you really have to do is to build a layer of abstraction between these instructions and a computer language. Also, since computers don’t forget things, he would only need to repeat his instructions once.

These days my dad is a COBOL programmer. Everybody dumps on COBOL, but in my mind it’s a language worth of a lot of respect. It has a syntax that is very English-like, something that makes reading COBOL code easy. Well, maybe it’s like reading some old-timer’s newsgroup post written in all caps, but it’s still much closer to English than most other computer languages.

At the time I was reading “Literate Programming” I was using ASP 3.0, IIS, and SQL Server 97. My task was to write a system that would account for booked and pending business. This is something that had to be done since the age of Mad Men. You see, the dealings of clients, account executives (like Pete Cambell), their bosses, account coordinators, creative department, etc are rather convoluted. But in the end, to get paid, you have to have a system that will track who brought in what business, who handled what, and how the commissions need to be split.

This is normally the realm of something called EAS (Enterprise Application Software). Back at the turn of the century, this area was still dominated by a company called SAP, but there were a few smaller players, like Salesforce.com that tried to package these applications. Any sane IT manager looks to see if an EAS solution can be purchased first. It turned out that TV Guide’s buseness logic was impossible to shoehorn into any existing solution. SAP folks said – yeah, no problem, we’ll build you what you want, but our prices start at $1M, and then there are consultant fees. ERM world is a crazy place, you can read about some true craziness in “Cube Farm”, an account of one hapless developer’s adventures at Lawson Software. It’s a truly riveting book, and I fell that every developer out there should read it. It’s literally Lovecraftian in nature, that book.

In any case, it fell to me to develop the application from scratch. Inspired by Knuth, I decided to write some semi-literate code. Me and a project manager, Brad, went to the clients and interviewed them at length, documenting their existing process (aka the most complicated set of spreadsheets you’ve ever seen). In the past, before cheap computers, all you needed was a Joan Holloway, but I believe they stopped making them.

Brad went on to go back and forth with a very terse document about 5 pages in length that described how the new system would work. He would sit down with the clients and go through the narrative, step by step, confirming that this is what they wanted. Meanwhile I created an object oriented library that made dealing with the database, creating forms and navigation elements much easier. This is similar to to what you might find in a CMS like Drupal, only a little cruder.

When the document shaped up, I created the database schema, and then I took a big chunk of the document and pasted it into one huge comment block. I proceeded to break off chunks of that block and writing the code around it. Interestingly enough, as time went on, the project manager started helping me to write the code: enough of scary database abstration was hidden by simple classes and method, and there were tons of self-evident examples all around to copy and paste. I switched to writing reports that involved cubes, rollups and other fancy stuff. Stored procedures that did the reports also received comments from the document that described the reports.

This wasn’t a monolythic system – I was writing it for 2 years or so, releasing a chunk after chunk. In the end it was handed off to another developer, the whole transfer took only a couple of hours. There weren’t any major bugs, maintanence issues (I believe I received only one phone call about it after several years of continuous use). All in all I was pretty pleased with this approach and can absolutely recommend it.

I believe this is the reason why so many English majors become excellent programmers: if you can write for people, you can write for computers. Sometimes there are reasons why you can’t do both at the same time, but there’s no reason not to find some middle ground.

Homer Simpson’s Toothpick Method of Blogging

There’s something that has been bothering me for a while, something that I call “Homer Simpson’s toothpick school of blogging”. In one of the Simpsons episodes Homer is marauding a grocery store at brunch, making a meal out of free samples. He proceeds to eat a few non-sample items by proclaming that “if it has a toothpick in it, it’s free” and sticking his toothpic into a variety of items. He even drinks a beer, piercing it with a toothpick. The most successful blogs are basically like that: they either paraphrase or directly quote juiciest pieces of online articles. There might be a little bit of commentary (the snarkier – the better), but the meat of these blogs is in the quotes.

This is known as “curating” – the successful toothpickers have excellent taste in content. The people they quote and take images from are very glad to receive traffic from these A-listers. BoingBoing.net, kottke.org, daringfireball.net are like that: short, high volume (once you get the hang of it, it does not take much to turn that interesting site in your firefox tab into a pithy little wrapper around a juicy quote), very enjoyable. More so than mechanized versions of the same thing like digg.com and stumbleupon.com. For one, submitters don’t do a very good job of quoting or paraphrasing, and you find yourself clicking on links more. Very successful blogs stick their toothpics into so much content that you don’t really need to click through to the originals much: I can read BoingBoing, Gothamist or Lifehacker without clicking too much – the juiciest stuff is already there. In fact Gothamist seems to be almost completely pulled from from New York Times and New York Post headlines. It’s a bit like a segment on some NY TV news stations where they read the latest headlines from local papers.

Now, there isn’t anything unethical about quoting and paraphrasing – it’s all squarely in the realm of fair use. These blogs are a bit like suckerfish that attach themselves to whales or sharks in that they benefit immensely from their hosts. Well, actually, unlike suckerfish they repay the favor by driving traffic.

In fact, I owe most of my readers to the low point in my blogging career, when after failing to submit my post about the Starbucks Siren to BoingBoing through their official black hole form, I begged Cory Doctorow to post it in a personal email. He did, I received tons of traffic and literally thousands of links from BB readers. Now that article shows up at the very top of Google search results for Starbucks logo.

Therein lies a problem: good content on the Internet does not always bubble up to the top on it’s own. Blogosphere is a bit like the Black Sea, which has a layer of very active and vibrant biosphere at low depths. But it’s very deep, and below 200 meters the depths are full of poisonous hydrogen sulfide, which luckily does not circulate very much (unless there’s a particularly strong storm). Think about digg.com or StackOverflow.com– at the top stuff circulates, gets upvoted and downvoted. But below, there’s a poisonous cesspool of Sturgeon’s Law’s 90 percent. And most of the time, new and worthwhile content starts not at the top, but at the bottom, or flutters briefly in above the mediocrity and the bad, does not get noticed and gets buried.

Speaking of StackOverflow, Joel Spolsky and Jeff Atwood recently touched on the topic of blogging success in their excellent podcast. They were discussing Steve Yegge’s retirement from blogging, and tried to pinpoint what it meant to be a successful blogger. “Perhaps one metric of success is getting people you respect and admire to link to your writing in an organic, natural way (that is, without asking them to).” I am a miserable failure on this front. Sure, I have some high profile readers, but their link love is rare, while I’m not really below begging for links.

Jason Kottke, an A-list blogger and a primo toothpick sampler, was reflecting on the monetary success. He likened business blogging to shining shoes: there might be some individuals who can get rich by running a chain of shoe shining stores (Jason Calacanis, Nick Denton), and maybe even some individual outstanding shoeshiners (Dooce) who can make a decent living, but for the majority of shoeshiners it’s not a very good career choice.

I’ve read somewhere about my hometown’s “king of shoeshiners”, a very colorful character. He was the best shoeshiner Odessa has ever seen, famous and loved by all, but he died poor and miserable. On his monument there was a short quote: “life is waksa” (waksa is a Russian word for shoe polish with a connotation of something pitch-black).

For me blogging takes a good deal of effort. In the immortal words of E.B. White “writing is never ‘fun'”. (White almost rejected an assignment to write an article that became the finest piece ever written about New York when an editor suggested that he might ‘have fun’). What makes blogging less fun for me is looking at server statistics, number of comments, ad revenue, and thinking about payoff and success. And feeling like that I maybe should have done something else with my time.

My high school Economics teacher, Mr. Oster, taught me one very valuable concept: “opportunity cost“. Whenever you make a decision do something, you almost always pay the opportunity cost – the difference in value you might have gotten by doing something better. Oh, there could be hundreds of things that have a better payoff than not very successful blogging.

I personally do not blog for money, and certainly don’t blog professionally (the ads on my site cover my hosting expenses). Well, not yet, anyway – I am preparing stuff for a commercial venture that I’ll soon announce. I blog in order to meet people (hanging out a Web 2.0 events and meetups would probably have been more productive), but mostly to get things out of my head. In that sense I’m a bit like Louise Bourgeois. I’ve recently seen an exhibition of her work, and I’m pretty sure that if she did not create all those sculptures and paintings, the inspiration for them (which must have been glipses of extra dimensions, cellular automata that drive our reality, and super disturbing things that can’t even be described) would have made her a raving lunatic and not a lucid and sane 97 year old woman that she is.

I don’t really intend on changing the format of deadprogrammer.com – the intricate, long, winding, interconnected posts about obscure topics. I probably would have had a lot more success if I just kept a photo blog about New York City. If I’d just stick to one popular topic and posted every day – I know I would have attracted a lot more readers. Instead, I’m going to start a new, for-profit blog. You’ll hear about it soon. I think I should be able to make some shekels with my mad shoeshining skills. And while I agree with Mr. White about writing not being fun, the fund is in having written.

Cognitive Filtering and Bayesian RSS

I hope one thing from the future will become popular in 2009: cognitive filtering. If the Internet was Dr. Dorian from the hit tv show “Scrubs”, I would be Dr. Cox with his list of things he cares very little about.

I got this idea from a science fiction book. In John C. Wright’s Golden Age Trilogy the singularity happened and people can upgrade and back up their wetware in any way they can afford. They still had the same problem that Henry Kuttner described in his short story “Year Day” – an overbearing amount of very innovative ads that masquerade as information and other spam. The trick in Golden Age was cognitive filtering: configurable software that removed any manifestations of anything an owner considered unpleasant: ads, sounds, pictures, symbols, and even people.

I like Twitter, and I like Robert Scoble. But I am tired of Robert’s relentless posts about friendfeed (sometimes I’m not even sure if he works with me at Fast Company or at friendfeed). Filtering this out would not be too hard – I could just ignore any post that has “friendfeed” in it. In fact, a Bayesian filter for Google reader, Facebook, and Twitter after a bit of training could do this automatically: I’d just flag posts that annoy me and the filter would analyze the words in the post, figure out which ones occur together more frequently in the posts that annoy me and hide future annoying posts based on that.

To take this a bit further, I would also like a Bayesian filter that would find me good posts from the firehydrant rss flow based on the ones I already like. There seem to be a few of these out there, but I find it hard leaving Google Reader.

Back to Livejournal

I did a bit of posting in Livejournal’s community related to my hometown and received more comments that I accumulated at deadprogrammer.com in a couple of years. The posts are in Russian, but I believe they will be interesting to a significant portion of my readers, especially the comment part where I debate merits (or lack thereof) of speedos, man purses, and other things.

Одесские наблюдения

Возвращение

Монета

Два вопроса

Привоз

The rest of you please enjoy these two pictures from one of the posts:

Paid ReviewMe Post: Phone Spam Filter

These days a controversial company RevieMe.com became downright unethical – they make it abundantly clear that they became a link purchasing company. On the other hand Phone Spam Filter is a site I don’t mind sharing Google juice with, so it’s a quick and fun way to add a 50 bucks to my Kindle fund. Here’s my review:

The goal of this site is pretty simple: Phone Spam Filter is asking you to snitch on telemarketers. You search for a phone number that you received a marketing call from and then complain about it. Besides getting a little relief from venting at the phone spammers, you get a bit of satisfaction from knowing that you added them to a blacklist. Nothing good can come out of this for the dinner-interrupting bastards. Meanwhile it’s a good place to find out if mysterious phone numbers that show up on your phone are from run of the mill telemarketers or not.

The even cooler thing is that they have an API that can help you block calls from this blacklist if you have an Asterisk PBX or are willing to install some Windows software and have a modem connected to a phone line. While Asterisk is pretty awesome, running Windows and having a modem connected to a phone line is a horrible idea these days – there are dozens of viruses that want nothing more than make a few 1-900 phonecalls. In the future Phone Spam Filter guys are hoping to add integration with VOIP providers.

The Phonespamfilter technology is not as cool as JWZ-endorsed audio-cock technology (“their computer’s speakers should create some sort of cock-shaped soundwave and plunge it repeatedly through their skulls”), but I guess it’s a start.

They also have sites in Australia, New Zealand, France, and UK

Being Dead Wrong

I like to think that I have a great intuition and am very good at predicting things. I also sometimes feel that I suffer from the Cassandra syndrome, as people don’t listen to my prediction as much as I would like them to.

This made me think about the times when I made ridiculously bad predictions. Here’s a list of what comes to mind off the bat:

1. When I was young I thought that programmers will soon write a computer program for writing computer programs, and that computer programming as a profession does not have much of a future.

2. I thought that architectural drawings will always have to be done by hand, as you can’t print out plans on dot matrix printers (the only printers I’ve seen at the time). I thought, sure, you can program some straight lines and such, but you’ll never get beautiful detailed drawings with all kinds of details.

3. I thought that Handspring would become the dominant player on the handheld market the same way that IBM did: by opening up the peripheral standards.

4. I thought that Diamond Rio would be huuuge and that Diamond Multimedia would become the hottest company ever because they were first on the market with an mp3 player.

5. I thought that Apple would just shrivel up and die, and if not, that I would certainly never completely switch to Macs.

Whewww, man. Those are some doozies. How about you, my readers?

Link Love 1

Stack Overflow: What do programmers listen to when they write code?

The best note taking tool in the world, Evernote, finally released an API. I really, really love Evernote.

Jesse Reklaw has a new Slow Wave book, The Night of Your Life, out. You can get signed copies from Slow Wave website or buy them at Amazon. Slow Wave is in the top 5 of my favorite comics, and Jesse drew the cat and programmer graphic used in the masthead of my site.

For some reason, my former co-worker Sean posts more New York City photos than I do these days.

State of the blog 2008

A while ago Merlin Mann gave a pretty neat definition of what makes blogs good:

” 1. Good blogs have a voice.
2. Good blogs reflect focused obsessions.
3. Good blogs are the product of “Attention times Interest”
4. Good blog posts are made of paragraphs.
5. Good “non-post” blogs have style and curation.
6. Good blogs are weird.
7. Good blogs make you want to start your own blog.
8. Good blogs try.
9. Good blogs know when to break their own rules. “

My list would have been very similar, and I am pretty sure deadprogrammer.com is a good blog. God knows I try :)

On the other hand, I seem to have about the same thousand readers I had 4 years ago. I was kind of hoping that my writing and photography would work on their own, bringing me a steady stream of new readers. It does not seem to be working. Every time I try to put some extra effort into a post, my hopes of somebody linking to it, digging it, twittering about it, or even just leaving me a thoughtful comment are dashed. After a few years of this, it gets harder and harder for me to write. My posting frequency is not what it used to be.

Several hundred readers still subscribe to my blog from Livejournal. The majority of the rest resulted from a single Boing Boing post. I don’t really think it’s the most interesting post I ever wrote, but to this day it brings in about 25% of my search engine traffic, and the majority of outside links.

Back then I actively tried submitting my posts to Boing Boing. Boing Boing submission form can rival magazine publisher’s black hole. I finally practically begged Cory Doctorow in a personal email to take a look at at the Starbucks post (which the submission form vaporized previously), and I did get that link.

I tried to get Jason Kottke interested in my blog, especially since it seems to me that a lot of the stuff that I write about is very much up his alley, but after a chat or two I got tired of pitching. It just does not fell right for me to buttonhole busy A-list bloggers – hey, look, I wrote this, you might like it.

Anyhoo, what’s the point of all this? Apparently I suck at PR and self-promotion, and I would really like some help here from the audience. Do you know somebody who’d enjoy reading Deadprogramemer.com? Please, tell them, especially if it’s an A, B, C, D or E list blogger. Please?

Surf Naked

I had to clean out my in laws’ Windows laptop this weekend, and this was my first major contact with Windows in weeks, outside of testing IE6 quirks on Parallels. The horror that is Sharepoint is only a distant memory. Even more distant (and not as unpleasant) memory of MS Commerce Server surfaced recently.

I received this shirt as a gift from Microsoft developers, some of whom actually wrote major parts of Commerce Server and Site Server when I visited Redmond. They were very competent, knowledgeable and sociable. I remember being surprised by seeing a lot of Perl books in one office, and being told that for parsing log files even the mighty Beast of Redmond relied on Perl.

I did not get to see Lake Bill or the Microsoft museum, but I did have a Blibbet Burger at the cafeteria. My favorite piece of swag was this polo shirt which I almost completely wore out. It is about to completely disintegrate, so I decided to scan the logo from it.