CRUD Ain’t Hard

And now for a little exercise in armchair software architecture — the most despicable coder’s pastime. Dear non-coding readers: despite its name, this blog is still mostly not about programming. Just skip this post or something. Dear coders, many of you will probably disagree with me. I am not a very good or accomplished coder myself, and you probably should not be taking your advice from me. But then again, I could be right, so keep your mind open.

You might have been aware of the very popular, but uptime-challenged social networking tool called Twitter. They have one of the best problems to have: too many very active users. The site is so popular that it constantly goes down and displays and “over capacity” screen that the users have nicknamed The Fail Whale.

Rapidly writing and displaying short chunks of text with high concurrency on the web is not one of them unsolvable problems in programming. It’s not easy, but with right people and tools Twitter could be rewritten inside a month. Twitter founders should do some soul searching. Meanwhile the critical mass has already been reached, the niche for bloggers who want to SMS instead of blogging is big, and even horrible uptime can’t this service. I use it myself.

There is a lot of speculation in the blogocube about whether the reason behind the Fail Whale is the wrong choice of technology — the highly hyped and sexy Ruby on Rails and if it can “scale”. Or is it just simple incompetence?

To me Ruby on Rails falls into a class of technologies that are affected by what I call “the VRML syndrome.” Basically, if I wait long enough the hype will go away, the recruiters will stop posting job listings requiring 4 years of experience in a 4 month old technology, books as fat as my two fists will stop being published, and I will not have to learn it.

What’s the problem with Ruby on Rails? Well, it’s the same problem that slightly affects the content management system that I am currently working with (Drupal), and is the reason why I completely gave up using Microsoft web technologies which are saturated with this shit. See, software craptitechts all of a sudden decided that writing CRUD applications is too difficult for regular developers, and complicated GUI tools and frameworks need to be created to help the poor things. CRUD stands for “Create, Read, Update, Delete” and is just a funny way to say “a browser-based application chock-full’o forms”.

The default way to build these is to rather simple. You hand-code the html forms, then you write functions or classes to deal with the form input — validators and SQL queries for creating, updating and deleting. Then you write some code that will query the database and display the saved data in various ways: as pages, xml feeds, etc. None of this is difficult or non-trivial. Bad coders don’t do a good job of validation and input sanitizing resulting in the Little Bobby Tables-type situation, but these things are not very hard to learn and there are great libraries for this.

Ruby on Rails makes it very easy to create CRUD apps without hand-coding forms or writing SQL. RoR goes to great lengths to abstract out SQL, not trusting the developers to do it right. SQL is more functional than procedural, and thus a difficult thing for many programmers to grasp, but it’s not that hard. Really. SQL is located far enough levels from the machine that abstracting it out becomes a horrible thing due to the Law of Leaky Abstractions. Even when you have full control of SQL queries optimizing them is sometimes hard. When they are hidden by another layer it becomes next to impossible.

In short, RoR makes something that is easy (building CRUD apps) trivial, and something that’s hard – optimizing the database layer next to impossible.

In Drupal there are two modules, CCK and Views that allow you to create CRUD entirely through web interfaces. This is a feature that exist in just about every major CMS, it’s just that in Drupal it’s a little buggier and overcomplicated than necessary. These are fine for small websites and are really useful to amateurs. The problem arises when these are used for high traffic websites.

I think that a lot of people will agree with me that writing HTML and SQL queries using GUI tools is amateur hour. You just can’t make a good website with Microsoft Front Page. You can’t, you can’t, you can’t. But in Drupalland it’s all of a sudden fine to use Views to build queries for high traffic sites. Well, it’s not. Dealing with Views and Views Fast Search has been an ongoing nightmare for me. Hell is not even other people’s code in this case. It’s other people’s Views.

RoR, Views, CCK are one level of abstraction higher than you want to be when building a high performance application. The only way the can be an “Enterprise” tool if your enterprise is a) run by a morons that require 100 changes a day AND b) has very few users. In short, if it’s an app for the HR department of a company with 12 employees – knock yourself out. If you are building a public website for millions of people – forget about it.

Your, Deadprogrammer.

P.S. Yes, I know, you can abstract just about everything and reduce your software application to a single button labled “GENERATE MONEY”. You have to be a very smart LISP developer for that.

New Billboard Day Effect : How to Advertise More Effectively on Your Blog

Advertising. “The Engine of Commerce”. Ideally, it should work like it does in the Simpsons episode 2F12 “Homer the Clown”:

“In the middle of driving down the highway, Homer skids to a halt in front of a billboard.

Homer: [gasps] It must be the first of the month: new billboard day!

Homer: [reading] “This year, give her English muffins.” Whatever you say, Mr. Billboard! [skids off]
[stops suddenly at another billboard for barbeque sauce]
[cars collide behind him and explode]

Homer: [reading] “Best in the West.” Heh heh heh, that rhymes!
[looking at the next one] “Clown college”? You can’t eat that.

At the power plant, Homer piles his purchases (including MSG, “Best in the West”, and English muffins) at his work station. “Well, I got everything I was supposed to get. I’m not going to enroll in that clown college, though…that advertisement had absolutely no effect on me whatsoever. In his daydream, he imagines himself sleeping and dreaming of himself eating a sandwich. The billboard for the clown college batters its way into his thoughts. The Krustys on the billboard start dancing to circus music.”

Of course, Homer enrolls in the clown college. Having never enrolled in a clown college because an ad told us to, we all go on thinking: “advertisement had absolutely no effect on me whatsoever.” It can’t possibly be true: bajillion dollar industries, such as advertising don’t simply exist if they are not effective.

During the dot com bubble even large companies mostly failed to earn much from banner ads. Even the heaviest online ad campaigns did not seem very effective and suffered horrible clickthrogh rates. Online ad companies escalated the war for clickthroughs by inventing obnoxious popunder, popover and floater ads. The more the ad was like a flash-bang grenade mistakenly used by NYPD on an elderly woman, the better. For instance, many sites started using larger sizes of vertical banners known as “Skyscraper.” That was not enough though – extreme, flash-driven skyscraper ads with movies and sound, capable of crashing browsers and known as “Godzilla” and “Pagekiller” started to appear.

The founders of Google decided to address this issue, and as a result, made bazillions of dollars. As a former googler remembered:

” Besides, Larry and Sergey hated these kinds of advertising. In fact they hated most kinds of advertising as inefficient, dishonest and a total waste of people’s (meaning their) precious time.”

We all know that AdWords and AdSense, Google’s advertising programs managed to earn so much money through unobtrusive, mostly text ads. The winning strategy was “relevancy”. Google’s server would read in the page where the ad were to appear, and serve up a relevant ad.

For instance, after parsing pages on chupaqueso.com, a site dedicated to a cheese snack invented by web cartoonist Howard Tayler, in theory shows ads about cheese. And after reading about chupaqueso’s cheesy goodness, I might indeed be in the mood to buy some cheese online.

On the other hand, the AdSense algorithm is not too efficient. On some pages in the abovementioned site it serves ads like this:

Yes, indeed, amongst Howard Tayler’s readers there are a lot of computer geeks. I know I am not a typical web user, but I am a pretty typical web developer. And I have zero desire to “Boost XML app performance.” I also have all the “ODBC drivers” that I need.

Many of you, my readers, are bloggers or have regular web sites with AdSense ads. Look at them. How many you’d say are “inefficient, dishonest and a total waste of people’s … precious time”?

I say – about 99.5%. And clickthrough ratios are pretty horrible. People try to tweak them by playing around with ad types, look and feel, positioning and excluding advertisers, but it’s all rather ineffective.

Google’s ads only pay if people click on them. In the TV, billboard, magazine and the type of advertising that people tattoo on their bodies there’s no such things as clicks. You get paid depending on how many people see the ad. It works really well if you need to make people remember your company’s name or logo.

Side Note:
When I was little, in Odessa ( Ukraine, Not Texas) somebody scribbled in almost every public phone booth “[Some girl’s full name] is a whore.” In a city of about a million people this worked like a charm. The mindshare that that advertisement delivered must have been off the charts.

These 99.5% of unclickable ads can be divided into two categories: a) ad campaigns that build brand’s awareness almost for free and b) those that indeed waste everyone’s time and money.

I don’t think I ever clicked on any Vonage ads, even though I’ve seen thousands of them. They worked without any clicks — if I did not also know that their customer service sucks and reliability is horrible, I’d have their VOIP service now.

The ads that nobody ever cares about still do get some clicks. When people come by a useful and interesting site, they tend to click on random ads so that the site owner would get some revenue. This is the untraceable portion of a much scarier phenomenon called click fraud. I am not even going to address this here.

In short, I feel that even though Google’s ads are a step in the right direction, AdSense sucks, especially for a blog with a smallish audience, such as mine. The useless, stupid ads that clog AdSense are a waste, even though they might generate a few “pity clicks.” Only half of my ad revenue for the site came from AdSense last year. The rest came from my experiment that I think will be of great interest to everyone.

My thinking went like this: I want to serve ads that are extremely relevant to my blog posts and interesting to my audience. Even more importantly, they must be selling something that I would be interested in. Ads I’d click on.

When you have limited advertising space, the problem with AdSense is that it often tries to sell things that your readers don’t want. What you want to do is advertise things that people aready want. As an example of such salesmanship, let me direct you to a post on the very popular waiterrant.net, where The Waiter describes selling dessert to calorie-conscious women:

“”Ladies,” I say sweetly, “We have some excellent desserts tonight.”

“Oh, nothing for me,” Bubbly Blonde replies.
“No dessert,” Severe Brunette says, holding up her hand.
“Me neither,” Lawyer Babe says firmly.

The fourth woman, a Soccer Mom type, looks at her companions and sighs. She wants dessert.

I see the longing for chocolate in Soccer Mom’s eyes. She’s my weak link. My in.

“Would anyone like some coffee?” I ask. Suggesting coffee is the first stage in selling dessert to calorie resistant ladies.”

“The ladies pay the bill, tip well, and leave. As I watch them go I think about how I got them to order dessert. To be a good salesman you have to have a seductive quality about you. Don’t believe me? Look at pharmaceutical reps.”

That’s what I want to do! This means that I need to find something that will be the equivalent of selling chocolate dessert to Soccer Mom types.

I believe that my 1000 readers are a lot like myself. And what do I spend a huge amount of money on every year? Books, movies, cds and gadgets. Also I purchase some rather esoteric items on eBay too, but the majority of my spending happens squarely at Amazon.com. My wishlist there is humongous, and in fact, I spent my advertising revenue there.

Luckily, Amazon has a pretty generous associate program. You can link to any of the products they sell and get a cut of the sale price, if the sale happens as a result of your clickthrough. In fact, you get a cut of the entire shopping cart amount (I am not sure, this could be only the items that were added after the click). In any case, it’s decent money, and most importantly, a great selection of new and even used items to sell.

What to sell, of course depends on your audience. I found some success selling items that tempt me. In fact, many times it’s the items that I am planning to buy or already bought.

In some cases, relevancy is important. My article with pictures from Fog Creek’s party sold 4 or 5 of Joel’s books. It was a combination of a very desirable in this particular audience product with a closely related article. Interestingly enough, I tried to sell the toy that you can see in the picture as well, but none sold. As I own both books and don’t own the toy, this seems logical.

I might have tried selling flat panel monitors and Aeron chars (WOW, Amazon sells them too! ) ,that make Joel’s office so nice (in fact, at home I have the same exact dual monitor setup, an Aeron chair and a window with a view, and I had id before Joel wrote about his bionic office). These are big ticket items though, and the likelihood of someone buying them on a whim is lower. But then again, so are rewards.

The relevancy does not matter as much as I thought, though. For instance, I advertised “Make” magazine subscriptions and Shure E2c headphones, and sold a few.

In fact, I think that the approach to selecting products should be somewhat similar to the one that Kevin Kelly uses for selecting items on his website Cool Tools:

“Cool tools really work. A cool tool can be any book, gadget, software, video, map, hardware, material, or website that is tried and true. I am chiefly interested in stuff that is extraordinary, better than similar products, little-known, and reliably useful for an individual or small group.”

In short, advertising video iPods is good, advertising “The world’s greatest 3D IM” is not!

Side Note:
My former co-worker won a $300 gift certificate for a certain gadget catalog in a contest. Now, he’s a guy who spends a lot of money on gadgets, like the uber geek that he is. I mean, he owns planetofthegeeks.com domain. But despite that, he had a lot of trouble picking something to spend $300 of free money on in that catalog! Not only was everything overpriced, but there were very few things he’d be interested in owning!

The great thing about selling items from Amazon is that you know that the prices there are very Wal-Mart-like, and most of your readers already shop there. Some people prefer not to patronize Amazon because of software patents or other issues, but there are “organic” alternatives, like for example Think Geek (in fact they sell through Amazon too).

The one gripe that I have with Amazon is the difficulty in creating the links. The tools that they provide want you to use iFrames to create image wrapped links, which of course do not work well in RSS Readers. This brings me to my final point, the specifics of blog advertisement.

A blog is a two-sided entity: it generates page views from people who don’t use RSS aggregators and those who come in from search engine referrals. And then there are the views from within RSS aggregators, in case you are serving up the entire text of the article in the feed. Some blogs don’t do this, serving up only the title or a title and a teaser. The thinking is, readers will click through to the page where they will see ads and thusly generate revenue. Some do this because they don’t serve ads and want to limit their traffic, and yet some do it because they use a default setting in their blogging software and don’t know better.

The great thing about my advertising scheme is that you can serve ads in-feed. A New York blog Gothamist, for example serves atrociously uninteresting ads that repeat. At some point they had a long run of a flashing ad for something that made me unsubscribe from the feed. If they started selling interesting items, they could greatly increase their advertising revenue.

Advertising my way does not detract from regular content and isn’t cheesy. It is clearly marked, unlike those fake editorials in magazines and newspapers. Advertisement can be entertaining in itself! Since the early years of Sears, Roebuck and Co. catalog, people look through catalogs like Levenger, Victoria’s Secret Penzeys Spices and Think Geek for fun! My wife has a lot of gardening catalogs that she looks through now and then. After finishing an interesting post, readers would not mind learning about an interesting gadget or book they might want. In fact, they might already be in the mood to buy it! There is no reason to serve partial RSS feeds with this type or advertising.

P.S. I turned off comments to this article because for some reason it attracts a ridiculous number of spam comments. If you would like to contact me, see about the author section. I also changed to a different way of displaying Amazon’s related items.

Monzy is Baaaaaaack!

The one, the only, the magnificent Dan Maynes-Aminzade started posting again. It looks like he does not believe in (or simply does not have time to set up) those newfangled xml feeds or html anchors, but he’s back and posting again. Scroll down to Monday, March 28, 2005 entry on monzy.com and be dazzled with his frat party science and engineering skills. Ice luge technology is not as simple as it seems.

Coming soon to deadprogrammer.com : “Deadprogrammer Visits Japan Part III : Day of the Tentacle”; “Dumpster Diving at the Flatiron”; “Ancient pre-Joel on Software Text Found at the Strand”; “Deadwood Train” and much, much more. Do not miss!

First Post!

I finally installed MovableType, converted my Livejournal entries and created a very rough preliminary design featuring a graphic made for me on commission by the very talented Jesse Reklaw of Slow Wave fame.

The masthead, secondary pages, xml feeds and all the shiny features like categories, pings, trackback, etc still need a lot of work. This is why I suggest you don’t subscribe to XML feeds yet – there likely to be a lot of change in them.

MovableType has a feature for timing the release of entries that actually works well (unlike Livejournal’s) and since I write posts in bursts on weekends, I’ll try to time them so that they will spread through the week, making Deadprogrammer.com an almost daily blog.

I have a few dozen topics for posts right now, but I am also working on lostindication.com (my photography portfolio), organizing my notes (of which I have a few notebooks), finishing deadprogrammer.com, working on a few other interesting projects. Since I have a day job, the progress is slow.