On Facebook Advertising

Two things happened recently: Facebook’s IPO fizzled and there was a slew of articles and blog posts about how Facebook ads are ineffective. The majority of the articles follow the same song and dance: we spent $250, got some likes from random people and nothing changed. The one standout it GM dropping Facebook ads. GM’s budget was 10 million dollars, and the way I understand it, this is their equivalent of $250 for a small business.

While I personally dislike Facebook tremendously and want it to fail, I think that the FB ad-bashing articles are off base. Facebook ads are a great deal. I am not an advertising professional, but I drank with a lot of them and I think picked up a reasonable amount of knowledge about the industry. I also stayed at … hmm, can’t remember the name of the hotel, but this is a reference to the TV ads where people say “well, I’m not a [professional that has to be very smart and skilled] but I stayed at [name of the hotel]”.

There are many, many bad deals in advertising. There are big-name websites that have “respectability” plus a huge saleseforce. These guys can charge $15, $20, $30, sometimes even $50 per 1000 banner impressions. cost for 1000 impressions is called CPM: cost per M where M is the Roman numeral 100. At the higher price you get gigantic custom ads known by a variety of names: browser crashers, godzillas, superskyscrapers, page fuckers, etc. There are also “sponsorships” where dumb companies buy little badges on new sites with next to no traffic.

How do these things get sold? Well, the agency people who control how the ad budget is spent are often young and get a lot of free drinks. Also, ad sales people are very good at what they do.

Then there are cut rate ad networks that have 5-20 cents CPM. The traffic for these comes from all kind of low quality sites: lyrics websites, guitar tab sites, all kinds of web farms, dating sites, porn and near-porn sites and the like. Big sites sometimes buy traffic from these sources when they don’t have enough “inventory”, but they’ve already sold a lot of high CPM ads. These are a pretty bad deal, and I suspect much of the traffic is simulated by bots. It is cheap and there’s a lot of it though. Targeting options are pretty weak with the exception of plentyoffish.com: POF allows for very detailed targeting. If you want to target only Canadian redheads between the ages of 18-22 – you can.

Google ads are a better deal: you are targeting people that are searching for something. Google has pretty sophisticated algorithms for detecting fraud, but I suspect click fraud is still rampant. Google’s predominant model is charging per click instead of per 1K impressions, and you have to compete with other people for hot search keywords. For instance, ambulance chasers pay ridiculous money for “mesothelioma” keyword. Mesothelioma is a type of cancer caused by asbestos, and layers apparently can make crazy money suing on behalf of people who have it. A click on an ad can fetch as much as $10 or more. There are many other expensive keywords where even a single click is worth paying actual humans to click on them from time to time. There are also pity clicks and punitive clicks: sometimes people click ads to support sites that they like and to punish ones advertisers they don’t like.

Facebook ads by my estimation cost about 1/10th of Google’s, almost in the low quality network territory. The targeting is amazing though: Facebook knows a lot about what people are interested in and lets you have a sniper-like precision of putting your ad in front of them. Do you want New Yorkers who are into planted aquariums, fishing and knife sharpening? If there are 10 people like that you can reach them on Facebook for a few bucks. Unlike Google users who are searching for something, Facebookers are there for ogling hot people and playing Farmville. They tend to ignore the ads, but you can get an even better deal by buying CPC ads. Massive click fraud is difficult on Facebook, most of these clicks are real.

There are many things about advertising in general that are mysterious: starting with the “if I almost never click on ads then who does” to “do tv and magazine ads actually make me buy anything”. Are funny ads where you remember the joke but not what was advertised worth the money? Are people swayed by car commercials? How about those Coca Cola billboards? How about those ads in New York taxi cabs and ads that annoy in general – are they effective? QR codes ( http://wtfqrcodes.com/ ) – are they the advertising equivalent of “free public wifi” zombies? I have a hunch, but I don’t really know.

What I do know is that compared to other options FB advertising is a pretty good deal at current pricing.

On Learning to Code. Or Not.

Alert! Jeff Atwood wrote an excellent post about the “learn to code” movement.

He starts with a tirade full of incredulity about Mayor Bloomberg’s New Years resolution to learn to code with Codeacademy.

“Fortunately, the odds of this technological flight of fancy happening – even in jest – are zero, and for good reason: the mayor of New York City will hopefully spend his time doing the job taxpayers paid him to do instead.”

Let’s put aside the princely sum of $1 that His Honor collects from the job. Let’s even put aside that Mayor Bloomberg is doing exactly what he’s supposed to be doing – promoting New York’s bustling tech industry. More to put aside: our Mayor happens to be a technology pioneer with a ridiculous IQ.

This all comes down to a very difficult question: should people learn nerdy things when they have little use for them, just for the sake of learning.

I remember a Livejournal discussion that was hashed over and over in the Russian-speaking community. A math teacher was stumped by a question from his student: why was she supposed to learn about trigonometry when she wanted to become a beautician. The teacher did not come up with a good answer, but the livejournalers did dig up some awesome reasons. One well meaning pro-education-for-the-sake-of-education zelot said something to this effect: well, if you work with nail polish, tangents and cotangents figure prominently in formulas that deal with reflectiveness of thin films. That will lead to a greater understanding of how and why nail polish looks the way it does.

On the surface it may seem that Mayor Bloomberg has about as much need to know how to code as much as a beautician needs to know about sines and cosines.

There’s more: executives who learned a little bit about writing code at some point tend to say the following phrase “oh, I don’t know much about writing code, just enough to be dangerous”. They say it with this look on their faces:

Jeff takes this further with the plumbing analogy: since almost everyone has a toilet, should everyone take a course at toiletacademy.com and spend several weeks learning plumbing?

Normally I’m against education for the sake of education. I once argued for a whole hour with a co-worker who felt that _any_ education is worth _any_ amount of money. I did not know at the time that he held degrees in Psychology of Human Sexuality, Biology, Sociology and Communications. He must have been on to something: he made an amazing career while mine took a nosedive soon after that discussion.

Here’s where Jeff is wrong (I know, this is shocking, Jeff being all wrong and such): it is better to push people to learn incongruous things then to tell them that this is a bad idea. Steve Jobs learned calligraphy in college and it turned out to be super useful. He might not have become a master calligrapher, but man, did that piece of esoteric knowledge change the world.

When I was in college I badly wanted to take a scientific glass blowing class, but did not. I deeply regret that.

Are there people who learned plumbing from This Old House annoying contractors? Yes. Are self-install refrigerator ice maker lines causing millions in water damage? Yes. Is the world better off because Richard Trethewey taught it some plumbing? Absolutely.

If anything, attempting to learn to code will make people more compassionate towards coders. I do believe that people who are not already drawn to programming are not likely to become programmers, more than that, they are not likely to sit through a whole RoR bootcamp or worse. Learn to code movement is not likely to lure in bad programmers, but it might give people some understanding of what coders go through and maybe be more hesitant to have loud yelling-on-the-phone sessions near their cubes. Mayor Bloomberg, who enforces open workspace policies everywhere he works, might understand why programmers need offices. Jeff, let His Honor code a bit.

How To Speed Up the Web (And Gain A little Privacy)

Let’s face it: Facebook “like buttons” are dragging the pageload speeds down. They are just the pits, and they are slow mostly because the Great Zuck want to know what you are up to. Screw that – install WidgetBlock for Chrome, your pages will load faster, your browser will crash less, and Zuck will know less about you.

This has been a public service announcement. Also, don’t forget to back up – hard drives only live for a few years.

[update] I seem to be singling out Facebook buttons. And I do – they are the worst offenders in terms of performance. On the other hand Widgetblock works for all kinds of widgets.


I, for one, welcome our new social overlords

Disclamer: I thought that Google Wave was an excellent idea, so you can safely disregard my blathering here.

Here’s what I’m picturing in my head: Google has approached Facebook and Twitter on the playground. Twitter stole a piece of Facebooks lunch, but can’t really hold onto it. After a few threats and a bit of running around and a few ineptly thrown punches Google got itself into position to really clean Facebook’s clock and take its lunch. Foursquare and Groupon which earlier evaded Google’s punches in the most ebarracing for Google way possible are likely to be lunchless later. It is rather strange that Google does not go after scrawny TV Guide and White Pages – it looks like their lunches are not that tasty.

Yes, it’s just another social network. Yes, Google has a track record of failing fast and frequently (which if I remember correctly is a “good thing”). But remember, a bunch of incompetent coders received such an applause, press coverage and a whole evem some money to build a Facebook alternative. And finally mighty Skynet is doing the same thing. I think the company behind the mighty Skynet and the future parent of our robotic overlords has a chance against a bunch of compiled spaghetty PHP.

P.S. Zuckerberg and his approach to privacy creeps me out, so I have deleted my Facebook account and turned it into a blank account used only for work (writing Facebook apps, testing and such). I’m completely fed up with the character limit on Twitter – it’s nothing more than a feed from my blog. But I do want to share photos, and I do want to post shorter, non-blog-worthy thoughts. I’m really rooting for Skynet here.

Gmail and tracking numbers

Hunting around for UPS, Fedex,USPS, and Japan Post tracking numbers in Gmail is no fun. I really wish there was a way to aggregate all the shipping numbers in a single Gmail plugin which would at a glance tell me where all the crap that I ordered is at any given moment. Google already knows how to tell a tracking number from all other strings, and there are apps for iOS that aggregate tracking (unfortunately you have to manually type in all the tracking numbers). A Gmail plugin that would keep track of tracking numbers would be great – maybe anyone with a bunch of mythical %20 percent time at Google will implement this…

The New Blogging Manifesto – Or a 3 Ways (4 Ways) To Make Blogging Easier

I noticed that twitter sapped much of my blogging mojo, and I am not happy about that. Wasting a perfectly good photo from my iphone on a twitpic feels painful: it takes an extra effort to view it, and a triple effort to annotate. Here’s my message to Twitter: images should be seen but link urls should not. It’s the other way around, you wildly successful jerks.

Castrating my thoughts with a character limit is unpleasant as well. How much information do I need to sacrifice for the ease of posting? Twitter is like Procrustes, a Greek mythological dude who would chop off the legs of his guests to fit the length of his bed. Twitter’s procrustean limits mess with my procrastination. See, a painful pun like this is impossible on Twitter.

Facebook has much saner character limit and link/image handling, but I really don’t want to place my junk in the “walled garden” of “a host of a party who goes through the pockets of the coats his guest hang up” (I don’t remember the source of the second metaphor, but I like it a lot). I got tired of twiddling settings every time Zuck’s army decided to opt me into yet another privacy nightmare. I dumped my old account and created a new one that I only use for work-related testing and development.

So, over the weekend I redesigned deadprogrammer.com. Here are my new rules for blogging:

1) The blog post input form goes on the front page. I’m basically aping WordPress’ P2 theme. Having a post form staring you in the face instead of being a few clicks away is amazing. It changed the way WordPress developers blog, and I’m hoping it will do the same for me (it seems to be working).

2) Big images. I’m tired of small images. The screens are big, the bandwidth is cheap, almost everybody has a fast connection, my camera takes amazing pictures that lose much of their life when squeezed into 600 pixel width. Then New standard width is 1000 pixels.

3) The P2-style post form is the first step on removing friction out of posting. But that’s a topic for another post – I need to keep my missives manageable. I’ll break things up: there will be pithy posts, and medium length ones, and then there will be long David Foster Wallacian ones (I just need to figure out the best way to do footnotes).

4) Facebook and Twitter will get posts from my RSS feed. That’s all they are good for.

iphone4 antenna rant

There are two types of people in this world: those who say that crushed bedbugs smell like expensive Cognac, and those who say that Cognac smells like bedbugs.

I really wish the whole Apple antennagate would be over. Because I’m tired of explaining my position on it to everybody who drools over my iphone.

I have 3 points to make:

1) The office where I work has super ridiculously terrible ATT reception despite being in direct line of view from a brutalist style building containing ATT switches. So does an underground subway station where I change trains on the way to work. The rest of NYC has simply terrible ATT reception.

I’ve tried at the office, I’ve tried underground. I’ve tried cupping any which way. I tried moistening my palms. I can’t get to get a reliable bar drop thing. Sometimes, in fact, I got more bars. Oh, and that underground subway station? None of the older iphones ever had reception there.

This is all clearly a a product of me being a “fanboi”, and being under the heavy influence of Steve Job’s reality distortion field. I think they have mobile generators transmitting that.

2) Sometimes getting more bars when cupping iphone4 gives me an idea that now it is possible to make a case that will actually boost iphone reception by possibly attaching a yagi or another type of big directional antenna to the case. Maybe even something gigantic, like the apartment tower antenna or one of those huge homemade antennas that I used to make for crystal radios.

3) Right now I’m vacationing in the Ukraine. Here you can buy a sim card for $1 that gets you a phone number. On every corner there are agents and machines that allow you to refill these sim cards with ridiculously small amounts of money that get you ridiculously cheap minutes and data. There’s no standing in line while ATT employees are chatting / checking their social networks, scratching their asses, etc. There’s no hassle with forms, accounts, etc. You spend a little money, you get your phone to work. Most phones are unlocked. Reception is excellent everywhere. I hear they are working on making iPhone’s mini sims working with iPhones. People are ready to pay just about any price for the 4th iPhone. Apple would do much better with a simple business model: phones for money here first, and then in the US for us, who have to deal with ridiculous contracts and lock-ins. Grandstanding politicians would gain my vote if they did a little trustbusting in the cellular service industry.

Are Tables Important?

I was talking to a former co-worker about Inc Magazine’s cover story about Markus Frind and his very profitable, but godawfully ugly dating website plentyoffish.com.

My co-worker (a programmer) loaded up the website. He took a quick look around and opened the source of the ratings page. Giggling like Bevis he could not believe what he saw: a gradient bar that was coded as [gasp!] an HTML table with bgcolor attributes.

It looked like this:

And was coded like that:

<table border=0 cellspacing=0 cellpadding=0 width=100%>
<tr height=5><td bgcolor=#204080><img width=1 height=5 border=0>
</td><td bgcolor=#202F70><img width=1 height=5 border=0></td>
<td bgcolor=#3F2060><img width=1 height=5 border=0></td>
<td bgcolor=#5F2050><img width=1 height=5 border=0></td>
<td bgcolor=#7F1F4F><img width=1 height=5 border=0></td>
<td bgcolor=#90103F><img width=1 height=5 border=0></td>
<td bgcolor=#B0102F><img width=1 height=5 border=0></td>
<td bgcolor=#CF0F1F><img width=1 height=5 border=0></td>
<td bgcolor=#E0000F><img width=1 height=5 border=0></td>
<td bgcolor=#F00000><img width=1 height=5 border=0></td>

He was going on and on and on about how tables are bad, and mwu-ha-ha-ha — look at this.

I was fully expecting him to take umbrage at the logo, the overall look and feel of the site, at the grotesquely skewed photo thumbnails. But no, all he was seeing is that Mr. Frind “used a table”.

I tried to tell my co-worker that despite “tables” or ugliness this website generates tens of millions of dollars of profit to its creator, that it has as much web traffic as Yahoo while being served a small handful of very powerful servers, that it was created and maintained by a single person who gets to keep most of the profits – but to no awail. The kid could not get over “tables”.

A famous hacker JWZ once was asked about his feelings about “an open source groupware system”. In a famous rant that followed he produced some of the best advice importance that I’ve ever seen:

“So I said, narrow the focus. Your “use case” should be, there’s a 22 year old college student living in the dorms. How will this software get him laid?”

While I’ve never heard of HTML tables (not the furniture kind) playing any role in getting laid, plentyoffish.com must have resulted in a mind boggling amount of action.

Plentyoffish.com, being a technological and aestetical abomination that it is, is firmly rooted in the lower, fundamental layers of Maslow’s Hierarchy and my Web Heirarchy.

At the most basic people need oxygen, water, food, to take a dump/whiz, sleep, sex, and a predictability in environment.

On the web people need hypertext, images, search, speed, and community features. If you provide all of these for a topic that is important to people, you will be successful. Start thinking about “html tables vs divs” first, and likely you won’t get to the important stuff.

Doing it another way – saying, look, I’ll do a site just like plentyoffish but prettier and without HTML tables does not work very well: Frind’s competiors at okcupid.com who set out to do just that are not succesful in toppling plentyoffish.

Ugliness for the sake of ugliness is not a good thing. In the long run people want things to be pretty, like Apple products and not ugly like Microsoft products. But taste, being pretty high up in the pyramid of needs only becomes a factor after all the basic needs are met.

December 11th – National Backup Awareness Day

Something horrible just happened to Jeff Atwood aka CodingHorror.

“ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images.”

“I had backups, mind you, but they were on the virtual machine itself :(“

It’s a times like these we start wishing for a time machine, a cosmic undo button or reversible computing.

Jeff’s blog was read by tenth of thousands of programmers and system administrators for many years. It contains information that is very valuable for these people, and represents an unthinkable amount of hours spent by Jeff. An agency rate for somebody like Jeff is between $250 and $500 an hour, but this is like appraising a priceless family heirloom.

I am not going to go through the motions of telling everybody how to backup things, about how important offisite backups are, how disk drives are fragile, how I don’t trust virtual servers, how raid is not a backup strategy, and how version control is not backup strategy, etc, etc. JWZ wrote a good article about backups.

Here are things I want to say. First, we are all not backed up sufficiently and likely have already lost data that we would want back.

I can’t find my grandmother’s recipe book (I still hope it’s only lost), my wife’s first email to me, my first web page through which she found me, my first job search web page that had a picture of the Twin Towers and said how I wanted to work there, my early school grading papers, a rare book about fishing in the Black Sea, a stamp from the Orange Republic that used to be in my father’s stamp album, the password to my very short-numbered ICQ account. A lot of stuff.

All of our digital information is susceptible to an electromagnetic pulse, fire, flood. Spinning platter hard drives are particularly bad – they have very short lifespans measured in low single digit years. CDs are even worse – aluminum inside them rots (I have a cd with a lot of outlook emails that reads as a blank filled with 1s).

So the first thing that I would like to mention is that if you never simulate a failure, you’ll never know if your stuff can be replaced. It’s not an easy thing to practice, though – restores and failovers are tricky to do.

A few jobs ago we were getting a fancy new load balancer set up. It was up and running, and supposedly we had failover: if one of the servers died, we would not even need to do anything, the backup servers would pick up the slack. I suggested that we should test it by pulling the network plug on one of the machines off hours. My boss would not allow that, saying that we could possibly break things. My argument that it’d be better if something like that happened when we were ready it would not be as bad if it happened when the actual failure would occur. When the actual failure did occur the load balancer did not switch, and we had an outage that was a good deal longer (it happened at night).

Load balancers are not backup solutions, but this story highlights an irrational streak in system administration: nobody wants to practice failure: it’s just too nerve-wracking, and a lot of hard work. It’s much easier to assume that somebody up the line did everything correctly: set up and tested backups, startup scripts, firewalls and load balancers. Setting up and validating backups and testing security are thankless jobs.

This brings me to a another point. The act of taking a backup is not risk free in itself. The biggest data losses that I suffered happened to me in the process of setting up backups. As an example I’ll bring up the legendary story about Steve Wozniak (whom I met yesterday):

The Woz was creating a floppy driver under an extreme time pressure, not sleeping much and feeling sick. The end result was a piece of software of unimaginable beauty: it bypassed a good deal of clunky hardware, and thanks to a special timing algorithm, was fast and quiet. When other disk drives sounded like a machine gun (I dealt with a few of those when I was young), Woz’s purred like a kitten. Finally he wrote the final copy onto a floppy, and decided to make a backup of it. Being dead tired, he confused the source and destination drives, and copied an empty floppy onto the one with the precious driver. Afterward he proceeded to burnish his place at the top of engineering Olympus by rewriting the thing from memory in an evening.

It’s really the easiest thing in the world to confuse the source and destination of a backup, destroying the original in the act of backup! The moral of the story?

Do as much backing up as possible, while being careful not to destroy your precious data in the process. Have an offsite backup. Print out your blog on paper if it’s any good. In fact, print out as much stuff as you can. Your backup strategy should be like a squirrel’s: bury stuff in as many places as possible (well, except sensitive information, which is a whole other story in itself).

8 Pieces of Architectural Advice for CMS

I have some advice for those in the business of building large websites with content management systems.

1) Do not implement search yourself.

Your CMS sucks at search, and so do you. I see this again and again and again. Everyone is implementing search on large websites instead of using Google. Developers are afraid of looking unprofessional. Managers are answer yes to the question “do you want advanced/faceted search” (the correct answer is no – user’s don’t like it and don’t use it). As a result a lot of resources (both server and developer) go into implementing something that Google is awesome at. Even some very smart people, like Jeff Atwood roll their own search, and their users end up going to google.com and typing “foo site:stackoverflow.com”.

Users are very happy with Google CSE, and don’t mind the text ads. Those text ads – well, that’s revenue that you would otherwise would not have, however small this is. If you absolutely can’t do Google CSE – buy their search appliance. If you can’t do that either – well, you better be using Solr.

2) Do not implement comments yourself (unless comments are what you do for a living).

It is extremely difficult to get comments right. Users absolutely abhor comments. Spammers – well, they love it. Luckily, you can just go and get DISQUS to do all the heavy lifting for you. The time saved on using DISQUS can be used on building something else, meanwhile users absolutely love leaving comments through it, while spammers hate it.

3) Physically separate your admin interface from the stuff that is going to be used by your users.

Maciej Ceglowski has some words of advice about not having your blog hacked: cache your output in flat files and hide the admin interface. The benefits of this are tremendous: cached files are fast and secure. You will need to do some fancy footwork to serve up parts that change a lot, but you can do it the same way DISQUS and Google CSE do it – through the magic of AJAX.

4) Sanity check: calculate the amount of RAM in the home computers of all of your interns. Compare that to the amount of RAM in your server farm. Who wins?

5) Use a CDN and/or caching proxy, don’t be cheap. These things will save your butt when Yahoo and Digg will come a-knocking at the same time. I’m not even going to mention Memcached – you can’t get big without it at all.

6) Fight WYSIWYG editors. These things are the worst. They are the Devil. They are a security hole. You never get what you see. People paste from Word. Do I need to go on?

The best middle of the road solution is something like Markdown.

Do not underestimate the user’s ability to learn a few simple rules. When I worked at TV Guide there was this movie database application. Very non-technical editors were using a very scary-looking Unix-based interface at an amazing speed. When I rewrote it as a web interface, it became more “user-friendly”, but they could not enter stuff as fast as before.

7) Make sure you have good backups

8) I know you won’t be able to follow my advice, I know I can’t either. Life is a constant compromise.