Let me tell you about web development and fishing, my two great passions. Here's a fishing story. When I was a kid, i fished off the long piers in the Black Sea. I did not catch all that much, and I mostly thought that was because of my lack of skill and resources: I though if I could cast further, have a better fishing rod, or be able to go out on a boat, maybe I could catch more. Then I noticed that one fisherman was catching huge quantities of fish.

He had an interesting technique. Instead of using a single rod and switching from a place to a place, he's bring ten. Each one was cheap and simple bamboo rod. He'd bait them, and drop the hook in shallow water in clear water, where the sea floor was covered with concrete blocks with holes used to stabilize the sand. I tried fishing near those holes before, but never caught anything. He'd set up his ten rods, and then just wait. An interesting thing happened: after about an hour the fish started biting, and were mostly just catching themselves: all he had to do was walk from rod to a rod and take off the fish. Sometimes just a single hole would be producing, then he would take that rod and catch fish after fish from the same place.

This technique is called deadsticking: you leave the bait motionless, and thus exposed to the fish for much longer periods of time. Most fish grab the bait and run: you don't even need to set the hook, the fish catches itself. When on the boat the same technique often works. Having a number of rods fishing all the time gives you two benefits: it shows you the hot spots and exposes your hooks to more fish.

I see this again and again: a company redesigns a website, changes the core technology used to build it, spends a lot of money, and then the traffic and search rankings fall, and thus revenues fall.

I am pretty sure I know the cause of this: broken links. Any redesign of a website of just about any complexity, especially when technology changes breaks a lot of links. Search engines are like fish: they do not like things moving from a place to a place in an unnatural manner. A fisherman once told me: hey, do you think a Tautog (a kind of fish) ever seen a dead fiddler crab jump three feet up and down? Fish do like movement, jiggling the bait often entices them to bite. But the important thing is, the jiggling can't be too vigorous and take the bait out of the view! Google likes to see changing content, but if the location of the content darts around - you betcha boots you are going to see your Pagerank take a hit.

The best thing to do when faced with with less traffic from Google is not to redesign the site again, but to dead stick: fix all the broken links, keep the site stable, and better yet, bring in more rods - build more sites.

In my time I've seen a large number of websites and careers that were set back by CMS switches and redesigns.

Further reading: The Russian Tea Room Syndrome and Deadprogrammer's Hierarchy of Web Needs.

Something horrible just happened to Jeff Atwood aka CodingHorror.

"ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images."

"I had backups, mind you, but they were on the virtual machine itself :("

It's a times like these we start wishing for a time machine, a cosmic undo button or reversible computing.

Jeff's blog was read by tenth of thousands of programmers and system administrators for many years. It contains information that is very valuable for these people, and represents an unthinkable amount of hours spent by Jeff. An agency rate for somebody like Jeff is between $250 and $500 an hour, but this is like appraising a priceless family heirloom.

I am not going to go through the motions of telling everybody how to backup things, about how important offisite backups are, how disk drives are fragile, how I don't trust virtual servers, how raid is not a backup strategy, and how version control is not backup strategy, etc, etc. JWZ wrote a good article about backups.

Here are things I want to say. First, we are all not backed up sufficiently and likely have already lost data that we would want back.

I can't find my grandmother's recipe book (I still hope it's only lost), my wife's first email to me, my first web page through which she found me, my first job search web page that had a picture of the Twin Towers and said how I wanted to work there, my early school grading papers, a rare book about fishing in the Black Sea, a stamp from the Orange Republic that used to be in my father's stamp album, the password to my very short-numbered ICQ account. A lot of stuff.

All of our digital information is susceptible to an electromagnetic pulse, fire, flood. Spinning platter hard drives are particularly bad - they have very short lifespans measured in low single digit years. CDs are even worse - aluminum inside them rots (I have a cd with a lot of outlook emails that reads as a blank filled with 1s).

So the first thing that I would like to mention is that if you never simulate a failure, you'll never know if your stuff can be replaced. It's not an easy thing to practice, though - restores and failovers are tricky to do.

A few jobs ago we were getting a fancy new load balancer set up. It was up and running, and supposedly we had failover: if one of the servers died, we would not even need to do anything, the backup servers would pick up the slack. I suggested that we should test it by pulling the network plug on one of the machines off hours. My boss would not allow that, saying that we could possibly break things. My argument that it'd be better if something like that happened when we were ready it would not be as bad if it happened when the actual failure would occur. When the actual failure did occur the load balancer did not switch, and we had an outage that was a good deal longer (it happened at night).

Load balancers are not backup solutions, but this story highlights an irrational streak in system administration: nobody wants to practice failure: it's just too nerve-wracking, and a lot of hard work. It's much easier to assume that somebody up the line did everything correctly: set up and tested backups, startup scripts, firewalls and load balancers. Setting up and validating backups and testing security are thankless jobs.

This brings me to a another point. The act of taking a backup is not risk free in itself. The biggest data losses that I suffered happened to me in the process of setting up backups. As an example I'll bring up the legendary story about Steve Wozniak (whom I met yesterday):

The Woz was creating a floppy driver under an extreme time pressure, not sleeping much and feeling sick. The end result was a piece of software of unimaginable beauty: it bypassed a good deal of clunky hardware, and thanks to a special timing algorithm, was fast and quiet. When other disk drives sounded like a machine gun (I dealt with a few of those when I was young), Woz's purred like a kitten. Finally he wrote the final copy onto a floppy, and decided to make a backup of it. Being dead tired, he confused the source and destination drives, and copied an empty floppy onto the one with the precious driver. Afterward he proceeded to burnish his place at the top of engineering Olympus by rewriting the thing from memory in an evening.

It's really the easiest thing in the world to confuse the source and destination of a backup, destroying the original in the act of backup! The moral of the story?

Do as much backing up as possible, while being careful not to destroy your precious data in the process. Have an offsite backup. Print out your blog on paper if it's any good. In fact, print out as much stuff as you can. Your backup strategy should be like a squirrel's: bury stuff in as many places as possible (well, except sensitive information, which is a whole other story in itself).

There's something that has been bothering me for a while, something that I call "Homer Simpson's toothpick school of blogging". In one of the Simpsons episodes Homer is marauding a grocery store at brunch, making a meal out of free samples. He proceeds to eat a few non-sample items by proclaming that "if it has a toothpick in it, it's free" and sticking his toothpic into a variety of items. He even drinks a beer, piercing it with a toothpick. The most successful blogs are basically like that: they either paraphrase or directly quote juiciest pieces of online articles. There might be a little bit of commentary (the snarkier - the better), but the meat of these blogs is in the quotes.

This is known as "curating" - the successful toothpickers have excellent taste in content. The people they quote and take images from are very glad to receive traffic from these A-listers. BoingBoing.net, kottke.org, daringfireball.net are like that: short, high volume (once you get the hang of it, it does not take much to turn that interesting site in your firefox tab into a pithy little wrapper around a juicy quote), very enjoyable. More so than mechanized versions of the same thing like digg.com and stumbleupon.com. For one, submitters don't do a very good job of quoting or paraphrasing, and you find yourself clicking on links more. Very successful blogs stick their toothpics into so much content that you don't really need to click through to the originals much: I can read BoingBoing, Gothamist or Lifehacker without clicking too much - the juiciest stuff is already there. In fact Gothamist seems to be almost completely pulled from from New York Times and New York Post headlines. It's a bit like a segment on some NY TV news stations where they read the latest headlines from local papers.

Now, there isn't anything unethical about quoting and paraphrasing - it's all squarely in the realm of fair use. These blogs are a bit like suckerfish that attach themselves to whales or sharks in that they benefit immensely from their hosts. Well, actually, unlike suckerfish they repay the favor by driving traffic.

In fact, I owe most of my readers to the low point in my blogging career, when after failing to submit my post about the Starbucks Siren to BoingBoing through their official black hole form, I begged Cory Doctorow to post it in a personal email. He did, I received tons of traffic and literally thousands of links from BB readers. Now that article shows up at the very top of Google search results for Starbucks logo.

Therein lies a problem: good content on the Internet does not always bubble up to the top on it's own. Blogosphere is a bit like the Black Sea, which has a layer of very active and vibrant biosphere at low depths. But it's very deep, and below 200 meters the depths are full of poisonous hydrogen sulfide, which luckily does not circulate very much (unless there's a particularly strong storm). Think about digg.com or StackOverflow.com- at the top stuff circulates, gets upvoted and downvoted. But below, there's a poisonous cesspool of Sturgeon's Law's 90 percent. And most of the time, new and worthwhile content starts not at the top, but at the bottom, or flutters briefly in above the mediocrity and the bad, does not get noticed and gets buried.

Speaking of StackOverflow, Joel Spolsky and Jeff Atwood recently touched on the topic of blogging success in their excellent podcast. They were discussing Steve Yegge's retirement from blogging, and tried to pinpoint what it meant to be a successful blogger. "Perhaps one metric of success is getting people you respect and admire to link to your writing in an organic, natural way (that is, without asking them to)." I am a miserable failure on this front. Sure, I have some high profile readers, but their link love is rare, while I'm not really below begging for links.

Jason Kottke, an A-list blogger and a primo toothpick sampler, was reflecting on the monetary success. He likened business blogging to shining shoes: there might be some individuals who can get rich by running a chain of shoe shining stores (Jason Calacanis, Nick Denton), and maybe even some individual outstanding shoeshiners (Dooce) who can make a decent living, but for the majority of shoeshiners it's not a very good career choice.

I've read somewhere about my hometown's "king of shoeshiners", a very colorful character. He was the best shoeshiner Odessa has ever seen, famous and loved by all, but he died poor and miserable. On his monument there was a short quote: "life is waksa" (waksa is a Russian word for shoe polish with a connotation of something pitch-black).

For me blogging takes a good deal of effort. In the immortal words of E.B. White "writing is never 'fun'". (White almost rejected an assignment to write an article that became the finest piece ever written about New York when an editor suggested that he might 'have fun'). What makes blogging less fun for me is looking at server statistics, number of comments, ad revenue, and thinking about payoff and success. And feeling like that I maybe should have done something else with my time.

My high school Economics teacher, Mr. Oster, taught me one very valuable concept: "opportunity cost". Whenever you make a decision do something, you almost always pay the opportunity cost - the difference in value you might have gotten by doing something better. Oh, there could be hundreds of things that have a better payoff than not very successful blogging.

I personally do not blog for money, and certainly don't blog professionally (the ads on my site cover my hosting expenses). Well, not yet, anyway - I am preparing stuff for a commercial venture that I'll soon announce. I blog in order to meet people (hanging out a Web 2.0 events and meetups would probably have been more productive), but mostly to get things out of my head. In that sense I'm a bit like Louise Bourgeois. I've recently seen an exhibition of her work, and I'm pretty sure that if she did not create all those sculptures and paintings, the inspiration for them (which must have been glipses of extra dimensions, cellular automata that drive our reality, and super disturbing things that can't even be described) would have made her a raving lunatic and not a lucid and sane 97 year old woman that she is.

I don't really intend on changing the format of deadprogrammer.com - the intricate, long, winding, interconnected posts about obscure topics. I probably would have had a lot more success if I just kept a photo blog about New York City. If I'd just stick to one popular topic and posted every day - I know I would have attracted a lot more readers. Instead, I'm going to start a new, for-profit blog. You'll hear about it soon. I think I should be able to make some shekels with my mad shoeshining skills. And while I agree with Mr. White about writing not being fun, the fund is in having written.