Something horrible just happened to Jeff Atwood aka CodingHorror.

"ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images."

"I had backups, mind you, but they were on the virtual machine itself :("

It's a times like these we start wishing for a time machine, a cosmic undo button or reversible computing.

Jeff's blog was read by tenth of thousands of programmers and system administrators for many years. It contains information that is very valuable for these people, and represents an unthinkable amount of hours spent by Jeff. An agency rate for somebody like Jeff is between $250 and $500 an hour, but this is like appraising a priceless family heirloom.

I am not going to go through the motions of telling everybody how to backup things, about how important offisite backups are, how disk drives are fragile, how I don't trust virtual servers, how raid is not a backup strategy, and how version control is not backup strategy, etc, etc. JWZ wrote a good article about backups.

Here are things I want to say. First, we are all not backed up sufficiently and likely have already lost data that we would want back.

I can't find my grandmother's recipe book (I still hope it's only lost), my wife's first email to me, my first web page through which she found me, my first job search web page that had a picture of the Twin Towers and said how I wanted to work there, my early school grading papers, a rare book about fishing in the Black Sea, a stamp from the Orange Republic that used to be in my father's stamp album, the password to my very short-numbered ICQ account. A lot of stuff.

All of our digital information is susceptible to an electromagnetic pulse, fire, flood. Spinning platter hard drives are particularly bad - they have very short lifespans measured in low single digit years. CDs are even worse - aluminum inside them rots (I have a cd with a lot of outlook emails that reads as a blank filled with 1s).

So the first thing that I would like to mention is that if you never simulate a failure, you'll never know if your stuff can be replaced. It's not an easy thing to practice, though - restores and failovers are tricky to do.

A few jobs ago we were getting a fancy new load balancer set up. It was up and running, and supposedly we had failover: if one of the servers died, we would not even need to do anything, the backup servers would pick up the slack. I suggested that we should test it by pulling the network plug on one of the machines off hours. My boss would not allow that, saying that we could possibly break things. My argument that it'd be better if something like that happened when we were ready it would not be as bad if it happened when the actual failure would occur. When the actual failure did occur the load balancer did not switch, and we had an outage that was a good deal longer (it happened at night).

Load balancers are not backup solutions, but this story highlights an irrational streak in system administration: nobody wants to practice failure: it's just too nerve-wracking, and a lot of hard work. It's much easier to assume that somebody up the line did everything correctly: set up and tested backups, startup scripts, firewalls and load balancers. Setting up and validating backups and testing security are thankless jobs.

This brings me to a another point. The act of taking a backup is not risk free in itself. The biggest data losses that I suffered happened to me in the process of setting up backups. As an example I'll bring up the legendary story about Steve Wozniak (whom I met yesterday):

The Woz was creating a floppy driver under an extreme time pressure, not sleeping much and feeling sick. The end result was a piece of software of unimaginable beauty: it bypassed a good deal of clunky hardware, and thanks to a special timing algorithm, was fast and quiet. When other disk drives sounded like a machine gun (I dealt with a few of those when I was young), Woz's purred like a kitten. Finally he wrote the final copy onto a floppy, and decided to make a backup of it. Being dead tired, he confused the source and destination drives, and copied an empty floppy onto the one with the precious driver. Afterward he proceeded to burnish his place at the top of engineering Olympus by rewriting the thing from memory in an evening.

It's really the easiest thing in the world to confuse the source and destination of a backup, destroying the original in the act of backup! The moral of the story?

Do as much backing up as possible, while being careful not to destroy your precious data in the process. Have an offsite backup. Print out your blog on paper if it's any good. In fact, print out as much stuff as you can. Your backup strategy should be like a squirrel's: bury stuff in as many places as possible (well, except sensitive information, which is a whole other story in itself).

If the Seal of New York City were designed today, it would not have a sailor and a Native American on it. It would have a cab driver and a food cart vendor.

Cab driving and food vending wood seem like the two of the most democratic enterpreneurial options, the foundation of which is the public streets New York City: you just wheel out your vehicle and try to make some commerce happen. The only thing that you need is a license. The one for cab driving is called a "medallion", costs $766K, and as an investment vehicle outperformed just about any commodity and stock index. The food cart licesnses are also very expensive. Plus you are hounded by NYPD, Department of Sanitation, and who knows what else. Cab drivers and food cart vendors are some of the hardest working and most prosecuted businesmen in the city, but sometimes they have their own victories, big and small.

You don't need to go any further than the Metropolitain Museum of Art to see two interesting examples. Right in front of the museum there's a collection of food carts. They all are very typical carts, none of them are of the fancy variety. There are two types represented - the basic "dirty water hot dog" cars and "street meat" carts. But there's one important difference - they all have stickers that say "Disabled Veteran", and there's usually an actual veteran somewhere nearby.

In the past years the space in front of the museum was either empty or occupied by one or two carts licensed by the Department of Parks. Then one day Dan Rossi, a disabled veteran, discovered a 19th century state law that allows disabled veterans to sell food in areas that are off-limits to others. The location in front of the museum is particularly lucrative because there are no affordable restaurants as far as an overweight tourist can walk. This hack is a small, but significant victory for food vendors. They are still ticketed mercelesly by NYPD, have to work crazy hours, and deal with the need to urinate in some kind of a miraculous way. At least they got an article in the New York Times written about them.

Across the road from the veteran's carts is a mansion that belongs to billionaire Tamir Sapir, a former cab driver.

Mr. Sapir's legend starts in Georgia, USSR. He found an interesting niche business: filling out complicated emigration forms for the Soviet Jews. At some point he was persuaded by his mother to give up his excellent life (it was a very lucrative business, from what I understand) and emigrate to Israel himself. He found himself in the middle of the Yom Kippur War, and quickly emigrated to the United States. He worked hard to earn enough money to leave rural Kentucky for New York, and then even harder to buy a cab medallion (which was a lot more affordable in those days). Then he risked everything again by putting up that medallion as collateral for a loan that he needed to open up an electronics store with a partner.

In the 80s there was a bit of a thaw in Sovet-American relations - Perestroyka and whatnot. There was a significant amount of people visiting the US - diplomats, scientists, sailors, and those invited by relatives. These people were allowed to exchange a small sum of rubles into dollars at the official rate - if I remember correctly, 60-something kopeks to a dollar.

What these lucky tourists wanted the most was electronics. In particular - vcrs, doule deck cassette players, and Walkmen. They had the money to buy these things, but here's a problem: they needed 220 volt round plug devices, and more than that, VCRs needed to support the SECAM standard. You could not just walk into any store and find these: American market was all 110V and NTSC.

Every child in Odessa back then knew all of this, as well as that if you found yourself in New York City with some money, all you needed to do was trudge over to Timur's (this was before he changed his name) store in Manhattan and find 220V SECAM VCRs.

Mr. Sapir was making a mint, but more importantly he was making connections with the Soviet ministers, diplomats, and future oligarchs. A little later he was invited back to the USSR, and made more connections there. These connections allowed him to play on the Soviet deregulation arbitrage market.

You see, when the Soviet Union was transitioning to the market economy all prices were regulated except those for commidities like metals, oil, and fertilizer. Those with connections could buy these commodities for already devalued rubles and sell them abroad for hard currency, making millions of dollars. All you needed was connections, which Mr. Sapir had.

He made millions, but the game became very dangerous as people tougher than NYC cabbies entered it. Mr. Sapir did not continue his career as a commodity exporter. Instead he invested his millions into New York City skyscrapers. The real estate market bottomed out, and you could buy a whole skyscraper for 10 million dollars or so. He bought a whole bunch of them. The price of Manhattan real estate exploded, and he became a billionare.

He bought a mansion across from the Metropolitain Museum to house his collection of carved ivory (for some reason this was a very popular area of collecting in the Soviet Union), has a yacht that used to be stuffed with a collection of exotic animal taxidermy that could rival Mr. Burn's wardrobe or Amy's car from Futurama.

Well, the two lessons here are: 1) you have to take risks and 2) you have to find a niche. The rest is luck.

A couple of my friends created a new url shortener. Wait, stop booing. There's a twist - it has the coolest url ever - http://wi.nr. And you can win something or other by just using it. And you get statistics. Well, I guess it's about it. Did I mention these are my friends that are doing that?

Why should you care how short your url is? Well, it's basically because the retards at Twitter don't allow for inline urls (if they will one day, url shorteners would die like they deserve to), and if you want your stuff retweeted, you need to leave a couple of characters for RT and the username. Of course url shorteners are evil in general, and people at Twitter are incompetent technologically, but very lucky. And being lucky is more important than being competent.

The funny thing is, I absolutely legitemately won their first $5 Amazon gift certificate.