December 11th – National Backup Awareness Day

Something horrible just happened to Jeff Atwood aka CodingHorror.

“ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images.”

“I had backups, mind you, but they were on the virtual machine itself :(“

It’s a times like these we start wishing for a time machine, a cosmic undo button or reversible computing.

Jeff’s blog was read by tenth of thousands of programmers and system administrators for many years. It contains information that is very valuable for these people, and represents an unthinkable amount of hours spent by Jeff. An agency rate for somebody like Jeff is between $250 and $500 an hour, but this is like appraising a priceless family heirloom.

I am not going to go through the motions of telling everybody how to backup things, about how important offisite backups are, how disk drives are fragile, how I don’t trust virtual servers, how raid is not a backup strategy, and how version control is not backup strategy, etc, etc. JWZ wrote a good article about backups.

Here are things I want to say. First, we are all not backed up sufficiently and likely have already lost data that we would want back.

I can’t find my grandmother’s recipe book (I still hope it’s only lost), my wife’s first email to me, my first web page through which she found me, my first job search web page that had a picture of the Twin Towers and said how I wanted to work there, my early school grading papers, a rare book about fishing in the Black Sea, a stamp from the Orange Republic that used to be in my father’s stamp album, the password to my very short-numbered ICQ account. A lot of stuff.

All of our digital information is susceptible to an electromagnetic pulse, fire, flood. Spinning platter hard drives are particularly bad – they have very short lifespans measured in low single digit years. CDs are even worse – aluminum inside them rots (I have a cd with a lot of outlook emails that reads as a blank filled with 1s).

So the first thing that I would like to mention is that if you never simulate a failure, you’ll never know if your stuff can be replaced. It’s not an easy thing to practice, though – restores and failovers are tricky to do.

A few jobs ago we were getting a fancy new load balancer set up. It was up and running, and supposedly we had failover: if one of the servers died, we would not even need to do anything, the backup servers would pick up the slack. I suggested that we should test it by pulling the network plug on one of the machines off hours. My boss would not allow that, saying that we could possibly break things. My argument that it’d be better if something like that happened when we were ready it would not be as bad if it happened when the actual failure would occur. When the actual failure did occur the load balancer did not switch, and we had an outage that was a good deal longer (it happened at night).

Load balancers are not backup solutions, but this story highlights an irrational streak in system administration: nobody wants to practice failure: it’s just too nerve-wracking, and a lot of hard work. It’s much easier to assume that somebody up the line did everything correctly: set up and tested backups, startup scripts, firewalls and load balancers. Setting up and validating backups and testing security are thankless jobs.

This brings me to a another point. The act of taking a backup is not risk free in itself. The biggest data losses that I suffered happened to me in the process of setting up backups. As an example I’ll bring up the legendary story about Steve Wozniak (whom I met yesterday):

The Woz was creating a floppy driver under an extreme time pressure, not sleeping much and feeling sick. The end result was a piece of software of unimaginable beauty: it bypassed a good deal of clunky hardware, and thanks to a special timing algorithm, was fast and quiet. When other disk drives sounded like a machine gun (I dealt with a few of those when I was young), Woz’s purred like a kitten. Finally he wrote the final copy onto a floppy, and decided to make a backup of it. Being dead tired, he confused the source and destination drives, and copied an empty floppy onto the one with the precious driver. Afterward he proceeded to burnish his place at the top of engineering Olympus by rewriting the thing from memory in an evening.

It’s really the easiest thing in the world to confuse the source and destination of a backup, destroying the original in the act of backup! The moral of the story?

Do as much backing up as possible, while being careful not to destroy your precious data in the process. Have an offsite backup. Print out your blog on paper if it’s any good. In fact, print out as much stuff as you can. Your backup strategy should be like a squirrel’s: bury stuff in as many places as possible (well, except sensitive information, which is a whole other story in itself).

Apple Confidential 2.0: The Definitive History of the World’s Most Colorful Company

Apple Confidential examines the tumultuous history of America’s best-known Silicon Valley start-up–from its legendary founding almost 30 years ago, through a series of disastrous executive decisions, to its return to profitability, and including Apple’s recent move into the music business. Linzmayer digs into forgotten archives and interviews the key players to give readers the real story of Apple Computer, Inc. This updated and expanded edition includes tons of new photos, timelines, and charts, as well as coverage of new lawsuit battles, updates on former Apple executives, and new chapters on Steve Wozniak and Pixar.

iWoz: Computer Geek to Cult Icon: How I Invented the Personal Computer, Co-Founded Apple, and Had Fun Doing It

The mastermind behind Apple sheds his low profile and steps forward to tell his story for the first time.

Before cell phones that fit in the palm of your hand and slim laptops that fit snugly into briefcases, computers were like strange, alien vending machines. They had cryptic switches, punch cards and pages of encoded output. But in 1975, a young engineering wizard named Steve Wozniak had an idea: What if you combined computer circuitry with a regular typewriter keyboard and a video screen? The result was the first true personal computer, the Apple I, a widely affordable machine that anyone could understand and figure out how to use.

Wozniak’s life—before and after Apple—is a “home-brew” mix of brilliant discovery and adventure, as an engineer, a concert promoter, a fifth-grade teacher, a philanthropist, and an irrepressible prankster. From the invention of the first personal computer to the rise of Apple as an industry giant, iWoz presents a no-holds-barred, rollicking, firsthand account of the humanist inventor who ignited the computer revolution. 16 pages of illustrations.

Doppelgangers

I was having dinner with my childhood friend and his girlfriend. The girlfriend, who met me for the first time, told me that I looked a lot like another friend of hers. Once again I was reminded of my many doppelgangers.

Yes, I seem to have a pretty generic look. You know, we all share a common ancestor about a thousand years ago, but some people, like me, probably share one a lot sooner. A bit like all these WWII era inhabitants of a French town visited by a son of a WWI era American soldier in Bill Mauldin‘s famous cartoon:


“This is th’ town my pappy told me about.”

Once, walking around in Brooklyn I’ve noticed a guy who looked a bit familiar, but I could not immediately understand why. I noticed that he was looking at me a little strangely too, as if trying to figure something out. His wife, on the other hand figured it out a in a split second – I could tell by her rounded eyes and pointed finger. If you’d put peyos and kipa on me, you would not be able to tell me and her husband apart. The three of us chatted for a bit about this and went on our separate ways, a little bit shaken by the experience.

Then, one day I thought that Travis Ruse, the very talented subway photographer, finally captured me on cmos. I’ve never met Travis, but I always thought that eventually either I’d spot him on the train or I’d end up in one of his pictures. This picture confused me a great deal. I could not tell if it was me in it or not for 15 minutes or so. Only small details were wrong: I don’t wear t-shirts under dress shirts, my glasses at the time had brown frames and I am slightly fatter. The blue shirt, pants and shoulder bag strap are spot on.

Update:
Long after I wrote this one, I ended up at the same company as Travis Ruse.

Update:
Here’s a picture of me with Steve Wozniak. There’s some resemblance, wouldn’t you say?

There are Always Leaks

There are those movies that keep you actively thinking about them for days and weeks after you see them. Primer, which I watched with my wife yesterday is one of those. If you are one of those who are afraid of “spoilers” – this is your warning, although I believe it’s really impossible to create a “spoiler” for this movie. You’ll watch it once, twice, three times, then with director’s commentaries, then read the entire message board and still will not be able to figure it out entirely.

Primer is a story about time travel paradoxes, but not really. It’s about innovation, competition, trust and inability to see the entire picture.

Without the science fiction element, the movie is about garage innovators. The core of innovative group is almost always two people. Sometimes it starts out with more people, but then boils down to two. Jobs and Wozniak, Hewlett and Packard, Gates and Allen. You need to have your John and your Paul, George and Ringo are not that important. So you have these two people who together are destined to create great things. Can they trust each other? Would they do screw each other over?

We know for a fact that the alpha geeks are often ruthless. Steve Jobs gets a design job from Atari, gives it to Steve Wozniak, promising 50/50 spilt, and after Woz delivers the work gives him $300 while pocketing a few grand, saying that the fee was $600? When Apple becomes a success he deserts Wozniak. Then gets forced out himself. Then he majorly screws over founders of Pixar. Then takes back Apple. Typical preppy high school drama, except with higher stakes. And realize this – he does all that instead of enjoying his money and free time.

Anyway, the movie has two protagonists, Abe and Aaron, engineers talented like Woz and a bit less ruthless than Jobs. Abe creates a time machine that can use to travel back in time to the moment when the machine is powered up for the first time and then explains its use to his friend. That opens endless possibilities for them: make money in the stock market, prevent bad stuff from happening. Which they do for a while, but then their competitive instincts kick in. Can you really trust your partner not to go into the past and put you out of commission?

Worse of all — if you go back in time and then prevent your second self from entering the time machine all of a sudden there are two of you. The biblical names of the characters are significant in this context – Abraham – the “Father of Many” and Aaron – the “Bearer of Martyrs”. They become involved and a four-dimensional battle for control with each other and their paradox-born doppelgangers. “Failsafe machines” — extra time boxes set up in hidden locations that allow for extra “entry points” or “save points” become important weapons in this game. Can you really trust yoursef becomes the real question.

Abe and Aaron are competitive and very, very smart. They create a crazily complicated situation, with time machines, time machines inside time machines, doubles that have all recorded audio track of the timeline provided to them by future selves, extra timelines and resets via failsafe machines. “Are you hungry? I haven’t eaten since later this afternoon” sounds absolutely normal in the context.

A similar, buth much less complicated situation transpired in Stanislaw Lem’s 7th voyage of the Star Diaries of Ijon Tichy..

That sci-fi story went like this: Ijon’s rocketship ends up in a “space storm” with a broken rudder. Fixing a rudder is a two person job, but luckily the space storm brings together Ijons from different times. All he really needs to do is put on a space suit, wait for a later him wearing a space suit to appear, cooperate and fix the rudder. Instead he ends up arguing with his future and past selves, hitting and being hit by them and eating his own supplies of chocolate. Here’s a quote from what seems to be a full text of the story that somebody probably illegally posted on the web:

“I came to, sitting on the floor of the bathroom; someone was banging on the door. I began to attend to my bruises and bumps, but he kept pounding away; it turned out to be the Wednesday me. After a while I showed him my battered head, he went with the Thursday me for the tools, then there was a lot of running around and yanking off of spacesuits, this too in one way or another I managed to live through, and on Saturday morning crawled under the bed to see if there wasn’t some chocolate left in the suitcase. Someone started pulling at my foot as I ate the last bar, which I’d found underneath the shirts; I no longer knew just who this was, but hit him over the head any how, pulled the spacesuit off him and was going to put it on–when the rocket fell into the next vortex.

When I regained consciousness, the cabin was packed with people. There was barely elbowroom. As it turned out, they were all of them me, from different days, weeks, months, and one–so he said–was even from the following year. There were plenty with bruises and black eyes, and five among those present had on spacesuits. But instead of immediately going out through the hatch and repairing the damage, they began to quarrel, argue, bicker and debate. The problem was, who had hit whom, and when. The situation was complicated by the fact that there now had appeared morning me’s and afternoon me’s–I feared that if things went on like this, I would soon be broken into minutes and seconds–and then too, the majority of the me’s present were lying like mad, so that to this day I’m not altogether sure whom I hit and who hit me when that whole business took place, triangularly, between the Thursday, the Friday and the Wednesday me’s, all of whom I was in turn. My impression is that because I had lied to the Friday me, pretending to be the Sunday me, I ended up with one blow more than I should have, going by the calendar. But I would prefer not to dwell any longer on these unpleasant memories; a man who for an entire week does nothing but hit himself over the head has little reason to be proud.”

One other main themes of the movie is the inability to know certain things no matter how smart you are. Too many things are open to too many interpretations. The geeks on the web are obsessively putting together timelines, diagrams and theories of what really went on. I don’t even think that the author of the screenplay completely understands the whole sequence of events. And he directed and played in the film! How many timelines are there? How many Abes and Aarons? What do they mean by “recycling” the machines? What the hell happened with Tom Granger?

There is also an interesting recursive theme in the movie: cheapness. The actor/director/screenwriter, shooting on what is described as $7000 budget and making it look very good, has done some ingenious things. So do the inventor in the movie – he keeps his day job instead of throwing it away to follow the dream, too cheap to have a steak for lunch, and even at some point he cuts copper tubing needed for the project out of a refrigerator. I don’t know if building a time machine is that much more difficult than making such an awesome movie on a 7K budget.

By the way, if you are looking for hints about the movie, the commentary track on the DVD is a pretty horrible place to start. It’s full of jems like “That sound effect – yeah [background laugh] – that was George Forman Grill”.

All I know, is that I want an Emiba Devices t-shirt. And a garage.