December 11th – National Backup Awareness Day

Something horrible just happened to Jeff Atwood aka CodingHorror.

“ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images.”

“I had backups, mind you, but they were on the virtual machine itself :(“

It’s a times like these we start wishing for a time machine, a cosmic undo button or reversible computing.

Jeff’s blog was read by tenth of thousands of programmers and system administrators for many years. It contains information that is very valuable for these people, and represents an unthinkable amount of hours spent by Jeff. An agency rate for somebody like Jeff is between $250 and $500 an hour, but this is like appraising a priceless family heirloom.

I am not going to go through the motions of telling everybody how to backup things, about how important offisite backups are, how disk drives are fragile, how I don’t trust virtual servers, how raid is not a backup strategy, and how version control is not backup strategy, etc, etc. JWZ wrote a good article about backups.

Here are things I want to say. First, we are all not backed up sufficiently and likely have already lost data that we would want back.

I can’t find my grandmother’s recipe book (I still hope it’s only lost), my wife’s first email to me, my first web page through which she found me, my first job search web page that had a picture of the Twin Towers and said how I wanted to work there, my early school grading papers, a rare book about fishing in the Black Sea, a stamp from the Orange Republic that used to be in my father’s stamp album, the password to my very short-numbered ICQ account. A lot of stuff.

All of our digital information is susceptible to an electromagnetic pulse, fire, flood. Spinning platter hard drives are particularly bad – they have very short lifespans measured in low single digit years. CDs are even worse – aluminum inside them rots (I have a cd with a lot of outlook emails that reads as a blank filled with 1s).

So the first thing that I would like to mention is that if you never simulate a failure, you’ll never know if your stuff can be replaced. It’s not an easy thing to practice, though – restores and failovers are tricky to do.

A few jobs ago we were getting a fancy new load balancer set up. It was up and running, and supposedly we had failover: if one of the servers died, we would not even need to do anything, the backup servers would pick up the slack. I suggested that we should test it by pulling the network plug on one of the machines off hours. My boss would not allow that, saying that we could possibly break things. My argument that it’d be better if something like that happened when we were ready it would not be as bad if it happened when the actual failure would occur. When the actual failure did occur the load balancer did not switch, and we had an outage that was a good deal longer (it happened at night).

Load balancers are not backup solutions, but this story highlights an irrational streak in system administration: nobody wants to practice failure: it’s just too nerve-wracking, and a lot of hard work. It’s much easier to assume that somebody up the line did everything correctly: set up and tested backups, startup scripts, firewalls and load balancers. Setting up and validating backups and testing security are thankless jobs.

This brings me to a another point. The act of taking a backup is not risk free in itself. The biggest data losses that I suffered happened to me in the process of setting up backups. As an example I’ll bring up the legendary story about Steve Wozniak (whom I met yesterday):

The Woz was creating a floppy driver under an extreme time pressure, not sleeping much and feeling sick. The end result was a piece of software of unimaginable beauty: it bypassed a good deal of clunky hardware, and thanks to a special timing algorithm, was fast and quiet. When other disk drives sounded like a machine gun (I dealt with a few of those when I was young), Woz’s purred like a kitten. Finally he wrote the final copy onto a floppy, and decided to make a backup of it. Being dead tired, he confused the source and destination drives, and copied an empty floppy onto the one with the precious driver. Afterward he proceeded to burnish his place at the top of engineering Olympus by rewriting the thing from memory in an evening.

It’s really the easiest thing in the world to confuse the source and destination of a backup, destroying the original in the act of backup! The moral of the story?

Do as much backing up as possible, while being careful not to destroy your precious data in the process. Have an offsite backup. Print out your blog on paper if it’s any good. In fact, print out as much stuff as you can. Your backup strategy should be like a squirrel’s: bury stuff in as many places as possible (well, except sensitive information, which is a whole other story in itself).

Zombie-free Mac Children’s Games

I was born at the beginning of the age of information. I welcome the content deluge.

I’m not a snob. I do not discriminate amongst the sources of content, gladly consuming books, television, movies, music, magazines, websites, wikis, and blogs. I like to think that thanks to technologies like ebook readers, blog aggregation, suggestion engines at Amazon and Netflix, and Tivo I limit my input to only the stuff that is “awesome” on the “Normal people” scale.

I remember the time when the flow of information available to me was limited to my father’s sizable library and a few hours a week of interesting TV culled from the 3 horrible channels of Soviet television, and really don’t miss it.

My 4 year old daughter is swimming in the sea of information together with me. We read books to her (the quality of children’s books these days is amazing), she watches dvd and tivo’d shows, youtube videos on a laptop. She really wants to play with a computer as well.

Unfortunately the only game that I have is “Plants Vs. Zombies“. We play it together usually as a reward for good behavior. She enjoys the “zen garden” part of the game, as well as the regular “zombie” part. This, of course led to the questions on the nature of zombies (uhh), their diet (brains), the nature of brains, and the absence of female zombies in the game (uhhh).

When Natalie was younger and I used to have a PC, there was a whole bunch of craptastic PC games (one even with a special keyboard, if I remember) that we used to play. These crashed often and were pretty retarded.

Now that I have a Mac, I’m looking for some better, zombie-free games suitable for a 4 year old. Finding good computer games is much more difficult than finding good children’s books. Do you have any suggestions?

Deadprogrammer Visits Odessa : Part II : Balconies and Yards

When I think about Odessa, I often remember the title of Robert Heinlen’s novel, “The Door into Summer“. Odessa is a summer city. If I were to pick one word to describe it in the summer, it would be sun-dappled.

The soil in Odessa is pretty bad, and there aren’t any local sources of fresh water. Because of that there are only three types of trees that thrive there: acacia, sycamore (locally known as “shameless” trees because they shed their bark), and horse chestnut. Acacia is one of the symbols of Odessa, and horse chestnut is that of Kiev. These trees are thriving in Odessa, and many hundred year old specimens provide a lot of shade.

The tree shade and the bright sun bathe everything in these spots of light, like in Renoir’s La Moulin de la Galette.

Here’s a very old acacia on the corner of the street where I grew up. Notice the parking sign: in Odessa the sidewalks are so wide that cars can be parked on them. Also, they are now selling melons out of cages that are locked for the night. Acacia blossoms cover all the sidewalks and produce and intoxicating aroma.

As I mentioned before, Odessa was built on the grand scale and by the best architects. Even the lesser buildings are very decorated. Atlases were very popular.

Apparently the current location of Odessa used to be under the Black sea, and the bedrock consists of a somewhat soft yellow sedimentary rock formed out of shells of sea creatures. It’s a type of limestone, I guess. This rock turned out to be a perfect building material: cheap and abundant, soft enough to cut and carve, but durable enough to build with, not too heavy, very good for thermal and sound isolation. So many buildings were built this stone that the quarries below the city formed a humongous labyrinth known as “catacombs“. It was used by contraband smugglers and WWII guerrilla fighters. It would have been super easy to build a subway in Odessa, but somehow it never happened.

There are two architectural features of the old limestone buildings that became very important in the Odessa way of life: balconies and courtyards. Just like the wide sidewalks, these are the artifacts of the town mostly built in 1800s, in the era of horse buggies, no air conditioning, and gas light. Balconies provided a breath of fresh air, and internal courtyards let architects let in light into all apartments.

Some balconies are very charming, with cast iron railings and pleasing shapes and aging gracefully.

Some did not survive and were replaced by ugly shitboxes.

Some are unsafe Frankenstein monsters.

Some are huge, with fancy statues, and in horrible state of disrepair.

Some are maintained.

Some are elegant, curved around gorgeous bay windows. Notice the hanging laundry.

Some instead of laundry are draped in grapevines.

I have no idea how these grapevines survive: they grow in the street, are peed on by dogs and are never watered.

Some of the most gorgeous buildings with the most awesome balconies are in such state of disrepair, that in New York they would have been immediately boarded up, but yet there are people living there…

These old buildings are what New York realtors are referring to as “pre-war”: they have very tall ceilings, big rooms, parquet floors, fireplaces, and other bourgeoisie niceties. After the revolution most of these apartments where turned into communal flats: instead of a single family with help, now 5-7 families were crammed into it. The process kind of reversed itself in the 90s, with rich people buying out communal dwellers, but some still remain: here’s a picture of the doorbells on one door. You can visit a site about communal living in Russia. Interestingly enough, despite the Marxist spirit, in communal apartments even the doorbells were hooked up to separate electric meters, and the excessive use of a lightbulb in bathroom was a major point of contention.

These buildings are old: many have horse hitches out front:

Once you tied your horse down, you enter the building through a gate.

Sometimes the gate is old and beautiful.

Sometimes it was replaced by a horrible modern monstrosity painted with signs like “angry dog” and “there’s no toilet here”.

Sometimes it’s missing altogether.

The gate leads you through a passageway. What will you find in that passageway?

A bunch of semi-destroyed soviet-era mailboxes. I am not sure if they are still in use. Here’s one of the better preserved specimen.

The passageway, a utility space, often has interesting ceilings, which are almost always mostly ruined.

Some are less ornate though.

Often there are very cool windows within the passageways.

The passageway will lead you into the yard. There will be entrances to apartments, parked cars, laundry.

Here’s something that used to be very common, but is actually a rarity now: drying plastic bags.

Some more cars and a kid riding a bike.

Grapevines, cats, old cars.

Restored all cars and walls that are in bad need of restoration.

Sleeping dogs and more drying laundry.

A non-functioning fountain.

A little garden. This one used to be my grandmother’s.

A granny reading a newspaper.

Peeling walls: I distinctly remember this particular wall from my childhood, it used to look exactly the same. It’s a testament to the quality of work of the old builders that with next to zero maintenance these buildings survive.

A seemingly functional pigeon coop.

An old tree. These old trees move move in the wind in a very hypnotic manner. Being so tall, they can be used to forecast the size of waves in the sea by the speed at which the tops of them move.

A decorative drinking well. I’m not sure, but it kind of looks like it could have been a real well once.

A woman feeding cats who will tell you to stop taking stupid pictures like the dumbass that you are and go and do something productive.

Walls with WWII bullet holes. Odessa is one of the few cities granted the designation of hero city for valiant resistance. When the Germans and Romanians entered the city there was a lot of executions.

A sad, sad sight: an ad for apartments in a newly constructed building.

Deadprogrammer visits Odessa : Part I : Introduction

Global Warming

A couple of cherry and plum trees in Brooklyn Botanic Garden blossomed prematurely. I took these pictures last Saturday, January 6th.

This reminded me of the article that I’ve seen in the Fantastic Story Magazine for Winter 1953, in fact the same magazine from which I liberated the masthead of my website.



Dr. Bartlett’s company is still in business and “this theory about the more carbon dioxide the warmer” is still around too.

Time to Get on the Top of the Rock
or
Can You Smell What’s Cooking at the Top of the Rock?

There’s one thing that I hate about Chrysler Building. It does not have a public observation deck. I feel that any major skyscraper needs to have two things at the top: a restaurant and an observation deck.

To the owners the public areas at the top are usually a pain in the ass and rarely pay for themselves. Having a lot of people from the street come up to the very top of the building, adding to the overall traffic is not fun for building management from the security standpoint. I remember reading about the co-op board of a posh building at 30 Central Park South trying to evict Nirvana, a top floor Indian restaurant with amazing views of Central Park. They complained about the traffic and the cooking smells.

Because of this most buildings that used to have observation decks and restaurants closed them. Over a the Tishman Building Top of the Sixes turned into an exclusive cigar club. At the City Services Building the public observation deck, which was actually planned as the owner’s penthouse, was turned into a closed lounge for AIG brass. An at the Chrysler Building, the observation deck became Schrafft’s Restaurant, then it morphed into posh and private Cloud Club, and then closed altogether.

When I was younger, I was not particularly attracted to the views from high vantage points. But for some reason at the turn of the Millennium found me deeply fascinated with skyscrapers and views from them. To this day I can’t forgive myself not visiting the observation deck at WTC. I probably did not have enough money to eat at the perished Windows on the World, but not visiting Top of the World still fills me with remorse.

Because of this upsetting tendency of restriction and destruction public spaces high in the skies, I find this very joyous news: the rocket-like 30 Rockefeller Plaza will be opening an observation deck to the public. It’s going to be called “Top of the Rock.”

Rockefeller Center has an interesting distinction of being one of the very few Rockefeller family projects that carry their name (the other big one being Rockefeller University). Most people also think that it’s built and named after John D. Senior, the Mr. Burns prototype and semi-crazy hander outer of nickels, when in fact, it was the mellow John D. Junior who built it.

Since the Rockefeller name is not that popular, it’s common to see “Rockefeller Center” to be shortened to sexier “Rock Center”, as for example the menu of Yummy Sushi contains several sushi combos named Rock X, where X is a number of the combo.

As much as I tried, I could not find out a more definite date than “sometime this fall”, but one of the free booklets that you can get in the lobby of the GE (former RCA) building has a little blurb and the logo. I like the logo. It’s all Art Deco-ey.

Reverse Hanlon’s Razor

While I am a big fan of Occam’s Razor, which I find very useful in most cases, I find Hanlon’s Razor to be false on most occasions.

“Never attribute to malice that which is adequately explained by stupidity” – states this “law”. It’s very frequently used by many people to explain why something is one way, when very clearly it should be done completely differently.

When I worked at Nathan’s clam bar, annoyed patrons frequently suggested that the pint containers of condiments should be always on the counter and not behind it at all times. My co-worker with decades of experience told me that it was not a good idea when he noticed that I started leaving the condiments on the counter. I chose to disregard this piece of advice as it was coming from someone who was opening clams for a little bit over minimum wage for decades. I was happy with this arrangement because I did not have to waste time on moving the condiments between waves of customers, and the customers were happy as they always had a few jars within easy reach at all times.

The reason for hiding the condiments became painfully clear to me pretty soon. Two drunks started an argument right at the counter. My lineup of pint containers of cocktail sauce and horseradish became perfect amunition for their condiment fight.

Reasons for presence and abscence of features is frequently governed by