December 11th – National Backup Awareness Day

Something horrible just happened to Jeff Atwood aka CodingHorror.

“ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images.”

“I had backups, mind you, but they were on the virtual machine itself :(“

It’s a times like these we start wishing for a time machine, a cosmic undo button or reversible computing.

Jeff’s blog was read by tenth of thousands of programmers and system administrators for many years. It contains information that is very valuable for these people, and represents an unthinkable amount of hours spent by Jeff. An agency rate for somebody like Jeff is between $250 and $500 an hour, but this is like appraising a priceless family heirloom.

I am not going to go through the motions of telling everybody how to backup things, about how important offisite backups are, how disk drives are fragile, how I don’t trust virtual servers, how raid is not a backup strategy, and how version control is not backup strategy, etc, etc. JWZ wrote a good article about backups.

Here are things I want to say. First, we are all not backed up sufficiently and likely have already lost data that we would want back.

I can’t find my grandmother’s recipe book (I still hope it’s only lost), my wife’s first email to me, my first web page through which she found me, my first job search web page that had a picture of the Twin Towers and said how I wanted to work there, my early school grading papers, a rare book about fishing in the Black Sea, a stamp from the Orange Republic that used to be in my father’s stamp album, the password to my very short-numbered ICQ account. A lot of stuff.

All of our digital information is susceptible to an electromagnetic pulse, fire, flood. Spinning platter hard drives are particularly bad – they have very short lifespans measured in low single digit years. CDs are even worse – aluminum inside them rots (I have a cd with a lot of outlook emails that reads as a blank filled with 1s).

So the first thing that I would like to mention is that if you never simulate a failure, you’ll never know if your stuff can be replaced. It’s not an easy thing to practice, though – restores and failovers are tricky to do.

A few jobs ago we were getting a fancy new load balancer set up. It was up and running, and supposedly we had failover: if one of the servers died, we would not even need to do anything, the backup servers would pick up the slack. I suggested that we should test it by pulling the network plug on one of the machines off hours. My boss would not allow that, saying that we could possibly break things. My argument that it’d be better if something like that happened when we were ready it would not be as bad if it happened when the actual failure would occur. When the actual failure did occur the load balancer did not switch, and we had an outage that was a good deal longer (it happened at night).

Load balancers are not backup solutions, but this story highlights an irrational streak in system administration: nobody wants to practice failure: it’s just too nerve-wracking, and a lot of hard work. It’s much easier to assume that somebody up the line did everything correctly: set up and tested backups, startup scripts, firewalls and load balancers. Setting up and validating backups and testing security are thankless jobs.

This brings me to a another point. The act of taking a backup is not risk free in itself. The biggest data losses that I suffered happened to me in the process of setting up backups. As an example I’ll bring up the legendary story about Steve Wozniak (whom I met yesterday):

The Woz was creating a floppy driver under an extreme time pressure, not sleeping much and feeling sick. The end result was a piece of software of unimaginable beauty: it bypassed a good deal of clunky hardware, and thanks to a special timing algorithm, was fast and quiet. When other disk drives sounded like a machine gun (I dealt with a few of those when I was young), Woz’s purred like a kitten. Finally he wrote the final copy onto a floppy, and decided to make a backup of it. Being dead tired, he confused the source and destination drives, and copied an empty floppy onto the one with the precious driver. Afterward he proceeded to burnish his place at the top of engineering Olympus by rewriting the thing from memory in an evening.

It’s really the easiest thing in the world to confuse the source and destination of a backup, destroying the original in the act of backup! The moral of the story?

Do as much backing up as possible, while being careful not to destroy your precious data in the process. Have an offsite backup. Print out your blog on paper if it’s any good. In fact, print out as much stuff as you can. Your backup strategy should be like a squirrel’s: bury stuff in as many places as possible (well, except sensitive information, which is a whole other story in itself).

Entrepreneurship Heros II: Night at the Museum

If the Seal of New York City were designed today, it would not have a sailor and a Native American on it. It would have a cab driver and a food cart vendor.

Cab driving and food vending wood seem like the two of the most democratic enterpreneurial options, the foundation of which is the public streets New York City: you just wheel out your vehicle and try to make some commerce happen. The only thing that you need is a license. The one for cab driving is called a “medallion”, costs $766K, and as an investment vehicle outperformed just about any commodity and stock index. The food cart licesnses are also very expensive. Plus you are hounded by NYPD, Department of Sanitation, and who knows what else. Cab drivers and food cart vendors are some of the hardest working and most prosecuted businesmen in the city, but sometimes they have their own victories, big and small.

You don’t need to go any further than the Metropolitain Museum of Art to see two interesting examples. Right in front of the museum there’s a collection of food carts. They all are very typical carts, none of them are of the fancy variety. There are two types represented – the basic “dirty water hot dog” cars and “street meat” carts. But there’s one important difference – they all have stickers that say “Disabled Veteran”, and there’s usually an actual veteran somewhere nearby.

In the past years the space in front of the museum was either empty or occupied by one or two carts licensed by the Department of Parks. Then one day Dan Rossi, a disabled veteran, discovered a 19th century state law that allows disabled veterans to sell food in areas that are off-limits to others. The location in front of the museum is particularly lucrative because there are no affordable restaurants as far as an overweight tourist can walk. This hack is a small, but significant victory for food vendors. They are still ticketed mercelesly by NYPD, have to work crazy hours, and deal with the need to urinate in some kind of a miraculous way. At least they got an article in the New York Times written about them.

Across the road from the veteran’s carts is a mansion that belongs to billionaire Tamir Sapir, a former cab driver.

Mr. Sapir’s legend starts in Georgia, USSR. He found an interesting niche business: filling out complicated emigration forms for the Soviet Jews. At some point he was persuaded by his mother to give up his excellent life (it was a very lucrative business, from what I understand) and emigrate to Israel himself. He found himself in the middle of the Yom Kippur War, and quickly emigrated to the United States. He worked hard to earn enough money to leave rural Kentucky for New York, and then even harder to buy a cab medallion (which was a lot more affordable in those days). Then he risked everything again by putting up that medallion as collateral for a loan that he needed to open up an electronics store with a partner.

In the 80s there was a bit of a thaw in Sovet-American relations – Perestroyka and whatnot. There was a significant amount of people visiting the US – diplomats, scientists, sailors, and those invited by relatives. These people were allowed to exchange a small sum of rubles into dollars at the official rate – if I remember correctly, 60-something kopeks to a dollar.

What these lucky tourists wanted the most was electronics. In particular – vcrs, doule deck cassette players, and Walkmen. They had the money to buy these things, but here’s a problem: they needed 220 volt round plug devices, and more than that, VCRs needed to support the SECAM standard. You could not just walk into any store and find these: American market was all 110V and NTSC.

Every child in Odessa back then knew all of this, as well as that if you found yourself in New York City with some money, all you needed to do was trudge over to Timur’s (this was before he changed his name) store in Manhattan and find 220V SECAM VCRs.

Mr. Sapir was making a mint, but more importantly he was making connections with the Soviet ministers, diplomats, and future oligarchs. A little later he was invited back to the USSR, and made more connections there. These connections allowed him to play on the Soviet deregulation arbitrage market.

You see, when the Soviet Union was transitioning to the market economy all prices were regulated except those for commidities like metals, oil, and fertilizer. Those with connections could buy these commodities for already devalued rubles and sell them abroad for hard currency, making millions of dollars. All you needed was connections, which Mr. Sapir had.

He made millions, but the game became very dangerous as people tougher than NYC cabbies entered it. Mr. Sapir did not continue his career as a commodity exporter. Instead he invested his millions into New York City skyscrapers. The real estate market bottomed out, and you could buy a whole skyscraper for 10 million dollars or so. He bought a whole bunch of them. The price of Manhattan real estate exploded, and he became a billionare.

He bought a mansion across from the Metropolitain Museum to house his collection of carved ivory (for some reason this was a very popular area of collecting in the Soviet Union), has a yacht that used to be stuffed with a collection of exotic animal taxidermy that could rival Mr. Burn’s wardrobe or Amy’s car from Futurama.

Well, the two lessons here are: 1) you have to take risks and 2) you have to find a niche. The rest is luck.

A couple of my friends created a new url shortener. Wait, stop booing. There’s a twist – it has the coolest url ever – And you can win something or other by just using it. And you get statistics. Well, I guess it’s about it. Did I mention these are my friends that are doing that?

Why should you care how short your url is? Well, it’s basically because the retards at Twitter don’t allow for inline urls (if they will one day, url shorteners would die like they deserve to), and if you want your stuff retweeted, you need to leave a couple of characters for RT and the username. Of course url shorteners are evil in general, and people at Twitter are incompetent technologically, but very lucky. And being lucky is more important than being competent.

The funny thing is, I absolutely legitemately won their first $5 Amazon gift certificate.

Semi-literate Programming

I recently finished “Coders at Work“, a series of interviews with famous programmers.

On one hand, reading a book like this is a downer: it’s very clear to me that I occupy a place that is very close to the median of the bell curve, and the skill level of programmers is a very steep non-linear curve in itself. I’ll never be as good as JWZ or Brad Fitzpatrick. But I knew that before, and I am ok with it. On the other hand, this book inspired me to read more code.

The programmers in the book disagree on many points, but they mostly agree on the importance of writing readable code and educating yourself by reading other people’s code. I make my living writing in scripting languages, and I haven’t written a line of C or C++ since college. But there’s nothing preventing me from downloading and taking a look at the source of Apache, PHP, MySQL.

It’s important for me to understand “how the sausage is made” in the PHP stack, and as it turns out, what happens between Apache PHP and MySQL in term of requests and timeouts is not as simple as one might think. I asked at StackOverflow about this, but all the diagrams that people pointed me at were of the very rudimentary type: “look, here’s a happy cow, it goes to Bovine University, look – it’s all shrink wrapped on the supermarket shelf” instead of “sausage farm/slaughterhouse/truck/factory tour, starting with cow insemenation”.

When I downloaded the source code of mod_rewrite, arguably the most useful Apache module in the world, I was amazed to find out that it’s only 5000 lines of C with comments.

The book ends with the interview of Donald Knuth, and another two major questions that the interviewer is asking everyone is – “have you read Knuth’s books and have you tried literate programming”. It was interesting to find out that most of the famous programmers use Knuth’s the same way that I do. The books sit on my bookshelf, I look at them, I sometimes try to read them, I skip most of the math. They serve as a constant reminder to me that I suck at computer science even more than I suck at programming, and luckily there are people out there who know all of this stuff who are not idiots like me.

Here’s a photo of my cubicle at TV Guide circa 2002, Knuth’s books are holding a place of honor next to the mini fridge. By the way, taking pictures of the places where you work and live is something that you should not forget to do: years from now nobody will care about those pictures of flowers, shadows, and sunsets, but

I’ve read the book about Literate Programming at the time, and was rather inspired by it. Ok, maybe I didn’t read it and more like skimmed it. I don’t think I understood what real literate programming is.

The way I understand it, Literate Programming is a way to write programs as a narrative that is readable to computers and humans. My father, in his former career a site supervisor (a type of a contractor) is very fond of giving very detailed instructions to me, the same way he used to give instructions to construction workers. His instructions usually are exaustive algorithms, with error handling. I think that his instructions, expressed as a flow of conciousness, would work not only on me and construction workers, but on computers as well, and are similar to what Donald Knuth has in mind. All you really have to do is to build a layer of abstraction between these instructions and a computer language. Also, since computers don’t forget things, he would only need to repeat his instructions once.

These days my dad is a COBOL programmer. Everybody dumps on COBOL, but in my mind it’s a language worth of a lot of respect. It has a syntax that is very English-like, something that makes reading COBOL code easy. Well, maybe it’s like reading some old-timer’s newsgroup post written in all caps, but it’s still much closer to English than most other computer languages.

At the time I was reading “Literate Programming” I was using ASP 3.0, IIS, and SQL Server 97. My task was to write a system that would account for booked and pending business. This is something that had to be done since the age of Mad Men. You see, the dealings of clients, account executives (like Pete Cambell), their bosses, account coordinators, creative department, etc are rather convoluted. But in the end, to get paid, you have to have a system that will track who brought in what business, who handled what, and how the commissions need to be split.

This is normally the realm of something called EAS (Enterprise Application Software). Back at the turn of the century, this area was still dominated by a company called SAP, but there were a few smaller players, like that tried to package these applications. Any sane IT manager looks to see if an EAS solution can be purchased first. It turned out that TV Guide’s buseness logic was impossible to shoehorn into any existing solution. SAP folks said – yeah, no problem, we’ll build you what you want, but our prices start at $1M, and then there are consultant fees. ERM world is a crazy place, you can read about some true craziness in “Cube Farm”, an account of one hapless developer’s adventures at Lawson Software. It’s a truly riveting book, and I fell that every developer out there should read it. It’s literally Lovecraftian in nature, that book.

In any case, it fell to me to develop the application from scratch. Inspired by Knuth, I decided to write some semi-literate code. Me and a project manager, Brad, went to the clients and interviewed them at length, documenting their existing process (aka the most complicated set of spreadsheets you’ve ever seen). In the past, before cheap computers, all you needed was a Joan Holloway, but I believe they stopped making them.

Brad went on to go back and forth with a very terse document about 5 pages in length that described how the new system would work. He would sit down with the clients and go through the narrative, step by step, confirming that this is what they wanted. Meanwhile I created an object oriented library that made dealing with the database, creating forms and navigation elements much easier. This is similar to to what you might find in a CMS like Drupal, only a little cruder.

When the document shaped up, I created the database schema, and then I took a big chunk of the document and pasted it into one huge comment block. I proceeded to break off chunks of that block and writing the code around it. Interestingly enough, as time went on, the project manager started helping me to write the code: enough of scary database abstration was hidden by simple classes and method, and there were tons of self-evident examples all around to copy and paste. I switched to writing reports that involved cubes, rollups and other fancy stuff. Stored procedures that did the reports also received comments from the document that described the reports.

This wasn’t a monolythic system – I was writing it for 2 years or so, releasing a chunk after chunk. In the end it was handed off to another developer, the whole transfer took only a couple of hours. There weren’t any major bugs, maintanence issues (I believe I received only one phone call about it after several years of continuous use). All in all I was pretty pleased with this approach and can absolutely recommend it.

I believe this is the reason why so many English majors become excellent programmers: if you can write for people, you can write for computers. Sometimes there are reasons why you can’t do both at the same time, but there’s no reason not to find some middle ground.

The Capacitor Plague

I woke up from a nap to a loud pop and a smell of burning plastic. The source turned out to be one of the most precious and important to me digital devices: a ReadyNAS NV+, a small silver box with over a terabyte of hard drives that store my backups, music, and photos.

Network attached storage (NAS) is an engineering compromise. It’s a storage solution that lets you keep a bunch of drives in a self-contained device. It’s redundant: you can lose a drive (which is a statistical certainty) and not lose your data. There are also handy usb ports that let you connect usb drives and a button to run backup jobs onto these drives. It also serves as a print server, and in theory it can be used as a streaming media server. On the other hand it’s slow (gigabit networks are not fast enough when you need gigs of data fast), a complete nightmare to use with photo managers like Picasa, and an even worse nightmare if you want to use it as a Time Capsule.

I’ve spent a lot of time babysitting my ReadyNAS NV+: changing the defective RAM that it shipped with, updating the buggy firmware, finding the right drives for it (some don’t have the right temperature sensors). Don’t get me started on what it took to make it work with Mac’s Time Machine.

And after all that, the one box that was supposed to keep my precious digital archives safe was smoking. This was preceeded by a few days of weird performance issues and a couple of hangs. The power supply finally died a horrible death, and I realized that once again I was falling victim (or “mugu” as Nigerians say) to faulty capacitors.

According to Wikipedia, the name of this phenomenon is “Capacitor Plague“. There is an epidemic of failure in electrolytic capacitors from certain shady manufacturers. Electrolytic capacitors are usually found in power supplies. They are little aluminum cylinders filled with special film and electrolytic liquid or gel. Power supplies get very hot, and the liquid part of the capacitors, the electrolyte, always wants to either dry up or explode. The formula for the electrolyte is very hard to get right.

The rumor is that one or a few companies resorted to industrial espionage to steal electrolyte formulations. They weren’t entirely successful – they either got an incomplete formula or just plain Brawndo.

Spectrum Online did some digging:

“According to the source, a scientist stole the formula for an electrolyte from his employer in Japan and began using it himself at the Chinese branch of a Taiwanese electrolyte manufacturer. He or his colleagues then sold the formula to an electrolyte maker in Taiwan, which began producing it for Taiwanese and possibly other capacitor firms. Unfortunately, the formula as sold was incomplete.
“It didn’t have the right additives,” says Dennis Zogbi, publisher of Passive Component Industry magazine (Cary, N.C.), which broke the story last fall. According to Zogbi’s sources, the capacitors made from the formula become unstable when charged, generating hydrogen gas, bursting, and letting the electrolyte leak onto the circuit board. Zogbi cites tests by Japanese manufacturers that indicate the capacitor’s lifetimes are half or less of the 4000 hours of continuous ripple current they are rated for.”

Wastefulness of today’s society masks the problem: most people don’t perform autopsies on their dead $70 DVD players or $500 computers, they just use that as an excuse to buy the new hottness. The techies with (or without) spare time and soldering skills do the following: fill bulleten boards with tales of saving their devices by soldering in new capacitors; search for instructions on how to solder and purchase capacitors; and curse creatively after doing it for the 5th time.

The unique thing about the capacitor plague is how easy it is to identify: the capacitors literally blow their tops, venting electrolyte through the special stress relief indentations. It’s also unique in that anybody with a soldering iron has a very good chance of fixing it: the caps are easy to locate and solder. In the age when most electronic components are of the “surface mount” type (the size of a sesame seed) or chips with dozens legs as fine as silk, soldering in a two legged capacitor is very refreshing.

Here’s a nest of capacitors from my busted power supply: two in the left corner are clearly popped, the one on the right is probably ok:

In the last couple of years the following devices that I own fell prey to faulty caps: a cheap off-brand dvd player, a speed control on my Dodge Caravan’s air conditioner, a Netgear network hub, a huge and expensive Air King window fan, and now, my ReadyNAS. The interesting thing is that the problem exists in both high end and low end products, as well as in high tech and low tech ones (I did not know there were electronic components in the window fan).

I am out of warranty on my ReadyNAS because I bought it in May of 07. The following passage leads me to believe that the shitty capacitors are a problem that they are aware of and (maybe) fixed in newer releases of the hardware (they could not offer a 5 year warranty if they used the same capacitors – they’d just go broke).

“Please be aware that ReadyNAS purchased prior to August 21, 2007 carries a one-year limited warranty. Extended warranty purchased for these ReadyNAS will be honored by NETGEAR. ReadyNAS NV+ and 1100 purchased August 21, 2007 and later have a 5-year limited warranty, and the ReadyNAS Duo has a 3-year warranty.”

The brand name of the popped capacitors reads “Fuhjyyu”. It lead me to the an urban dictionary entry that says that Fuhjyyu is either

“1) Chinese word for feces.


(2) Brand name of abysmal quality capacitors that are installed on logic boards, switching power supplies and various other electronic components.”

There’s also a post from a guy who implores ReadyNas to stop using those capacitors.

Then there’s – a global capacitor gripefest that is too depressing to read.

You can see a nice gallery of busted caps over here

There are broader implications of this: coupled with the fragile lead free solder, leaky capacitors don’t only cause kajillions of dollars of damage, but will also make electronics of our era impossible to use in the near future. The aluminum in burnable cds and dvds are rotting too, destroying the record of our time.

Three Firsts

I always thought that the quote went “I’ll try anything once” and it was Andy Warhol who said it. Apparently the quote is “I’ll try anything once, twice if I like it, three times to make sure” and it belongs to Mae West.

Living in New York, amongst other things, pushes you to try something for the first time ever almost every single day. Here are three things I recently tried for the first time, mostly under pressure from New York City.


Probably every New Yorker that looks even remotely Semitic in appearence has been repetidly asked “Are you Jewish?” by the Hasidim. If you answer yes, you’ll get a Billy Mays-worthy pitch to pray/light Shabbat candles/put on at Teffilin. I will admit to occasionally denying my membership in the tribe when in a hurry, but most of the time my answer is “a little bit”, followed by a firm sticking to plain cowardly agnosticism.

Ever since I wrote a long and rambling post about Tefillins, I meant to put one on. So this one time, after being approached by a young Hasid in the Atlantic station passageway, and customarily declining his pamphlet, I accepted his halfhearted offer to help me lay Teffilin. He was particularly surprised – I don’t really think he gets to help a lot of people perform this particular mitzvah a lot.

He produced a Tefillin set from a black shopping bag and a loaner kipah from a pocket, helped me put it on and say the necessary prayers right there on the BMT’s Atlantic Avenue platform, amongst the hustle and bustle of people and trains. It felt strange, yet somehow very comforting – performing this ritual in one of the most familiar places to me.

He also gave me a pamphlet in which the Lubavitcher Rebbe explains to a computer science professor that Tefillin is a symbolic representation of computers. He was very glad to be able to accomplish such an epic mitzvah.


I was walking through Union Square farmers’ market, already having sampled and bought a package of organic bacon hawked by an upstate hippie (I’m not a very observant Jew as you might have already noticed in this post). I was passing by a little stall providing free samples of wine made by hippies somewhere upstate. That bit of hippie bacon called for some wine, but I did not want to fight the mob of greedy Manhattan housewifes for a tiny sip, but then I heard a magical phrase – “We also have dandelion wine”.

I never really finished reading “Dandelion Wine” by Ray Bradbury, and always thought that there was no such thing – how can you make wine out of bitter yellow flowers? Apparently this is how. I bought a bottle. All I have to say is that dandelion wine tastes just like they say it should: like summer and childhood. I also bought some salad corn shoots that I’ve read about in New York Times. Those tasted like raw corn kernels.


You know that nobody really drives in New York because there are too many cars there. I’ve spent many happy car-less years here, but the arrival of a baby forced me to buy a car (a minivan, in fact). Parking is a very sore topic around these parts. I was never willing to splurge on a garage, and had to subject myself to the indignities of alternate side parking regulations.

There’s a whole book about parking in NYC – Calvin Trillin’s brilliant Tepper Isn’t Going Out. I bought it only because I have a friend named Tepper, but ended up immensely enjoying it. Which is what I can’t say about parking in the street.

Well, recently, I finally broke down and shelled out $200 for a spot in a garage. The feeling on the “alternate” days is rather novel – hey, I don’t need to move a car! Also new – not worrying about what those loud teenagers are probably doing to my poor car, or if there’s a used car window repair place that just received a shipment of my car’s specific windows (did you notice how they always have a used window for your car ready, no matter how obscure, when you go to a nearest car window shop for a mysteriously shattered one?). It’s a new and pleasant feeling.

What about you?

The Legend of Kovalevsky’s Tower

I decided that I’ll take a break from a blogging break to tell a story that has two of my favorite things: symbolism and legends of my hometown. In particular, it’s the legend of the Kovalevsky’s Tower.

A fountain and a tower are two very commonly used symbols. A fountain is a symbol for life and creativity, one with a mostly positive meaning. A tower is a darker symbol, one about overreaching achievement, and fall. In Tarot a Tower card is an ill omen. Yet a tower is a very attractive symbol. I am immensely drawn to towers, just look at how frequently I write about them.

There aren’t any skyscrapers in my hometown. There are fountains though. Not just the regular water spritzing ones: there are three very important neighborhoods called The Big, Middle, and Little Fountain. You see, Odessa, although on the sea, is located very far from any rivers that can provide potable water. In the olden days much of the water was procured from three wells, the Big, Middle, and Little Fountains, which were located relatively far from the city center, near the sea. An alternative way to collect water from rooftops was available, but as there’s a lot of dust in the air, people referred to this less tasty water as “this is no fountain”, an expression that survived to this day referring to something sub par.

If you take a tram from the city center, you’ll pass through the many stations of the Little, Middle and Big Fountains which follow the sea shore and are mostly filled with dachas. My grandparents used to have a dacha at the 13th station of the Big Fountain, a little plot of land with a house and outhouse that my father and grandfather built themselves. When we left, it was sold for about $7000. I am told that the land that we used to own would be worth about $1,000,000 today.

The last station is 16th. There you would switch to an ancient little tram with two “heads” (it was needed because there was only a single track, and one tram would service the whole line going back and forth). This tram, nicknamed push-pull would take you to yet another neighborhood called Kovalevsky’s Dacha. There were more dachas there, a large Russian Orthodox monastery, and natural, sand-less beaches with steep rocky cliffs (the rest of the shore used to be like that as well, but the cliffs where dynamited and sand washed on).

There is a legend connected to Kovalevsky and his dacha. This legend is a little bit like Rashomon, as there are many different ways it’s told.

In one version, a wealthy merchant, Timofey Kovalevsky decides to solve Odessa’s water problem by building the first water line. He finds a huge water well beyond Big Fountain and starts laying pipe into the city. At the source he builds a huge tower. He finishes the project, but he is financially ruined due to insufficient revenues. Either the water is of poor quality (not like Fountain water, because of the pipe sediment), or because people are afraid of technology and call his water “machine water”, or because around the same time another water line from Dniester river is completed. One way or another he is ruined, and commits suicide by jumping from the tower. In this version it is a story of pitfals of technology, its quick obsolescence and/or inferiority to older technology.

A famous version of the story comes from Kataev’s “Lonely White Sail”. In Kataev’s version Kovalevsky undertakes his water line project alone out of greed. A slew of bankers beg him to take them as partners, but he refuses, wanting a water monopoly. He builds the tower to get to the water, only to run out of money – the water is too deep, and the machinery and pipes are too expensive. He begs the bankers for money, but they refuse. Kovalevskiy keeps circling around the tower thinking about how to get money to complete the project, getting crazier and crazier, but one day gives up, climbs the tower and jumps off it. In Kataev’s version the legend is a cautionary tale of greed.

An even more disturbing version comes from Paustovsky’s “Slow Time”. Paustovsky tells the story differently. In it, Kovalesky is a very rich eccentric. He buys a plot of land far from the city and builds a dacha. Then he commissions a huge tower to be built, for no practical reason whatsoever. Several times he drinks tea at the top of the tower, and then commits suicide by jumping from the top. In another variant of this the purpose of the tower is mystifying the contractors, until the moment it’s complete: Kovalevsky has it built in order to jump from. This way it’s a story of depression, eccentricity, purposelessness, and suicide.

All stories mention that the tower was preserved until WWII, and did end up serving a purpose: it was a very good nautical navigational landmark. During WWII it was destroyed, similar to Tesla’s Wardenclyffe Tower on Long Island. Legend of Wardenclyffe Tower echoes many aspects of the legend of Kovalevsky’s Tower – the running out of funds, the mysterious purpose, etc.

This, of course is all very lame journalism and research. I am basically retelling the same story the way I heard it, the way I’ve read it on a number of websites, and in several books. I don’t have access to archives or any specialist literature that might shed any light on what really happened with Kovalevsky and his tower. I hope maybe commenters might eventually add some more information to this post.

I don’t know exactly where the tower used to be and where its remnants are right now. My understanding that it’s somewhere near the Big Fountain cape. The link provides the rough coordinates and a photo. Maybe it’s possible to find the location in Google Maps.

I was able to locate only a single photo that supposedly shows the tower on a website retelling one of the variants of the story. The tower is indeed humongous, and really does not look like a water tower. Maybe there’s a point to the the story about a pointless tower built for suicide.

[update] I recently found this old photo that provides a view from beyond the lighthouse:

From all the accounts that I’ve encountered the tower was destroyed. But what is this then? I’ll be in Odessa this summer, and I’ll mount an expedition to check this out.

The tower is described as an over-sized chess tower, a tower that looks like a lighthouse, a good navigational aid seen from the sea, and a ominous, dark building.

Here are some two old postcards (a photograph and its retouched version) of the Big Fountain cape. There is a lighthouse, and something that looks like a long slender tower. Could that be Kovalevsky’s tower?

This is what the natural beaches looked like all over Odessa’s coast. I remember beaches near Kovalevskiy’s dacha looking like that when I was little.

Another view of the Big Fountain cape, with a “shalanda” fishing boat. There’s something tower-like in the distance, but it’s very hard to tell.

A review of personal genomics profile from 23andme

What do I buy: a cleaning robot, an e-ink powered ebook, or a personal genetic test from This is one of those terrible trilemmas facing geeks who are not willing to sacrifice financial stability for having all the coolest gadgets. I chose a 23andme test, and haven’t regretted it.

The factoid that always is mentioned in the press about 23andme is that its co-founder Anne Wojcicki is married to Google co-founder Sergey Brin. This of course means that the data that 23andme is collecting is going to be used to help Google’s Skynet to gain the upper hand in the forthcoming war.

23andme’s service works like this: you give them some money (at the time when I write this the price is $399, but it used to be over $800), they send you a test tube into which you spit, some special preservative solution, and a return envelope. You follow some simple instructions and send your spit to a processing facility. Then you wait (this is the hard part). In a month or so (it depends how long the backlog of orders is) you recieve an email telling you that you can log into and take a look at the results.

The processing that they do in the lab creates a data file that represents an impressive number of genetic data points called SNPs or single-nucleotide polymorphisms. The way I understand it, SNPs are known variations that happen in DNA sequences. Much of the DNA stays the same for all people, but there are some variations in a single location. Let’s say there’s a string of DNA in humans that goes


and 6 positions there are the same for everyone, except the seventh, which is sometimes AT and sometimes GC. So basically it’s a single position on the one of the 23 chromosomes or mitochondrial DNA (I guess 23andmitochondriaandme was shot down as a possible name for the company) that statistically is different from person to person as opposed to long stretches of DNA that stay the same.

Scientists all over the world are studying the correlations that SNPs have with disease risks and traits. I guess these studies go something like this: you grab 500 programmers that use emacs editor and 500 that use vi. You have a hunch that editor preference is related to gene C4711. You sequence the dna of all your coders looking for SNPs in the gene C4711 that are one way in your vi-using alphas and another way in emacs-using sub-omegaloids. Then you find that in most of vi users SNP rs1729 is AA. Then you come out and say – AA in s1729 increases the odds of a programmer preferring vi by 47%.

The more people that a study has – the better the results. 23andme has genetics experts on staff that sift through mountains of these studies, rank them, and then tell the website people to add results to the interface. This way you can go to and see (as of right now) that tell you right away what your lifetime chances of getting 10 diseases. Some, like Parkinson’s, they don’t let you see without you agreeing to see it yourself. Sergey Brin was reported in the media to get some bad news from that particular one (mine came back “Typical Risk” which means that my chance of getting Parkinson’s is about 1.6%, same as everybody else).

A lot of people that I talked to about genetic testing told me that they’d “rather not know”. It’s true that some of the things that you might learn will make you worry without being able to do anything about it. On the other hand, there are some that you might be able to do something about.

“24.1 out of 100 people of European ethnicity who share this genotype will get Prostate Cancer between the ages of 35 and 79” vs “17.8 out of 100” on average. Here I learned that I have about 1 in 4 chance of getting prostate cancer. Prostate cancer is one of the relatively treatable ones if caught early, but with unpleasant diagnostic procedures. All of a sound “digital exam” does not sound so bad, and I guess I’ll have to bend over and cough a lot more often than I otherwise would. I’m not sure what coughing does, but the “digital” part has nothing to do with electronics or numbers.

There are also 10 traits that are available right now. The most interesting one for me was “Resistant to infection by the most common strain of HIV people usually encounter, though protection is not complete”. I apparently have two copies of something called “Delta32 version of CCR5”. “Although people with two copies of Delta32 are highly resistant to the most common type of HIV, they can be vulnerable to strains of the virus that do not use CCR5 to enter immune cells”.

This makes me a little bit more at ease when I donate blood, even though I know that the chances of getting infected with AIDS at a blood donation are already less than winning that Mega jackpot. On the other hand Dr. Asimov died of AIDS that he received through a transfusion. What’s more interesting, is that this is the same mutation that prominently figured in the media with the AIDS patient who was cured through a bone marrow transplant.

Then there’s a section that has 79 research reports. These get a rating fоr “research confidence”. I, for instance, have an elevated risk of obesity from research that has a three star rating.

Doctor House would have loved getting access to this kind of data for every one of his patients. There are risks for rare diseases mentioned on the show Ankylosing spondylitis, Hemochromatosis, Bloom’s syndrome, and everybody’s favorite – Lupus (which it’s never).

One of the tests included in 23andme is for muscle performance, the one that was in newspapers lately because it was offered by a few companies. The news angle was that sports obsessed parents paid hundred of dollars to find out if their kids have sprinting or marathon ability. Mine says: “One working copy of alpha-actinin-3 in fast-twitch muscle fiber. Many world-class sprinters and some endurance athletes have this genotype.” This sounds plausible – for a fat dude I have an uncanny ability to sprint, yet suck at long distances.

But wait, this is not all. For the same low price you also get some ancestral info. This comes in a form of maternal and paternal haplogroup. I am not sure I understand the what haplogroup is very well, and found 23andme’s infographics somewhat confusing. From what I understand a haplogroup is a number that is attached to a certain mutation (a few SNPs maybe?) that arose thousands of years ago and that has been statistically crosslinked to people living in certain geographical locations.

My paternal haplogroup is E3b1c1* – “populations: Ethiopians, Jordanians, Ashkenazi and Sephardic Jews”. My maternal haplogroup is H5a* – “populations: Lebanese, Polish, Irish”. Both of my ancestral branches are supposedly Ashkenazi, and I guess haplogroups are not a high enough resolution to prove or disprove Jewish ancestry. All they do is tell you that the progeny of your very remote ancestor was likely found in certain wide geographic locales “before the era of intercontinental travel.” What’s interesting is that there’s a small chunk of my maternal haplogroup’s population in Siberia.

23andme is a true Web 2.0 service (in the good sense of that expression). There’s a blog called The Spittoon. In between educational (sometimes interesting, sometimes boring) posts about genetics they have a section called “SNPwatch“. There they write about new research and usually provide a link to raw data in your 23andme profile. So for instance there was an article titled “Variants in Genes for Carcinogen Transporters Linked to Lung Cancer” about a mutation that prevents lungs from clearing out cancer-causing chemicals. Then you can click on two links and see if you have that mutation (to my relief I don’t).

There are some social network features: you can compare your DNA profile to your relatives. I’m not sure if hilarity might ensue because I think you can figure out possibility/impossibility of paternity and maternity, like Dr. House did in that episode. You can even compare your DNA to other users of 23andme – you have to accept a “friend” request for that. I’m not really sure what’s the fun in that. You can participate in surveys that might be used for research (there isn’t one yet about vi vs. emacs yet, but there are some other ones).

It does not look like the $400 fee even covers the cost of gathering the data (there no further fees as of now) or paying for staff/website maintenance, and ongoing research. I’m not really sure what the business model here is (and that makes me a little nervous. Businesses like this remind me of an old Yiddish joke:

Moyshe is selling boiled chicken eggs. He buys them for a kopek each and sells them for a kopek each. Chaim asks – “Moishe – where’s the profit in this?” “What, are you stupid?” – answers Moyshe. “I get to keep the broth”.

My guess is that the “broth” here is an opportunity to conduct groundbreaking research and maybe sell the anonymized data. That, and helping Google’s Skynet find and assassinate John Connor.

I don’t really worry too much about being denied insurance in the grim meathook future that surely is coming just because I used 23andme. There’s legislation against that on the books right now, and if insurance companies will be able to deny coverage based on genetics they’ll be much more likely to get this info by requiring a blood test.

I really wish I could get the data from my grandparents (none of whom are sadly alive). I will buy kits for my entire family though as soon as I’ll be willing to part with another chunk of disposable income (I do want that dang Kindle first though).

McCain and Palin Condoms

I do not like the Democratic and Republican candidates, and planning to spend my vote in the only way that it can make a difference in New York: voting for a third party candidate. I haven’t figured out which one – they all seem like losers too…

While tax plans that both Obama and McCain are proposing are not ideal for entrepreneurs, they themselves are lending a helping hand: yesterday in Times Square I’ve seen them help a dude market street meat –

and this enterprising young man was selling $10 McCain-Palin condoms –

The Man Who Owns the News: Inside the Secret World of Rupert Murdoch

If Rupert Murdoch isn’t making headlines, he’s busy buying the media outlets that generate the headlines. His News Corp. holdings—from the New York Post, Fox News, and most recently The Wall Street Journal, to name just a few—are vast, and his power is unrivaled. So what makes a man like this tick? Michael Wolff gives us the definitive answer in The Man Who Owns the News.

With unprecedented access to Rupert Murdoch himself, and his associates and family, Wolff chronicles the astonishing growth of Murdoch’s $70 billion media kingdom. In intimate detail, he probes the Murdoch family dynasty, from the battles that have threatened to destroy it to the reconciliations that seem to only make it stronger. Drawing upon hundreds of hours of interviews, he offers accounts of the Dow Jones takeover as well as plays for Yahoo! and Newsday as they’ve never been revealed before.

Written in the irresistible stye that only an award-winning columnist for Vanity Fair can deliver, The Man Who Owns the News offers an exclusive glimpse into a man who wields extraordinary power and influence in the media on a worldwide scale—and whose family is being groomed to carry his legacy into the future.