Wednesday, October 30, 2013

Beware Architecture Ghosts that go BOO!

As the Cowardly Lion once remarked, “I do believe in ghosts, I do, I do, I do believe in ghosts!” In the technology space, ghosts are fateful decisions, implicit or explicit, that are buried within a solution until the right set of circumstances raise them from the dead.

For instance, in January 1990, AT&T experienced a cascade of failures in their long distance network switches. It seems that each switch is capable of shutting itself down upon receipt of a special message. This, in and of itself isn’t entirely bad - there may be instances where you would want to shut down a switch. Unfortunately the “special message” is routinely initiated by a switch when it is itself recovering from a failure.

So, a system fails for some reason, and is brought back online whereupon it signals to its neighbors that it just recovered from a failure and sends the “special message.” Each neighbor in turn, crashes. Upon recovery - they each send out the same special message, and so on, and so on.

No one designed this in a deliberate sort of way, however the architecture was likely discussed, possibly debated, and … at … some … point … decided, i.e. thinking died. Thus a ghost was introduced into the system to remain forever dormant until that one special day.

Architecture ghosts are real, they even have names. We call them Sev-One and Sev-Two. These ethereal entities are the antithesis of reliability, availability, and predictability. Architecture ghosts are usually the result of tight coupling, but can be created out of two insidious verbal incarnations that include the demonic phrases of “there’s not enough time” and its evil corollary “that will never happen.” Speak these at your own peril.

As our thoughts turn to pumpkins, candy, and things that go beep in the night - let us learn from the architecture ghosts of our past. When Sev-One or -Two jump out to say Boo, let’s do a little Ghost Busting, and fix the real issue.

Follow by Email