What took Second Life down?

Last Monday early the Second Life™ grid went dark. I have heard the total number of concurrent users was 2.

The details of what happened were published Friday, The Road to Downtime Was Paved with Good Intentions.

The TL:DR is a complex plan had been devised to allow replacement of the routers that connect the SL system to the Internet without taking SL down. The complexity came from figuring out how to make the switch over without taking SL down. At home, you would just concede that replacing a router would interrupt your Internet service and deal with it.

The Art of Rain

The Art of Rain

The SL system has a boatload of routers and considerable redundancy. Meaning some number of those routers should be able to fail with the system falling back to the remaining operational ones without users ever noticing. Something the equipment manufacturer, network consultant, and Lindens had not imagined went sideways and cut the system off from users when it shouldn’t have. While they are not giving out exact details, the newly discovered way for the system to fail has now been built into the Linden planning to hopefully never cause a problem again.

So, in the middle of a complex project, the system was down. What to do? The decision was made to turn the system off, replace the rest of the routers as per the original goal but without all the fancy keep live footwork then turn it back on. Easier, but not what they had hoped for.

Telling users when they were planning to make this change was sort of a secret. What we were told time-wise was vague. That was deliberate. When the bad guys know something like this, they know it is the best time to attack. Sort of like why 98% of all mass shootings are in gun free zones. They may be crazy but they aren’t stupid. The Lindens were doing their best to avoid an attack while weakened.

So, now SL is back up with all new routers.

On a side note… these new routers were needed because the previous routers were near their end-of-life AND… they are needed for the move to the cloud.

Leave a Reply

Your email address will not be published. Required fields are marked *