What took Second Life down?

Last Monday early the Second Life™ grid went dark. I have heard the total number of concurrent users was 2.

The details of what happened were published Friday, The Road to Downtime Was Paved with Good Intentions.

The TL:DR is a complex plan had been devised to allow replacement of the routers that connect the SL system to the Internet without taking SL down. The complexity came from figuring out how to make the switch over without taking SL down. At home, you would just concede that replacing a router would interrupt your Internet service and deal with it.

The Art of Rain

The Art of Rain

The SL system has a boatload of routers and considerable redundancy. Meaning some number of those routers should be able to fail with the system falling back to the remaining operational ones without users ever noticing. Something the equipment manufacturer, network consultant, and Lindens had not imagined went sideways and cut the system off from users when it shouldn’t have. While they are not giving out exact details, the newly discovered way for the system to fail has now been built into the Linden planning to hopefully never cause a problem again. Continue reading

DDoS Attacks of February 2018

The previously reported (Second Life DDoS Attack) DDoS is mostly over… for now. Oddly, I am having to search for news of the attack(s). It is there, but the media does not seem interested. On the net, you can find information. Like:

I suspect the reason for the lack of coverage is this type of information is too techy for most newscasters and writers to handle… maybe even understand.

Warriors... come out to play

Warriors… come out to play

From Internet searches, we can see a lot of people knew this type of attack was coming way before it happened. But, those that failed to protect their servers from misuse apparently didn’t. This failure of people to prepare for a known possibility is rampant in society.

Those of us that want solid Internet connections can take some steps to mitigate the impact such attacks have. The attacks often target the DNS servers, the units that translate secondlife.com into something the computer can use, IPv6 = 2001:578:3f::30 or IPv4 = 216.82.8.56. The bad actors get the most impact for their clicks by taking down DNS servers as without them your computer can’t find the machine addresses and cannot connect. Fail. Continue reading

News Update 2018 w09

Earlier I reported (ref) on the DDoS attack and linked to the Linden post on the subject. Things seem to be getting better. Concurrent use was back to the mid 40k users.

In the Server-Scripting UG meeting, Simon Linden told us this has been on-going for about 24 hours. April Linden tells us the same in the SL Blog post.

However, from a credible source, I understand the SL Forum went nuts with fake news. So, be aware there are nuts posting on the forum that enjoy creating havoc.  The sad part is some believe what they are saying and think they know something. Others know they are deliberately making mischief.

How do you tell which is real? If you go into the Answers section of the forum and look at the 3 to 6 posts threads, you will find a small number of people consistently answering questions. Those people are mostly trustworthy. When they don’t know, they have been known to say so…

So, the statement the asset servers went down and all SL content was lost was totally bogus. I’m not sure why anyone would make that statement. There are those always wanting to harm the Lab. Others think it fun to scare people. Others seem to believe whatever pops into their head.

I understand some new people will believe anything about SL and the Lab. But, I have the idea that with all the fake news, propaganda, and spin people would be more skeptical of what they hear. Silly me.

Second Life News 2018 w09

The last few days we have had some people complaining about login issues. I’m one of them. But, the problem is so erratic and apparently random I can’t put any useful information in a JIRA.

This morning there was some issue with logins. Only 17,000+ were on when I logged in at 10:45 Am SLT. On my trackers next update, 19,000+ were on. That 17k is an extraordinarily small number. The low for the day is about 7,000… but it may have been zero. My tracker only grabs data every 10 minutes. A lot can happen in the computer world and RL in 10 minutes.

Lauressa - Close-up

Lauressa – Close-up

I currently show 3,000 to 2,000 people logging in every 10 minutes. This is indicative of some grid wide problem.

Status only says they are investigating intermittent login issues.

Update 11:00 AM SLT – Seems the problem is a DDOS attack. See Unscheduled DDoS. This is not a Linden Lab specific attack. Early indications, often wrong, suggest it is possibly an attack on America. I would not be surprised to trace it back to North Korea. They will likely react to the sanctions just placed on the Democratic People’s Republic of Korea. So, appease them now and face a nuclear attack later or deal with them now?

Did you know the average duration of a mass shooting is three minutes? What is the response time of the police in your neighborhood, to your sports arena, movie theater, or school? Continue reading

Internet Outage

The outage is large enough to be making national news. Amazon cloud servers went down. That affects many web sites and services like Second Life. The SL asset services depend on Amazon’s cloud, the S3 services – AFAIK.

SL login is supposedly not affected. The login servers are  in one of the Lab’s locations or on co-location hosts other than Amazon. But, once logged in you may not rez, the region may not rez completely, and your inventory may not populate.

Do NOT try to change or fix your appearance. You will likely confuse your viewer and make things worse. Do as little inventory-asset related tasks as possible.