Sudden Lag on the Radar

Toysoldier Thor has been relentlessly tracking down the causes of his in-world event crashes and sudden massive lag. Region owners and event managers have been collecting and providing data on the problem in the forum. See my previous article: New Lag Problem. See the forum thread here: Major Increase in Instant SIM LAG & Crash during large events.

Simon Linden acknowledged the effort and is starting to get people on the trail of the problem.  From Toysoldier:

From Simon Linden…
” Thanks for the reports, Toy … there’s some good information there that seems to shows the simulator getting into trouble when there are network problems. ”

I think now that we have a possible smoking gun… the LL staff can focus their investigation on the NETWORK.

Having more sim /region owners post the same performance statistics during one of these extreme lag events would help us further confirm the root cause.

If you are seeing this problem, read the thread and post the requested information from your occurrence of the event. Filing a JIRA works too.

#SL Large Group Edit Update Wk40

Wednesday we got three packages rolled to the Release Candidate (RC) channels. The changes for Large Group Editing were in the package that rolled to the Le Tigre channel. While the packages rolled to Blue Steel and Magnum worked well, something went wrong in the Le Tigre roll out.

Large Groups – Image by: KittyKat3756 – Flickr

Something in the code is miscalculating the prim cost/count and forcing the return of large numbers of prims. You can imagine the screams. Very quickly the Lindens realized they had a problem. While they didn’t know the cause as of Thursday, they do know a massive number of prims were being returned.

The Le Tigre regions were rolled back and the package running on Blue Steel, a maintenance package, was rolled over Le Tigre. Unfortunately that roll could not undo the damage of returned prims. So, the damaged regions had to be restored to the region state previous to the damaging roll out.

Read more

#SL News 1 Week 40

Not much news this week. Plus I have been trying to resuscitate dying laptops. Seems to be a rash of those this week.

UPDATE: 1:54 PM PT – Le Tigre had serious problems and is being rolled back to the previous version. The regions have to be rolled back to get an earlier version of the server software. That is an automated process. Next all the regions in Le Tigre have to be manually updated to a previous SIM state prior to the problems. There are a over 1,000 regions to do. It is going to take some time.

Main Channel

There was no roll of a new server package to the main channel. Last week no Release Candidates made it to the RC channels, so there was nothing to roll.

There is a the New Lag Problem. This problem is pretty much invisible to the Lindens, meaning it is not showing up in their stats. Nor is it an easily reproducible problem. Both of those things make it invisible to the Lindens for all practical purposes. However, those running events are pulling their hair out dealing with the problem.

So, if you are an event manager or promoter, you will want to record the date and time, nearest minute, the lag hit, get as close as you can. Attend the Server Beta and Server & Scripting user group meetings (Time & Location) and provide at least the server version, region name, and date time of the problem.

Read more

New Lag Problem

In the forum and at Friday’s Server & Scripting User Group meeting the issue of sudden crippling lag came up. People are reporting regions with lots of free script time dropping to 0.3 Physics Frames per Second (FPS). One cannot move within the region nor can they TP out of the region. After a couple of minutes people are logged out of Second Life™.

Regions were events are occurring seem to get hit more often than other regions. It may be the high number of people. But, that may simply be their getting reported more often because people are around to notice. There are a few reports here and there in the forum from people asking about the problem. Those regions are not event regions nor do they always have a large number of avatars.

Read more

#SL News Final for Week 36

Wind

Andrew Linden has been working on Second Life™ Wind. You may not know it needed some help, and may still. Wind is rather random and quickly changes direction and speed. Andrew did some work on it. It should change less often now.

Sever-Scripting Meeting September 2012

Andrew Linden told us, “While working on the interest list stuff I spent some time optimizing how updates to the terrain and wind are prepared for the viewer. While I was at it I decided to see if I could tweak the wind, which has been described here at these user groups as “useless”.  So that it might be a little more what people are looking for.

[It is still] probably not useful for the sailing community, but I think I’ve managed to slow down its variation, so that it doesn’t get so wild sometimes. It will eventually be up on ADITI for people to play with if they like. I know wind isn’t really used for much except for some flexi prims that bend to the wind.”

Read more

#SL News Update 3 Week 36

Last Saturday Oskar Linden found SVC-8208 and the roll out plans for this week (36) changed. So, we missed a roll out to the main channel, which is old news now. This week things are regrouping and we are in hopes that the bandwidth fixes will make it to the main channel in week 37. I’ll know more later today (Friday), but I won’t get it on the blog until late Saturday.

Server Beta Group

Oskar was able to reproduce the error (8208) and get the crew started figuring it out and fixing it. Because the code deals with L$ the Lindens considered it serious enough not to proceed with a roll out.

Since Monday was a holiday in the US, Monday’s work and Tuesday’s testing all had to be crammed into Tuesday. We saw this code roll to the Magnum RC channel Wednesday.

Read more