Server

This continues to be a problem for event managers. Toysoldier is getting more and more Lindens to look at the forum thread for BUG-355 – Increased Instant Sim Lag & Avi Crashes During Major Events – sim network issue. (Limited access)

Andrew did not realize people were putting in the tech info needed to actually find the problem in the logs. I hope he has realized and taken a look at the thread: Increase in Instant SIM LAG & Crashes During Larger Events – Network Source?

Simon has responded in the thread: M7808.

Thanks for the reports, Toy … there’s some good information there that seems to shows the simulator getting into trouble when there are network problems.

If you are seeing the problem in your travels through the grid, read the thread so you know what to report and add your information. Please do not add noise. Just saying, “I saw that. Me too.” doesn’t help and actually hurts the process.

Toysoldier asks:

Next time one of your sims encounters one of these major lag events…. take a snapshot of the Advanced Performance screen and post it here.  The important stats to look at are related to the NETWORK stats. 

What has been initially noticed (and we would like more evidence of it from stale sims) is that some event triggers the the sim or the server kernel’s network driver or something related to the network to not process network packets at a normal flow.  Even at a low avi count and with other metrics looking normal, the network stats show unusually high pending downloads and very high un-acked packets.

But, we need more examples (snapshots of the performance stats) of a sim when it has fallen into this state.  Also make sure you mention date/time and the name of the sim. 

The thread: Increase in Instant SIM LAG & Crashes During Larger Events – Network Source?

Oskar Linden posted in the SL Forum requesting help testing some network changes. I suspect this is related to the Sudden Lag I’ve been writing about. See: Networking Optimization Pile On Test – October 11th

Linden Lab has made some changes to the way regions handle network traffic. We need your help to test. This will help us insure that regions are communicating appropriately over the network. Mainly we are concerned with agents entering via direct login, teleport, and region crossing. As well as other functions such as IMs, Voice, and Group chat.

Tomorrow, October 11th, at 4PM Pacific time (right after the server beta user group) we will conduct these tests with as many people as we have. Testing will take place on ADITI and require out of world communication we will be coordinating via IRC. This will require an IRC client connected to EFNET in the channel #sltest. If you don’t know how to do that you have until tomorrow afternoon to figure it out. 🙂

The details on the tests are here:

 – https://wiki.secondlife.com/wiki/Networking_Optimization_Pile_On_Tests

I hope you can come and help us test these new changes.

__Oskar

The test procedure is described in the SL Wiki. See: Networking Optimization Pile On Tests.

These tests will be on the ADITI grid.

There was a roll to the main channel this week. It seems to have gone well. At least all the regions get a restart after 3 weeks or so. The package was the one running on the Blue Steel RC Channel. It is a minor update and should have no visible effect on the simulator behavior. This is a release needed to enable a coming release.

Magnum

Will get a new package. It will be the main channel code plus the changes to allow better use of the new hardware. Not knowing exactly what hardware the Lab is upgrading to… its hard to speculate on what the change may be. Presumably, if they are running Intel, they may be adding more threading features for the iCore series chips. If they run AMD, I have no clue.

Read more

Toysoldier Thor has been relentlessly tracking down the causes of his in-world event crashes and sudden massive lag. Region owners and event managers have been collecting and providing data on the problem in the forum. See my previous article: New Lag Problem. See the forum thread here: Major Increase in Instant SIM LAG & Crash during large events.

Simon Linden acknowledged the effort and is starting to get people on the trail of the problem.  From Toysoldier:

From Simon Linden…
” Thanks for the reports, Toy … there’s some good information there that seems to shows the simulator getting into trouble when there are network problems. ”

I think now that we have a possible smoking gun… the LL staff can focus their investigation on the NETWORK.

Having more sim /region owners post the same performance statistics during one of these extreme lag events would help us further confirm the root cause.

If you are seeing this problem, read the thread and post the requested information from your occurrence of the event. Filing a JIRA works too.