The week of 6/12-18 there was not a main server roll out. The week of 6/19-25 we did get a roll out on the main grid. The roll out had the group chat fixes.
Group chat has been receiving attention. The Utility Servers were updated over a week ago. Now the region simulators are updated, a minor fix.
One of the Group Chat Problems found was the Utility Server for chat had to send chat messages to multiple people in a region. If ten people were in a region the server had to send ten copies of chat updates to the region. This change allows the utility server to send the sim one message and the simulator distributes the chat updates. This sim change is not something that makes a lot of difference to most users. But, any load taken off the utility servers running group chat helps.
The previous update to the utility servers has made a significant difference to group chat for everyone. Enough so that this alarming JIRA was filed: https://jira.secondlife.com/browse/SVC-7031
The above image is a plot of chat messages (messages + group status updates) per minute sampled over one day. Each dot is one of the 16 utility nodes.
It shows the total number of messages being delivered per minute by each of the 16 group chat nodes. Previously any one node could handle at most 15000 messages a minute. You’ll notice that there are a lot of dots above that line. Now we can handle more like 30000 per minute. So, for most nodes, most of the time we have enough capacity [and] that’s why a lot of people saw [group chat] lag go away.
Now some nodes are still peaking up near 30K. Those would be ones serving some really big groups. You may still get [group chat] lag sometimes if you’re group is on one of those nodes at peak usage.
Overall, the good news is that we think we’ve caught up with group chat usage in terms of capacity. The bad news is that we’ve just caught up. This is appreciated by higher ups, so we are considering more long term improvements.
To fix the problem more permanently is going to take some architectural changes, which will take longer than what we’ve done so far.
Chat should be better, but we realize that we have’t got [group chat] lag beat yet.
Doubling the hardware to increase capacity is not going to work. The system is into the area of diminishing returns with the current system. So, a group that happens to be on an overloaded group chat node is going to suffer chat lag. There is no simple immediate fix. The current system does not allow a group-chat to move to a new chat node if the current node overloads.
The increase in capacity has helped everyone. The recent changes can really help a group where members are together in a region. When a group has members spread across the grid they are more likely to experience chat lag.
One of the changes to SL that is likely contributing to more chat load is how the SLV2 group chat works. The group chat load has nothing to do with how many members are in a group. So, the increased group limits are not the problem. Nor are all the old members that seldom logon. Only the number of people active in a group chat affects the load.
Because groups tend to remain open down in the lower right corner and out of our face, I suspect more people are leaving the groups open. I tend to do so. If you’re not participating in the chat, close the group. I try to remember to do that now.
Some viewers do not allow you to easily close a group for the secession. In the series one viewers the trick was to detach a group chat from the main chat floater then close it. That would close the group for the remainder of the secession. Chat does not work that way in many of the new viewers. So, if you’re logged onto SL you’re participating in all your group chats.
I’m not sure that there is a consistent way to temporarily close a group’s chat. Some viewers have a combined chat floater and others have a floater for each group. The minimize and close buttons seem to be identical in some viewers. In others the close button seems to close the chat for the remainder of the session. In others it doesn’t and any additional comments in a group chat will appear. So, check with your viewer’s support group and learn how to close group chat for the secession.
Blue Steel Release Channel
This channel has been running a package of fixes labeled Server-Maint. It includes a couple of new features; a error message level for the HTTP throttle and PRIM_OMEGA for use with llGetPrimitiveParams and related functions. Plus more bug fixes.
Le Tigre Release Channel
This channel continues to run the Mono2 upgrade. For weeks this has been in a testing/fix/retest cycle. Much of the Mono2 upgrade is about performance. The scripts our avatars wear cause lag when we enter or leave a region. While region crossings have greatly improved, there are still problems. The Mono2 update is being changed and improved.
This upgrade was held in testing because under certain conditions the upgrade kept a region from restarting, a really bad thing. Consider doing a region restart on your region and it never comes back up.
We discovered yesterday that certain avatars with a lot of attached scripts cause other scripts in the region to pause for an extended period [, as in minutes,] when they log out.[Part of discussion omitted.] Well, the worst case, when the worst avatar logs out of the worst region, is a 5 minute pause.
Whether one logs off SL or just TP’s to a new region the result is about the same. Considerable effort was put into solving the problem of new scripts rezzing and stalling the region. Now the pause is caused when scripts are being cleared. It is not really a sim freeze as the past freeze on rez was. Sim FPS remains high or changes very little in this case. In this case other scripts are stalled for a time. So, I suppose you can walk, but your weapon won’t fire.
The worse case avatar is one with so many scripts they have problems teleporting. It is not so much Mono scripts as what is done in the script. Maestro was testing with 1024 Mono compiled scripts attached and it was way better than another avatar with 100 scripts attached. The problem scripts are large and have lots of event handlers. Those that take over controls are somewhat worse than those that do not. Probably the single largest factor is the amount of memory being used by the script.
The Lindens do not have enough information yet to completely understand the problem. So, their not saying much more, for now.
Fancy scripts… it is hard to say what Maestro and Oskar were hinting at. Some comments may suggest complex genitals are some of the worst for this problem. Since most people are not going to give up sex… it is likely there will be a fix. Maestro thinks they have a solution that is promising.
Magnum Release Channel
The Mesh Prep upgrade is still in testing in this channel. Many of these changes have to do with Mesh Cost and Resource Equivalence equations. Mesh is disabled on a region by region basis. AFAIK all regions in this channel have mesh upload disabled.
This release also includes changes to physics and has the llCastRay() scripting function. The function is disabled in the release channel.
Hidden in the physics changes is stuff that affects vehicles.
So we made some more physics tweaks. Basically there will be some performance decreases for vehicles. These things can be tweaked on the content creator side.
Vehicles in SL are pretty much a hack anyway. We’ve been trying to move towards a model that works better with Havoc. Once mesh is out some things will become easier.
Last week vehicles had bounce problems. Those that have non-convex prim contact points have the worst problems. This week’s upgrades improved on the bounce problem. Quoting Maestro, “…some of the horses, for some reason they have a path cut and dimple which makes the physics shape behave poorly.”
The roll out plan has not been announced. Oskar expects the sim side stuff to be rolled out to the main grid with Mesh disabled. When they are ready to announce mesh they will enable the entire grid. But, the Mesh Team may have other plans.
The Server Beta Group meets at 3PM SLT Fridays in Morris, ADITI.