We learned a bit more about Tuesday’s lag fest. It was a hardware issue in the Phoenix collocation facility. A network link and its redundant backup both failed. The Lindens got it fixed about 5PM. See my first announcement of the problem: Second Life Problems.
Rumor is Meeroos got loose in the server room and gnawed the cables.
Rollouts
Wednesday the Multi-Threaded Region-Crossing code rolled out to all three RC channels. The Lindens saw a sudden uptick in problems, which was mostly server crashes. Another problems was the loss of avatar attachments after several region crossings. In some cases after a relog the attachments were still missing.
After about 5 hours the Lindens decided to ‘turn off’ the region crossing code. You may remember some months ago the Lindens added a new Configuration Process that lets them turn new features on and off without having to roll a feature back. This time it saved having to down the regions with a roll back.
The current thinking is that the problem has to do with script serialization. Regions with high script loads seemed to be the ones failing. The test regions in ADITI did not have heavy script loads. So, the problem did not show up in ADITI QA testing. That hole has been filled. The test regions in ADITI now include some regions with heavy script loads.
Coyot Linden is seeing if he can get the heavy script regions named the Hollywood channel… you know… because of all the bad scripts… yuk, yuk. (I had to hear it, so I am passing it along.)
The Interest List package ran into problems and never made it out.
A problem with it went like this:
- UserA owns an object which is on the ground.
- UserB looks at the object and UserA, then turns his back to both the item and UserA.
- UserA wears the object as an attachment, then TPs away.
- UserB turns around and looks back. UserB sees the object on the ground, still.
Obviously the object should be gone and not show on UserB’s viewer.
Maestro Linden says a new maintenance package headed for an RC has some cleanup around the estate access code, and adds server support for ‘neck’ and ‘avatar center’ attachment points… say what?
For some reason the code server side to deal with neck and avatar center attachments points was never added to the servers. That has caused some errors… or so I hear. I haven’t seen them. But, the code will get added in the next couple of weeks.
There is a Threaded Rez Project that Baker Linden is working on. This is the one that moves object rez into separate threads. That prevents the region from freezing up while large mesh objects download. That code is being tested in the DRTSIM-194 channel in ADITI. One region in the channel is: CCMTEST26.
The code has improved region performance. But, there are a couple of bugs in it; a crash mode and a memory leak. Once those are fixed this should move through QA and make it to an RC pretty quickly. But, that is a Linden Quickly™ so think weeks.
Materials System
A bit about the coming Materials System is out. We now know that using Material normal and/or specular maps will cause the object to move into the new Land Impact accounting system, just as using custom physics shapes on non-mesh linksets moves them into LI system.