For the Lindens this was a bit of a hectic week. Originally Creative Tools was to roll the main grid, it did. A maintenance package was to roll to Blue Steel, it did. Pathfinding was to roll to Le Tigre and Magnum. That last one did not happen. Read on for the details.
Oskar says at the last minute they decided to put Creative Tools (AKA: Advanced Creative Tool, AKA Experience Tools) on Le Tigre and enable the Tools a number of regions in the channel. Pathfinding was updated to a new release with a number of fixes in Magnum.
Oskar explained some of the scheduling and work load around the roll outs. He says the weeks when they have three RC’s rolling out are really confusing because they have so much to watch, monitor, and track.
Monday and Tuesday the Lab’s software build system was not building. By the time they got the build system fixed they were almost out of time for the final QA checks. Oskar pointed out that they have very tight timelines with no room for failures. When something goes wrong it takes additional effort on the Lindens’ part to stay on schedule.
RC releases are rolled out on Wednesday. Wednesday and Thursday they monitor the new software. By Friday morning they are choosing which release to promote to the main channel next week. Then the candidate’s code has to be merged into the next releases for the three RC packages. Any merge issues have to be resolved. The code is then compiled and moved to QA as a deploy in ADITI where smoke tests and function verification tests are run. This work all has to happen and complete by Tuesday night to be ready for the Wednesday morning roll out.
QA in ADITI takes a full day to complete. There is little room for problems and having the build system down is a serious wrench in the works.
Oskar said, “Oftentimes we have to make a lot of decisions on the fly and then adjust the communications.” I take that to mean that the Deploys posts can get out of sync with what is happening.
The Lab has a new, or at least recently new, deploy-service with new features. Many are aware that it is slower than the previous deploy-system. But, each week it is supposedly getting faster. In the system is the ability to enable and disable features in the new server code. The idea is that if a problem revealed itself, they could simply disable the feature without having to down the regions and roll a previous version of the server software.
When running a live reconfiguration on Le Tigre the deploy-enable process failed. So, about half of Le Tigre has the Creative Tools enabled and the other half doesn’t. Coyote Linden is working to figure out what’s up with that.
Regions running Pathfinding have region crossing issues because of a Havok, physics engine, version mismatch. This is a known problem that ‘won’t be fixed’ as it will resolve itself when all regions run the same version.
However, the Lab is watching for problems and mitigating them as best possible. The regions experiencing the most problems are moved into a server running the same PF version. Oskar says so far, as of Thursday, only 7 regions have needed to be moved.
The bug where sculpty and prim attachment caused region crossing failures has been fixed. So, only mesh attachments are a problem and that is expected behavior.
It is being reported when crossing from any Blue Steel region to any non-Blue Steel region the avatar jumps about 20 meters into the destination region.
SVC-8050 – Crossing between Blue Steel sims and sims running different simulator versions causes loss of animation offsets.
Maestro Linden says they are seeing some oddities in Blue Steel. Driving across Blue Steel regions to non-Blue Steel creates some odd animation affects. For now they expect to just roll Blue Steel as is to the main channel rather than try to make a fix that will be useless in a couple of days.
Maestro has done some region crossing testing between 2 Pathfinding regions, then between 2 main channel version regions. He gets 50.9ms average vs 51.1ms average, crossing on a mesh motorcycle 40 times each. So, crossing time is essentially the same.
Crossing Stuff Coming
Oskar says that some new crossing oriented code is nearly ready for RC testing. I’m guessing this is some of the infrastructure code that adds multi-threaded crossing support, which was some time ago referred to as Phase II of Multi-Threaded Crossing.
The Pathfinding update should bring the new Land Impact Cost algorithm online. You can play with that in the Magnum sandboxes. I have yet to test it. But, it is supposed to reduce costs by capping some Prim Equivalence counts.
Pathfinding will be in RC at least one more week according to Oskar. I take that to mean they have found some things that need to be fixed.
Terrain Editing Oddity
In the Pathfinding regions terrain editing is as it always was but with a new oddity. The visual terrain and the physics model of the terrain are now different things. So, if you lower terrain to make a lake, you will be able to walk on the previous terrain surface until the Havok physics model of the terrain updates. You can now walk on water.
It takes the region some time to update the physics model. It can possibly take 10 to 20 minutes.
There is a work around. I have yet to try it. But with the Pathfinding Project Viewer one can rez a prim to get a Navmesh update button similar to the Stand button. Provided you are the only one in the region rezzing prims or editing terrain, clicking the button should trigger a recalc of the physics model. Depending on the complexity of the region the recalc can take from seconds to minutes to complete.
See: Pathfinding Quick Start.
It is nice to see the pace pick up a bit with the Release Channels. Having 3 different packages out is work for the Lindens, but it increases the odds we will see one roll to the main channel.
It is exciting to see Pathfinding and Advanced Creator Tools getting close to release. I’m looking forward to playing with both in my home on the main grid.