Thursday we had a meeting with Lorca, Falcon, and Stinson Linden. We got some questions answered. Motor Loon had a good whine about Pathfinding, the keyword being good.
The Lindens have a couple of days of stats to look at now. The result is the average performance of the grid has not changed. This means that while most regions are NOT optimized, they are not pulling down performance.
Falcon pulled up the Pathfinding code and explained what the code is doing in English. What he said follows.
“Here’s how it works for the record and let me grab the code for reference.
First, the navmesh is updated to account for dynamic obstacles and the time required to do so is measured. The time used is compared to the maximum which is predetermined (4ms for full regions, 1ms for homesteads). If more than that time has been used, then the navmesh update step will be skipped for enough subsequent frames that the average time per frame is reduced to 4ms or 1ms.
You can see this in the “Num Skipped Frames” (I think?) item in the AI Stats section of the server stats floater.
With an optimized region the time used by the dynamic navmesh update should be near zero and we’ve only seen it become an issue in very very few regions.
Next, we perform path searches for some number of characters. There are additional throttles in place for those path searches. If we went over that time, we throttle down how many we try next frame. If we were well under that time, we increase how many we’ll do next frame.
We increase the number of path searches if we aren’t doing all [searches] every frame if <1ms is being spent on them and we decrease the number if >2ms is spent on them.
[Next] Then we handle advancing the characters physics. If the character has spent too long on this step in the past (more than 50us) we will skip updates in the future.“
To Optimize or Not
Regarding wheter one optimizes or not, Falcon says, “It [optimizing] will never be worse to set objects to one of the walkable/obstacle settings. It may or may not be better depending on the circumstances.
There are a number of future ideas we have, at the Lab, for neat new things we could do someday that will rely on that same data [walkable/obstacle settings].
So, you might as well mark it [optimize] if you can.
[…] Here’s what I would advise:
- Look at performance on your region. Is it noticeably worse than before? If not, you should still optimize for reasons of future features that might use that data, but you can do so whenever you get around to it.
- If performance is worse, look at the Sim Stats floater. Is AI Time high? If so, are silhouette steps being skipped? If the answer here is yes, then you have two choices:
- optimize your content by setting as many objects as possible to one of the static types (walkable, obstacle, etc.) or
- disable dynamic pathfinding.”
Lorca Linden said, “However, (b) is not generally necessary.
Again I want to stress that we have good metrics on average grid wide performance, and we saw no appreciable impact on this average grid wide performance when we released pathfinding to the Grid earlier this week.”
Falcon said, “Also, and I don’t think this has been said before at one of these meetings: The view of the physics world you see in the navmesh floater is an extremely accurate representation of how the physics engine sees it (with the exception of a 10cm size difference on boxes…long story for another day).”
Motor Loon was one of the region managers that decided to optimize his region. He found a gotcha in how the PF Tools work. Linksets that include phantom prims to reduce Land Impact are not handled properly. The navmesh/linkset list is not telling the user that it will turn off the phantom property of prims in linksets. I am fuzzy on why that happens. It seems like a bug to me. But, it apparently is a complex fix. So, we are likely to see some other fixes to help mitigate the problem before the phantom change bug gets resolved.
Also, a number of Linden Scripting Language (LSL) functions will not work once an object is set to a STATIC part of the Navmesh. Any function that moves a prim does not work when it is in a static object.
The problem comes when the object begins shouting error messages, like: Object: llSetLocalRot() does not work on objects that contribute to the Navmesh. The viewer is reporting most of these error shouting objects at position <0,0,0>, which is useless as the objects are not there. This is a definite bug.
What is happening is problems in multiple places. Using the PF Tools to optimize, one cannot see which linksets have scripts. Lorca is looking at having that information added to the tool.
When the linkset has a child object with a script that moves it, the error is reported as coming from an object located at <0,0,0>. Lorca talking with Kelly Linden finds that changing things so the root prim’s real position is reported is possible. These changes will fix most of the problems Motor ran into and will mitigate most of the issues until a more complex fix can be implemented.
The Sort of Fix
The Lindens will improve the tools and eventually get the phantom switching issue resolved. In the mean time if you have made changes and are trying to find objects shouting errors there is an easier process than going object by object through the entire region.
Stinson Linden reminded us we can use a ‘halves’ process to find the item. Using the process one can generally find a specific item out of a million items in 20 tries worst case.
If you converted your items and got the error, revert half of them back to a dynamic/movable type. If you still get the error, you know which half it is in. Repeat until you find it. It is still tedious, but nowhere near as bad as a one-by-one.
Falcon asked a question that I think is representative of how some of these bugs make it into a release. Falcon asked, “You were on the Magnum RC for weeks, some of you on the Second Life RC PF for months. This process hasn’t changed. Why didn’t you try this during the beta and provide feedback?”
There is no doubt in my mind the question is sincere. The community had weeks to test PF and the PF Tools. But, we did not find the problem. The Lindens did not anticipate these issues because it was not in the range of actions they anticipated. I’ll explain that a bit.
The Lindens are not profuse builders. Nor do they have the incentive to worry about Land Impact cost when they do build. So, their test builds tend to focus on the things they want to test, that they ‘think’ are representative, and they also tend to conform to the Best Practices when building, keyword being ‘tend’.
Residents on the other hand have a primary interest in mitigating Land Impact costs. Conforming to Best Practices is mostly a nice idea rather than a rule. So, we tend to push the envelope on things and use hacks or whatever means will reduce Land Impact costs. Residents can get really creative.
The result is the two groups build differently. What we do often surprises the Lindens. They, or anyone, cannot anticipate what tens of thousands of creative minds will come up with. So, the Lindens are going to miss things. Beta testing is supposed to catch those surprises. But, as I answered Falcon, “Falcon, as for why… it is just human nature to test just the things we want to do to see if they work. The idea that the phantom switch [problem] was likely to happen is not likely to come up for us.”
I doubt most users would get into testing region optimization until they had a reason to do so. Once it becomes something they want to do they start working with it. The few people that tried optimization did not run into the problem. It was not until people started optimizing everything in sight that it was found and that did not happen until after the release roll out. This is just part of how software development works when done by humans.
No one did anything ‘wrong’. People just did what people do. In hindsight it is an obvious problem and something that could have been tested and caught. As soon as humans develop perfect foresight these types of problems will stop… don’t expect that to happen any time soon.
So, a considerable portion of the SL population is spooked by the FS/PH post that regions will take an 18% performance hit from PF, if the regions are not optimized. Some people did not catch the ‘optimized’ part just the 18% hit. Some stopped reading at that point and reacted. While technically possible and factually accurate as to a possibility, it is also unrealistic in normal scenarios.
If those region owners had bothered to do some testing of their own, they would have had some facts to base the decision on. So, the result is many region owners are depriving their self and their residents of new features for no reason. But, they are free to do that.