One of my readers is into sailing. Sailors have been having some problems with region crossings. We all have. Region crossings are an old problem that the Lab has worked on for years. It is a complex problem unique to Second Life™. The current Havok upgrade arriving with Pathfinding is a concern for the sailors because of know problems, which lead to the following in-world note.
Hi. I’m a reader of your server and scripting blogs, an active sailor, and Head Race Director for one of the clubs.
I was talking yesterday to our Estate Owner who is contemplating a bulletin to our members warning about what I think some are calling “Hell week”.
I have not been in world as much as normal over the last couple of months because of RL. So, I’m not fully up to speed on what’s going on. I’ve done some spelunking around and think I’ve got a rough notion of the issues at a superficial level anyway. I’ve drafted a note (which I’m dropping on you) that we might send round to our members. I would really appreciate it if you could take a look at it and set me straight if I have got the situation wrong.
I’ll be checking Oskar’s forum for specifics on this week’s roll, I haven’t seen anything yet.
I won’t relate the attached note card. The preliminary bulletin ties Havok and Pathfinding to closely together, but it was pretty much right on and the sentiments understandable. There is a problem with Havok Physics Engine upgrades. The problem comes from how Havok handles moving-mesh-objects or dynamic objects like a vehicle or boat. The Havok people are constantly improving their system, as is every company that makes physics engines for games.
In Second Life™ we have a problem with Havok causing region crossing problems, a somewhat unique problem to SL. Lots of information must pass from region to region as objects cross. Much of that information is sent to update the Havok engine in the region seeing the arrival of new objects. The Lindens are constantly working to improve that information transfer.
Currently we are running Phase I of the Multi-Threaded Region Crossing Code. Phase It does nothing we can notice. It is like seeing a new water pipeline before the water is turned on. It doesn’t do much. Phase II is somewhere in development and the QA pipeline. When Phase II arrives we should start to see improved region crossings. Some water will eventually come out the faucet.
All the Linden region crossing code has to work with Havok. Havok is third party software and is not something the Lindens will modify. It is normal not to make changes within products that have to be made with each upgrade of the product, especially something as complex and highly optimized as Havok.
As Havok upgrades the Lindens have to modify their code that handles the aspects of Second Life using Havok. Those changes are tested in the QA channel and then ADITI, the preview grid. Once those changes look good the changes move on to the release channels in AGNI, the main grid. When Havok updates arrive they arrive on only a few regions (release channel) and that creates the crossing problems.
The Lindens are considering ways to minimize the impact of Havok upgrades while in testing on the main grid’s release channels. The problem is Second Life needs to advance with technology and the Havok updates are problematic because of what the Havok people do. The Lab has a no-win decision to make: advance the SL tech and have problems or stop upgrades and fall further behind technologically.
The Havok update problem highlights one of the community problems too. A number of peeps in the community want to avoid change. Another number of peeps want the latest and greatest. It is a dilemma of sorts.
My experience is many only consider their personal immediate concerns. That is a practical free market premise that works pretty well in general and it is human nature, meaning it is hard to avoid. But, it can lead one to being short sighted. That short sighted thing can be on both the residents’ and the Lindens’ side of the problems. More effective solutions can often be found when taking a broader look at problems, immediate personal and long term. There is also the issue of what is best for the community. As we consider what is best for the community we start to deal with the ideas of minorities and majorities and things become political.
Smart political operatives think through the practical realities and figure out what is possible in the short term and long term. Into that context they insert the needs and goals of their party (group) and figure out what can be achieved and what must be endured as a plan progresses.
There is no doubt those making automotive race tracks and sailing competitions have challenges and have had problems for some time. The Havok upgrades are another annoyance that makes planning events difficult. To be told that once or twice a year regions crossing will be worse than usual is a bitter pill. Remembering that the pill will make things better in the long may help. But, the pill still tastes bad.
Over time region crossings have improved. They will continue to improve. The upgrade disruption is necessary to advance. The Lab is unlikely to be diverted from the process. But, the Lab may not understand how much of a problem those upgrades can be. Planning around unpredictable upgrades lasting for unpredictable periods is pretty much impossible. It is planned events that pull and keep some number of people in SL.
Getting the Lab to expend the resources needed to mitigate the Havok update problems, whether by rearranging the release channels (no easy feat) or hard scheduling testing and roll outs of Havok, is the only place I believe residents have a chance of influencing the Lab. Showing up at Oskar Linden’s Thursday server meeting in ADITI is probably the most effective communication point. Having a filed JIRA Feature request with a good plan for mitigating Havok update problems would be a good move. Oskar would probably help someone come up with the plan, or at least provide feedback on it.
I’m one who has noticed the return of sim crossing problems. The question I have in my mind is, why upgrade HAVOC until the Lab has a fix or workaround? This is not an unheard of practice.
There is not going to be a fix or work-around for an inherent problem in Havok upgrades. Havok™ is backward compatible. So, like most upgrades, it supports old games. The inherent problem in all software is compatibility from version to version. It is common to provide legacy or backward support. BUT… no one provides forward compatibility. It just is not needed nor is it generally possible. One can’t write old software to support new software without knowing what is going to be in the new software. If you knew, you would probably put it in. But, often new ideas that have not been thought of during the old version’s development cycles are used in the new version. No one wants to go back and update old software. It is easier to upgrade users to the new stuff.
Second Life has a unique problem: side-by-side regions running different versions. So, crossing from an old Havok version region to a new Havok version region is going to be a problem. It is likely always going to be a problem. The Lindens are considering what can reduce the problems. So, far the available mitigations are apparently too expensive in terms of labor, hardware, planning, or something. So, the Lab is deciding dealing with the crossing problems for a couple of weeks per year is cheaper than any currently available solution.
Remember. Solving the special crossing issue for a Havok upgrade is a minor issue. Fixing the general crossing problem is much more important. With limited resources the Lab has to prioritize. Pulling people off the Multi-Threaded Crossing Upgrade to work on Havok crossing issues would be counterproductive.
The current region crossing failure rate I experience is pretty low. Less than 1%, when I eyeball the maps.
Yes, it still needs work. But rolling out that Havok upgrade is a known nuisance to users. It isn’t a bug. We and the Lindens know a crossing is going to go wrong if it is from new to old, and those boundaries will go from 1% to 100% failure.
Limited resources? Well, maybe it makes sense to pay a bit more attention to managing roll-outs. Does the testing gain anything from a scattering of RC regions in the Blake Sea? Is this really a never to be repeated problem? Have the Lindens been getting away with being stupid about the logistics of a roll-out, because they’ve never had to roll out code that in known to break things?
I did a test this evening in a 30 prim speedboat and had very good success till I hit one sim that seemed to hurl me back to olden-time, sim crossing netherworld. I travelled over 10 sims at highspeed without problem though. The Blake Sea seems ok for now.
OK, I am confused. The problem appears to be with region crossings from the new version of Havok to the old. So how can it matter that the old version cannot be written in knowledge of the new version?
Anyway, this problem is going to repeat. And Linden Labs are going to have to find ways to reduce the shocks. Suggestions have been made, but there seems to be no time to implement anything. And if they haven’t tweaked the new code used to do the roll-outs, it is going to be a horribly slow process even if nothing goes wrong. It looks as if Linden Labs have ended up with a bunch of medium-risk changes all happening at the same time, and so far none of them have quite gone right.
My own inclination would be to hold back on Pathfinding until the roll-out system worked better. It is something you need to work well if something goes wrong. When a roll-out has been taking 12 hours for Main Channel, it’s likely that a roll-back to get out of a mess will take that long. That looks like a risky option to me.
And if Havok version is a predictable issue–we didn’t get this problem with Mesh, but I thought that needed a new Havok version–somebody needs to be looking at ways to minimise the effects.
1: Do the hooks exist in LSL to allow something similar to the Banline HUD to warn of different Server versions?
2: Is the distribution of RC regions in areas such as the Blake Sea something that should be changed?
3: I know it was suggested that a whole continent be set to the same RC for testing Pathfinding, so that region crossing wouldn’t have version-specific issues. Simple, obvious, and it couldn’t be done. Is it something that can never be done, or does it need more time than was deemed available?
3a: One of the things they did to ease region crossing was to put map-nearby regions in net-nearby sim instances. It seems that the logic behind assigning a region to a sim server, during region restarts, erodes that structure. Is such remapping a good idea at times like this, a useful preparatory step, or does it increase the risk?
4: Is Pathfinding worth it?
Projecting our inclinations on the Lab doesn’t work well. We don’t have the information and stats that the Lab has.
Mesh used Havok in the upload process to calculate the convex hulls for uploads. Havok upgrades were separate upgrades at the time, AFAIK. For whatever reason they have tied the Havok upgrade to Pathfinding. They are using this Havok in different ways than they have in the past. So, supposedly something in the new uses is tied to how we build and that is definitely tied into Havok.
AFAIK the Blake sea is not in an RC… I should go look.
#3 I have no clue why they give up on the continent idea. I suspect that idea is a problem for those that have volunteered to be in release channels.
3$ That decision was made long ago. One can’t decide those things from outside the Lab. We don’t have their plans or stats. We know what we want and would like. But, trying to put that in absolutes as in is or sin’t worth it, is silly. Feature worth is very subjective.
Pingback: There may be trouble ahead … and they’re already calling it Hell Week « Prim Perfect