Some transcripts got posted and there are some bits of news.
You may remember that the Lindens started balancing the regions by running adjacent regions in the same physical server, or at least physically nearby servers. They also made changes to the system so that popular regions would not load into the same server. The change improved performance and up time.
Since there was no new software roll to the main channel this week, they took the opportunity to do more balancing work. More busy regions were moved in to non-busy hosts. Oskar Linden says this makes a huge difference. They were going to get us some neat graphs and do a blog post, but that has never happened.
Over time as regions restart they drift… meaning the balance is lost. I suppose as busy regions crash more often they get moved to next available server more often and would tend to collect together. While rebalancing is a somewhat automated process Oskar says it takes some handholding to get it done well.
You may remember part of the balancing act was improving the region conductor or concierge service. This is the service that finds a crashed region and brings it up in a new host. The idea is to have it avoid putting two busy regions in the same server. Coyot Linden explains, “This is kind of hard [to do] because usually by the time the conductor sees a region should be put on a host, the region is down and wants to come up. If it is down, it is hard to see how busy it is…”
So, I suppose that is why busy regions can drift into the same server.
Maestro Linden said, “It [an idling region] would be doing its usual work for 22ms, then sleep 100ms after each frame. The full frame time is roughly 100ms. But it gives a bit extra back to scripts… So a frame would start every 122ms, so the FPS goes to ~8.2.”
“Child agents wake up a sim. So if any avatar sees into the sim, it’s non-idle.”
Simon Linden added, “Some of that additional 100ms is given to scripts — about 25, I believe. … up to 25, I mean. Physical things will slow down, and time dilation will be high … just like a slow region.”
“It’s more like a loop lasting 100ms: sleep a few ms, check for network activity, repeat.”
Simon has said the region can break out of Idle while the 100ms sleep loop is running. This means spin up is almost instant.
Homeless asked, “If the avatar has his back to the sim, will it remain idle?”
Maestro Linden answered, “Only if you drop the network connection to the sim, Homeless. Usually that doesn’t depend on your camera angle, though.”
Simon Linden added, “If that network activity is a ‘get out of idling’ event, (incoming http, for example) it will break out at that point.”
“This is less about saving energy than trying to free the CPU from doing un-needed work and having it available for other regions or back-end processes on the same server.”
Maestro Linden elaborates, “If you have a clock with hands that move with key framed motion, it might run behind when idle.” Exactly what Maestro means probably depends on the script the clock is running.
Simon added, “The time in key framed motion is stretched by lag … it’s not real-time.”
“The sim has some threading, and should get more in the future as we go after lag-causing problems. There are also other non-simulator processes running on the server that communicate with the back-end. Code that launches regions, talks to databases, handles capabilities, etc. Those will benefit [from Region Idling].”
Breed-able Pets and Other Fun Monsters
There is some concern by pet and animal owners that Region Idling will be a problem for them.
So, far the Lindens have not seen a problem in testing. Simon points out, “I can’t promise that [meaning promise breed-able pets won’t be affected]. If people have poorly scripted breed-ables, they will be affected. For example, if someone programs a breed-able to eat every 162000 frames (1 hour at 45 fps) then they would eat much less on an idling region.”
Before you decide if you breed-able is fated for disaster, you should do some testing. I suspect breed-able designer figured out that a laggy region would cut into their feed sales. It would be in their interest to assure that feed consumption was not tied to region performance. However, if they did not take that into consideration, the pet owner would be the winner.
I hear a number of pets handle food consumption via calls to external servers. The breed-ables’ designers run the external servers and those are not affected be the Lab’s Region Idle. They continue to run at 100% available CPU cycles. Outgoing and incoming HTTP calls spin a sim region up to full speed. So, the feed-hunger cycle should be unchanged.
Presumably the breed-able is asking the external server if they are hungry. If so and the pet needed to get to food, then the time they need to get to food would be longer. So, how long does one have to feed a pet before it starves? How long does it take your pet to reach food? If it takes 5 minutes with a region at 45 FPS then at 8 FPS it will take about 28 minutes… maybe. That depends on how their movement is scripted. I don’t know of any pets that starve in an hour, so they seem safe to me, which some find unfortunate. But, that is another issue.
Idling Feature Request
Some want to have a feature in the Estate Managers Console to disable Region Idle. The thought is those running science simulations and eco-systems would need it. I doubt that.
The llGetEnv(), llGetRegionFPS(), and llGetRegionTimeDilation () functions in the Linden Scripting Language (LSL) provide features that should allow scripter’s to sense region performance and adapt. If they are not using those functions now, it is likely the scripts are written in ways that will not be affected by Idling. If they do use those features, they would already be compensating for the slow region frame rates.
Some assume that most people would use a console feature to disable Region Idling when there is no reason to do so. This would have a negative effect on the entire grid. So, I doubt we will see such a feature unless someone can post clear user cases for it and justify a benefit that out weights the negative impact.
Bots & Idling
If you have bots in your region, it is never going to go into Region Idle mode. These are the bots that have an AI program running on your computer controlling an avatar like bot in-world. The SL system sees them as it would any avatar. So, regions with those bots will stay spun up.
There is concern about how Region Idling will affect scripted operations. The Region Idle code is currently running in Blue Steel. If you have a concern, go test. Get any problems you find into a JIRA. If the Lindens are not hearing of or seeing any problems, this feature may roll to the main grid on Tuesday.
The Region Crossing Refactor Phase I code is testing on the other two release channels. If it holds up and there are no problems, this will be the code they roll out. In which case, you’ll have another week to test Region Idling. Don’t count on that extra week. The Crossing code is complex and has proven tough to sort out. It has been on and off the release channels a number of times. I would not be surprised if it revealed new problems and remained in testing.