Debugging a Second Life Region’s Lag

I suspect many of us have visited a region and had our FPS (Frames per Second) rate drop to low single digits. Innula Zenovka is managing a region that has the problem and started looking for help figuring out what was wrong with it. I suppose managers like Innula get handed some real messes to deal with.

Innula says her viewer usually runs at about 30 FPS. But, in the problem region it drops to 2 to 3 FPS. I hate it when that happens.

See the thread: Advice, please, on reading consoles and debugging a laggy sim.

There is some good information and tips in the thread.

Ctrl-Alt-Shift-T Read Out

Ctrl-Alt-Shift-T Read Out

One tip is to select an item you think may be using a number of large textures. A simple right click on it will do. Then press Ctrl-Alt-Shift-T. This will send a series of notifications. You will get a list of the textures used in the item as a list ordered by each object in a link set. The notifications tell you the size of the texture and which face it is applied to.

You can see the result in the image: Ctrl-Alt-Shift-T Read Out. 

Another tip comes from Mr Ogynist. In searching to find lag problems he found that it was possible to find the problem by walking around in the region and seeing when rubber banding started. I think it would be easier and more precise to use the Viewer Stats and watch for FPS rate drops and PING spikes.

He found that when a large texture was directly in front or behind the avatar the viewer slowed down as it loaded and processed the textures. If the item were to the left or right, the viewer sort of ignored it and there was less lag, higher FPS. So, you can sort of play warmer colder to find the problem.

Having a building in between made no difference. That is something that Interest List improvements should have changed. Since Mr Ogynist doesn’t say when they were experimenting in finding the problem it is not possible to say whether this is still accurate or recent Interest List changes have changed that.

If you are wondering how to figure out if it is behind or in front of you, it is a bit tricky but there are ways. Reducing draw distance and moving forward and backward may reveal the direction. Moving left or right a distance and then turning the avatar right and left may reveal it.

The problem is complicated if there are lots of large textures causing problems. Using the Texture Console (Ctrl-Shift-3) will show you if single or multiple large textures are a problem.

Penny Patton points out her experience is scripts and textures are big lag producers.

OrinB pointed to a unique type of script scanner in the Market Place. See Parcel Script Time Scanner Version 1.1. It sells for L$225.

  • Scan entire parcel floor to ceiling for objects using script time.
  • Get name, location and script time of each object.
  • Get script time and object totals for entire parcel.
  • Super simple to use. Just rez, click and wait.
  • SLURLs included in the report for easy object finding.

For parcel owners that are not estate managers this would be a handy tool.

Maggie points people to: Understanding your Statistics Menu – Help Improve your Region’s Performance. The statistics menu is access via Ctrl-Alt-1.

Trinity Dejavu points out the lag caused by regions having to communicate lots of updates to all the people in a region. So, lots of avatars with scripts that update things in SL, like changing color, textures, or moving cause lots of updates to be sent.

When you see the region stats degrade you know those communication limits are being reached. Otherwise, scripts don’t have that much effect, well in-region scripts. Avatar scripts are different requiring the regions to transfer data from region to region as avatars tp in and out. The server works to move that information and to load or unload the scripts. So, avatar scripts have a tendency to cause lag spikes.

Some time ago there were huge spikes caused by arriving avatars. The Lab has greatly reduced that problem. The server update in RC this week (37) has an improvement that allows the region to multi-thread data transfer of departing avatar data. So, we should see some reduction in lag spikes from region departing avatars.

Still avatars with 3mb or more of attached scripts place a heavy and noticeable load on regions as they enter and leave. Having a script weight counter and using it is a sign of an avatar with an enlightened self-interest.

Darien Caldwell brings up the concept of script ‘flywheeling’. Once upon a time this was a problem. I remember when Kelly Linden was talking about the measures put in place to penalize those scripts. I think this was in the days when Kelly still was having his scripting meetings. So, this is almost a non-issue in today’s Second Life.

Darien mentions Babbage Linden explaining why scripts can be a problem and there is no way to prevent flywheeling steeling CPU share. That is pre 2010 information. See: Requiem for Babbage Linden.

See: #SecondLife News Week 16 (2012).

The idea of flywheeling was to have the script do something when it first started that was CPU intense to gain a larger allocation of script time. It then ran in its normal mode doing its thing much faster because of having a larger share of the script time. Occasionally the script would spin the flywheel again. This did impact other scripts. But these in-region scripts are all limited to a set amount of script time and now flywheeling scripts are generally ineffective. The gain from a flywheel spin is pretty much used up by the flywheel code itself. Removing the advantage to the remaining code.

Watching the script time, free times, and percent of scripts being run in Viewer Statistics tells you whether or not a region’s script performance is being degraded by scripts.

There are still ways to gain more script time and abuse the time slicing algorithm. So, some consider flywheeling to still be an issue. Most of the problems now come from avatar attached scripts, like animated tails. But, the recent changes make it difficult to degrade overall region performance attributable to the script.

Many animated tails use sculpty animation. The sculpt map is changed to create an animation effect. This forces the system to broadcast updates at each map change. That throws a load on the system. As shapes change everyone has to be updated and the physics engine has to up date too. Updating the physics engine places an exceptional load on the region server.

Animated mesh places more load on the viewer. Animated mesh is usually a series of shapes as sculpties are. But, all the shapes are loaded all the time. So, the physics engine does not have to be updated when the shape appears to change. It appears to change shape because each shape is made visible in turn by a texture or transparency change, which generates a viewer update notice that has to be sent to the viewers. But, that does not affect the physics engine. But all those polygons do increase the load on the viewer and such up some bandwidth.

6 thoughts on “Debugging a Second Life Region’s Lag

  1. An important point for those readers here who are not sophisticated in their understanding of lag:
    Your Lag is not the same as Region Lag. Region Lag will contribute to your experience of lag but there are many other contributing factors. Among them:
    Your settings.
    Your network connections (all the way through LL’s network to the servers and back).
    Your Hardware.
    What is around you in the region, especially other avatars.
    Etc.

  2. Ctrl-Alt-Shift-T is handy, thanks for mentioning it. I wonder if it messages you about all the textures on the object, or just the unrepeated: I do care more if my boat have hundred of 1024 textures, that if it have just one 1024×1024 texture repeated a hundred of times. Maybe server side there is no difference.. but for GPU memory for sure there is.

    • It does list the faces the texture appears on in a single prim. But, I don’t see any reference for the same texture appearing on multiple prims.

  3. So – since the topic of textures has come up- here are TWO questions that drive me nuts…
    1) if someone rezzes a prim – for this example a cube – and applies the SAME texture to every face, and the cube only needs a texture on 1 face- does the server and or viewer have to work harder to redisplay the texture multiple times even though its the same exact asset?

    2) If a texture is applied to a prim surface that is never displayed (burined under other prims) does it “count” – and would it be better off deleted?

    Thanks

Leave a Reply

Your email address will not be published.