Second Life: How to Fix CDN Problems

Well, this may or may not fix your problem. But, it is a chance to improve things.

System Failure - Survival Horror Game

System Failure – Survival Horror Game by Loverdag, on Flickr

The problem some are seeing is texture and mesh fetching errors, which means regions and/or avatars are failing to render. When that happens things may be odd shaped or never appear and they stay gray. If you look in your Second Life™ log files you’ll find errors like these: 

LOG Error Examples

newview/lltexturefetch.cpp(1575) : 2015-03-31T03:12:25Z INFO: LLTextureFetchWorker::doWork: HTTP GET failed for: http://bake-texture.agni.lindenlab.com/texture/4c4ab9a6-0d19-4428-957e-bd283729474c/lower/dbb84db1-e… Status: Easy_28 Reason: ‘Timeout was reached’

2015-03-13T02:01:48Z INFO: LLTextureFetchWorker::doWork: HTTP GET failed for: http://asset-cdn.agni.lindenlab.com/?texture_id=65e28e31-9700-7784-4181-796869041279 Status: Easy_28 Reason: ‘Timeout was reached’

Oz explains that above error messages indicates either a texture or mesh download failed.

Oz and Monty Linden have been responding to a forum thread: Rez Failures & CDN Issues during Peak SL times? 5:30pm SLT – 9pm ish? Nightly?

The URL with asset-cdn indicates the attempt that failed was from a request to a CDN server. The above URL referencing bake-texture is an avatar baked texture, which also uses the same CDN service. The name difference according to Oz is just a legacy thing that evolved as the system grew.

If your having problems and are fairly certain it is CDN, Monty suggests you try these troubleshooting steps: (I’ve added my spin)

  1. In the viewer change these Debug Settings:
    1. Mesh2MaxConcurrentRequests (set to 3)
    2. TextureFetchConcurrency (set to 3)
  2. Shut down the viewer.
  3. Examine your log for texture failures. You are interested in the number of them. If you search on ‘HTTP GET failed’ using find all your editor often gives you a count.
  4. Clear the viewer cache – while this is normally a last resort step, it is needed here to create a better viewer log for troubleshooting.
  5. Restart the viewer.
  6. Go to a region you are having problems with.
  7. Give it 10 or 15 minutes then close the viewer.
  8. Examine to log again looking for failures. 404 Not Found errors are unrelated to CDN issues. Well, may be… I have thousands of those.
  9. If that has NOT helped, repeat this process and add on more setting.
    1. HttpPipelining (set to false) – Monty says, “Combined with the previous two, it may be a very slow load but I’m interested in error-free first.

Your log file is in:

Win 7 & 8: C:\Users\[Win_Login_ID]\AppData\Roaming\SecondLife\logs\

If this change gives you have way less errors, you have a work around. Otherwise, you have information to give to your ISP.

About this work around Monty says:

In no way is this intended to dismiss the problems people are seeing.  They are real and work is progressing behind the firewall on getting the experience up to expectations.  I’m just offering an expedient treat-the-symptom approach.  Something to alleviate some pain now and get sufferers closer to where they want to be.  It may also give some insight into where we might go with service monitoring or adaptive behavior in the client.

The CDN failures are geographically and time-of-day sensitive. As one of the people said, …lots of textures and avatars are now failing to load. An hour ago all was working fine.

Not all of us are seeing these problems. But, in SoCal there are days when regions load really slow. It very much looks to me like cold CDN caches. But, my home is repeatedly slow to load, which I just don’t understand, unless the CDN in my area is clearing cache every 20 or 30 minutes.

I think it is going to take some time for the Lab to figure this out. From their side it is impossible to see. They are dependent on users to find and report the problems. Knowing what to look for in the logs, gives us some idea what to report and a way to decide if our problem is CDN related.

4 thoughts on “Second Life: How to Fix CDN Problems

  1. Is the CDN server throttling? Does the stuff they supposedly hold as rezzed use up space they need during peak times so it purges? Can they set aside guaranteed space for a price?

    I am Ignorant about such things 🙂 but the gray – even on SL Go bugs me.

    • Different CDN services do things differently. So, we can’t know. But, each of your questions is a possibility.

      I am surprised SL Go would have gray. Their servers should have the benefit of CDN and their own local cache. I expect that their servers that generate the images they push to their SL Go app are like out local computers and have a ‘viewer’ cache.

  2. Frankly, while CDN is certainly a big improvement on LL’s side (by alleviating the load on their servers and links to Internet), it’s far from an improvement for the SLers… Here, in France, I did not encounter any speed-up since CDN has been activated on the grid, contrarily to the obvious improvements brought sooner by HTTP pipelining, and in fact things are often slower…

    The problem, especially in countries where few SLers are roaming the same sims as yourself, is that you will invariably end-up hitting deprecated CDN caches that will have to reload from scratch before they can deliver contents to you…

    Add to this problem the connection limits of CDN servers which might not be as high as LL’s servers allow (thus the workarounds you describe here).

    There is another way to fool the CDN mechanism (and thus to work around occasional glitches in some countries), simply by making it so that the DNS request your computer sends appears to be sent from another country (the CDN server you will hit depends on a DNS reply, the DNS using your IP address to determine your country and thus the CDN server closest to you). This can be achieved via “proxyfication” of your DNS request; one easy way to proxy DNS requests is to use TOR. Simply make sure that your TOR configuration file contains the line ‘DNSPort 53’, and then configure your system DNS IP address to 127.0.0.1 (it will then connect to your own computer on port 53, which the TOR demon will serve). This will cause all DNS requests to be done via TOR, and the actual request will be issued by the TOR exit node (which may reside in another country and which you may change at any time by simply requesting a new TOR circuit or ‘new identity’ as it is called in Vidalia).

  3. Per Henri advise after months of Linden Labs blaming the ISP and the ISP saying they see no issues until data reaches the Linden CDN server….I tried IP spoof/change. Actually spoofed a Swiss one, figured go all out of the country.

    Low and behold… EVERYTHING pops into place.

Leave a Reply

Your email address will not be published. Required fields are marked *