#SL Group Edit Update Week 34

Baker Linden got us some new information this week in the Server-Scripting UG on the fix for Large Group Editing Problems.

For the last two months he has been trying to solve the problem of getting large data sets to transfer reliably via the current API (application programming interface) running over UDP (an Internet protocol – think of it as a spoken language), which is the protocol all viewers use now to get group member lists. It is when that UDP transfer fails that people have problems with group member lists failing to load. Those failures also create some simulator side issues that degrade performance.

Large Groups – Image by: KittyKat3756 – Flickr

The practical limit for use with UDP is a member list of about 10,000 members. Any group larger will not reliably download. On some connections even the smaller 10k list will have problems. Sometimes larger lists will download. Sometimes they won’t. Baker has optimized the SQL query that pulls the list and eliminated as many timing problems as possible. I get the impression that several SQL tricks were tried, like breaking up queries and batching group of members for transfer were tried. But, in the end it simply is not going to work. So, the new API is going to use HTTP for the transfer. Eventually the UDP service tube for group member lists will be eliminated.

Backward compatibility is going to consist of the servers only providing group member lists for groups less than 10k via UDP. So, older viewers will only be able to edit member lists for smaller groups. This is another nail in V1 code based viewer’s coffins. Plus this UDP download service for smaller groups will only be available while the services are transitioning to the new HTTP service.

Now that Baker has the server side working he has moved over to the viewer side. While he works on the viewer side the server code is going into QA next week, week 35.

He anticipated having working viewer code by end of work Friday (8/24). Initially the code will be for in-house testing by the Lab’s QA people. Sometime after that it will come out in a project viewer. If I understand this type of project’s QA, the code and project viewer will be made available to Third Party Viewer (TPV) developers. Their feedback will be used to make changes in the process and viewer. Servers in ADITI will be updated to run the new HTTP member list service to allow some testing. Then a project viewer will be released to the general user population for use on ADITI. Once things work well on ADITI simulators the simulator code will move to the release channels in AGNI and eventually the main channel.

How long this is going to take is undefined. It is not possible to know what the TPV developers will find nor object to nor how long it may take to fix those problems or concerns. Nor is it known what will be found in the general testing by users on the ADITI grid or eventually the AGNI release channels. At the absolute very best there are 3 or 4 weeks of testing and revisions ahead. I expect it to take longer. It may be 2 or more months depending one the problems encountered. All these unknowns are why the Lindens don’t give out ETA dates. But progress is being made and I can provide my guesses on ETA.

Initially there will be some rough spots. For instance, it takes some time for the larger group lists to download. A 40k member list produces about 5+mb of data. That can take from query to list load completion on the viewer side a couple of minutes or more. One needs a progress indicator of some kind. That indicator will be missing in the first pass. Users will have to have patience and understand what is happening. I can imagine how well that will work, not.

In the second or third pass HTTP 1.1 compression will be added for the member list download. That will speed things up. But, that compression step is apparently dependent on the HTTP Library now being worked on in the Shining Project. The HTTP Library, that I’ll call, version 1.0 is winding through QA now. It does not have all the planned features. But, it is already getting good reviews from those using the HTTP Project Viewer. These projects are intertwined and problems with either could slow the Group List Editing fixes.

Once the new code reaches the AGNI release channels, where you are when you attempt to edit a group’s members list is going to make a difference. This is where there will be some problems. Not everyone that owns a group pays attention to what’s going on with Second Life™. While it would be great if the Lab sent an email to each group owner explaining what is happening. But, I doubt even half of those getting the email would read it and the SQL query and email process would be a PITA to build. It probably is not worth the effort.

So, we’ll have transition issues as people bump into the new wall. BUT… if you own a group with (trivia alert) 112,000 members (largest group in SL) and use the project’s viewer from a release channel you will be able to edit the members list. Provided you have the patience to wait for the download into the viewer to complete. I have not tried it, but the SQL and download process could take several minutes, like 5 to 10… projecting a straight line based on info currently available. Probably less, but still long enough people will run out of patience.

Auto-Clean

The Lab has looked at and considered editing the list by ‘last login date.’ But, the feature is considered too risky. It is far too easy to goof entering a date and wipe out a group list. Imagine how you would feel if you cleaned out 111,996 members of 112,000 member group. HFS!!!

There will be some features for group owners to edit the list. But, it is likely to remain a tedious process because the Lindens and Baker in particular do not want to provide an overly powerful tool to users that would create more problems than it solves. This does not mean that TPV developers won’t.

Group Notices

If you are wondering of this will improve the group notice process, Baker doubts it. Group notices do not use any of the new code being provided. So, that is a project for another day.

Summary

You may have noticed how the Lab communicates is changing. Baker Linden is a recent new hire at the Lab. He has been discussing his work regularly for the last 3 or so months. This group editing fix, it’s not really a project in the Linden sense of a project, has been explained step-by-step. He has been open, with limits, about the challenges of adapting to how the SL system works and how the Lab does things.

This openness is providing insight to users on the human side of software development. He has spent, and I’m sure some, not me, would say ‘wasted’, a couple of months trying to make this fix backward compatible with older viewers. Now he is dropping the compatible criteria because of hard system limits in the UDP protocol’s nature and moving forward.

He has now gained some familiarity with the server side development. To complete the project he is now learning how to handle the viewer side.

That a new hire is taking on an old problem suggests something about where the Lab is workload-wise. This group edit thing has been a problem for a long time. It has only been since Baker’s hire that the Lab has had someone with the time to work on the problem. I may read too much into that whole speculation, but it seems reasonable to me. I think most of the staff is working on priority projects to move SL forward and the plates seem pretty full.

4 thoughts on “#SL Group Edit Update Week 34

  1. Not implementing a membership purge for fear of wiping off a group’s membership is not convincing. There are many ways this could be implemented, like setting a minimum value to remove all those whose last login is at least 1, or even 2 years. Even if someone makes a mistake nothing would be lost because those people didn’t login in a long time anyway. I would even make it so that the group owner can set an automatic purge to avoid any possible error. And if you are afraid people can make mistakes, just show a popup with the number of people being purged once the feature is enabled and a request for confirmation. I bet that those huge groups would have their membership reduced by half if a membership purge was implemented.

    As a matter of fact, the surprising thing about groups is that they don’t have a membership purge, which is a de facto standard in any forum package. The problem is that the whole group infrastructure is archaic and inefficient. It’s simply unbelievable that a group owner can’t even change the group name. There are a bunch of groups with mistyped names just because once you type the name and hit return it is carved in stone.

    As for Baker’s process on finding a solution and the Linedns having their hands full, I agree with your observations.

    A little, unrelated rant about the wasted time. Those who label 3 months of work in trying to find a solution as wasted usually are the bean counters, people who don’t understand that software development at times can be a lengthy process of trial and error, mostly when working on legacy code not written by the programmer who works on it (in this case the task is aggravated by being new in the company and having to understand how the system works). All the bean counters want is that the job is finished by yesterday. I meet such people in my daily job all the time. Usually their level of incompetence is equivalent to their arrogance.

    • On implementing a purge… the Linden experience is risky powerful commands create more problems than they solve. It doesn’t seem to work well with this group of people. We’ll see what they implement and then TPV peeps can develop what they want. For a group of 112k one needs some serious tools. But, I don’t see any way to talk a Linden into that.

      Baker did some checking on how many people would drop out on groups based on last login dates. It is not that many. It was a surprisingly small number, percent. But, if interactions between players are what keep people in SL, then it makes a bit of sense.

      There will be some type of purge tool. I expect it to be tedious to reduce problems… problems for the Lab.

      Someone can add a feature request for an automatic purge.

      The group name change not existing is a real pain. Never being able to reuse a name is a pain. I can understand not reusing a name for months as a reasonable security thing, but never? Really?

      Arrogance and ignorance seem to define incompetence… such people seem to spontaneously begin talking about what they know the least about… which is the best argument I have heard yet for creationism. In the evolution model we would have killed off such people in primitive times removing the traits from the gene pool… which obviously didn’t happen.

  2. I would rather see people who have not logged in for a long time moved to an Inactive subset. People fairly frequently leave SL (or a particular account) for rather long periods of time. When they come back it is good to have them return to the group.

  3. Thank you so much for this update!! So many in the community are really frustrated by this issue and I am glad to see some real activity on the project. This also touches so many people in so many ways they cannot perceive

    I will post this URL to the JIRA.

    On a very quiet sim, after a fresh relog i can sometimes manage to load my list of about 25,000 and was able to manually remove all the old non-logins and i also found a handfull that seemed to contain corrupt data and removed them as well. I brought my list down to about 14,000 and my greeter bot was then able to load the list. Manual removal was not too bad and you can select multiple records and remove them at one time…however I do agree (and i called for this in the JIRA) that an Auto-Removal system be added. It is too bad that those records were removed and lost…I am hoping that with the possible return of old people and the additon of new people through the inclusion of Second Life to STEAM will see a boost in past members returning…either way, this system will need to be ready to go and welcome/manage these old and new people.

Leave a Reply

Your email address will not be published. Required fields are marked *