Baker Linden got us some new information this week in the Server-Scripting UG on the fix for Large Group Editing Problems.
For the last two months he has been trying to solve the problem of getting large data sets to transfer reliably via the current API (application programming interface) running over UDP (an Internet protocol – think of it as a spoken language), which is the protocol all viewers use now to get group member lists. It is when that UDP transfer fails that people have problems with group member lists failing to load. Those failures also create some simulator side issues that degrade performance.
The practical limit for use with UDP is a member list of about 10,000 members. Any group larger will not reliably download. On some connections even the smaller 10k list will have problems. Sometimes larger lists will download. Sometimes they won’t. Baker has optimized the SQL query that pulls the list and eliminated as many timing problems as possible. I get the impression that several SQL tricks were tried, like breaking up queries and batching group of members for transfer were tried. But, in the end it simply is not going to work. So, the new API is going to use HTTP for the transfer. Eventually the UDP service tube for group member lists will be eliminated.
Backward compatibility is going to consist of the servers only providing group member lists for groups less than 10k via UDP. So, older viewers will only be able to edit member lists for smaller groups. This is another nail in V1 code based viewer’s coffins. Plus this UDP download service for smaller groups will only be available while the services are transitioning to the new HTTP service.
Now that Baker has the server side working he has moved over to the viewer side. While he works on the viewer side the server code is going into QA next week, week 35.
He anticipated having working viewer code by end of work Friday (8/24). Initially the code will be for in-house testing by the Lab’s QA people. Sometime after that it will come out in a project viewer. If I understand this type of project’s QA, the code and project viewer will be made available to Third Party Viewer (TPV) developers. Their feedback will be used to make changes in the process and viewer. Servers in ADITI will be updated to run the new HTTP member list service to allow some testing. Then a project viewer will be released to the general user population for use on ADITI. Once things work well on ADITI simulators the simulator code will move to the release channels in AGNI and eventually the main channel.
How long this is going to take is undefined. It is not possible to know what the TPV developers will find nor object to nor how long it may take to fix those problems or concerns. Nor is it known what will be found in the general testing by users on the ADITI grid or eventually the AGNI release channels. At the absolute very best there are 3 or 4 weeks of testing and revisions ahead. I expect it to take longer. It may be 2 or more months depending one the problems encountered. All these unknowns are why the Lindens don’t give out ETA dates. But progress is being made and I can provide my guesses on ETA.
Initially there will be some rough spots. For instance, it takes some time for the larger group lists to download. A 40k member list produces about 5+mb of data. That can take from query to list load completion on the viewer side a couple of minutes or more. One needs a progress indicator of some kind. That indicator will be missing in the first pass. Users will have to have patience and understand what is happening. I can imagine how well that will work, not.
In the second or third pass HTTP 1.1 compression will be added for the member list download. That will speed things up. But, that compression step is apparently dependent on the HTTP Library now being worked on in the Shining Project. The HTTP Library, that I’ll call, version 1.0 is winding through QA now. It does not have all the planned features. But, it is already getting good reviews from those using the HTTP Project Viewer. These projects are intertwined and problems with either could slow the Group List Editing fixes.
Once the new code reaches the AGNI release channels, where you are when you attempt to edit a group’s members list is going to make a difference. This is where there will be some problems. Not everyone that owns a group pays attention to what’s going on with Second Life™. While it would be great if the Lab sent an email to each group owner explaining what is happening. But, I doubt even half of those getting the email would read it and the SQL query and email process would be a PITA to build. It probably is not worth the effort.
So, we’ll have transition issues as people bump into the new wall. BUT… if you own a group with (trivia alert) 112,000 members (largest group in SL) and use the project’s viewer from a release channel you will be able to edit the members list. Provided you have the patience to wait for the download into the viewer to complete. I have not tried it, but the SQL and download process could take several minutes, like 5 to 10… projecting a straight line based on info currently available. Probably less, but still long enough people will run out of patience.
The Lab has looked at and considered editing the list by ‘last login date.’ But, the feature is considered too risky. It is far too easy to goof entering a date and wipe out a group list. Imagine how you would feel if you cleaned out 111,996 members of 112,000 member group. HFS!!!
There will be some features for group owners to edit the list. But, it is likely to remain a tedious process because the Lindens and Baker in particular do not want to provide an overly powerful tool to users that would create more problems than it solves. This does not mean that TPV developers won’t.
If you are wondering of this will improve the group notice process, Baker doubts it. Group notices do not use any of the new code being provided. So, that is a project for another day.
You may have noticed how the Lab communicates is changing. Baker Linden is a recent new hire at the Lab. He has been discussing his work regularly for the last 3 or so months. This group editing fix, it’s not really a project in the Linden sense of a project, has been explained step-by-step. He has been open, with limits, about the challenges of adapting to how the SL system works and how the Lab does things.
This openness is providing insight to users on the human side of software development. He has spent, and I’m sure some, not me, would say ‘wasted’, a couple of months trying to make this fix backward compatible with older viewers. Now he is dropping the compatible criteria because of hard system limits in the UDP protocol’s nature and moving forward.
He has now gained some familiarity with the server side development. To complete the project he is now learning how to handle the viewer side.
That a new hire is taking on an old problem suggests something about where the Lab is workload-wise. This group edit thing has been a problem for a long time. It has only been since Baker’s hire that the Lab has had someone with the time to work on the problem. I may read too much into that whole speculation, but it seems reasonable to me. I think most of the staff is working on priority projects to move SL forward and the plates seem pretty full.