Misplaced Pages

Revision as of 19:49, 9 December 2005

• ]

Template messages allow certain standard text to be included on many pages, usually with the idea that in the future, any changes to that text block can be changed in one place.

"Meta-templates" as used in this article are those that are created and used to keep other templates in a standard format. They are extremely useful and convenient for many purposes.

However, note that Wikimedia developers agree that templates in templates can be severely problematic to server performance and that, as such, they should be avoided for technical reasons on pages that are at all popular. Jamesday has noted that they create a noticeable performance problem in general.

It is hoped that future versions of MediaWiki will not have this problem to this degree, if developers can be found to work on it.

Usage on Misplaced Pages

Convenience is often cited as the reason for using meta-templates, much like all templates. Certainly, it offers a central place with which to effect changes across a wide number of articles.

Examples of meta-templates currently in use on Misplaced Pages:

• Template:Message box - used currently on many top-of-page notice templates, such as Template:NPOV and Template:FAC.
• Template:Metastub & Template:MetaPicstub - used as part of the Misplaced Pages:WikiProject Stub sorting and Misplaced Pages:Stub categories to facilitate topical stub messages.

How a page is built and cached

Here's some technical background which may be of use. Jamesday 07:52, 13 Feb 2005 (UTC) and clarified by LarryLACa 00:22, 11 November 2005 (UTC) (Clarifications show in 's).

[Re: Rendering impact: The first time a page is viewed, i.e. after it has been removed from caches, the page request causes several steps:]

• Each item in the base page portion is requested from the database (images and CSS aren't in the main part). The page you edit, each template, each template included in the template and so on. Two templates, two database records to be retrieved. One template on its own, one read, one template including another, two. Plus the one for the base page.
• Once that and the rest of what is called the parsing is done, the page is saved in the parser cache. That's kept in RAM in memcached.
• Finally the skin is applied and the page is passed on to the Squids, which cache it in RAM and on disk (to get larger capacity but at slower access time) for all who aren't logged in (will only be useful if it's the normal skin) and send it on to the person who originally requested it.

• Whenever any part of a page is changed, be it the page itself or a template or image used in it:
  • The page is marked as changed ("touched") and will be regenerated next time it is requested. Both the Squid and parser caches have it removed. Necessary so people see the correct version.
  • The marking process involves a database update to every affected page, which for many pages can produce "replication lag" , with outdated information displayed to those using the page. This (the marking process) also shows up as slower response times when the lag is less than ten seconds. The effect is minimised by affecting small numbers of pages and maximised by affecting a large number, in part because the wait of up to ten seconds makes batches in the few thousand range effectively invisible except for delay in page load times. Touching about 18,000 pages currently takes a database slave with 4GB of RAM, 6 drive RAID 10 array and write caching disk controller some 90 seconds (that's from a real touch operation).
    • Assuming even distribution across 18,000 pages, Each of the 8 edits would flush from cache one eighth of the pages in each edit. On the database side, lets look at that 90 second case:
    • 90/8 = 11.25 seconds, call it 12.
    • Those who view in the first 2 seconds will wait ten seconds then see out of date information.
    • Those who view in the remaining ten seconds will see delay of up to ten seconds and then completely current data.
    • So, splitting it has removed most of the visible lag and visible problem.

The Squids, because of the limitations in the way they can work, with much less work per page, are inherently the fastest way of serving the pages and just 4 machines can serve some 75% of all hits to the site. But they are restricted in what they can serve. Next step is using the parser cache via the apache web servers. That allows all of the user settings for logged in people but uses more web server CPU so it's much less efficient.

We could switch everyone to using the apaches but that would be far less efficient and would require something like 4-5 times as many apaches and database servers as we have today, far more than the 4 machines gained by not using them as squids. And the page views would be slower, because it's an inherently slower process.

While all template use causes an extra database read and flush all pages using it, meta-templates are a special case because they use twice the number of database queries and can cause flushing of many more pages than other templates. If a meta-template is used in only a few or perhaps a few dozen templates which are fundamentally unrelated, it's debatable whether the extra equipment costs are worth it, compared to the relatively modest work involved in updating the individual templates. The replication lag issue can't so easily be addressed - that work is done by however many database servers are purchased. We're trying to reduce the effect but updates affecting many pages are inherently more problematic in this area than those affecting fewer.

Harmful effects

The impacts of meta-templates include not only direct server load effects but also indirect effects, e.g. creating vulnerabilities (Vandalism) or creating opportunities for inefficient development (Scope creep).

Server load

In normal cases, a single template entry is checked each time a page is displayed. When using a meta-template, each non-cached view adds one additional call. That is, one template within another doubles the database work of using the template compared to putting it all in one.

For templates that include parameters, this operation also adds incrementally to the processing time.

Each time a page is saved, all of its links from it are refreshed, including the reference to the template (and meta-template) being used. So when I update a page like Republic of Canada, the database creates all the normal links, and one to Template:Canada-stub and one to Template:MetaPicstub. Look at Special:Whatlinkshere/Template:MetaPicstub - none of those articles are actually calling it directly – they are just "false links". Because all links are purged and re-created on page save, it takes a lot more time and server resources than even the database read in the first place.

When a template is changed, all pages involving that template are marked as invalid and uncachable (by changing the database entry of every page which uses the template) and must be purged from the cache servers. On the next view of each of those pages, the apache web servers must also carry out the operation of rebuilding the cache page, increasing the load on them and slowing down the site. When individual work templates are involved, the flushing is limited to only the individual work concerned. When a meta-template is changed, the effect purges the cache of thousands of pages.

Vandalism

Meta-templates, which would be featured on a very high percentage of pages, are an excellent denial-of-service attack vector, since changing it or any component used in it would flush a substantial percentage of the site caches, which are critical to site performance and normally serve some 75-80% of all hits. Making even one subtle change, like the addition of a space, causes the effect.

In addition to the specific update issues for each template, the template's Whatlinkshere, image description pages and all other components involved in the template become denial of service attack vectors, since they can involve reading very large numbers of database entries or updating very large numbers of pages when they are changed.

Of course, more obvious forms of direct vandalism are possible when using these meta-templates.

Scope creep

The existence of some skeleton templates has resulted in an uncontrolled expansion of related meta-templates. (A skeleton template, as used here, is a template used by copy or subst to create another template.) This is because creating a new one (meta-template) is so simple given the existence of the skeleton. The primary example of this is Misplaced Pages:Stub categories, which lists templates created using the Template:Metastub & Template:MetaPicstub. There are literally hundreds of stub templates, many of which have been created on the spur of the moment for use on only a few articles and are not even documented on the main page. Although each new one was probably created in good faith, the sheer number of these templates has grown tremendously. Already, many of these new "child" templates have come up for deletion.

One of the almost paradoxical effects of this growth is that, as more and more "child" templates are created, there is often less of a chance for any particular change to be made to the meta-template, because of the wider impact.

Alternatives

1. MediaWiki needs developers. Metatemplates that did not cause the server hit would be of great benefit to the project, if someone can work out how to implement them in such a fashion. m:How to become a MediaWiki hacker
   • Please read "The touching process involves a database update to every affected page". There are changes in 1.5 which will help. How much is currently not known. Jamesday 14:55, 6 August 2005 (UTC)
2. Design, document, and implement - To give an example in the case of Misplaced Pages:Sister projects, a proposal was made to use a meta-template. In this area, it is much better to decide on a common look and format, document that standard for easy reference in case new related templates are needed, and then implement it across the few templates being used. When changes are needed, this gives one central place for discussion and revision. After there is consensus for the change, interested editors can quickly apply them. Creating a page which displays each template also helps to locate templates which don't follow the agreed upon format.
3. Make use of CSS - Some meta-templates serve only to produce a specific visual format – such as size, position, color. If these were identified, CSS classes could be added to the site's global stylesheets. The meta-templates could then be replaced with the CSS classes in relevant pages. This would accomplish the same purpose – maintaining uniform style across the site – without placing a burden on the server.
   • This would be very useful. I've previously discussed difficulties with using CSS, notably lack of project and page-specific CSS and the inability to use the span tag, which is part of the blocked HTML set. Jamesday 14:55, 6 August 2005 (UTC)
4. Use lists, not templates and categories - Some templates and categories are used with the goal of helping editors find articles under a specific topic area of interest. For this use, it is often more valuable to create a list, which can be annotated and prioritized. Many WikiProjects already maintain an area for reporting articles that need work.
5. Fix the problem, don't mark it with a template - If more effort was made correcting the problems which usually call for the use of a notice or template, fewer of them would be needed, less often.
6. Use {{subst: }} when creating the daughter templates. This copies the text message to the daughter template (substitution) rather than causing a transclusion, thus eliminating the second template call. (See the descriptions of the use of subst: here and here.) Of course, this means that when the metatemplate is changed, the daughter templates won't update, and won't be tracked by the What links here feature, which probably defeats the point of using a meta-template in the first place: If all of the daughter templates are to be updated, this would then have to be done by hand, expending a considerable amount of time, work and bandwidth. It's very unclear that this is better than just listing the templates and documenting the format in use on a project page for easy copy-and-pasting - see "Design, document and implement", above.
   • You can also do this on the end user pages. Where it is unimportant that the latest version is displayed this is a useful approach. Having the very latest version of say a stub message on a page, when that status will change only if the page is edited and the message can then be updated, doesn't seem to matter very much, since the message that it is a stub is the important part. Jamesday 14:55, 6 August 2005 (UTC)

See also

• Misplaced Pages:Transclusion costs and benefits

• Misplaced Pages style guidelines
• Misplaced Pages how-to