Better multilingual entry support

[azurelunatic]

Title:
Better multilingual entry support

Area:
entries, search

Summary:
Allow entries to be tagged with the language(s) that they are composed of. This can be used to power more interesting things around the site.

Description:
Entries composed of written or spoken material (text, images of writing, audio, video) usually have one or more languages in which the material is presented. Allowing entries to be voluntarily tagged by their owners to describe the language(s) they are using might allow some interesting features to be developed based on entry tagging.

If a particular spelling appears in more than one language, specifying the language of the entry in site search could help find the thing someone's looking for.

Statistics on actual use of the site by users who speak different languages might be helpful to staff, especially if the technical barriers to offering the site in translation are overcome.

It could help users better connect with people who speak their same language, especially users whose preferred language is in a minority on the site.


What would the user interface be like? A whole long list of possible languages could a) be unwieldy, b) might also leave out languages used by actual site users (sign languages and constructed languages spring to mind as languages that might be left out of even a fairly exhaustive list of languages, and entries with embedded video might have sign language, and fannish communities are reasonably likely to include Tengwar and Klingon, and goodness knows there are probably more use cases that I know nothing of).

One way to do it might be like the tags interface, where something can be typed in, and attempt to autofill from a preset list, but accept new entries gracefully. If designed properly, unique data entered here on public entries could be logged, collated, and presented to an administrator on a regular basis for review; items that are found to be actual common languages not present on the list could then be entered.

Any site function that involves searching by language should allow for synonyms -- three different people might use "tlhIngan Hol", "pIqaD", and "Klingon" to mean the same language -- to say nothing of the typos. There should be a way to bundle known synonyms and known typos -- and also a way to override this bundling.

Another challenge is that people might not tag all their entries (to say nothing of back entries). How hard/expensive would it be to autodetect languages? Failing autodetection, could a default be set by user, like the last language they used?

Poll #7733 Better multilingual entry support

Open to: Registered Users, detailed results viewable to: All, participants: 66

View Respondents

This suggestion:

View Answers

Should be implemented as-is.
38 (57.6%)

Should be implemented with changes. (please comment)
4 (6.1%)

Shouldn't be implemented.
2 (3.0%)

(I have no opinion)
20 (30.3%)

(Other: please comment)
2 (3.0%)

Comments

[voldsom]

Regarding the 80/20 split? Is that a general 'this is how good detection algorithms are', or is that based on existing Dreamwidth content. Being contrary, I guess my concern is over whether general Dreamidth posts fall into or outside the standards being measured against. If 95% of posts fall into the 80% category, then okay; but if 50% of the posts fall into the 20% category it becomes less practical. I should also poke around the algorithms to try and learn how they handle multiple languages.

It's possible that you could handle most of the 20% by using the user's list of languages from their profile page (*), either simply (we can't be 100% certain of the auto detect, so let's use the default / not set it) or smarter (auto detect says it's X or Y, the user speaks B, M and X, so X it is).

Personally, while I agree that the post entry page is already busy enough, I would prefer to see it as an enterable field. I think that ultimately, the user should be able to control it. Actually, let me take a step back. I think it depends on how it's to be used.

I think the already linked suggestion for languages on profiles is a good precursor to this, and has all the same questions for how you manage languages and the entry and listing thereof.

If this is something that is supposed to be a user choice and/or will be displayed on individual user entries, then it needs to be something that the user can enter, so it becomes the users choice of whether they specify a single language or multiple, or default to leaving it blank and none at all. Leaving it blank is fine for user display here, but if it's to be used for other tools, you still have the problem of selecting a default language for the entry, which may mean forcing the user to have a primary language, and then other languages, which I'm not keen on.

If it's to be an invisible field, that's more used in the background for allowing more choices on the Latest Things page (I'd love to be able to browse for Japanese / Korean posts), then auto detection is probably good enough, within the provisos above. It takes the onus away from the user of having to key in the same language every single time, but it also takes away their choice.