make search respect umlauts

[yvi]

Title:
make search respect umlauts

Area:
journal search

Summary:
Treat ä/ö/ü and other vowel 'variants' as different characters from a/o/u in the journal search.

Description:
Contrary to popular opinion, the special vowel (and sometimes also consonant) thingies in other languages (thingies is the technical term, really!) are not just fancy versions with a few dots or lines added, but different characters. I only know some things about German, but ä/ö/ü can be written as ae/oe/ue, but not as a/o/u.

I am sure people with knowledge about linguistics are getting a high blood pressure by now, but I really don't know a lot of these things. But what I know is that when I search for "Münster" in the journal search having 75% of the results be about The Munsters the Irish province is not exactly helpful and confusing at first because it's not the word I searched for. If there is any way this can be separated, it would be much appreciated.

Downside being that some people just write ä/ö/ü as a/o/u because they don't know how to do it on their keyboard. I used to type on an American keyboard and just learned the [Alt]+[numbers] combinations, but maybe that's just not widespread.

Poll #2205 make search respect umlauts

Open to: Registered Users, detailed results viewable to: All, participants: 43

View Respondents

This suggestion:

View Answers

Should be implemented as-is.
20 (46.5%)

Should be implemented with changes. (please comment)
7 (16.3%)

Shouldn't be implemented.
8 (18.6%)

(I have no opinion)
8 (18.6%)

(Other: please comment)
0 (0.0%)

Comments

[archane]

Based on my understanding of how searches work, there are two potential problem with implementing this. First, as you mention, use of accented characters is limited, and allowing both accented and non-accented results in the report means that I can have something which is meaningful and comprehensive rather than leaving out half of the results.

Second, my (admittedly limited) understanding of search functions is that to implement a search for special characters, the search protocol must know what character set you're working from, and while you're using Microsoft's [Alt] key to create accented characters, other people are using Apples [⌘] key, and some of us use html entity references. Ensuring that all of these options—but not the unaccented character—appear in the results could potentially be a programmer's nightmare.

[yvi]

Ensuring that all of these options—but not the unaccented character—appear in the results could potentially be a programmer's nightmare.

Apart from DW's search, I know no search that treats ä/ö/ü as a/o/u. Google and ebay treat them completely separately from what I can see, Yahoo and Amazon seem to match them with their ae/oe/ue variants. Other search engines do fuzzy matching for misspellings, so you get both results for Munster and Minster mixed in.

[yvi]

Second, my (admittedly limited) understanding of search functions is that to implement a search for special characters, the search protocol must know what character set you're working from, and while you're using Microsoft's [Alt] key to create accented characters, other people are using Apples [⌘] key, and some of us use html entity references.

In my understanding that's not necessary, as all pages on DW get UTF8 encoded no matter what you used to put the characters in.

(And as an completely unrelated aside, I have a German keyboard :) my father used to have an American keyboard, like, 10 years back, because it is a bit easier to use for programming, that's why I learned the Alt stuff)

[pseudomonas]

It's not clear what is being proposed here. A search for one variation should return all the likely variations in the text, otherwise you have to know exactly which variant is used - so a search for "naive" won't return anything if the author wrote "naïve", and searching for "café" won't find "cafe". "Ö" is an "oe" equivalent in German, but is not always such in English and in other languages.

[yvi]

Ö" is an "oe" equivalent in German, but is not always such in English and in other languages.

When is Ö being used in English? And if it is not used as 'oe', that would be a reason against just assuming that is as 'o' and instead just treating it as the same character, in my opinion.

[pseudomonas]

There are various categories;
German words, where ö -> oe, as you mention.
Loan-words and names where the diacritics are optional; the standard English spelling of Händel is Handel rather than Haendel, but a search should return all the variant forms.
Cases where the dots are a diaeresis rather than an umlaut: coöperate, indicating that co-o is broken into two syllables rather than one "coo". These are typically optional (and slightly archaic, but I like them)
Decorative uses like the Heavy metal umlaut - these are typically optional.

In all these cases it'd be better if one didn't miss results because the author had used a different spelling.

[alixtii]

+1

[pauamma]

Hmm: are ä ö ü different from a o u in all languages and writing systems that have them?

(Also, better than memorizing Unicode character codes, if you have a US keyboard, you could sett t to "US International" or its equivalent for your OS.)

[jaaaarne]

Yes. They mean different sound. Say, in my username the a's are actually ä's and are read more like ya's rather than a's. Hope that makes sense. :)

[yvi]

(Also, better than memorizing Unicode character codes, if you have a US keyboard, you could sett t to "US International" or its equivalent for your OS.)

I don't think my father would have liked that 10 years back when he had an American keyboard for making programming a bit easier and I was allowed to use his computer for accessing the net every two days for an hour :D

I still know the codes, though. 132 was ä, 148 was ö, 149 was ü and either 225 or 255 was ß :)

[aedifica]

I voted against because while I do know the difference between o-umlaut (or oe) and o, I have to look up the code every time I want to do it on my English keyboard--it's a non-trivial use of time, by the time it adds up. Also because not everyone uses the same characters to represent special characters, and I hate the thought of trying to do separate searches on o-with-umlaut, plain o, and oe every time I want to look for a word that sometimes contains an umlaut. (As it is I need to search on oe and plain o.)

[kyrielle]

I wonder if it could be made to include both exact-match and near-match as it does now, but float the exact-match to the top of the results.

[ninetydegrees]

That would be what I want too.

[msilverstar]

+1, definitely!

[thedivinegoat]

+1

[pseudomonas]

I like this option.

[cesy]

+1, with "near-match" defined as first oe and only then o.

[triadruid]

This is the only way in which I'll support this.

[lanterne_rouee]

Isn't it already differentiated? I have both 'carl barat' and 'carl barât' listed in my interests, for example. Clicking on each of them returns different lists of communities and interested users. I've always found this to be true of sites based on LJ's code. I guess journal search works differently?

I'd prefer if it was not differentiated (in any sort of search) though. It'd be simpler if it could catch everything so similar in one search, so you don't miss out on potentially relevant results.

[yvi]

I meant journal search, not interest search, which does indeed match exactly on these characters.

[lanterne_rouee]

Thanks for confirming that. I'd never noticed. :)

[schnurble]

If people don't know how to type ä, ö or ü, they can always type ae, oe and ue, because these are the correct replacements. So the search should match ä, ö, ü with ae, oe, ue and not with a, o, u just because people write it wrong in the first place.

There's also the character ß, which I've already seen replaced with a B, but no sensible search would search for a B just because people are creative.
So wen searching for special characters, it should search for what the character means and not what it looks alike.

[yvi]

So the search should match ä, ö, ü with ae, oe, ue and not with a, o, u just because people write it wrong in the first place.

At least Yahoo and Amazon seem to do that, and that's what 'd definitely prefer, too, yeah.

There's also the character ß, which I've already seen replaced with a B

...

There are no words for how wrong that is. If anything, it's a 'ss'. *boggles*

[cesy]

Yeeeeah. *blinks*

[pseudomonas]

Gawsh. I've seen people using ß for β (in non-greek phrases like ß-haemoglobin). It's all wrong.

[triadruid]

Those are the correct replacements in some languages/settings, but not all. And being prescriptive with our search results doesn't address that people are already doing this.

[inalasahl]

I like it the way it is. I like being able to search for "zoë" and getting "zoe," also. There are a lot of things the search does other than give you back the same exact characters you put in. I just did a search for [word]ing and got back results with [word]ed and [word], as well.