Autocomplete: Multi-lang search (based on user's lang)

[Joxit]

Transcription of #127 (comment)

What is this for ?

We want Pelias to send responses to queries written in other languages than English.
For example, a Dutch looking for Parijs (Paris in Dutch) will get Parijs, Frankrijk.

What should we do ?

• Use multi_match in autocomplete queries (done in Use default index + user-lang index in autocomplete #1300)
• Should search in name.$LANG index with higher boost
• Should search in name.default index with standard boost
• Should search in name.en index as fallback with lower boost (when $LANG is not en)
• Should return name.$LANG when available, default otherwise (done in Translate name to request language when available. #1301)
• Found something for matching parent e.g Parijs, Frankrijk.

Some use cases

Text	Lang	Result	Status
Parijs	nl	Parijs, Frankrijk (whosonfirst:locality:101751119 Paris)	KO
Londre	fr	Londres, Angleterre, Royaume-Uni (whosonfirst:locality:101750367 London)	KO
ブラジル	ja	ブラジル (whosonfirst:country:85633009 Brazil)	OK

cc @mihneadb Have you started working on it ? I can take the task if you want 😄

[mihneadb]

@Joxit I haven't, I was waiting for some clarifications in the other issue and I started working on some other stuff now. So sure, go ahead and take it, thanks a ton for doing this! :) 🥂

[Joxit]

It seems that in order to validate this issue, all the importers must support the multi-lang index.

At this time, only OSM supports it.
WOF will be supported with pelias/whosonfirst#446
Geonames needs alternateNamesV2 file to add multi-lang (we want that ?)
OpenAddresses and Polylines are unavailable

I think, the most important importer is WOF, the city/country search is the most common use case of the geocoder.

[mihneadb]

@Joxit FWIW there seems to be some level of support for that already - when looking for something, passing e.g. lang=en or lang=ru yields the same name but the city name is translated.

https://pelias.github.io/compare/#/v1/autocomplete%3Flang=en&text=red%20square%20moscow
vs
https://pelias.github.io/compare/#/v1/autocomplete%3Flang=ru&text=red%20square%20moscow
(see label)

I thought that data was based on WOF.

[Joxit]

Yes, this is done by pelias/placeholder which is a middleware and translate ElasticSearch responses for the user (using WOF ids).
This issue is about ES requests (and not responses).
That means, when you use lang=ru and search red square Москва, you will not found the correct venue (geonames:venue:6295575).

The data is present in WOF, but not indexed in ES, only the default name and English variant are currently indexed. That's why I opened pelias/whosonfirst#446 😄

[mihneadb]

Gotcha now, thanks! About that, I'm thinking we should also return Кра́сная пло́щадь if someone searches red square lang=ru, would you agree? I'm thinking this should be easier to achieve - building on what you pointed out about the middleware. I can make a PR if so.

[Joxit]

I think the API can return the name.{lang} index when it's available in OSM, but for Geonames, it will be a bit more tricky because we do not use it anywhere.
Maybe this can be added in placeholder ? But we will have conflicts with WOF data...

[mihneadb]

I was thinking about it at a higher level. Simplest seems to me to update geojsonify here: https://github.com/pelias/api/blob/master/helper/geojsonify.js#L55-L60

Instead of going for default, prioritize req.lang?

[mihneadb]

Hi everyone! Any update on this one? LMK if I can help some way.

[missinglink]

Hi @mihneadb, unfortunately, it's me whos the blocker here, I would like to land #1287 before merging this (It's a complex change but I'm planning on doing the final testing and merging next week).

It's really not ideal to hold back another PR, especially a community contribution, but it makes sense for us in this case because the PR I linked is a massive refactoring of how autocomplete queries are generated.

We are sometimes a little over-cautious with merging big PRs but it's our responsibility to ensure compatibility and reliability for organisations running Pelias in a production environment with user-facing traffic.

[missinglink]

Oh actually I thought this was another PR, but the same still applies to this one ;)

[mihneadb]

@missinglink thanks for the transparency! Looking forward to using the new parser! :)

[slvlirnoff]

@missinglink Any news on this?

[missinglink]

I've been sick this week but releasing the new parser is a top priority.