Warm tip: This article is reproduced from serverfault.com, please click

Solr: Searching with/without spaces in keywords

发布于 2020-12-04 15:19:48

I am experiencing an issue when spaces are introduced to keywords, for example:

  • We have a product with the title "Sony Playstation 4 Camera V2 PS4 (PSVR)"

  • Searching for "playstation" or "playstation camera" brings back this product

  • Searching for "play station" or "play station camera" does not bring back this product (notice the space)

Here is the fieldType being used:

    <fieldType name="text_en_splitting" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
        <analyzer type="index">
            <tokenizer class="solr.WhitespaceTokenizerFactory"/>
            <filter class="solr.HyphenatedWordsFilterFactory"/>
            <filter class="solr.EnglishMinimalStemFilterFactory"/>
            <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
            <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
            <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" preserveOriginal="1"/>
            <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
        <analyzer type="query">
            <tokenizer class="solr.WhitespaceTokenizerFactory"/>
            <filter class="solr.HyphenatedWordsFilterFactory"/>
            <filter class="solr.EnglishMinimalStemFilterFactory"/>
            <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
            <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
            <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" preserveOriginal="1"/>
            <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
    </fieldType>

How can I fix this, and make both "playstation" and "play station" match? This is only limited to PlayStation for my example, but it can happen to any search term e.g. "cyberpunk", "cyber punk". So solutions that require alot of manual work such as adding a synonym for play station => playstation are not feasible.

Things I have tried, but not managed to make work:

  • N-GRAM filter and tokenizer
  • Fuzzy search
  • Removing whitespace
  • Escaping whitespace
Questioner
Jacob-Fitzpatrick
Viewed
0
MatsLindh 2020-12-05 04:05:19

You can use a Shingle Filter to combine multiple tokens into one.

<analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.HyphenatedWordsFilterFactory"/>
    <filter class="solr.EnglishMinimalStemFilterFactory"/>
    <filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
    <filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
    <filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" preserveOriginal="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.ShingleFilterFactory"/>
</analyzer>

If you assume that the terms are spelled correctly when being indexed, you can apply this only when querying. It'll concatenate the tokens for you, effectively giving you multiple "merged" tokens:

play station camera => play, station, camera, playstation, stationcamera

.. given maxShingleSize=2. If you increase the max size to 3, this will also give you playstationcamera as a single token (in this case). If you have terms where people will possibly split a word multiple times, that might be necessary.

If you assume that your terms are indexed correctly, and this is only necessary on query time, your index won't change and you won't have to reindex (and the size won't change).

You might have to change the location of the filter around; your stemming filter will break this in mysterious places, since you'll end up concatenating previously stemmed terms.