Warm tip: This article is reproduced from serverfault.com, please click

Faster way to do this comparison of string in perl for foreign charsets

发布于 2020-11-28 08:36:56

I have the following code which works well:

foreach my $type (qw/Arabic Armenian Bengali Bopomofo Braille Buhid Canadian_Aboriginal Cherokee Cyrillic Devanagari Ethiopic Georgian Greek Gujarati Gurmukhi Han Hangul Hanunoo Hebrew Hiragana Kannada Katakana Khmer Lao Limbu Malayalam Mongolian Myanmar Ogham Oriya Runic Sinhala Syriac Tagalog Tagbanwa TaiLe Tamil Telugu Thaana Thai Tibetan/) {
    if ($page_title =~ /\p{Script_Extensions=$type}/i) {

        print qq|TITLE: $_->{domain} is not english ($type), so lets ignore it...\n| if $DEBUG > 0;

        last;
    }
}

All it does is look for certain charsets, so we can get rid of those we don't want. Now while it works, its a bit slow (as its doing a foreach() on each one). Is there a way to this in the single regex? (and also extract the matching set if possible)

UPDATE: I'm now trying as suggested with:

if ($page_title =~ /\p{Script_Extensions=Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Cherokee|Cyrillic|Devanagari|Ethiopic|Georgian|Greek|Gujarati|Gurmukhi|Hangul|Hanunoo|Hebrew|Hiragana|Kannada|Katakana|Khmer|Lao|Limbu|Malayalam|Mongolian|Myanmar|Ogham|Oriya|Runic|Sinhala|Syriac|Tagalog|Tagbanwa|TaiLe|Tamil|Telugu|Thaana|Thai|Tibetan}/i) {
    colored(qq|$page_title matches $1, so lets ignore... |, 'yellow on_magenta'), "\n";
}

and also:

if ($page_title =~ /\p{Script_Extensions=(Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Cherokee|Cyrillic|Devanagari|Ethiopic|Georgian|Greek|Gujarati|Gurmukhi|Hangul|Hanunoo|Hebrew|Hiragana|Kannada|Katakana|Khmer|Lao|Limbu|Malayalam|Mongolian|Myanmar|Ogham|Oriya|Runic|Sinhala|Syriac|Tagalog|Tagbanwa|TaiLe|Tamil|Telugu|Thaana|Thai|Tibetan)}/i) {
    colored(qq|$page_title matches $1, so lets ignore... |, 'yellow on_magenta'), "\n";
}

But I get an error:

Can't find Unicode property definition "Script_Extensions=Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Cherokee|Cyrillic|Devanagari|Ethiopic|Georgian|Greek|Gujarati|Gurmukhi|Hangul|Hanunoo|Hebrew|Hiragana|Kannada|Katakana|Khmer|Lao|Limbu|Malayalam|Mongolian|Myanmar|Ogham|Oriya|Runic|Sinhala|Syriac|Tagalog|Tagbanwa|TaiLe|Tamil|Telugu|Thaana|Thai|Tibetan" in regex; marked by <-- HERE in m/\p{Script_Extensions=Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Che at process-all-domains.cgi line 300.

Questioner
Andrew Newby
Viewed
0
ikegami 2020-11-29 15:22:32

First of all, I'm guessing there's no point in using /i.

As for solution, alternation is an option.

/
     \p{Script_Extensions=Arabic}
   | \p{Script_Extensions=Armenian}
   | \p{Script_Extensions=Bengali}
   | ...
   | \p{Script_Extensions=Thaana}
   | \p{Script_Extensions=Thai}
   | \p{Script_Extensions=Tibetan}
/x

Alternation allows us to use whitespace, but a faster solution would be a character class.

/
   [\p{Script_Extensions=Arabic}\p{Script_Extensions=Armenian}\p{Script_Extensions=Bengali}...\p{Script_Extensions=Thaana}\p{Script_Extensions=Thai}\p{Script_Extensions=Tibetan}]
/x

However, just 3 of the 41 properties takes up the entire width of the screen. As with the other solutions presented here, nothing stops you from building the pattern dynamically.

my $class_body =
   join '',
      map "\\p{Script_Extensions=$_}",
         qw(
            Arabic Armenian Bengali
            ...
            Thaana Thai Tibetan
         );

/[$class_body]/

But there's an alternative: (?[...])

use experimental qw( regex_sets );

/
   (?[ \p{Script_Extensions=Arabic}
   +   \p{Script_Extensions=Armenian}
   +   \p{Script_Extensions=Bengali}
   +   ...
   +   \p{Script_Extensions=Thaana}
   +   \p{Script_Extensions=Thai}
   +   \p{Script_Extensions=Tibetan}
   ])
/x

It's still experimental, but I consider it stable and likely to make it into language.