Press n or j to go to the next uncovered block, b, p or k for the previous block.
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | 6x 6x 326x 213x 14x | /**
* Shared text normalisation for RRM's inbound-message matching.
*
* This exists because two modules independently grew a `tokenise()` with the
* same name, in the same directory, doing subtly *different* things — one kept
* combining marks and one dropped them. Neither was wrong at its own call
* site, but two functions that look identical and disagree about Telugu is a
* trap with a long fuse. One implementation, and the stricter one wins.
*/
/**
* Characters deleted outright rather than turned into a space.
*
* Apostrophes (U+0027, U+2019): "don't message" and "dont message" have to end
* up as the same string, so the apostrophe must close the gap rather than open
* one. Phone keyboards produce the typographic form.
*
* Zero-width joiners (U+200C, U+200D) and variation selectors (U+FE00–U+FE0F)
* sit *inside* things. Replacing them with a space would cut a word in half,
* and a U+FE0F left behind by a stripped emoji would survive as a token of its
* own and push a short message past the opt-out length limit.
*
* Written as an alternation rather than a character class because a joiner
* inside a class is a lint error — in a class it would split emoji sequences.
*/
const DELETED_CHARS = /'|’|||[︀-️]/gu;
/**
* Everything that is not a letter, digit, combining mark or whitespace becomes
* a space.
*
* `\p{M}` is load-bearing and easy to leave out: Telugu vowel signs and the
* virama are marks, not letters (వద్దు is వ + ద + ్ + ద + ు), so dropping them
* would collapse "వద్దు" (no) and "వద్ద" (near) into the same string and let a
* script phrase list match the wrong words.
*/
const NON_TEXT = /[^\p{L}\p{N}\p{M}\s]/gu;
/**
* Lowercase, strip punctuation and emoji, collapse whitespace.
*
* Both sides of every comparison go through this — the inbound message and the
* phrase lists — so the lists can be written in plain form and compared
* literally.
*/
export function normaliseForMatching(text: string | null | undefined): string {
return (
(text ?? "")
// NFC first: the same Telugu word arrives composed differently from
// different keyboards, and everything downstream compares literally.
.normalize("NFC")
.toLowerCase()
.replace(DELETED_CHARS, "")
.replace(NON_TEXT, " ")
.replace(/\s+/g, " ")
.trim()
);
}
/** Splits an already-normalised string. Empty string yields no tokens. */
export function tokeniseNormalised(normalised: string): string[] {
return normalised === "" ? [] : normalised.split(" ");
}
/** Normalise and split in one step. */
export function tokenise(text: string | null | undefined): string[] {
return tokeniseNormalised(normaliseForMatching(text));
}
|