R: tokenize n-grams but not strip punctuations

I played around with this a little bit -- and I am thinking you should convert your [ and ] characters to something unique but alphanumeric. It seems like {quanteda} wants to parse tokens that contain or are adjacent to special characters in this way -- and not consider them part of the "word" per se. Since your concept of "[sad]" is a single word, then to tokenize it, just do something that distinguishes it from regular "sad".

I use gsub and search for patterns "\\[" and "\\]" respectively. [ is a regular expression special character so you need to escape it with two backslashes. I replace the first with the word "emoji" and the second with "" to form "emojisad"

Note how the period at the end of the sentence is handled. You said you stripped out punctuation, but this behavior seems like a "feature" not a bug.

library(quanteda, warn.conflicts = FALSE)
#> Package version: 2.1.2
#> Parallel computing: 2 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
#> 

test_string <- "I dislike lemons [sad]."

tokens_ngrams(tokens(gsub("\\[", "emoji", gsub("\\]", "", test_string))))
#> Tokens consisting of 1 document.
#> text1 :
#> [1] "I_dislike"       "dislike_lemons"  "lemons_emojisad" "emojisad_."

^{Created on 2020-12-06 by the reprex package (v0.3.0)}