nltk.TreebankWordTokenizer.span_tokenize

TreebankWordTokenizer.span_tokenize(s)

Identify the tokens using integer offsets (start_i, end_i), where s[start_i:end_i] is the corresponding token.

Return type:iter(tuple(int, int))