Tokenizer of UITextInput, what is it used for?

Question

I'm implementing a custom text input view, which adopts UITextInput protocol, when using UITextView, double tap a word make the word selected, I wonder how UITextInput use its tokenizer to tokenize the string, by now I didn't see any difference with or without assigning a tokenizer for UITextInput by overwrite [UITextInput -tokenizer] method.

Answer 1

It is used for things like keyboard navigation (when you have a hardware keyboard connected). For example, navigating between words while holding Option and using the arrow keys, going to start / end of line.

From Apple's Text Programming Guide for iOS:

Tokenizers are objects that determine whether a text position is within or at the boundary of a text unit with a given granularity. When queried by the text input system, a tokenizer returns ranges of text units with a given granularity or the boundary text position for a text unit with a given granularity. Currently defined granularities are character, word, sentence, paragraph, line, and document; enum constants of the UITextGranularity type represent these granularities. Granularities of text units are always evaluated with reference to a storage or layout direction.

The text input system uses the tokenizer in a variety of ways. For example, the keyboard might require the last sentence’s worth of context to figure out what the user is trying to type. Or, if the user is pressing the Option-left arrow key (on an external keyboard), the text system queries the tokenizer to find the information it needs to move to the previous word.

More: https://developer.apple.com/library/ios/documentation/StringsTextFonts/Conceptual/TextAndWebiPhoneOS/LowerLevelText-HandlingTechnologies/LowerLevelText-HandlingTechnologies.html