Inherits from NSObject
Conforms to FMTokenizerDelegate
Declared in FMTokenizers.h

Overview

This tokenizer extends the simple tokenizer with support for a stop word list.

Tasks

Properties

words

@property (atomic, copy) NSSet *words

Class Methods

tokenizerWithFileURL:baseTokenizer:error:

+ (instancetype)tokenizerWithFileURL:(NSURL *)wordFileURL baseTokenizer:(id<FMTokenizerDelegate>)tokenizer error:(NSError **)error
Discussion

Load a stop-word tokenizer using a file containing words delimited by newlines. The file should be encoded in UTF-8.

Declared In

FMTokenizers.h

Instance Methods

initWithWords:baseTokenizer:

- (instancetype)initWithWords:(NSSet *)words baseTokenizer:(id<FMTokenizerDelegate>)tokenizer
Discussion

Initialize an instance of the tokenizer using the set of words. The words should be lowercase if you’re using the FMSimpleTokenizer as the base.

Declared In

FMTokenizers.h