nika-v / tokenizer-challenge

Tokenizer-Challenge

Usage

./tokenizer_challenge.rb ./sample_text/grapes_of_wrath_28_copies.txt

or

cat ./sample_text/grapes_of_wrath_28_copies.txt | ./tokenizer_challenge.rb

Test File

I've included a test file (grapes_of_wrath_28_copies.txt) in the sample_text directory.