picky
https://github.com/floere/picky
HTML
Picky is an easy to use and fast Ruby semantic search engine that helps your users find what they are looking for.
Picky::Tokenizer#pretokenize
Pretokenizing. Does: * Split the text into words. * Cap the amount of tokens if max_words is set.
Edit
git clone [email protected]:floere/picky.git
cd picky
open server/lib/picky/tokenizer.rb
Contribute
# Make a new branchgit checkout -b -your-name--update-docs-Picky--Tokenizer-pretokenize-for-pr
# Commit to gitgit add server/lib/picky/tokenizer.rbgit commit -m "better docs for Picky::Tokenizer#pretokenize"
# Open pull requestgem install hub # on a mac you can `brew install hub`
hub fork
git push <your name> -your-name--update-docs-Picky--Tokenizer-pretokenize-for-pr
hub pull-request
# Celebrate!