sonic-pi

https://github.com/sonic-pi-net/sonic-pi

C++

Code. Music. Live.

Rouge::RegexLexer#stream_tokens

This implements the lexer protocol, by yielding [token, value] pairs.

The process for lexing works as follows, until the stream is empty:

1. We look at the state on top of the stack (which by default is
   `[:root]`).
2. Each rule in that state is tried until one is successful.  If one
   is found, that rule's callback is evaluated - which may yield
   tokens and manipulate the state stack.  Otherwise, one character
   is consumed with an `'Error'` token, and we continue at (1.)

Source | Google | Stack overflow

Edit

git clone [email protected]:sonic-pi-net/sonic-pi.git

cd sonic-pi

open app/server/ruby/vendor/rouge/lib/rouge/regex_lexer.rb

Contribute

# Make a new branch

git checkout -b -your-name--update-docs-Rouge--RegexLexer-stream_tokens-for-pr

# Commit to git

git add app/server/ruby/vendor/rouge/lib/rouge/regex_lexer.rbgit commit -m "better docs for Rouge::RegexLexer#stream_tokens"

# Open pull request

gem install hub # on a mac you can `brew install hub`

hub fork

git push <your name> -your-name--update-docs-Rouge--RegexLexer-stream_tokens-for-pr

hub pull-request

# Celebrate!