sonic-pi

https://github.com/sonic-pi-net/sonic-pi

C++

Code. Music. Live.

Parser::Base#tokenize

Currently, token stream format returned by #tokenize is not documented,
but is considered part of a public API and only changed according
to Semantic Versioning.

However, note that the exact token composition of various constructs
might vary. For example, a string `"foo"` is represented equally well
by `:tSTRING_BEG " :tSTRING_CONTENT foo :tSTRING_END "` and
`:tSTRING "foo"`; such details must not be relied upon.

Source | Google | Stack overflow

Edit

git clone [email protected]:sonic-pi-net/sonic-pi.git

cd sonic-pi

open app/server/ruby/vendor/parser-2.2.2.1/lib/parser/base.rb

Contribute

# Make a new branch

git checkout -b -your-name--update-docs-Parser--Base-tokenize-for-pr

# Commit to git

git add app/server/ruby/vendor/parser-2.2.2.1/lib/parser/base.rbgit commit -m "better docs for Parser::Base#tokenize"

# Open pull request

gem install hub # on a mac you can `brew install hub`

hub fork

git push <your name> -your-name--update-docs-Parser--Base-tokenize-for-pr

hub pull-request

# Celebrate!