robotstxt
https://github.com/temoto/robotstxt
Go
The robots.txt exclusion protocol implementation for Go language
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Go not yet supported3 Subscribers
Add a CodeTriage badge to robotstxt
Help out
- Issues
- TestAgent and Test (for the same user-agent) gives different results in case of temporary error when fetching the robots.txt file
- List allow & disallow
- Make the `AllowAll` and `DisallowAll` instances public
- Parse rules for a given user agent
- Reduce memory footprint of multiple logically same RobotsData acquired from different inputs
- Docs
- Go not yet supported