Skip to content

A Clojure(-script), Google-compliant robots.txt parser

License

Notifications You must be signed in to change notification settings

isker/clj-robots-parser

Repository files navigation

clj-robots-parser Build Status

What

A Clojure(-script) library to parse robots.txt files as specified by The Great Goog themselves. As robots.txt is woefully underspecified in the "official" docs, this library tolerates anything it doesn't understand, extracting the data it does.

It can use the extracted data to query whether a given user-agent is allowed to crawl a given URL.

Why

Why use Google's (much more stringent) documentation for handling robots.txt? In terms of SEO, googlebot is what you ought to care about the most.

About

A Clojure(-script), Google-compliant robots.txt parser

Resources

License

Stars

Watchers

Forks

Packages

No packages published