robotstxt (0.6.2)

0 users

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker.

https://github.com/ropensci/robotstxt
http://cran.r-project.org/web/packages/robotstxt

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Maintainer: Peter Meissner
Author(s): Peter Meissner [aut, cre], Oliver Keys [ctb], Rich Fitz John [ctb]

License: MIT + file LICENSE

Uses: future, future.apply, httr, magrittr, spiderbar, stringr, testthat, knitr, dplyr, rmarkdown, covr
Reverse suggests: rzeit2, spiderbar

Released 5 months ago.


6 previous versions

Ratings

Overall:

  (0 votes)

Documentation:

  (0 votes)

Log in to vote.

Reviews

No one has written a review of robotstxt yet. Want to be the first? Write one now.


Related packages: Rserve, XML, httpRequest, rjson, RCurl, OAIHarvester, RgoogleMaps, sendmailR, twitteR, scrapeR, RJSONIO, imguR, googleVis, plotGoogleMaps, factualR, ROAuth, Rook, osmar, whisker, FastRWeb(20 best matches, based on common tags.)


Search for robotstxt on google, google scholar, r-help, r-devel.

Visit robotstxt on R Graphical Manual.