Scrapinghub

Repositories

• scrapy-autounit

  Automatic unit test generation for Scrapy.
  Python BSD-3-Clause 12 32 11 1 Updated Sep 18, 2020

• scrapy-poet

  Page Object pattern for Scrapy
  Python BSD-3-Clause 10 26 5 4 Updated Sep 18, 2020

• scrapy-autoextract

  Scrapinghub AutoExtract API integration for Scrapy
  Python BSD-3-Clause 7 25 1 1 Updated Sep 18, 2020

• dateparser

  python parser for human readable dates
  Python BSD-3-Clause 313 1,557 136 (3 issues need help) 47 Updated Sep 17, 2020

• splash

  Lightweight, scriptable browser as a service with an HTTP API
  Python 421 3,090 293 23 Updated Sep 16, 2020

• shub

  Scrapinghub Command Line Client
  Python 57 97 33 (6 issues need help) 7 Updated Sep 16, 2020

• asgi_rabbitmq

  Forked from proofit404/asgi_rabbitmq

  RabbitMQ backend for ASGI
  Python 15 0 0 2 Updated Sep 15, 2020

• sample-projects

  Sample projects showcasing Scrapinghub tech
  Python 141 103 5 4 Updated Sep 14, 2020

• shublang

  Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
  Python BSD-3-Clause 3 5 26 (1 issue needs help) 4 Updated Sep 13, 2020

• spidermon

  Scrapy Extension for monitoring spiders execution.

  testing monitoring scraping crawling spiders monitoring-tool scrapinghub
  Python BSD-3-Clause 59 275 31 (1 issue needs help) 14 Updated Sep 12, 2020

• autopager

  Forked from TeamHG-Memex/autopager

  Detect and classify pagination links
  HTML 15 10 0 0 Updated Sep 9, 2020

• extruct

  Extract embedded metadata from HTML markup

  microformats semantic-web rdfa json-ld microdata opengraph
  Python 77 515 18 (3 issues need help) 12 Updated Sep 7, 2020

• shublangjs
  JavaScript 0 0 0 0 Updated Sep 3, 2020

• crawlera-headless-proxy

  A complimentary proxy to help to use Crawlera with headless browsers

  crawler proxy scraping
  Go MIT 15 59 3 (1 issue needs help) 1 Updated Sep 2, 2020

• shub-workflow
  Python BSD-3-Clause 8 6 2 1 Updated Aug 31, 2020

• scrapinghub-autoextract

  Python clients for Scrapinghub AutoExtract API
  Python BSD-3-Clause 4 16 5 1 Updated Aug 25, 2020

• number-parser

  Parse numbers written in natural language
  Python BSD-3-Clause 1 6 11 1 Updated Aug 25, 2020

• crawlera-clients

  Crawlera HTTPS clients collection
  Java 0 2 0 0 Updated Aug 19, 2020

• autoextract-poet

  web-poet definitions for AutoExtract
  Python 0 2 0 1 Updated Aug 18, 2020

• kafka-docker

  Forked from wurstmeister/kafka-docker
  Shell Apache-2.0 2,124 0 0 1 Updated Aug 17, 2020

• js2xml

  Convert Javascript code to an XML document
  Python MIT 16 113 1 1 Updated Aug 13, 2020

• frontera

  A scalable frontier for web crawlers
  Python 204 1,112 79 (10 issues need help) 22 Updated Aug 7, 2020

• scrapinghub-stack-scrapy

  Software stack with latest Scrapy and updated deps
  Dockerfile BSD-3-Clause 8 38 1 0 Updated Aug 6, 2020

• varanus

  A command line spider monitoring tool

  spider monitoring python36
  Python 5 6 2 1 Updated Jul 28, 2020

• python-crfsuite

  A python binding for crfsuite

  data-science crf crfsuite
  Python MIT 191 654 42 3 Updated Jul 22, 2020

• web-poet

  Web scraping Page Objects core library

  python web-scraping page-objects
  Python BSD-3-Clause 3 19 1 1 Updated Jul 18, 2020

• article-extraction-benchmark

  Article extraction benchmark: dataset and evaluation scripts
  Python MIT 6 36 1 0 Updated Jul 16, 2020

• autoextract-spiders

  Pre-built Scrapy spiders for AutoExtract
  Python BSD-3-Clause 5 12 2 3 Updated Jul 8, 2020

• epmdless_dist
  Erlang 0 2 0 0 Updated Jul 3, 2020

• woodpecker

  Forked from laszlocph/woodpecker

  An opinionated fork of the Drone CI system
  Go 19 0 0 0 Updated Jul 1, 2020