crawler

A simple website crawler with $ selector-type syntax.

Introduction

Provides an instance of a queue that will iterate concurrently until all items have been processed. Each iteration calls the provided handler with a jQuery like $ selector syntax for ease of accessing any relevant DOM nodes. A helper method trimText is also provided for a common task of selecting and trimming a node's text value. Crawling a website will continue for as long as items are queued.

Installation

npm i --save CentralPing/crawler

API Reference

Example

var crawler = require('crawler');
var queue = crawler.queue(options);
var requests = []; // Array of request objects

// Start crawling
queue.push(requests, function callback(error, response, $body, request) {
  // Do something
});

crawler~queue([options], [options.$]) ⇒ function

Kind: inner method of crawler
Returns: function - A queue managed by async's queue.

Param	Type	Default	Description
[options]	object
[options.userAgent]	string		This value is used to set the User Agent in the request header. Defaults to Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36
[options.maxRedirects]	number	5	The maximum number of redirects to follow.
[options.maxConcurrency]	number	10	An integer for determining how many requests should be run in parallel.
[options.$]	object		A jQuery like DOM selector. Defaults to using cheerio. Any alternative option must be a function that takes the raw body as an argument

crawler~trimText($node) ⇒ string | undefined

Extract and trims text from $node

Kind: inner method of crawler

Param	Type	Description
$node	object	jQuery-like node object

Requests

One or more requests can be pushed onto the queue at a time. Each request must be an object with a req property that contains a valid options object or URL for a request. Any global optons will be replaced by any matching request options (e.g. maxRedirects). The request object will also be passed back to the callback handler as the 4th argument.

// Single requests
queue.push({req: 'http://www.github.com'}, callback);
queue.push({req: {url: 'http://www.cnn.com', maxRedirects: 10}}, callback);

// Multiple requests
queue.push([
  {req: 'http://www.github.com'},
  {req: {url: 'http://www.cnn.com', maxRedirects: 10}}
], callback);

Callback

The callback is executed once per every request. Four arguments are passed to the callback: (error, response, $body, request).

• error: An error object from the request.
• response: The raw response object from the request.
• $body: The DOM of the response HTML.
• request: The original queued request object.

License

Apache 2.0