Stars: 146
Forks: 10
Pull Requests: 4
Issues: 3
Watchers: 4
Last Updated: 2023-03-08 09:44:30
The most integrated web scraper package for Laravel.
License: MIT License
Languages: PHP
The most integrated web scraping package for Laravel.
Scavenger provides the following features and more out-the-box.
php artisan scavenger:seek
Install via composer; in your terminal:
composer require reliqarts/laravel-scavenger
or require in composer.json:
{
"require": {
"reliqarts/laravel-scavenger": "^3.1"
}
}
then run composer update
in your terminal to pull it in.
(Optional) Publish package resources and configuration:
php artisan vendor:publish --provider="ReliqArts\Scavenger\ServiceProvider"
You may opt to publish only configuration by using the scavenger-config
tag:
php artisan vendor:publish --provider="ReliqArts\Scavenger\ServiceProvider" --tag="scavenger-config"
or only the migrations via the scavenger-migrations
tag:
php artisan vendor:publish --provider="ReliqArts\Scavenger\ServiceProvider" --tag="scavenger-migrations"
Scavenger is highly configurable. Once configured, the settings will be used for every scrape.
Below is an example of a typical config file structure, with comments explaining each setting.
<?php
return [
// debug mode?
'debug' => false,
// whether log file should be written
'log' => true,
// How much detail is expected in output, 1 being the lowest, 3 being highest.
'verbosity' => 1,
// Set the database config
'database' => [
// Scraps table
'scraps_table' => env('SCAVENGER_SCRAPS_TABLE', 'scavenger_scraps'),
],
// Daemon config - used to build daemon user
'daemon' => [
// Model to use for Daemon identification and login
'model' => 'App\\User',
// Model property to check for daemon ID
'id_prop' => 'email',
// Daemon ID
'id' => '[email protected]',
// Any additional information required to create a user:
// NB. this is only used when creating a daemon user, there is no "safe" way
// to change the daemon's password once he has been created.
'info' => [
'name' => 'Scavenger Daemon',
'password' => 'pass',
],
],
// guzzle settings
'guzzle_settings' => [
'timeout' => 60,
],
// hashing algorithm to use
'hash_algorithm' => 'sha512',
// storage
'storage' => [
// This directory will live inside your application's log directory.
'log_dir' => env('SCAVENGER_LOG_DIR', 'scavenger'),
],
// different model entities and mapping information
'targets' => [
// NB. the "rooms" target shown below is for example purposes only. It has all posible keys explicitly.
'rooms' => [
'example' => true,
'serp' => false,
'model' => 'App\\Room',
'source' => 'http://myroomslistingsite.1demo/section/rooms',
'search' => [
// keywords
'keywords' => ['professional'],
// form markup
'form' => [
// search form selector (important)
'selector' => '#form',
// input element name for search term/keyword
'keyword_input_name' => 'keyword',
'submit_button' => [
// text on submit button (optional)
'text' => null,
// submit element id, use if button doesn't have text (optional)
'id' => null,
],
],
],
'pager' => [
// link (a tag) selector
'selector' => 'div.content #page a.pagingnav',
],
// max. number of pages to scrape (0 is unlimited)
'pages' => 0,
// content markup: actual data to be scraped
'markup' => [
'title' => 'div.content section > table tr h3',
// inside: content to be found upon clicking title link
'__inside' => [
'title' => '#ad-title > h1 > a',
'body' => 'article .adcontent > p[align="LEFT"]:last-of-type',
// focus: focus detail on the following section
'__focus' => 'section section > .content #ad-detail > article',
],
// wrapper/item/result: wrapping selector for each item on single page.
// If inside special key is set this key becomes invalid (i.e. inside takes preference)
'__result' => null,
],
// split single attributes into multiple based on regex
'dissect' => [
'body' => [
'email' => '(([eE]mail)*:*\s*\w+\@(\s*\w)*\.(net|com))',
'phone' => '((([cC]all|[[tT]el|[Pp][Hh](one)*)[:\d\-,\sDL\/]*\d)|(\d{3}\-?\d{4}))',
'beds' => '([\d]+[\d\.\/\s]*[^\w]*([Bb]edroom|b\/r|[Bb]ed)s?)',
'baths' => '([\d]+[\d\.\/\s]*[^\w]*([Bb]athroom|bth|[Bb]ath)s?)',
// retain: whether details should be left in source attribute after extraction
'__retain' => true,
],
],
// modify attributes by calling functions
'preprocess' => [
// takes a callable
// optional third parameter of array if callable method needs an instance
// e.g. ['App\\Item', 'foo', true] or 'bar'
'title' => null,
],
// remap entity attributes to model properties (optional)
'remap' => [
'title' => null,
'body' => null,
],
// scraps containing any of these words will be rejected (optional)
'bad_words' => [
'office',
],
],
// Google SERP example:
'google' => [
'example' => true,
'serp' => true,
'model' => 'App\\GoogleResult',
'source' => 'https://www.google.com',
'search' => [
'keywords' => ['dog'],
'form' => [
'selector' => 'form[name="f"]',
'keyword_input_name' => 'q',
],
],
'pages' => 2,
'pager' => [
'selector' => '#foot > table > tr > td.b:last-child a',
],
'markup' => [
'__result' => 'div.g',
'title' => 'h3 > a',
'description' => '.st',
// the 'link' and 'position' attributes make use of some of Scavengers available properties
'link' => '__link',
'position' => '__position',
],
],
// Bing SERP example:
'bing' => [
'example' => true,
'serp' => true,
'model' => 'App\\BingResult',
'source' => 'https://www.bing.com',
'search' => [
'keywords' => ['dog'],
'form' => [
'selector' => 'form#sb_form',
'keyword_input_name' => 'q',
],
],
'pages' => 3,
'pager' => [
'selector' => '.sb_pagN',
],
'markup' => [
'__result' => '.b_algo',
'title' => 'h2 a',
'description' => '.b_caption p',
'link' => '__link',
'position' => '__position',
],
],
],
];
The targets
array contains a list of entities (to be scraped from) keyed by a unique target identifier. The structure is as follows.
model
: Laravel DB model to create from target.source
: Source URL to scrape.search
: Search settings. Use if a search is to be performed before target data is shown. (optional)
keywords
: Array of keywords to search for.keyword_input
: Keyword input text markup.form_markup
: CSS selector for search form.submit_button_text
: The text on the form's submit button.pager
: Next link CSS selector. To skip to next page.markup
: Array of attributes to scrape from main list. [attributeName => CSS selector]
__inside
: Sub markup for detail page. Markup for page which shows when article title is clicked/opened. (optional)dissect
: Split compound attributes into smaller attributes via REGEX. (optional)preprocess
: Array of attributes which need to be preprocessed. [attributeName => callable]
(optional)remap
: Array of attributes which need to be renamed in order to be saved as target objects. [attributeName => newName]
(optional)bad_words
: Any scrap found containing these words will be discarded. (optional)The following words may appear in context above.
Daemon
: User instance to be used by the Scavenger service.Scrap
: Scraped data before being converted to the target object.Target
: Configured source-model mapping for a single entity.Target Object
: Eloquent model object to be generated from scrape.This library is heavily inspired by, and dependent on, the Guzzle library, although several concepts may have been adjusted.