PhpDev.App
hedii/php-crawler

hedii/php-crawler

Stars: 129

Forks: 65

Pull Requests: 6

Issues: 33

Watchers: 23

Last Updated: 2021-05-20 21:26:39

#php-crawler #crawler #php #laravel #vuejs #vue #webscraping #webscraper #webcrawler

A php crawler that finds emails on the internets

License: MIT License

Languages: PHP, Vue, Shell, Blade

php-crawler

This repository has not been updated since a long time and I am not using it anymore. Please feel free to fork it and do whatever you want with it.

Build Status

A crawler application with a php backend using Laravel, and a js frontend using vuejs, that finds email addresses on the internets.

Given an entry point url, the crawler will search for emails in all the urls available from this entry point domain name. The emails are downloadable as a text file.

Multiple users can start searching for emails without being able see the other users' searches (searches are related to a single user).

Screenshots

Screenshot1

Screenshot2

Screenshot3

Screenshot4

Server requirements

  • PHP >= 7.2.0
  • OpenSSL PHP Extension
  • PDO PHP Extension
  • Mbstring PHP Extension
  • Tokenizer PHP Extension
  • XML PHP Extension

Installation

  • Create a mysql database (default name: crawler)
  • Install the project with composer:
composer create-project hedii/php-crawler crawler
cd crawler
  • Open the .env file, check the database credentials, and modify it if needed:
DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=crawler
DB_USERNAME=root
DB_PASSWORD=your_password_here
  • In the .env file, set the application url:
APP_URL=http://crawler.localhost
  • Build the crawler application
php artisan crawler:build
  • Point your web server's document / web root to be the public directory: /some/path/crawler/public. The index.php in this directory serves as the front controller for all HTTP requests entering your application. See Laravel documentation. I highly recommend using Laravel Valet if you are using a Mac. Otherwise, check Laravel Homestead.
  • Done

Usage

  • Navigate to your php-crawler website
  • Register a new account
  • Create a new search
  • Create more searches
  • Download the found emails

Testing

composer test

Contributing

All contributions are welcome :)

Please write some tests if you are adding or modifying features.

License

php-crawler is open-sourced software licensed under the MIT license.

OPEN ISSUES

See all

RELEASES

See all