Rails Web Scraper

Posted on 2/13/2022 by admin

Web Scraper Extension Chrome
Rails Web Scraper Login
Rails Web Scraper Download
Rails Web Scraper
Rails Web Scraper Tutorial
Open Source Web Scraper

What is Web Scraping?

Web scraping is a technique to extract a large amount of data from a website and display it or store it in a file for further use.

Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information from websites. The Mechanize library is used for automating interaction with websites. Web scraping is the act of pulling data directly from a website by parsing the HTML from the web page itself. It refers to retrieving or “scraping” data from a website. Instead of going through the difficult process of physically extracting data, web scraping employs cutting-edge automation to retrieve countless data points from any number.

It is used to crawl and extract the required data from a static website or a JS rendered website.

Few terms to get familiar with:

Nokogiri:
- Uses CSS selectors or XPath for web scraping.
Capybara:
- Allows JS-based interaction with the websites.
Kimurai:
- It is a web scraping framework in ruby.
- Combination of Nokogiri + Capybara.
- Allows scraping data for JS rendered websites and even static HTTP requests.

There are few tools available for web scrapings such as Nokogiri, Capybara and Kimurai. But, Kimurai is the most powerful framework to scrape data.

Kimurai

A web scraping framework in ruby works out of the box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows us to scrape and interact with JavaScript rendered websites.

Features :

Scrape JS based websites.
Supports Headless Chrome, Headless Firefox, PhantomJS or Simple HTTP requests(mechanize) engines.
Capybara methods used to fetch data.
Rich library for built-in helpers to make scraping easy.
Parallel Scrapping – Process web pages concurrently.
Pipelines: To organize and store data in one place for processing all spiders.

You can also scrape data from JS rendered websites, i.e. infinite scrollable websites and even static websites. Amazing right !!!

Read Also: Web scraping using Mechanize in Ruby on Rails

Static Websites:

You can use this framework in 2 ways:

Making a rails app and extract information with the help of models and controllers.
- Create a new rails app.

rails _5.2.3_ new web_scrapping_demo --database=postgresql

- Change the database configurations in app/config/database.yml as per the requirement to run in the development environment.
- Open rails console and create a database for the web app:

rails db:create

- Add gem ‘kimurai’ to Gemfile.
- Install the dependencies using:

bundle install

- Generate a model using the below command with the parent as Kimurai::Base instead of ApplicationRecord:

rails g model Web Scrapper --parent Kimurai::Base

- Perform database migrations for this generated model.

rails db:migrate

- Generate a controller using:

rails g controller WebScrappersController index

- Make a root path for the index action:

root 'web_scrappers#new'

- Add routes for WebScrapper model:

resources: web_scrapper

- Add a link to the index.html.erb file as shown below:

<%= link_to 'Start Scrap', new_web_scrapper_path %>

- Now add an action in the WebScrappersController to perform scraping:

def new
Web Scrapper.crawl!
end

Note: Here, crawl! Performs the full run of the spider. parse method is very important and should be present in every spider. The entry point of any spider is parse.

- Now add some website configurations in the model for which you need to perform scrapping.

Here,

@name = name of the spider/web scraper

@engine = specifies the supported engine

@start_urls = array of start URLs to process one by one inside parse method.

@config = optional, can provide various custom configurations such as user_agent, delay, etc…

Read the Case Study about – Web Scraping RPA (Data Extraction)

Note: You can use several supported engines here, but if we use mechanize no configurations or installations are involved and work for simple HTTP requests but no javascript but if we use other engines such as selenium_chrome, poltergeist_phantomjs, selenium_firefox are all javascript based and rendered in HEADLESS mode.

- Add the parse method to the model for initiating the scrap process.

Here, in the above parse method,

response = Nokogiri::HTML::Document object for the requested website.

URL = String URL of a processed web page.

data = used to pass data between 2 requests.

The data to be fetched from a website is selected using XPath and structures the data as per the requirement.

- Open the terminal and run the application using:

rails s

- Click on the link 'Start Scrap'
  - The results will be saved in the results.json file using save_to helper of the gem.

- Now, check out the stored JSON file, you will get the scraped data.

Hooray !! You have extracted information from the static website.

Making a simple ruby file for extracting the information.

- Open the terminal and install kimurai using the below-mentioned command:

gem install kimurai

- You can refer to the code written for the generated model and make a ruby file using it.
- Run that ruby file using:

ruby filename.rb

Dynamic Websites / JS rendered websites:

Pre-requisites:

Install browsers with web drivers:

For Ubuntu 18.04:

For automatic installation, use the setup command:

$ kimurai setup localhost --local --ask-sudo

Note: It works using Ansible. If not installed, install using:

Web Scraper Extension Chrome

$ sudo apt install ansible

Firstly, install basic tools:

sudo apt install -q -y unzip wget tar openssl
sudo apt install -q -y xvfb

For manual installation, follow the commands for the specific browsers.

You can use this framework in 2 ways: