Coding with python: Scrapy

Showing posts with label Scrapy. Show all posts

Sunday, 3 October 2021

How to extract javascript dictionaries scrapy

Why would you want to do that?

Well, if you are web scraping using Python, and Scrapy for instance, you may need to extract reviews, or comments that are loaded from JavaScript. This would mean you could not use your css or xpath selectors like you can with regular html.

Parse

Instead, in your browser, check if you may be able to parse the code, beginning with ctrl + f, and “json” and track down some JSON in the form of a python dictionary. You ‘just’ need to isolate it.

web-scraping javascript pages — view-source to find occurrences of “JSON” in your page

The response is not nice, but you can gradually shrink it down, in Scrapy shell or python shell…

scrapy-shell-response — Figure 1 – The response

Split, strip, replace

From within Scrapy, or your own Python code you can split, strip, and replace, with the built-in python commands until you have just a dictionary that you can use with json.loads.

x = response.text.split('JSON.parse')[3].replace("\u0022","\"").replace("\u2019m","'").lstrip("(").split(" ")[0].strip().replace("\"","",1).replace("\");","")

Master replace, strip , and split and you won’t need regular expressions!

With the response.text now ready as a JSON friendly dictionary you can do this:

import json
q = json.loads(x)
comment = (q[‘doctor’][‘sample_rating_comment’])
comment.replace(“\u2019″,”‘”)
print(comment)

The key thing to remember to use when parsing the response text is to use the index, to pick out the section you want, and then make use of “\” backslash to escaped characters when you are working with quotes, and actual backslashes in the text you’re parsing.

parsed-response — Figure 2 – The parsed response

Conclusion

Rendering to HTML using Splash, or Selenium, or using regular expressions are not always essential. Hope this helps illustrate how you can extract values FROM a python dictionary FROM json FROM javascript !

You may see a mass of text on your screen to begin with, but persevere and you can arrive at the dictionary contained within…

Monday, 6 August 2018

Tutorial HANDLING JAVASCRIPT IN SCRAPY WITH SPLASH

A common roadblock when developing spiders is dealing with sites that use a heavy amount of JavaScript. Many modern websites run entirely on JavaScript, and require scripts to be run in order for the page to render properly. In many cases, pages also present modals and other dialogues that need to be interacted with to show the full page. In this post we’re going to show you how you can use Splash to handle JavaScript in your Scrapy projects.

What is Splash?

Splash is our in-house solution for JavaScript rendering, implemented in Python using Twisted and QT. Splash is a lightweight web browser which is capable of processing multiple pages in parallel, executing custom JavaScript in the page context, and much more. Best of all, it’s open source!

Setting Up Splash

The easiest way to set up Splash is through Docker:

$ docker pull scrapinghub/splash
$ docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash

Splash will now be running on localhost:8050. If you’re using a Docker Machine on OS X or Windows, it will be running on the IP address of Docker’s virtual machine.

If you would like to install Splash without using Docker, please refer to the documentation.

Using Splash with Scrapy

Now that Splash is running, you can test it in your browser:

http://localhost:8050/

On the right enter a URL (e.g. http://amazon.com) and click 'Render me!'. Splash will display a screenshot of the page as well as charts and a list of requests with their timings. At the bottom you should see a text box containing the rendered HTML.

Manually

You can use Request to send links to Splash:

req_url = "http://localhost:8050/render.json"
body = json.dumps({
    "url": url,
    "har": 1,
    "html": 0,
})
headers = Headers({'Content-Type': 'application/json'})
yield scrapy.Request(req_url, self.parse_link, method='POST',
                                 body=body, headers=headers)

If you’re using CrawlSpider, the easiest way is to override the process_links function in your spider to replace links with their Splash equivalents:

def process_links(self, links):
    for link in links:
        link.url = "http://localhost:8050/render.html?" + urlencode({ 'url' : link.url })
    return links

Scrapy-Splash (recommended)

The preferred way to integrate Splash with Scrapy is using scrapy-splash. See here for why it's recommended you use the middleware instead of using it manually. You can install scrapy-splash using pip:

pip install scrapy-splash

To use scrapy-splash in your project, you first need to enable the middleware:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

The middleware needs to take precedence over HttpProxyMiddleware, which by default is at position 750, so we set the middleware positions to numbers below 750.

You then need to set the SPLASH_URL setting in your project's settings.py:

SPLASH_URL = 'http://localhost:8050/'

Don’t forget, if you’re using a Docker Machine on OS X or Windows, you will need to set this to the IP address of Docker’s virtual machine, e.g.:

SPLASH_URL = 'http://192.168.59.103:8050/'

Enable SplashDeduplicateArgsMiddleware to support cache_args feature: it allows to save disk space by not storing duplicate Splash arguments multiple times in a disk request queue. If Splash 2.1+ is used the middleware also allows to save network traffic by not sending these duplicate arguments to Splash server multiple times.

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

Scrapy currently doesn’t provide a way to override request fingerprints calculation globally, so you will also have to set a custom DUPEFILTER_CLASS and a custom cache storage backend:

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

If you already use another cache storage backend, you will need to subclass it and replace all calls to scrapy.util.request.request_fingerprint with scrapy_splash.splash_request_fingerprint.

Now that the Splash middleware is enabled, you can use SplashRequest in place of scrapy.Request to render pages with Splash.

For example, if we wanted to retrieve the rendered HTML for a page, we could do something like this:

import scrapy
from scrapy_splash import SplashRequest

class MySpider(scrapy.Spider):
    start_urls = ["http://example.com", "http://example.com/foo"]

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse,
                endpoint='render.html',
                args={'wait': 0.5},
            )

    def parse(self, response):
        # response.body is a result of render.html call; it
        # contains HTML processed by a browser.
        # …

The ‘args’ dict contains arguments to send to Splash. You can find a full list of available arguments in the HTTP API documentation. By default the endpoint is set to ‘render.json’, but here we have overridden it and set it to ‘render.html’ to provide an HTML response.

Running Custom JavaScript

Sometimes you may need to press a button or close a modal to view the page properly. Splash lets you run your own JavaScript code within the context of the web page you’re requesting. There are several ways you can accomplish this:

Using the js_source Parameter

You can use the js_source parameter to send the JavaScript you want to execute. The JavaScript code is executed after the page finished loading but before the page is rendered. This allow to use the JavaScript code to modify the page being rendered. For example, you can do it with Scrapy-Splash:

# Render page and modify its title dynamically
yield SplashRequest(
    'http://example.com',
    endpoint='render.html',
    args={'js_source': 'document.title="My Title";'},
)

Splash Scripts

Splash supports Lua scripts through its execute endpoint. This is the preferred way to execute JavaScript as you can preload libraries, choose when to execute the JavaScript, and retrieve the output.

Here’s an example script:

function main(splash)
    assert(splash:go(splash.args.url))
    splash:wait(0.5)
    local title = splash:evaljs("document.title")
    return {title=title}
end

You need to send that script to the execute endpoint, in the lua_source argument.
This will return a JSON object containing the title:

{
    "title": "Some title"
}

Every script requires a main function to act as the entry point. You can return a Lua table which will be rendered as JSON, which is what we have done here. We use the splash:gofunction to tell Splash to visit the URL. The splash:evaljs function lets you execute JavaScript within the page context, however, if you don't need the result you should use splash:runjsinstead.

You can test your Splash scripts in your browser by visiting your Splash instance’s index page (e.g. http://localhost:8050/). It’s also possible to use Splash with IPython notebook as an interactive web-based development environment, see here for more details.

It’s often the case that you need to click a button before the page is displayed. We can do that using splash:mouse_click function:

function main(splash)
    assert(splash:go(splash.args.url))
    local get_dimensions = splash:jsfunc([[
        function () {
            var rect = document.getElementById('button').getClientRects()[0];
            return {"x": rect.left, "y": rect.top}
        }
    ]])
    splash:set_viewport_full()
    splash:wait(0.1)
    local dimensions = get_dimensions()
    splash:mouse_click(dimensions.x, dimensions.y)
    -- Wait split second to allow event to propagate.
    splash:wait(0.1)
    return splash:html()
end

Here we use splash:jsfunc to define a function that will return the element coordinates, then make sure the element is visible with splash:set_viewport_full, and click on the element. Splash then returns the rendered HTML.

You can find more info on running JavaScript with Splash in the docs, and for a more in-depth tutorial, check out the Splash Scripts Tutorial.

We hope this tutorial gave you a nice introduction to Splash, and please let us know if you have any questions or comments!

Wednesday, 4 July 2018

Python | How To Crawl A Web Page with Scrapy and Python 3

Introduction

Web scraping, often called web crawling or web spidering, or “programatically going over a collection of web pages and extracting data,” is a powerful tool for working with data on the web.

With a web scraper, you can mine data about a set of products, get a large corpus of text or quantitative data to play around with, get data from a site without an official API, or just satisfy your own personal curiosity.

In this tutorial, you’ll learn about the fundamentals of the scraping and spidering process as you explore a playful data set. We'll use BrickSet, a community-run site that contains information about LEGO sets. By the end of this tutorial, you’ll have a fully functional Python web scraper that walks through a series of pages on Brickset and extracts data about LEGO sets from each page, displaying the data to your screen.

The scraper will be easily expandable so you can tinker around with it and use it as a foundation for your own projects scraping data from the web.

Prerequisites

To complete this tutorial, you'll need a local development environment for Python 3. You can follow How To Install and Set Up a Local Programming Environmentfor Python 3 to configure everything you need.

Step 1 — Creating a Basic Scraper

Scraping is a two step process:

1. You systematically find and download web pages.

2. You take those web pages and extract information from them.

Both of those steps can be implemented in a number of ways in many languages.

You can build a scraper from scratch using modules or libraries provided by your programming language, but then you have to deal with some potential headaches as your scraper grows more complex. For example, you'll need to handle concurrency so you can crawl more than one page at a time. You'll probably want to figure out how to transform your scraped data into different formats like CSV, XML, or JSON. And you'll sometimes have to deal with sites that require specific settings and access patterns.

You'll have better luck if you build your scraper on top of an existing library that handles those issues for you. For this tutorial, we're going to use Python and Scrapy to build our scraper.

Scrapy is one of the most popular and powerful Python scraping libraries; it takes a "batteries included" approach to scraping, meaning that it handles a lot of the common functionality that all scrapers need so developers don't have to reinvent the wheel each time. It makes scraping a quick and fun process!

Scrapy, like most Python packages, is on PyPI (also known as pip). PyPI, the Python Package Index, is a community-owned repository of all published Python software.

If you have a Python installation like the one outlined in the prerequisite for this tutorial, you already have pip installed on your machine, so you can install Scrapy with the following command:

· pip install scrapy

If you run into any issues with the installation, or you want to install Scrapy without using pip, check out the official installation docs.

With Scrapy installed, let's create a new folder for our project. You can do this in the terminal by running:

· mkdir brickset-scraper

Now, navigate into the new directory you just created:

· cd brickset-scraper

Then create a new Python file for our scraper called scraper.py. We'll place all of our code in this file for this tutorial. You can create this file in the terminal with the touch command, like this:

· touch scraper.py

Or you can create the file using your text editor or graphical file manager.

We'll start by making a very basic scraper that uses Scrapy as its foundation. To do that, we'll create a Python class that subclasses scrapy.Spider, a basic spider class provided by Scrapy. This class will have two required attributes:

name — just a name for the spider.
start_urls — a list of URLs that you start to crawl from. We'll start with one URL.

Open the scrapy.py file in your text editor and add this code to create the basic spider:

scraper.py

import scrapy

class BrickSetSpider(scrapy.Spider):

name = "brickset_spider"

start_urls = ['http://brickset.com/sets/year-2016']

Let's break this down line by line:

First, we import scrapy so that we can use the classes that the package provides.

Next, we take the Spider class provided by Scrapy and make a subclass out of it called BrickSetSpider. Think of a subclass as a more specialized form of its parent class. The Spider subclass has methods and behaviors that define how to follow URLs and extract data from the pages it finds, but it doesn't know where to look or what data to look for. By subclassing it, we can give it that information.

Then we give the spider the name brickset_spider.

Finally, we give our scraper a single URL to start from: http://brickset.com/sets/year-2016. If you open that URL in your browser, it will take you to a search results page, showing the first of many pages containing LEGO sets.

Now let's test out the scraper. You typically run Python files by running a command like python path/to/file.py. However, Scrapy comes with its own command line interface to streamline the process of starting a scraper. Start your scraper with the following command:

· scrapy runspider scraper.py

You’ll see something like this:

Output

2016-09-22 23:37:45 [scrapy] INFO: Scrapy 1.1.2 started (bot: scrapybot)

2016-09-22 23:37:45 [scrapy] INFO: Overridden settings: {}

2016-09-22 23:37:45 [scrapy] INFO: Enabled extensions:

['scrapy.extensions.logstats.LogStats',

'scrapy.extensions.telnet.TelnetConsole',

'scrapy.extensions.corestats.CoreStats']

2016-09-22 23:37:45 [scrapy] INFO: Enabled downloader middlewares:

['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',

...

'scrapy.downloadermiddlewares.stats.DownloaderStats']

2016-09-22 23:37:45 [scrapy] INFO: Enabled spider middlewares:

['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',

...

'scrapy.spidermiddlewares.depth.DepthMiddleware']

2016-09-22 23:37:45 [scrapy] INFO: Enabled item pipelines:

[]

2016-09-22 23:37:45 [scrapy] INFO: Spider opened

2016-09-22 23:37:45 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2016-09-22 23:37:45 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023

2016-09-22 23:37:47 [scrapy] DEBUG: Crawled (200) <GET http://brickset.com/sets/year-2016> (referer: None)

2016-09-22 23:37:47 [scrapy] INFO: Closing spider (finished)

2016-09-22 23:37:47 [scrapy] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 224,

'downloader/request_count': 1,

...

'scheduler/enqueued/memory': 1,

'start_time': datetime.datetime(2016, 9, 23, 6, 37, 45, 995167)}

2016-09-22 23:37:47 [scrapy] INFO: Spider closed (finished)

That's a lot of output, so let's break it down.

The scraper initialized and loaded additional components and extensions it needed to handle reading data from URLs.
It used the URL we provided in the start_urls list and grabbed the HTML, just like your web browser would do.
It passed that HTML to the parse method, which doesn't do anything by default. Since we never wrote our own parse method, the spider just finishes without doing any work.

Now let's pull some data from the page.

Step 2 — Extracting Data from a Page

We've created a very basic program that pulls down a page, but it doesn't do any scraping or spidering yet. Let’s give it some data to extract.

If you look at the page we want to scrape, you'll see it has the following structure:

There's a header that’s present on every page.
There's some top-level search data, including the number of matches, what we’re searching for, and the breadcrumbs for the site.
Then there are the sets themselves, displayed in what looks like a table or ordered list. Each set has a similar format.

When writing a scraper, it's a good idea to look at the source of the HTML file and familiarize yourself with the structure. So here it is, with some things removed for readability:

brickset.com/sets/year-2016

<body>

<a class="highslide plain mainimg" href=

"http://images.brickset.com/sets/images/10251-1.jpg?201510121127"

onclick="return hs.expand(this)"><img src=

"http://images.brickset.com/sets/small/10251-1.jpg?201510121127"

title="10251-1: Brick Bank"></a>

<h1><a href='/sets/10251-1/Brick-Bank'>Brick Bank</a></h1>

<a href='/sets/10251-1/Brick-Bank'>10251-1</a> <a href=

'/sets/theme-Advanced-Models'>Advanced Models</a> <a class=

'subtheme' href=

'/sets/theme-Advanced-Models/subtheme-Modular-Buildings'>Modular

Buildings</a> <a class='year' href=

'/sets/theme-Advanced-Models/year-2016'>2016</a>

</div>

</div>

<a href="#" onclick="return hs.previous(this)" title=

"Previous (left arrow key)">« Previous</a> <a href="#"

onclick="return hs.next(this)" title=

"Next (right arrow key)">Next »</a>

</div>

...

</article>

...

</article>

</section>

</body>

Scraping this page is a two step process:

1. First, grab each LEGO set by looking for the parts of the page that have the data we want.

2. Then, for each set, grab the data we want from it by pulling the data out of the HTML tags.

scrapy grabs data based on selectors that you provide. Selectors are patterns we can use to find one or more elements on a page so we can then work with the data within the element. scrapy supports either CSS selectors or XPath selectors.

We’ll use CSS selectors for now since CSS is the easier option and a perfect fit for finding all the sets on the page. If you look at the HTML for the page, you'll see that each set is specified with the class set. Since we're looking for a class, we'd use .set for our CSS selector. All we have to do is pass that selector into the response object, like this:

scraper.py

class BrickSetSpider(scrapy.Spider):

name = "brickset_spider"

start_urls = ['http://brickset.com/sets/year-2016']

def parse(self, response):

SET_SELECTOR = '.set'

for brickset in response.css(SET_SELECTOR):

pass

This code grabs all the sets on the page and loops over them to extract the data. Now let’s extract the data from those sets so we can display it.

Another look at the [source](view-source:brickset.com/sets/year-2016) of the page we're parsing tells us that the name of each set is stored within an a tag inside an h1 tag for each set:

brickset.com/sets/year-2016

<h1><a href='/sets/10251-1/Brick-Bank'>Brick Bank</a></h1>

The brickset object we’re looping over has its own css method, so we can pass in a selector to locate child elements. Modify your code as follows to locate the name of the set and display it:

scraper.py

class BrickSetSpider(scrapy.Spider):

name = "brickset_spider"

start_urls = ['http://brickset.com/sets/year-2016']

def parse(self, response):

SET_SELECTOR = '.set'

for brickset in response.css(SET_SELECTOR):

NAME_SELECTOR = 'h1 a ::text'

yield {

'name': brickset.css(NAME_SELECTOR).extract_first(),

}

Note: The trailing comma after extract_first() isn't a typo. We're going to add more to this section soon, so we've left the comma there to make adding to this section easier later.

You’ll notice two things going on in this code:

We append ::text to our selector for the name. That’s a CSS pseudo-selector that fetches the text inside of the a tag rather than the tag itself.
We call extract_first() on the object returned by brickset.css(NAME_SELECTOR) because we just want the first element that matches the selector. This gives us a string, rather than a list of elements.

Save the file and run the scraper again:

· scrapy runspider scraper.py

This time you'll see the names of the sets appear in the output:

Output

...

[scrapy] DEBUG: Scraped from <200 http://brickset.com/sets/year-2016>

{'name': 'Brick Bank'}

[scrapy] DEBUG: Scraped from <200 http://brickset.com/sets/year-2016>

{'name': 'Volkswagen Beetle'}

[scrapy] DEBUG: Scraped from <200 http://brickset.com/sets/year-2016>

{'name': 'Big Ben'}

[scrapy] DEBUG: Scraped from <200 http://brickset.com/sets/year-2016>

{'name': 'Winter Holiday Train'}

...

Let's keep expanding on this by adding new selectors for images, pieces, and miniature figures, or minifigsthat come with a set.

Take another look at the HTML for a specific set:

brickset.com/sets/year-2016

...

<h1><a href="/sets/10251-1/Brick-Bank"><span>10251:</span> Brick Bank</a> </h1>

...

<dl>

<dt>Pieces</dt>

<dt>Minifigs</dt>

...

</dl>

</div>

...

</div>

</article>

We can see a few things by examining this code:

The image for the set is stored in the src attribute of an img tag inside an a tag at the start of the set. We can use another CSS selector to fetch this value just like we did when we grabbed the name of each set.
Getting the number of pieces is a little trickier. There's a dt tag that contains the text Pieces, and then a dd tag that follows it which contains the actual number of pieces. We'll use XPath, a query language for traversing XML, to grab this, because it's too complex to be represented using CSS selectors.
Getting the number of minifigs in a set is similar to getting the number of pieces. There's a dt tag that contains the text Minifigs, followed by a dd tag right after that with the number.

So, let's modify the scraper to get this new information:

scraper.py

class BrickSetSpider(scrapy.Spider):

name = 'brick_spider'

start_urls = ['http://brickset.com/sets/year-2016']

def parse(self, response):

SET_SELECTOR = '.set'

for brickset in response.css(SET_SELECTOR):

NAME_SELECTOR = 'h1 a ::text'

PIECES_SELECTOR = './/dl[dt/text() = "Pieces"]/dd/a/text()'

MINIFIGS_SELECTOR = './/dl[dt/text() = "Minifigs"]/dd[2]/a/text()'

IMAGE_SELECTOR = 'img ::attr(src)'

yield {

'name': brickset.css(NAME_SELECTOR).extract_first(),

'pieces': brickset.xpath(PIECES_SELECTOR).extract_first(),

'minifigs': brickset.xpath(MINIFIGS_SELECTOR).extract_first(),

'image': brickset.css(IMAGE_SELECTOR).extract_first(),

}

Save your changes and run the scraper again:

· scrapy runspider scraper.py

Now you’ll see that new data in the program's output:

Output