{"id":813,"date":"2020-02-01T07:16:36","date_gmt":"2020-02-01T07:16:36","guid":{"rendered":"https:\/\/www.diggernaut.com\/blog\/?p=813"},"modified":"2020-05-03T17:20:09","modified_gmt":"2020-05-03T17:20:09","slug":"learning-how-to-scrape-the-data-from-ebay","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/learning-how-to-scrape-the-data-from-ebay\/","title":{"rendered":"Learning how to scrape the data from eBay"},"content":{"rendered":"

eBay is a very famous and popular marketplace. Very often, it is used by small sellers to sell goods, the same way as on Amazon. Therefore, the data from it can be used to assess a trading niche when entering a market with a new product.<\/p>\n

The eBay site is quite simple, does not use Javascript to display pages, and there should be no technical problems for scraping it. However, you need to know that there is a limit to the number of listings eBay shows per query. Therefore, if you want to collect all the results, you will have to build your queries in such a way that the search query or filtering returns as many listings as would not exceed the limit.<\/p>\n

Let’s say we want to sell e-readers. Let’s try to find the category we need and configure the filters. eBay has the Tablets & eReaders<\/a> category. That is what we need as a starting point. Open this page in Google Chrome<\/p>\n

\"eBay<\/a><\/figure>\n

However, tablets are also shown in this category, and we are only interested in e-readers. To do this, we need to configure the filter for the type of product. Unfortunately, there is no such filter in the main set, so you need to click on the \u201cMore Filters\u201d button, which will open a window with a list of all filters.<\/p>\n

\"eBay<\/a><\/figure>\n

There we need to find the \u201cType\u201d filter, select the \u201ce Reader\u201d option in it and click on the \u201cApply\u201d button.<\/p>\n

\"eBay<\/a><\/figure>\n

We see that there are significantly fewer products, but we are only interested in new devices. Therefore, we need to select the \u201cNew\u201d option in the \u201cCondition\u201d filter.<\/p>\n

\"eBay<\/a><\/figure>\n

We are not interested in auctions either, so we\u2019ll select the \u201cBuy It Now\u201d option above the listings block.<\/p>\n

\"eBay<\/a><\/figure>\n

Now, to save on page requests, increase the number of displayed results on the page. By default, eBay displays 50 listings. We can choose 200, which will save our costs of going through the entire catalog of the filtered category by 4 times. To do this, under the block with the results, you need to select the number of results to show: 200.<\/p>\n

\"eBay<\/a><\/figure>\n

After we set up all the filters and options, we need to copy the URL from the address bar:
\nhttps:\/\/www.ebay.com\/sch\/Tablets-eBook-Readers\/171485\/i.html?_dcat=171485&_fsrp=1&_sacat=171485&rt=nc&Type=eBook%2520Reader&LH_ItemCondition=1000&LH_BIN=1&_ipg=200<\/a><\/p>\n

This will be our starting URL.<\/p>\n

Next, we need to disable JS on the page. We will do this, as usual, using the extension for Google Chrome: Quick Javascript Switcher. Next, open the developer tools in Google Chrome by pressing Ctrl + Shift + I. Then, using the tool to select elements, we will find the blocks we need on the page and CSS selectors for them. Firstly, we are interested in the listing block, and secondly, the link to the next page.<\/p>\n

\"eBay<\/a><\/figure>\n

First, let’s collect all the blocks with listings, that is, define a CSS selector for them.<\/p>\n

\"eBay<\/a><\/figure>\n

CSS selector: ul> li.s-item<\/code>. To check, we will do a search in the \u201cElements\u201d section (by pressing Ctrl + F in it). And make sure that the selector selects all the listings. We will see that more than 200 elements have been selected, although there should be 200 exactly. This happened because eBay, in addition to regular listings, also shows us Sponsored ads.<\/p>\n

\"eBay<\/a><\/figure>\n

Filtering them out will not be easy, but we will try to do it. Click on one of the letters in the word \u201cSPONSORED\u201d and see which element opens in the \u201cElements\u201d window. We will see a set of span elements containing a random set of characters.<\/p>\n

\"eBay<\/a><\/figure>\n

We see that these elements have different classes. It is quite clear that \u201cSPONSORED\u201d is shown on the page due to the fact that one of the classes is shown, and the second is not. But we can\u2019t just take the class we need as a constant. Because, judging by the name of the class, it is not static, but dynamically generated. Therefore, we cannot rely on his name. However, if one of the classes is shown, and the second is not, somewhere on the page there should be CSS that sets this rule. To find it, do a search in the elements by the name of the class and find that the page contains the following code:<\/p>\n

<style type="text\/css">\nspan.s-m1yuhh {\n    display: inline;\n}\nspan.s-o2xlx7k {\n    display: none;\n}\n<\/style>\n<\/code><\/pre>\n
Our task now is to construct a selector for it and check it in the \u201cElements\u201d window, making sure that the selector selects only 1 element on the page.<\/p>\n
style:contains(“display: inline;”)<\/code> \u2013 such a selector will work just fine for our task.<\/p>\n
Now that we have found the desired element, we need to pull out the class that is shown from there. To do this, in the parse<\/strong> command we will use the filter<\/strong> option. Let’s see which regular expression helps highlight the name of the class we need:<\/p>\n
span\\.([^\\s\\{]+)\\s*\\{\\s*display\\:\\s*inline;<\/code><\/p>\n
Using this regular expression, we will extract the class name: s-m1yuhh<\/strong>.
\nNow that we have a class, we can go into the \nspan [role =” text ”]<\/code> element and remove all span<\/strong> elements with a class other than our class (which we will write to the class<\/em> variable beforehand). You can do this with the node_remove<\/strong> command:<\/p>\n
- node_remove: span:not(.<%class%>)\n<\/code><\/pre>\n
Then you need to use the parse<\/strong> command and compare the result in the register with the string \u201cSPONSORED\u201d using the if<\/strong> command.<\/p>\n
This way we can skip commercial listings.<\/p>\n
Now you can look into the listing block and select the fields that we will collect:<\/p>\n
\"eBay<\/a><\/figure>\n
From this page we will collect:<\/p>\n