{"id":750,"date":"2019-02-10T13:08:40","date_gmt":"2019-02-10T13:08:40","guid":{"rendered":"https:\/\/www.diggernaut.com\/blog\/?p=750"},"modified":"2020-02-01T07:46:56","modified_gmt":"2020-02-01T07:46:56","slug":"scraping-olx-classified-ads-making-a-one-stop-solution","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/scraping-olx-classified-ads-making-a-one-stop-solution\/","title":{"rendered":"Scraping OLX classified ads: making a one-stop solution."},"content":{"rendered":"

Probably many people know what OLX is. In Russia, the company was absorbed by Avito. However, OLX still exists in many other countries: Ukraine, Poland, Kazakhstan, and many others. A complete list of countries can be found on the main site OLX<\/a>.<\/p>\n

Modified on 01 Feb 2020. Added functionality to bypass cases when proxy is banned<\/strong><\/p>\n

Since all the OLX sites are usually built on the same framework, the web scraper that we code can theoretically work with the website in any country. There may be exceptions of course, but as a rule, everything should work. Therefore, we are going to use OLX Ukraine as a base, and after we have the web scraper ready, we will test it with other sites as well.<\/p>\n

So, we will begin with the catalog. We
\nshould understand how the navigation between the pages of one section works. Find where to get links to pages with a specific ad. Choose a random category: Children\u2019s clothing<\/a>. Open the page in Chrome, and enable the developer tools. Go to the Elements tab, and select a tool to inspect the item on the page (1). Click on the block we are interested in with the first item in the list (2), then, in the HTML code in the \u201cElements\u201d tab, you should be able to see selected HTML node (3).<\/p>\n

\"OLX:<\/a><\/figure>\n

We may be able to use td.offer<\/code> as the CSS selector to select a block, but first, we need to make sure of that. To do it, press CTRL + F, while you are in the HTML code of the \u201cElements\u201d tab. Let\u2019s type our selector in the search bar. If you do everything correctly, then you should see that 44 elements (1) have been found. To check if the selector has not taken something extra, simply use the up and down buttons (2) and see what nodes are selected. If you want to exclude ads in the top (which are promoted for money), you can use the following selector: td.offer: not (.promoted)<\/code>.<\/p>\n

\"OLX:<\/a><\/figure>\n

However, we do not need the block itself, but a link to the ad page. Therefore, let\u2019s open the HTML elements (1) and find the link we need (2). Thus, our selector for links to ads pages will be td.offer a.link.detailsLink<\/code>. We need to check and make sure that there are exactly 44 links. In different versions of OLX, there may be different formatting of blocks with ads, so we can use the a.link.detailsLink<\/code> selector for better compatibility.<\/p>\n

\"OLX:<\/a><\/figure>\n

Let\u2019s check the paginator. Doing the same as we found the elements with the ad, we are going to find the link to the next page in the paginator (3). And we get the selector a[data-cy="page-link-next"]<\/code>. Let\u2019s make sure that there is just single element with such selector on the page.<\/p>\n

\"OLX:<\/a><\/figure>\n

Now we have everything to describe the scraping logic for the specific category. To navigate through the pages of the category we are planning to use link pool<\/a>. This will allow us to use the same code for all pages of the category. Therefore, our scraper looks like:<\/p>\n

---\nconfig:\n    agent: Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/71.0.3578.98 Safari\/537.36\n    debug: 2\ndo:\n# Push start URL to the pool\n- link_add:\n    url: https:\/\/www.olx.ua\/detskiy-mir\/detskaya-odezhda\/dnepr\/\n# Iterate over the pool and load each link\n- walk:\n    to: links\n    do:\n    # Find the link to the next page\n    - find:\n        path: a[data-cy="page-link-next"]\n        do:\n        # Parse link from href attribute\n        - parse:\n            attr: href\n        # Add it to the pool\n        - link_add\n    # Find a link to an ad\n    - find:\n        path: a.link.detailsLink\n        do:\n        # Parse URL from href attribute\n        - parse:\n            attr: href\n        # We are not going to do anything with it for now\n<\/code><\/pre>\n
This code will go through all the pages of the catalog, go into blocks with ads and parse a link to the ad page from there.<\/p>\n
Now we need to describe the logic of data collection from the ad page. To do it, open any ad and find CSS selectors to blocks you want to extract same way as we did for catalog page.<\/p>\n