{"id":405,"date":"2018-02-12T09:39:55","date_gmt":"2018-02-12T09:39:55","guid":{"rendered":"https:\/\/www.diggernaut.com\/blog\/?p=405"},"modified":"2020-02-06T06:07:50","modified_gmt":"2020-02-06T06:07:50","slug":"how-to-scrape-pages-infinite-scroll-extracting-data-from-instagram","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/how-to-scrape-pages-infinite-scroll-extracting-data-from-instagram\/","title":{"rendered":"How to scrape pages with infinite scroll: extracting data with Instagram scraper"},"content":{"rendered":"

In this article, we will teach you how generally you can scrape the data from the websites with infinite scroll and build the Instagram scraper. All you will need to do it is just your browser, a free account on the Diggernaut platform, and your head and hands.<\/p>\n

Updated on 01.10.2020<\/strong><\/p>\n

Infinite scroll on the webpage is based on Javascript functionality. Therefore, to find out what URL we need to access and what parameters to use, we need to either thoroughly study the JS code that works on the page or, and preferably, examine the requests that the browser does when you scroll down the page. We can study requests using the Developer Tools, which are built-in to all modern browsers. In this article, we are going to use Google Chrome, but you can use any other browser. Just keep in mind that the developer tools may look different in different browsers.<\/p>\n

We will use an official Instagram<\/a> channel. Open this page in the browser, and run Chrome Dev Tools \u2013 developer tools that are built-in to Google Chrome. To do it, you need to right-click anywhere on the page and select the \u201cInspect\u201d option or press \u201cCtrl + Shift + I\u201d:<\/p>\n

\"Scraping<\/a><\/figure>\n

It will open the tool window, where we go to the Network tab, and in the filters, we select only XHR requests. We do it to filter out requests we don\u2019t need. After that, reload the page in the browser using the Reload button in the browser interface or the \u201cF5\u201d key on the keyboard.<\/p>\n

\"Scraping<\/a><\/figure>\n

Let\u2019s now scroll down the page several times with the mouse wheel. It will cause content loading. Whenever we scroll down to the bottom of the page, JS makes an XHR request to the server, receive the data and add it to the page. As a result, we should have several requests in the list that look almost the same. Most likely its what we are looking for.<\/p>\n

\"Scraping<\/a><\/figure>\n

To make sure, we have to click on one of the requests and in the newly opened panel go to the Preview tab. There we can see the formatted content that the server returns to the browser for this request. Let\u2019s get to one of the leaf elements in the tree and make sure that there are data about the images that we have on the page.<\/p>\n

\"Scraping<\/a><\/figure>\n

After making sure that these are the queries we need, let\u2019s look at one of them more carefully. To do it, go to the Headers tab. There, we can find information about what URL is used to make the request, what is the type of that request (POST or GET), and what parameters are passed with the request.<\/p>\n

\"Scraping<\/a><\/figure>\n

It\u2019s better to check query string parameters at Query String Parameters section. To see it, you need to scroll down the pane to the bottom:<\/p>\n

\"Scraping<\/a><\/figure>\n

As result of our analysis we get the following:<\/p>\n

Request URL:<\/strong> https:\/\/www.instagram.com\/graphql\/query\/
\nRequest type:<\/strong> GET
\nQuery string parameters:<\/strong> query_hash and variables<\/p>\n

Obviously, some static id is passed as query_hash, which is generated by JS or exist either on the page, cookie or some JS file. There are also some parameters, which defines what exactly you get from the server are passed in the JSON format as the variables query parameter.<\/p>\n

Now we need to understand where query_hash comes from. If we go to the Elements tab and try to find (CTRL + F) our query_hash e769aa130647d2354c40ea6a439bfc08<\/strong>, then we find out that it doesn\u2019t exist on the page itself, which means that it is loaded or generated somewhere in the Javascript code, or comes with cookies. Therefore, go back to the Network tab and put the filter on JS. Thus, we can see only requests for JS files. Sequentially browsing the request by request, we have to search for our id in the loaded JS files: just click on the request, then open the Response tab in the opened panel to see the content of JS and do a search for our id (CTRL + F). After several unsuccessful attempts, we find that our id is in the following JS file:<\/p>\n

https:\/\/www.instagram.com\/static\/bundles\/ProfilePageContainer.js\/031ac4860b53.js<\/p>\n

and the code fragment that surrounds the id looks like this:<\/p>\n

profilePosts.byUserId.get(n))||void 0===s?void 0:s.pagination},queryId:"e769aa130647d2354c40ea6a439bfc08",queryParams\n<\/code><\/pre>\n
So, to get query_hash, we need to:<\/p>\n
    \n
  1. Load main channel page<\/li>\n
  2. Find the URL to the file which filename contains ProfilePageContainer.js<\/em><\/li>\n
  3. Extract this URL<\/li>\n
  4. Load JS file<\/li>\n
  5. Parse the id we need<\/li>\n
  6. Write it into a variable for later use.<\/li>\n<\/ol>\n
    Now let\u2019s see what data is passed as variables parameter:<\/p>\n
    {"id":"25025320","first":12,"after":"AQAzEauY26BEUyDxOz9NhBP2gjLbTTD3OD1ajDxZIHvldwFwboiBnIcglaL6Kb_yDssRABBoUDdIls5V8unGC86hC2qk_IeLFUcH2QPTrY3f4A"}\n<\/code><\/pre>\n
    If we analyze all XHR requests that load data, we find that only the after<\/em> parameter changes. Therefore id<\/em> most likely is the id of the channel, which we need to extract from somewhere, first<\/em> \u2013 the number of records that the server should return, and after<\/em> is the id of the last record shown.<\/p>\n
    We need to find where we can extract the channel id. So the first thing we do is look for the text 25025320<\/strong> in the source code of the main channel page. Let\u2019s go to the Elements tab and do a search (CTRL + F) for our id. We  find that it exists in the JSON structure on the page itself, and we can easily extract it:<\/p>\n
    \"Scraping<\/a><\/figure>\n
    It seems everything is clear, but where do we get after<\/em> value for each subsequent data loading? It is straightforward. Since it gets changed with each new loading, it\u2019s most likely loaded with the data feed. Let\u2019s look at the loaded data again more carefully:<\/p>\n
    We will see the following data structure there:<\/p>\n
    data: {\n    user: {\n        edge_owner_to_timeline_media: {\n            count: 5014,\n            page_info: {\n                has_next_page: true,\n                end_cursor: "AQCCoEpYvQtj0-NgbaQUg9g4ffOJf8drV2RieFJw1RA3E9lDoc8euxXjeuwlUEtXB6CRS9Zs2ZGJcNKseKF9f6b0cN0VC3ck8rnTfOw5q8nlJw"\n            }\n        }\n    }\n}\n<\/code><\/pre>\n
    where end_cursor<\/em> looks like what we are looking for. Also, there is field has_next_page<\/em> which can be very handy for us so we could stop loading feeds with data if there is no more data available.<\/p>\n
    Now we\u2019ll write the beginning part of our Instagram scraper, load the main channel page and try to load the JS file with query_hash. Create a digger in your Diggernaut account and add the following configuration to it:<\/p>\n
    ---\nconfig:\n    agent: Firefox\n    debug: 2\ndo:\n# Load main channel page\n- walk:\n    to: https:\/\/www.instagram.com\/instagram\/\n    do:\n    # Find all elements that loads Javascript files\n    - find:\n        path: script[type="text\/javascript"]\n        do:\n        # Parse value in the src attribute\n        - parse:\n            attr: src\n        # Check if filename contains ProfilePageContainer.js string\n        - if:\n            match: ProfilePageContainer\\.js\n            do:\n            # If check is true, load JS file\n            - walk:\n                to: value\n                do:<\/code><\/pre>\n
    Set the Digger to the Debug mode. Now we need to run our Instagram scraper, and when the job is done, we are going to check the log. At the end of the log, we can see how the diggernaut works with JS files. It converts them into the following structure:<\/p>\n
    \n  <\/head>\n    \n      \n