{"id":581,"date":"2018-12-26T19:40:07","date_gmt":"2018-12-26T19:40:07","guid":{"rendered":"https:\/\/www.diggernaut.com\/blog\/?p=581"},"modified":"2019-01-16T19:11:18","modified_gmt":"2019-01-16T19:11:18","slug":"solving-google-recaptcha-v3-2captcha-has-been-integrated-to-the-diggernaut-platform","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/solving-google-recaptcha-v3-2captcha-has-been-integrated-to-the-diggernaut-platform\/","title":{"rendered":"Solving Google ReCaptcha v3: 2Captcha has been integrated to the Diggernaut platform"},"content":{"rendered":"

There is a new arrival in the ranks of services for solving captcha supported by us. Meet the 2Captcha<\/a>.<\/p>\n

From the functional side, 2Captcha looks similar to the AntiCaptcha. It supports all popular kinds of captcha from the box, has \u201cproxyless\u201d mode, which is in demand by our users. Prices are on the same level as AntiCaptcha has. However, there is one crucial thing that differs 2Captcha from other captcha solving services. The offer solution for the Google ReCaptcha v3, which is a new kind of Google captcha and already used on some popular websites. We tested it, and it works indeed!<\/p>\n

More details about usage of 3rd party providers for captcha resolving you can find in our documentation<\/a>.<\/p>\n

Below we describe how to use 2Captcha to solve Google ReCaptcha v3 using the example of the Russian traffic police website. And we will request information about registrations in the traffic police by VIN.<\/p>\n

For a start, let\u2019s go to [russian traffic police site](https: \/\/xn--90adear.xn--p1ai\/check\/auto\/) and make sure that there is ReCaptcha v3 used on the page with the search form. At first glance, the link to api.js from Google is not observed in the source code of the page, so automatic extraction of the site key and the action parameter will not work, and we need to find them manually. If we look at the Network tab in the developer console of your browser (Chrome \/ Firefox), then we will see that api.js is loaded, which means that the browser doesn\u2019t load it directly from the page source, but use some javascript. Therefore, our task will be to check all loaded JS files on the Network tab.<\/p>\n

\"Looking<\/a><\/figure>\n

Looking through the code of JS files, we will see that app.js<\/a> file has a fragment of the code that looks like something we are seeking.<\/p>\n

\/* reCaptcha *\/\nvar reCaptchaSiteKey = '6Lc66nwUAAAAANZvAnT-OK4f4D_xkdzw5MLtAYFL';\nvar onloadReCaptcha = function() {\n    grecaptcha.execute(reCaptchaSiteKey, {action: 'check_auto'})\n    .then(function(token) {\n        appVehicleCheck.reCaptchaToken = token;\n    });\n}\n<\/code><\/pre>\n
From this block we can take the site key (6Lc66nwUAAAAANZvAnT-OK4f4D_xkdzw5MLtAYFL) and the action parameter (check_auto).<\/p>\n
Next, we need to determine which query the page is doing when we click the search button. To do it, open the developer panel in the browser, go to the Network tab and set the filter so that only XHR requests are shown to us.<\/p>\n
\"Doing<\/a><\/figure>\n
We\u2019 can see that the following resource is being loaded: https: \/\/xn--b1afk4ade.xn--90adear.xn--p1ai\/proxy\/check\/auto\/history by the POST method. The VIN, the type of verification and the Google ReCaptcha v3 token are transferred to the server in the request.<\/p>\n
\"Getting<\/a><\/figure>\n
Thus, we have everything to build a web scraper. Please note, that to scrape this website you need to use Russian proxy, as they recently closed access to this data to foreign IPs.<\/p>\n
---\nconfig:\n    debug: 2\n    agent: "Mozilla\/5.0 (Windows NT 10.0; WOW64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/54.0.2840.99 Safari\/537.36"\n        proxy: PUT HERE YOUR RUSSIAN PROXY IN 1.1.1.1:8888 FORMAT\ndo:\n# SET VIN TO CHECK TO THE VARIABLE\n- variable_set:\n    field: vin\n    value: WVGZZZ7LZ8D050320\n# LOAD PAGE WITH SEARCH FORM\n- walk:\n    to: https:\/\/xn--90adear.xn--p1ai\/check\/auto\/\n    do:\n    # \u0420\u0415\u0428\u0410\u0415\u041c \u041a\u0410\u041f\u0427\u0423\n    - captcha_resolve:\n        provider: 2captcha\n        type: recaptchav3\n        apikey: HERE YOU SHOULD SET YOUR OWN 2CAPTCHA API KEY\n        sitekey: 6Lc66nwUAAAAANZvAnT-OK4f4D_xkdzw5MLtAYFL\n        minscore: 0.3\n        action: check_auto\n    # GO TO BODY BLOCK ON THE BAGE TO SWITCH TO THE BLOCK CONTEXT\n    - find:\n        path: body\n        do:\n        # READ VARIABLE captcha TO THE REGISTER\n        - variable_get: captcha\n        # IF THERE IS TOKEN, CAPTCHA WAS RESOLVED AND WE CAN DO QUERY AND GET DATA\n        - if:\n            match: \\S\n            do:\n            # DOING POST REQUEST TO GET THE DATA\n            - walk:\n                to:\n                    post: https:\/\/xn--b1afk4ade.xn--90adear.xn--p1ai\/proxy\/check\/auto\/history\n                    data:\n                        vin: \n                        captchaWord: ''\n                        checkType: history\n                        reCaptchaToken: \n                do:\n                # PARSE DATA AND SAVE IT TO THE OBJECT\n                - find:\n                    path: requestresult\n                    do:\n                    - object_new: vehicle\n                    - find:\n                        path: vehicle>enginevolume\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: engine_volume\n                    - find:\n                        path: vehicle>color\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: color\n                    - find:\n                        path: vehicle>bodynymber\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: body_number\n                    - find:\n                        path: vehicle>year\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: year\n                    - find:\n                        path: vehicle>enginenumber\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: engine_number\n                    - find:\n                        path: vehicle>vin\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: vin\n                    - find:\n                        path: vehicle>model\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: model\n                    - find:\n                        path: vehicle>category\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: category\n                    - find:\n                        path: vehicle>type\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: type\n                    - find:\n                        path: vehicle>powerhp\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: power_hp\n                    - find:\n                        path: vehicle>powerkwt\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: power_kwt\n                    - find:\n                        path: vehiclepassport>number\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: passport_number\n                    - find:\n                        path: vehiclepassport>issue\n                        do:\n                        - parse\n                        - object_field_set:\n                            object: vehicle\n                            field: passport_issued\n                    - find:\n                        path: ownershipperiod\n                        do:\n                        - object_new: ownership\n                        - find:\n                            path: from\n                            do:\n                            - parse\n                            - object_field_set:\n                                object: ownership\n                                field: from\n                        - find:\n                            path: to\n                            do:\n                            - parse\n                            - object_field_set:\n                                object: ownership\n                                field: to\n                        - find:\n                            path: simplepersontype\n                            do:\n                            - parse\n                            - object_field_set:\n                                object: ownership\n                                field: owner_type\n                        - find:\n                            path: lastoperation\n                            do:\n                            - parse\n                            - object_field_set:\n                                object: ownership\n                                field: operation_type\n                        - object_save:\n                            name: ownership\n                            to: vehicle\n                    - object_save:\n                        name: vehicle<\/code><\/pre>\n
If we run it on the Diggernaut platform (of course, you will have to use your 2Captcha API key in the web scraper code), then we will get the following data (here\u2019s an example of the dataset in JSON format)<\/p>\n
[{\n    "vehicle": {\n        "category": "\u0412",\n        "color": "\u0427\u0415\u0420\u041d\u042b\u0419",\n        "engine_number": "080571",\n        "engine_volume": "2967.0",\n        "model": "\u0424\u041e\u041b\u042c\u041a\u0421\u0412\u0410\u0413\u0415\u041d \u0422\u041eU\u0410R\u0415G",\n        "ownership": [\n            {\n                "from": "2008-05-22T00:00:00.000+04:00",\n                "operation_type": "11",\n                "owner_type": "Natural",\n                "to": "2011-12-10T00:00:00.000+04:00"\n            },\n            {\n                "from": "2011-12-10T00:00:00.000+04:00",\n                "operation_type": "06",\n                "owner_type": "Natural",\n                "to": "2017-02-02T00:00:00.000+03:00"\n            },\n            {\n                "from": "2017-02-02T00:00:00.000+03:00",\n                "operation_type": "03",\n                "owner_type": "Natural"\n            }\n        ],\n        "passport_issued": "\u0422\u0410\u041c\u041e\u0416\u041d\u042f: 10009191",\n        "passport_number": "78\u0422\u0423623619",\n        "power_hp": "224",\n        "power_kwt": "164.8",\n        "type": "21",\n        "vin": "WVGZZZ7LZ8D050320",\n        "year": "2008"\n    }\n}\n]\n<\/code><\/pre>","protected":false},"excerpt":{"rendered":"
There is a new arrival in the ranks of services for solving captcha supported by us. Meet the 2Captcha. From the functional side, 2Captcha looks similar to the AntiCaptcha. It supports all popular kinds of captcha from the box, has \u201cproxyless\u201d mode, which is in demand by our users. Prices are on the same level […]<\/p>","protected":false},"author":4,"featured_media":588,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[35,9],"tags":[],"class_list":["post-581","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-integrations","category-learning-meta-language"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/581","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/comments?post=581"}],"version-history":[{"count":6,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/581\/revisions"}],"predecessor-version":[{"id":709,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/581\/revisions\/709"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media\/588"}],"wp:attachment":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media?parent=581"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/categories?post=581"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/tags?post=581"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}