{"id":441,"date":"2018-02-28T20:29:51","date_gmt":"2018-02-28T20:29:51","guid":{"rendered":"https:\/\/www.diggernaut.com\/blog\/?p=441"},"modified":"2019-01-12T13:00:48","modified_gmt":"2019-01-12T13:00:48","slug":"bypass-captcha-diggernaut-web-scraping-platform-solving-google-recaptcha-v2","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/bypass-captcha-diggernaut-web-scraping-platform-solving-google-recaptcha-v2\/","title":{"rendered":"How to bypass captcha on Diggernaut web scraping platform: Solving Google reCaptcha v2"},"content":{"rendered":"

Google reCaptcha v2 is not a problem for our users anymore. We recently implemented integration with Death by Captcha service, and you can easily bypass such captcha now.<\/p>\n

Let’s see how reCaptcha v2 looks like:<\/p>\n

\"Bypass<\/a><\/figure>\n

If you see such captcha on the website you need to scrape, you can continue reading as we are going to give you guide how to implement it in your web scraper. Moreover, we are going to use a real example. We are planning to scrape this website: http:\/\/www.receita.fazenda.gov.br\/PessoaJuridica\/CNPJ\/cnpjreva\/Cnpjreva_Solicitacao2.asp<\/a>.<\/p>\n

To use this solution you need to have an account with Death by Captcha<\/a> service. This service is not free, solving 1000 such captchas going to cost you approx 2.89$ (at least this is an actual price on 02.28.2018).<\/p>\n

Diggernaut solve captcha automatically. You need to load the page with captcha and call special command captcha_resolve<\/strong> with specific parameters:<\/p>\n

provider<\/strong>: captcha solution provider, should be set to deathbycaptcha.com<\/em>\n
type<\/strong>: captcha type, should be set to nocaptchav2<\/em>\n
username<\/strong>: your death by captcha account username
password<\/strong>: your death by captcha account password<\/p>\n

Also please NOTE!<\/strong> To have such captcha successfully resolved, persons who resolve your captcha should be able to connect page with captcha using the same IP as your web scraper. So, the only way to achieve it is to use YOUR OWN PROXY SERVER<\/strong> in the digger configuration. Any IP outside of our network cannot access our rotating proxies, so they cannot be used for this operation.<\/p>\n

So base code for our scraper to resolve captcha would be:<\/p>\n

---\nconfig:\n    debug: 2\n    agent: Firefox\n    proxy: PUT YOUR OWN PROXY HERE\ndo:\n# We are going to retry calls until we get successfully resolved captcha (yes its not 100% successful rate), so we set variable we going to use in the walk command\n- variable_set:\n    field: repeat\n    value: "yes"\n# Loading page with captcha\n- walk:\n    to: http:\/\/www.receita.fazenda.gov.br\/PessoaJuridica\/CNPJ\/cnpjreva\/cnpjreva_solicitacao2.asp\n    repeat: \n    do:\n        # Resolving captcha\n    - captcha_resolve:\n        provider: deathbycaptcha.com\n        type: nocaptchav2\n        username: YOUR DBC USERNAME HERE\n        password: YOUR DBC PASSWORD HERE<\/code><\/pre>\n
Don’t run this code yet.<\/strong> If the captcha has been successfully resolved, you get token value in the captcha<\/strong> variable. So the first thing we are going to do after captcha resolve process is to check if we have some value in the variable and depending on check results either retry call or submit token to the server and get the results.<\/p>\n
---\nconfig:\n    debug: 2\n    agent: Firefox\n    proxy: PUT YOUR OWN PROXY HERE\ndo:\n# We are going to retry calls until we get successfully resolved captcha (yes its not 100% successful rate), so we set variable we going to use in the walk command\n- variable_set:\n    field: repeat\n    value: "yes"\n# Loading page with captcha\n- walk:\n    to: http:\/\/www.receita.fazenda.gov.br\/PessoaJuridica\/CNPJ\/cnpjreva\/cnpjreva_solicitacao2.asp\n    repeat: \n    do:\n        # Resolving captcha\n    - captcha_resolve:\n        provider: deathbycaptcha.com\n        type: nocaptchav2\n        username: YOUR DBC USERNAME HERE\n        password: YOUR DBC PASSWORD HERE\n        # Switching to the body block\n    - find:\n        path: body\n        do:\n                # Reading captcha variable value to the register\n        - variable_get: captcha\n                # Checking if register has not empty value\n        - if:\n            match: \\w+\n            do:\n                        # If its not empty, turn off repeat mode for walk command\n            - variable_set:\n                field: repeat\n                value: "no"\n                        # Submit token along with other form data we need to submit to get information we need: we are getting information about the company using its CNPJ number\n            - walk:\n                to:\n                    post: http:\/\/www.receita.fazenda.gov.br\/PessoaJuridica\/CNPJ\/cnpjreva\/valida_recaptcha.asp\n                    data:\n                        origem: comprovante\n                        cnpj: 05754558000186\n                        g-recaptcha-response: \n                        submit1: Consultar\n                        search_type: cnpj\n                do:\n                - find:\n                    path: 'div#principal'\n                    do:\n                    - object_new: item\n                    - find:\n                        path: td:haschild(font:contains('NÚMERO DE INSCRIÇÃO')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: registration_number\n                    - find:\n                        path: td:haschild(font:contains('DATA DE ABERTURA')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: registration_date\n                    - find:\n                        path: td:haschild(font:contains('NOME EMPRESARIAL')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: company_name\n                    - find:\n                        path: td:haschild(font:contains('CÓDIGO E DESCRIÇÃO DA ATIVIDADE ECONÔMICA PRINCIPAL')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: primary_code\n                    - find:\n                        path: td:haschild(font:contains('CÓDIGO E DESCRIÇÃO DAS ATIVIDADES ECONÔMICAS SECUNDÁRIAS')) b\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_push:\n                            object: item\n                            field: secondary_codes\n                    - find:\n                        path: td:haschild(font:contains('CÓDIGO E DESCRIÇÃO DA NATUREZA JURÍDICA')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: legal_code\n                    - find:\n                        path: td:haschild(font:contains('LOGRADOURO')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: street\n                    - find:\n                        path: td:haschild(font:contains('BAIRRO\/DISTRITO')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: district\n                    - find:\n                        path: td:haschild(font:contains('MUNICÍPIO')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: municipal\n                    - find:\n                        path: td:haschild(font:contains('TELEFONE')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: phone\n                    - find:\n                        path: td:haschild(font:contains('E-MAIL')) b\n                        slice: 0\n                        do:\n                        - parse\n                        - space_dedupe\n                        - trim\n                        - object_field_set:\n                            object: item\n                            field: email\n                    - object_save:\n                        name: item<\/code><\/pre>\n
You should get the following data when you run it.<\/p>\n
[{\n    "item": {\n        "company_name": "LIST - LOGISTICA INTEGRADA, SERVICOS E TRANSPORTES LTDA",\n        "district": "COROADO",\n        "legal_code": "206-2 - Sociedade Empresária Limitada",\n        "municipal": "MANAUS",\n        "phone": "(92) 3622-7885",\n        "primary_code": "52.12-5-00 - Carga e descarga",\n        "registration_date": "26\/06\/2003",\n        "registration_number": "05.754.558\/0001-86",\n        "secondary_codes": [\n            "49.30-2-02 - Transporte rodoviário de carga, exceto produtos perigosos e mudanças, intermunicipal, interestadual e internacional",\n            "77.39-0-99 - Aluguel de outras máquinas e equipamentos comerciais e industriais não especificados anteriormente, sem operador",\n            "77.19-5-99 - Locação de outros meios de transporte não especificados anteriormente, sem condutor",\n            "52.29-0-99 - Outras atividades auxiliares dos transportes terrestres não especificadas anteriormente",\n            "49.30-2-01 - Transporte rodoviário de carga, exceto produtos perigosos e mudanças, municipal",\n            "52.50-8-03 - Agenciamento de cargas, exceto para o transporte marítimo"\n        ],\n        "street": "R PROFESSORA RAYMUNDA MAGALHAES"\n    }\n}\n]\n<\/code><\/pre>\n
This particular solution can be used in “Data on demand” scenario when someone enters a number on your website. Your website sends a command to the Diggernaut API to run digger with a specific incoming parameter.  API then return results to your website and its render results to your user. Please note, that captcha resolve process is not very fast. For example, resolve process for reCaptcha v2 may take more than a minute.<\/p>\n
However, in any case, we are hoping that this article was useful for you and you now know, how to bypass recaptcha v2 with Diggernaut and Death by Captcha. We are planning to implement other captcha solutions soon, so please keep tuned.<\/p>","protected":false},"excerpt":{"rendered":"
Google reCaptcha v2 is not a problem for our users anymore. We recently implemented integration with Death by Captcha service, and you can easily bypass such captcha now. Let’s see how reCaptcha v2 looks like: If you see such captcha on the website you need to scrape, you can continue reading as we are going […]<\/p>","protected":false},"author":4,"featured_media":447,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,9,2],"tags":[],"class_list":["post-441","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-codeproject","category-learning-meta-language","category-web-scraping"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t