For paid subscribers, it became possible to set the proxy type for use in diggers.
You can choose between data center, residential, IPv6, and Tor proxy pools. The default pool is always the data center, this pool contains mainly US proxies. As a rule, all proxies in this pool are fast and reliable. Residential proxies work much slower and are not as reliable, but they allow you to collect information from sites that are not allowed to access through data center proxies. IPv6 can be used for sites that support this protocol (eg Instagram, Google, Yandex). Tor is the slowest option, and also you cannot access some websites through Tor. On the other hand, Tor gives access to the darknet, which means you now can scrape the darknet. Users with a free account only have access to the Tor pool. However, you can always use your proxies, and now not only HTTP. We’ve added support for SOCKS4 and SOCKS5 proxies. So even with free accounts, there have more proxy options now. More details about the proxy settings can be found in our documentation.
We have added few new static variables to provide more flexibility when creating digger configurations.
You can use the
responseCode variable to process the errors returned by the source website server. There is a three-digit numeric code of the server response stored in this variable after each request. For example, 200 – if everything is OK, or 500 – if an error occurred on the server, 503 – if access to the website is denied, and so on (for more details about response codes, you can read here). Then you can build logical constructs in the configuration to bypass the errors that occur if the website actively resists scraping or is simply unstable.
filename variable can be useful to you if you transfer binaries and images using a digger. This variable will be automatically assigned the value of the name of the last saved file and you can put it to the dataset object to associate a specific record with the file.
If you are using
update mode when saving objects, the
object_saved variable may come in handy. It allows you to find out whether the object was saved or not, and, depending on the outcome, make certain actions. The variable can be “yes” or “no”.
You can find more details about information on static variables here.
We have expanded the capabilities of working with Selenium on our Diggernaut platform.
scrollto command, you can scroll the page to the specific element. Thus, the element will be visible in the browser window and it will be possible to perform various actions with it. For example, you can scroll to the button you want to click on. The
fetch_content command to access their content. In this case, the HTML content of the frame will be saved to the register, so you can create a block from the register content and work with it. You can refer to our documentation to find out how to work with Selenium.