Stage One: preparation.<\/h4>\n
This is test configuration you can use as a sample for education, its fully working example and doesn\u2019t break any TOS. So its safe to use it in any country.<\/p>\n
---\ndo:\n - walk:\n to: 'https:\/\/www.diggernaut.com\/sandbox\/'\n do:\n - find:\n path: 'div:nth-child(6) .result-content'\n do:\n - find:\n path: h3\n do:\n - parse\n - variable_clear: name\n - variable_set: name\n - find:\n path: p\n do:\n - parse\n - variable_clear: description\n - variable_set: description\n - find:\n path: 'tbody > tr'\n do:\n - parse\n - object_new: item\n - variable_get: name\n - object_field_set:\n object: item\n field: name\n - variable_get: description\n - object_field_set:\n object: item\n field: descr\n - object_new: date\n - find:\n path: '.col5 > .nowrap:nth-child(1)'\n do:\n - parse\n - object_field_set:\n object: date\n field: date\n - find:\n path: '.col5 > .nowrap:nth-child(2)'\n do:\n - parse\n - object_field_set:\n object: date\n field: date\n joinby: " - "\n - find:\n path: .col6\n do:\n - parse\n - normalize:\n routine: replace_substring\n args:\n \\s*(\\d+)A: 's' \n - object_field_set:\n object: date\n field: time\n - object_save:\n name: date\n to: item\n - object_save:\n name: item<\/code><\/pre>\nResulting dataset has the following structure:<\/p>\n
item: [\n {\n "name":"",\n "descr":"",\n "date": [\n {\n "date":",\n "time":""\n }\n ...\n ...\n ]\n },\n ...\n ...\n ]\n<\/code><\/pre>\nLet\u2019s prepare our database. Digger can save data to MySQL, Microsoft SQL and PostgreSQL. So we need to create database and tables for our data.<\/p>\n
MYSQL<\/h4>\n CREATE DATABASE `digger`;\n\n CREATE TABLE `items` (\n `id` INT(11) NOT NULL AUTO_INCREMENT,\n `name` VARCHAR(50) NULL DEFAULT NULL,\n `descr` TEXT NULL,\n PRIMARY KEY (`id`)\n )\n COLLATE='utf8_general_ci'\n ENGINE=InnoDB;\n\n CREATE TABLE `dates` (\n `id` INT(11) NOT NULL AUTO_INCREMENT,\n `item_id` INT(11) NOT NULL,\n `date` VARCHAR(50) NULL DEFAULT NULL,\n `time` VARCHAR(50) NULL DEFAULT NULL,\n INDEX `Index 1` (`id`),\n INDEX `FK__items` (`item_id`),\n CONSTRAINT `FK__items` FOREIGN KEY (`item_id`) REFERENCES `items` (`id`) ON UPDATE NO ACTION ON DELETE NO ACTION\n )\n COLLATE='utf8_general_ci'\n ENGINE=InnoDB;\n<\/code><\/pre>\nMS SQL<\/h4>\n CREATE DATABASE `digger`;\n\n CREATE TABLE `items` (\n id INT NOT NULL IDENTITY(1,1) CONSTRAINT pk_items_pid PRIMARY KEY,\n `name` VARCHAR(50),\n `descr` TEXT \n );\n\n CREATE TABLE `dates` (\n id INT NOT NULL IDENTITY(1,1) CONSTRAINT pk_dates_pid PRIMARY KEY,\n item_id int CONSTRAINT fk_item_id FOREIGN KEY(item_id) REFERENCES items(id),\n date TEXT,\n time TEXT \n );\n<\/code><\/pre>\nPostgreSQL<\/h4>\n CREATE DATABASE digger;\n\n CREATE TABLE items (\n id SERIAL PRIMARY KEY,\n name varchar(50),\n descr text,\n );\n\n CREATE TABLE dates (\n id SERIAL PRIMARY KEY,\n date varchar(50),\n time varchar(50),\n item_id integer REFERENCES items (id),\n );\n<\/code><\/pre>\nNow let\u2019s compile our digger.<\/p>\n![\"\"](\"https:\/\/www.diggernaut.com\/blog\/wp-content\/uploads\/2016\/10\/compile.jpg\")
<\/a><\/figure>\n![\"\"](\"https:\/\/www.diggernaut.com\/blog\/wp-content\/uploads\/2016\/10\/compilemysql.jpg\")
<\/a><\/figure>\n![\"\"](\"https:\/\/www.diggernaut.com\/blog\/wp-content\/uploads\/2016\/10\/compilemssql.jpg\")
<\/a><\/figure>\n![\"\"](\"https:\/\/www.diggernaut.com\/blog\/wp-content\/uploads\/2016\/10\/compilePostgre.jpg\")
<\/a><\/figure>\nDownload it and run. Once the job is finished, you can see data in your DB. So now you can have your data flow directly from digger to your database without a hassle.<\/p>","protected":false},"excerpt":{"rendered":"
Do you want to save data to RDBMS? It\u2019s not that hard. So today I\u2019m going to show you how easy it can be done using Diggernaut\u2019s Compile Service. Stage One: preparation. This is test configuration you can use as a sample for education, its fully working example and doesn\u2019t break any TOS. So its […]<\/p>","protected":false},"author":5,"featured_media":85,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,2],"tags":[],"class_list":["post-83","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-learning-meta-language","category-web-scraping"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/83","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/comments?post=83"}],"version-history":[{"count":5,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/83\/revisions"}],"predecessor-version":[{"id":681,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/83\/revisions\/681"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media\/85"}],"wp:attachment":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media?parent=83"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/categories?post=83"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/tags?post=83"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}