{"id":171,"date":"2016-11-21T15:12:19","date_gmt":"2016-11-21T15:12:19","guid":{"rendered":"https:\/\/blog.diggernaut.com\/?p=171"},"modified":"2019-01-12T18:15:14","modified_gmt":"2019-01-12T18:15:14","slug":"using-json-schema-to-validate-your-data","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/using-json-schema-to-validate-your-data\/","title":{"rendered":"Using JSON schema to validate your data"},"content":{"rendered":"

Recently we added a couple of neat functions which let you work with data more efficiently. So one of these functions is JSON schema support. JSON schema can be used in many cases, e.g., if you need to ensure that digger still works appropriately and data you are getting is still in good state, or if you need to get just specific records and skip others. For example, if you are gathering some events, you may want to get the only event that not canceled or has open slots, if a website has information about it, you can easily set rules in a JSON scheme to pick only records you need.<\/p>\n

So what is JSON schema? As json-schema.org<\/a> states: \u201cJSON Schema is a vocabulary that allows you to annotate and validate JSON documents.\u201d. I would recommend you to learn more about it from the above site, as we are not going to cover syntax and JSON schema usage in this article. You can quickly learn it and play with it in debug mode at Diggernaut<\/a> without paying a dime for it.<\/p>\n

So how can you set JSON schema for a digger? First, you need to login to your Diggernaut account, then go to Projects > Diggers, find digger you need and click on \u201cConfig\u201d button.<\/p>\n

\"jsonschema1\"<\/a><\/p>\n

It opens editor panel where you usually put in digger config. You can see that it has 2 additional tabs now. You need to click on the \u201cValidator\u201d tab.<\/p>\n

\"jsonschema2\"<\/a><\/p>\n

Then you have to put your JSON schema and click on the \u201cSave\u201d button.<\/p>\n

\"jsonschema3\"<\/a><\/p>\n

Next time your digger is running, it applies your JSON scheme for data validation. To understand it better, you may want to look into digger config we used for tests:<\/p>\n

---\nconfig:\n    debug: 2\ndo:\n  - link_add: 'https:\/\/diggernaut.com\/sandbox\/'\n  - walk:\n      to: links\n      do:\n        - sleep: 1\n        - find:\n            path: .result-content\n            do:\n              - variable_clear: name\n              - variable_clear: descr\n              - find:\n                  path: h3\n                  do:\n                    - parse\n                    - variable_set: name\n              - find:\n                  path: p\n                  do:\n                    - parse\n                    - variable_set: descr\n              - find:\n                  path: table\n                  do:\n                    - find:\n                        path: 'tbody > tr'\n                        do:\n                          - object_new: item\n                          - variable_get: name\n                          - object_field_set:\n                              object: item\n                              field: name\n                          - variable_get: descr\n                          - object_field_set:\n                              object: item\n                              field: descr\n                          - find:\n                              path: .col2\n                              do:\n                                - parse\n                                - object_field_set:\n                                    object: item\n                                    field: number\n                          - find:\n                              path: .col3\n                              do:\n                                - parse\n                                - object_field_set:\n                                    object: item\n                                    field: short_descr\n                          - find:\n                              path: .col4\n                              do:\n                                - parse\n                                - object_field_set:\n                                    object: item\n                                    field: location\n                          - find:\n                              path: .col5\n                              do:\n                                - object_new: date\n                                - find:\n                                    path: ' .nowrap:nth-child(1)'\n                                    do:\n                                      - parse\n                                      - object_field_set:\n                                          object: date\n                                          field: start\n                                - find:\n                                    path: ' .nowrap:nth-child(2)'\n                                    do:\n                                      - parse\n                                      - object_field_set:\n                                          object: date\n                                          field: end\n                                - object_save:\n                                    name: date\n                                    to: item\n                          - find:\n                              path: .col6\n                              do:\n                                - object_new: time\n                                - find:\n                                    path: ' .nowrap:nth-child(1)'\n                                    do:\n                                      - parse\n                                      - object_field_set:\n                                          object: time\n                                          field: start\n                                - find:\n                                    path: ' .nowrap:nth-child(2)'\n                                    do:\n                                      - parse\n                                      - object_field_set:\n                                          object: time\n                                          field: end\n                                - object_save:\n                                    name: time\n                                    to: item\n                          - find:\n                              path: .col7\n                              do:\n                                - parse\n                                - object_field_set:\n                                    object: item\n                                    field: days\n                          - find:\n                              path: .col8\n                              do:\n                                - parse:\n                                    filter:\n                                      - "\\\\s*\\\\$\\\\s*(\\\\d+)\\\\\/"\n                                      - "\\\\s*\\\\$\\\\s*(\\\\d+)"\n                                - object_field_set:\n                                    object: item\n                                    type: int\n                                    field: member_fee\n                                - parse:\n                                    filter:\n                                      - "\\\\s*\\\\\/\\\\s*\\\\$\\\\s*(\\\\d+)"\n                                      - "\\\\s*\\\\$\\\\s*(\\\\d+)"\n                                - object_field_set:\n                                    object: item\n                                    type: int\n                                    field: non_member_fee\n                          - find:\n                              path: .col9\n                              do:\n                                - parse\n                                - object_field_set:\n                                    object: item\n                                    field: ages\n                          - find:\n                              path: .col10\n                              do:\n                                - parse\n                                - object_field_set:\n                                    object: item\n                                    field: is_available\n                          - find:\n                              path: .ajaxLoad.info-icon.tooltips\n                              do:\n                                - parse:\n                                    attr: href\n                                - walk:\n                                    to: value\n                                    do:\n                                      - find:\n                                          path: 'tr:nth-of-type(2) td:nth-of-type(2)'\n                                          do:\n                                            - parse\n                                            - object_field_set:\n                                                object: item\n                                                field: gender\n                          - object_save:\n                              name: item\n        - find:\n            path: .next a\n            do:\n              - parse:\n                  attr: href\n              - link_add<\/code><\/pre>\n
And JSON scheme we used for it:<\/p>\n
{\n    "$schema": "http:\/\/json-schema.org\/draft-04\/schema#",\n    "title": "Activities",\n    "description": "Park district activities",\n    "type": "object",\n    "properties": {\n        "item": {\n            "type": "object",\n            "properties": {\n                "number": {\n                    "description": "The unique identifier for an activity",\n                    "type": "string"\n                },\n                "name": {\n                    "description": "Activity name",\n                    "type": "string"\n                },\n                "descr": {\n                    "description": "Activity description",\n                    "type": "string"\n                },\n                "gender": {\n                    "description": "Gender specification for an activity",\n                    "type": "string"\n                },\n                "short_descr": {\n                    "description": "Activity short description",\n                    "type": "string"\n                },\n                "ages": {\n                    "description": "Allowed ages",\n                    "type": "string"\n                },\n                "days": {\n                    "description": "Weekdays when activity takes place",\n                    "type": "string"\n                },\n                 "member_fee": {\n                    "description": "Fee for members",\n                    "type": "number"\n                },\n                 "non_member_fee": {\n                    "description": "Fee for non-members",\n                    "type": "number"\n                },\n                 "is_available": {\n                    "description": "Shows if activity is still available",\n                    "type": "string"\n                },\n                 "location": {\n                    "description": "Location where activity takes place",\n                    "type": "string"\n                },\n                 "dates": {\n                    "type": "array",\n                    "items": {\n                        "type": "object",\n                        "properties": {\n                             "start": {\n                                "description": "Start date for activity session",\n                                "type": "string"\n                            },\n                             "end": {\n                                "description": "End date for activity session",\n                                "type": "string"\n                            }\n                        },\n                        "required": ["start","end"]\n                    },\n                    "minItems": 1,\n                    "uniqueItems": true\n                },\n                 "time": {\n                    "type": "array",\n                    "items": {\n                        "type": "object",\n                        "properties": {\n                             "start": {\n                                "description": "Start time for activity event",\n                                "type": "string"\n                            },\n                             "end": {\n                                "description": "End time for activity event",\n                                "type": "string"\n                            }\n                        },\n                        "required": ["start","end"]\n                    },\n                    "minItems": 1,\n                    "uniqueItems": true\n                }\n           },\n            "required": ["number","name","gender"]\n        }\n    },\n    "required": ["item"]\n\n}\n<\/code><\/pre>","protected":false},"excerpt":{"rendered":"
Recently we added a couple of neat functions which let you work with data more efficiently. So one of these functions is JSON schema support. JSON schema can be used in many cases, e.g., if you need to ensure that digger still works appropriately and data you are getting is still in good state, or […]<\/p>","protected":false},"author":4,"featured_media":178,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,23],"tags":[7,20,6,11],"class_list":["post-171","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-scraping","category-website-and-api","tag-data","tag-data-extraction","tag-scraping","tag-web-scraping"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/171","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/comments?post=171"}],"version-history":[{"count":4,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/171\/revisions"}],"predecessor-version":[{"id":671,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/171\/revisions\/671"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media\/178"}],"wp:attachment":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media?parent=171"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/categories?post=171"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/tags?post=171"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}