{"id":55,"date":"2016-09-21T19:22:42","date_gmt":"2016-09-21T19:22:42","guid":{"rendered":"https:\/\/blog.diggernaut.com\/?p=55"},"modified":"2019-01-12T21:23:10","modified_gmt":"2019-01-12T21:23:10","slug":"what-to-do-when-server-respond-with-json","status":"publish","type":"post","link":"https:\/\/www.diggernaut.com\/blog\/what-to-do-when-server-respond-with-json\/","title":{"rendered":"What to do when server respond with JSON?"},"content":{"rendered":"
[\n  {\n    "id": 1,\n    "name": "Leanne Graham",\n    "username": "Bret",\n    "email": "Sincere@april.biz",\n    "address": {\n      "street": "Kulas Light",\n      "suite": "Apt. 556",\n      "city": "Gwenborough",\n      "zipcode": "92998-3874",\n      "geo": {\n        "lat": "-37.3159",\n        "lng": "81.1496"\n      }\n    },\n    "phone": "1-770-736-8031 x56442",\n    "website": "hildegard.org",\n    "company": {\n      "name": "Romaguera-Crona",\n      "catchPhrase": "Multi-layered client-server neural-net",\n      "bs": "harness real-time e-markets"\n    }\n  },\n  ...\n  ...\n  ...\n  ...\n  ]\n<\/code><\/pre>\nFirst, we create a digger, switch it to debug mode, and set Firefox as the name of the browser.<\/p>\n
---\nconfig:\n    debug: 2\n    agent: Firefox<\/code><\/pre>\n<\/p>\n
Let\u2019s add do<\/strong> block and use walk<\/strong> command with the address of the test RestAPI service inside. Also, I am going to show you a little trick that should be very useful in the future. The essence of the trick is to stop the execution of digger at some point you need. To do this, add the command stop<\/strong> to the do<\/strong>  block.<\/p>\n
---\nconfig:\n    debug: 2\n    agent: Firefox\ndo:\n- walk:\n    to: http:\/\/jsonplaceholder.typicode.com\/users\n    do: \n    - stop<\/code><\/pre>\nRun digger, and wait for the completion. As expected, the digger finished with an error.<\/p>\n
<\/p>\n
Let us move to the digger\u2019s log and see why it happened.<\/p>\n
<\/p>\n
Unsupported operator used: stop<\/code><\/p>\n
The system does not support this command, it raised an exception, and stopped the execution of digger, as a digger is in debug mode, and the number of errors cannot be more than 1.<\/p>\n
Now let\u2019s see how server response was handled by the digger.<\/p>\n
<\/p>\n
Its how it looks in text editor.<\/p>\n
\n    <\/head>\n    \n        \n                \n                      \n                    Gwenborough<\/city>      \n                            \n                        -37.3159<\/lat>        \n                        81.1496<\/lng>      \n                    <\/geo>      \n                    Kulas Light<\/street>      \n                    Apt. 556<\/suite>     \n                    92998-3874<\/zipcode>    \n                <\/address>    \n                     \n                    harness real-time e-markets<\/bs>      \n                    Multi-layered client-server neural-net<\/catchphrase>     \n                    Romaguera-Crona<\/name>    \n                    <\/company>    \n                Sincere@april.biz<\/email>    \n                1<\/id>    \n                Leanne Graham<\/name>    \n                1-770-736-8031 x56442<\/phone>    \n                Bret<\/username>    \n                hildegard.org<\/website>  \n            <\/element>  \n                \n                      \n                    Wisokyburgh<\/city>      \n                            \n                        -43.9509<\/lat>        \n                        -34.4618<\/lng>      \n                    <\/geo>      \n                    Victor Plains<\/street>      \n                    Suite 879<\/suite>      \n                    90566-7771<\/zipcode>    \n                <\/address>    \n                      \n                    synergize scalable supply-chains<\/bs>      \n                    Proactive didactic contingency<\/catchphrase>      \n                    Deckow-Crist<\/name>    \n                <\/company>    \n                Shanna@melissa.tv<\/email>    \n                2<\/id>    \n                Ervin Howell<\/name>    \n                010-692-6593 x09125<\/phone>    \n                Antonette<\/username>    \n                anastasia.net<\/website>  \n            <\/element>  \n            ...\n            ...\n            ...\n            ...\n            ...\n            ...\n            ...\n            ...\n            ...\n        <\/body_safe>\n    <\/body>\n<\/html>\n<\/code><\/pre>\nLet\u2019s delete the command stop<\/strong> from out config and go to each element<\/strong> tag and extract and collect data from it. It\u2019s simple. You can do it the same way as you do it with standard HTML, so I\u2019ll describe it very briefly using comments inside config. Comments are marked with a # character.<\/p>\n
---\nconfig:\n    debug: 2\n    agent: Firefox\ndo:\n- walk:\n    to: http:\/\/jsonplaceholder.typicode.com\/users\n    do: \n    # Lets find all element tags and traverse through them\n    - find:\n        path: element\n        do:\n                Let's create an object to store data on each iteration for the element tag\n        - object_new: user\n        #Collecting data\n        #We will extract not all data, just few fields\n        - find:\n            path: id\n            do:\n            - parse\n            #Saving value of the register to the field "id" of the object "user" (futher we omit such comments as all operations are similar to this)\n            - object_field_set:\n                object: user\n                field: id\n        - find:\n            path: name\n            do:\n            - parse\n            - object_field_set:\n                object: user\n                field: name\n        - find:\n            path: username\n            do:\n            - parse\n            - object_field_set:\n                object: user\n                field: username\n        #Lets make full address merged to single field (we will use joinby option for it)\n        - find:\n            path: address\n            do:\n            - find:\n                path: street\n                do:\n                - parse\n                - object_field_set:\n                    object: user\n                    field: address\n            - find:\n                path: suite\n                do:\n                - parse\n                - object_field_set:\n                    object: user\n                    field: address\n                    joinby: ", "\n            - find:\n                path: city\n                do:\n                - parse\n                - object_field_set:\n                    object: user\n                    field: address\n                    joinby: ", "\n            - find:\n                path: zipcode\n                do:\n                - parse\n                - object_field_set:\n                    object: user\n                    field: address\n                    joinby: ", "\n        #Save object "user"\n        - object_save:\n            name: user<\/code><\/pre>\nSo we built our config and can now switch digger to active mode and run it.<\/p>\n
<\/p>\n
When it\u2019s done, we can download data and check if all worked well. I downloaded data as JSON, and it looks this way in the text editor:<\/p>\n
{\n    "user": [{\n        "username": "Bret",\n        "id": "1",\n        "name": "Leanne Graham",\n        "address": "Kulas Light, Apt. 556, Gwenborough, 92998-3874"\n    },\n    ...\n    ...\n    ...\n     {\n        "username": "Moriah.Stanton",\n        "address": "Kattie Turnpike, Suite 198, Lebsackbury, 31428-2261",\n        "id": "10",\n        "name": "Clementina DuBuque"\n    }],\n    "session": "session_595",\n    "digger": "digger_120"\n}\n<\/code><\/pre>\nThat\u2019s all, if you have any questions, feel free to ask in comments, I will answer.<\/p>","protected":false},"excerpt":{"rendered":"
Diggernaut makes it easy to work with JSON format by converting it to XML. So now I\u2019ll show you how you can work with it. I found one of the public JSON Rest API we use as a sample. Link. It\u2019s a sample of their server response: [ { “id”: 1, “name”: “Leanne Graham”, “username”: […]<\/p>","protected":false},"author":5,"featured_media":58,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,2],"tags":[],"class_list":["post-55","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-learning-meta-language","category-web-scraping"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/55","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/comments?post=55"}],"version-history":[{"count":11,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/55\/revisions"}],"predecessor-version":[{"id":686,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/posts\/55\/revisions\/686"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media\/58"}],"wp:attachment":[{"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/media?parent=55"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/categories?post=55"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.diggernaut.com\/blog\/wp-json\/wp\/v2\/tags?post=55"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}