What to do when server respond with JSON?

3 min read

Diggernaut makes it easy to work with JSON format by converting it to XML. So now I’ll show you how you can work with it.

I found one of the public JSON Rest API we use as a sample. Link.

It’s a sample of their server response:

[
  {
    "id": 1,
    "name": "Leanne Graham",
    "username": "Bret",
    "email": "Sincere@april.biz",
    "address": {
      "street": "Kulas Light",
      "suite": "Apt. 556",
      "city": "Gwenborough",
      "zipcode": "92998-3874",
      "geo": {
        "lat": "-37.3159",
        "lng": "81.1496"
      }
    },
    "phone": "1-770-736-8031 x56442",
    "website": "hildegard.org",
    "company": {
      "name": "Romaguera-Crona",
      "catchPhrase": "Multi-layered client-server neural-net",
      "bs": "harness real-time e-markets"
    }
  },
  ...
  ...
  ...
  ...
  ]

First, we create a digger, switch it to debug mode, and set Firefox as the name of the browser.

---
config:
    debug: 2
    agent: Firefox

ALT

Let’s add do block and use walk command with the address of the test RestAPI service inside. Also, I am going to show you a little trick that should be very useful in the future. The essence of the trick is to stop the execution of digger at some point you need. To do this, add the command stop to the do block.

---
config:
    debug: 2
    agent: Firefox
do:
- walk:
    to: http://jsonplaceholder.typicode.com/users
    do: 
    - stop

Run digger, and wait for the completion. As expected, the digger finished with an error.

ALT

Let us move to the digger’s log and see why it happened.

ALT

Unsupported operator used: stop

The system does not support this command, it raised an exception, and stopped the execution of digger, as a digger is in debug mode, and the number of errors cannot be more than 1.

Now let’s see how server response was handled by the digger.

ALT

Its how it looks in text editor.

<html>
    <head></head>
    <body>
        <body_safe>
            <element>    
                <address>      
                    <city>Gwenborough</city>      
                    <geo>        
                        <lat>-37.3159</lat>        
                        <lng>81.1496</lng>      
                    </geo>      
                    <street>Kulas Light</street>      
                    <suite>Apt. 556</suite>     
                    <zipcode>92998-3874</zipcode>    
                </address>    
                <company>     
                    <bs>harness real-time e-markets</bs>      
                    <catchphrase>Multi-layered client-server neural-net</catchphrase>     
                    <name>Romaguera-Crona</name>    
                    </company>    
                <email>Sincere@april.biz</email>    
                <id>1</id>    
                <name>Leanne Graham</name>    
                <phone>1-770-736-8031 x56442</phone>    
                <username>Bret</username>    
                <website>hildegard.org</website>  
            </element>  
            <element>    
                <address>      
                    <city>Wisokyburgh</city>      
                    <geo>        
                        <lat>-43.9509</lat>        
                        <lng>-34.4618</lng>      
                    </geo>      
                    <street>Victor Plains</street>      
                    <suite>Suite 879</suite>      
                    <zipcode>90566-7771</zipcode>    
                </address>    
                <company>      
                    <bs>synergize scalable supply-chains</bs>      
                    <catchphrase>Proactive didactic contingency</catchphrase>      
                    <name>Deckow-Crist</name>    
                </company>    
                <email>Shanna@melissa.tv</email>    
                <id>2</id>    
                <name>Ervin Howell</name>    
                <phone>010-692-6593 x09125</phone>    
                <username>Antonette</username>    
                <website>anastasia.net</website>  
            </element>  
            ...
            ...
            ...
            ...
            ...
            ...
            ...
            ...
            ...
        </body_safe>
    </body>
</html>

Let’s delete the command stop from out config and go to each element tag and extract and collect data from it. It’s simple. You can do it the same way as you do it with standard HTML, so I’ll describe it very briefly using comments inside config. Comments are marked with a # character.

---
config:
    debug: 2
    agent: Firefox
do:
- walk:
    to: http://jsonplaceholder.typicode.com/users
    do: 
    # Lets find all element tags and traverse through them
    - find:
        path: element
        do:
                Let's create an object to store data on each iteration for the element tag
        - object_new: user
        #Collecting data
        #We will extract not all data, just few fields
        - find:
            path: id
            do:
            - parse
            #Saving value of the register to the field "id" of the object "user" (futher we omit such comments as all operations are similar to this)
            - object_field_set:
                object: user
                field: id
        - find:
            path: name
            do:
            - parse
            - object_field_set:
                object: user
                field: name
        - find:
            path: username
            do:
            - parse
            - object_field_set:
                object: user
                field: username
        #Lets make full address merged to single field (we will use joinby option for it)
        - find:
            path: address
            do:
            - find:
                path: street
                do:
                - parse
                - object_field_set:
                    object: user
                    field: address
            - find:
                path: suite
                do:
                - parse
                - object_field_set:
                    object: user
                    field: address
                    joinby: ", "
            - find:
                path: city
                do:
                - parse
                - object_field_set:
                    object: user
                    field: address
                    joinby: ", "
            - find:
                path: zipcode
                do:
                - parse
                - object_field_set:
                    object: user
                    field: address
                    joinby: ", "
        #Save object "user"
        - object_save:
            name: user

So we built our config and can now switch digger to active mode and run it.

ALT

When it’s done, we can download data and check if all worked well. I downloaded data as JSON, and it looks this way in the text editor:

{
    "user": [{
        "username": "Bret",
        "id": "1",
        "name": "Leanne Graham",
        "address": "Kulas Light, Apt. 556, Gwenborough, 92998-3874"
    },
    ...
    ...
    ...
     {
        "username": "Moriah.Stanton",
        "address": "Kattie Turnpike, Suite 198, Lebsackbury, 31428-2261",
        "id": "10",
        "name": "Clementina DuBuque"
    }],
    "session": "session_595",
    "digger": "digger_120"
}

That’s all, if you have any questions, feel free to ask in comments, I will answer.

Leave a Reply

Your email address will not be published. Required fields are marked *