Entity Manipulations

Data Objects

Data objects are used to organize the structure of the stored data. Before you can save data to an object, you must create an object. This is done with the object_newcommand, which takes a single argument - the name of the object. After the object is created and until it is saved with the object_save command, you can work with the fields of the object. You can write values and a list of values ​to the fields with the command object_field_set. You can also save other object to the object field using a special parameter in the object_save command. If you use the JSON validation scheme in your digger, the object_save command object will check your object for validity. If there are validation errors, the object will not be saved. Also for checking an object without saving object to the DB, there is a command object_check, which simply checks the object for validity and in the case of a positive result it executes commands from the do block, and in the case of a negative one, from the else block.

Allowed commands:

          # CREATE OBJECT
- object_new: item
          
          # SET VALUE OF THE FIELD OF THE OBJECT
- object_field_set:
    # OBJECT NAME
    object: item
    # FIELD NAME
    # CAN BE SET AS STRING: somefield
    # OR WITH USING VARIABLES/ARGUMENTS/COUNTER VALUE: somefield_<%somevar%>
    field: somefield
          
          # SET VALUE OF THE FIELD OF THE OBJECT AS SPECIFIC DATA TYPE
- object_field_set:
    # OBJECT NAME
    object: item
    # FIELD NAME
    # CAN BE SET AS STRING: somefield
    # OR WITH USING VARIABLES/ARGUMENTS/COUNTER VALUE: somefield_<%somevar%>
    field: somefield
    # SETTING FIELD TYPE (CAN BE: float, date OR int)
    type: float
    # YOU CAN ALSO SET PRECISION FOR FLOAT TYPE
    precision: 2
          
          # PUSHING VALUE TO THE FIELD AS ELEMENT OF ARRAY
- object_field_push:
    # OBJECT NAME
    object: item
    # FIELD NAME
    # CAN BE SET AS STRING: somefield
    # OR WITH USING VARIABLES/ARGUMENTS/COUNTER VALUE: somefield_<%somevar%>
    field: somefield
    # CHECK FOR UNIQUENESS (OPTIONAL)
    # IF SUCH VALUE ALREADY EXIST IN THE ARRAY, IT WILL NOT BE PUSHED
    unique: yes
          
          # SAVING OBJECT TO THE FIELD OF OTHER OBJECT
- object_save:
    # OBJECT NAME
    name: item
   # TARGET OBJECT NAME
    to: anotherobj
# OR
- object_save:
    # OBJECT NAME
    name: item
    # TARGET OBJECT NAME
    to: anotherobj
    # SAVE OBJECT AS MAP (OPTIONAL, WITHOUT THIS OPTION OBJECT WILL BE PUSHED AS AN ELEMENT OF ARRAY, WHICH ALLOW YOU TO PUSH SEVERAL OBJECTS TO THE FIELD)
    as: map
          
          # SAVING OBJECT TO THE DB
- object_save:
    # OBJECT NAME
    name: item
          
            # CHECKING OBJECT
  - object_check:
      # OBJECT NAME
      name: item
      do:
      # IF CHECK RESULTS WERE POSITIVE WE DO THIS BLOCK
      - register_set: All is good
      else:
      # IF CHECK RESULTS WERE NEGATIVE WE DO THIS BLOCK
      - exit
        

The process of saving of objects to an object can occur in two scenarios:
save a list of objects in a field or save one object as a hashmap.

Saving of a list of objects into an object is performed as follows:

              # CREATE BLOCK FROM HTML STRING
- register_set: '<ul>
                    <li>A</li>
                    <li>B</li>
                    <li>C</li>
                    <li>D</li>
                </ul>'
- to_block
- find:
    path: ul
    do:
    # CREATE OBJECT WITH NAME `item`
    - object_new: item
    - find:
        path: li
        do:
        # CREATE ANOTHER OBJECT WITH NAME `sub-item`
        - object_new: sub-item

        # PARSE TO REGISTER
        - parse

        # SAVE REGISTER VALUE TO THE OBJECT `sub-item` FIELD `sometext`
        - object_field_set:
            object: sub-item
            field: sometext

        # SAVE OBJECT `sub-item` TO THE OBJECT `item`
        - object_save:
            name: sub-item
            to: item

    # SAVE OBJECT `item` TO THE DB
    - object_save:
        name: item
              
              {
    "item": {
        "sub-item": [
          {
            "sometext": "A",
          },
          {
            "sometext": "B",
          },
          {
            "sometext": "C",
          },
          {
            "sometext": "D",
          }
        ]
    }
}
              

Saving object to other object as hashmap:

              # CREATE BLOCK FROM HTML STRING
- register_set: '<ul>
                    <li>A</li>
                    <li>B</li>
                    <li>C</li>
                    <li>D</li>
                </ul>'
- to_block
- find:
    path: ul
    do:
    # CREATE OBJECT WITH NAME `item`
    - object_new: item
    - find:
        path: li
        do:
        # CREATE ANOTHER OBJECT WITH NAME `sub-item`
        - object_new: sub-item

        # PARSE TO THE REGISTER
        - parse

        # SAVE REGISTER VALUE TO THE OBJECT `sub-item` FIELD `sometext`
        - object_field_set:
            object: sub-item
            field: sometext

        # SAVE OBJECT `sub-item` TO THE OBJECT `item` AS HASHMAP
        - object_save:
            name: sub-item
            to: item
            as: map

    # SAVE OBJECT `item` TO THE DB
    - object_save:
        name: item
              
              {
    "item": {
        "sub-item" : {
            "sometext": "D"
    }
}
              

Please note:
In this example, the script goes through all li tags, but in the dataset there is only the last element with the value D falls into the data. This is because the field will be consistently overwritten by the values ​​of all the elements sequentally. As a result, only the value of the last element will remain in the field.

An example of the correct creation and saving objects:

              # EVERY TIME DIGGER ITERATE SAME BLOCK - IT EXECUTES SAME CODE AND APPLY SAME LOGIC
# THATS WHY WITH EACH NEW ITERATION DIGGER WILL CREATE NEW OBJECT
- find:
    path: .somepath
    do:
    # CREATE OBJECT WITH NAME `someobj`
    - object_new: someobj

    # PARSE TEXT TO THE REGISTER
    - parse

    # SET `somefield` FIELD OF THE `someobj` OBJECT WITH THE VALUE OF THE REGISTER
    - object_field_set:
        object: someobj
        field: somefield

    # JUMP TO SOME OTHER BLOCK, PARSE TEXT CONTENT AND SAVE IT TO SOME OTHER FIELD OF THE OBJECT `someobj`
    # PLEASE NOTE: IF DIGGER FIND SINGLE ELEMENT USING CSS SELECTOR - THEN VALUE OF FIELD WILL BE PROPER,
    # IF DIGGER FINDS SEVERAL ELEMENTS, IT WILL ITERATE OVER ALL OF THEM SEQUENTALLY AND ONLY CONTENT OF LAST ELEMENT WILL BE ACTUALLY KEPT IN THE FIELD
    # WE WILL OVERVIEW IT IN DETAILS LATER, NOW LETS JUST SWITCH TO THE SINGLE BLOCK
    - find:
        path: li:nth-of-type(1)
        do:
        - parse

        # SET `anotherfield` FIELD OF THE `someobj` OBJECT WITH THE VALUE OF THE REGISTER
        - object_field_set:
            object: someobj
            field: anotherfield

    # LETS NOW USE CSS SELECTOR THAT SELECTS SEVERAL ELEMENTS,
    # WHILE WE ITERATING WE WILL CREATE NEW OBJECTS AND PUT THEM TO OUR MAIN OBJECT `someobj`
    - find:
        path: .anotherpath
        do:
        # CREATE NEW OBJECT
        - object_new: anotherobj

        # SET FIELD `somefield` OF THE OBJECT `anotherobj`
        - parse
        - object_field_set:
            object: anotherobj
            field: somefield

        # SAVE OBJECT `anotherobj` TO THE OBJECT `someobj`
        - object_save:
            name: anotherobj
            to: someobj

    # SAVE MAIN OBJECT `someobj`
    - object_save:
        name: someobj
              
              # EXAMPLE OF HOW YOU SHOOULD NOT DO IT :)
- find:
    path: .somepath
    do:
    # CREATE OBJECT `someobj`
    - object_new: someobj

    # PARSE TEXT TO THE REGISTER
    - parse

    # SET FIELD `somefield` OF THE OBJECT `someobj` WITH THE VALUE OF THE REGISTER
    - object_field_set:
        object: someobj
        field: somefield

    # LETS USE CSS SELECTOR THAT WILL FIND SEVERAL ELEMENTS,
    # WHILE WE ITERATING WE WILL CREATE NEW OBJECTS AND PUT THEM TO OUR MAIN OBJECT `someobj`
    - find:
        path: .anotherpath
        do:
        # CREATE NEW OBJECT
        - object_new: anotherobj

        # SET FIELD `somefield` OF THE OBJECT `anotherobj`
        - parse
        - object_field_set:
            object: anotherobj
            field: somefield

        # SAVE OBJECT `anotherobj` TO THE OBJECT `someobj`
        - object_save:
            name: anotherobj
            to: someobj

    # SWITCH TO OTHER BLOCK AND SAVE PARSED DATA TO SOME OTHER FIELD OF THE OBJECT `someobj`
    - find:
        path: li:nth-of-type(1)
        do:
        - parse

        # SET FIELD `anotherfield` OF THE OBJECT `someobj` WITH THE VALUE OF THE REGISTER
        - object_field_set:
            object: someobj
            field: anotherfield

        # SAVE OBJECT `someobj`
        - object_save:
            name: someobj

    # AS YOU SEE MAIN OBJECT IS SAVED IN DIFFERENT SCOPE THAN OBJECT OPENING SCOPE
    # IT COULD LEAD TO THE POTENTIAL PROBLEM
    # IF SOME PAGES WILL NOT HAVE ANY ELEMENTS WITH SELECTOR `li:nth-of-type(1)`, OBJECT WILL NOT BE SAVED
    # AND YOU WILL LOSE DATA FOR SUCH PAGES
    # SO IT WILL WORK ONLY IF SUCH SELECTOR HAS AT LEAST 1 ELEMENT ON PAGE
    # THATS WHY YOU SHOULD ALWAYS SAVE YOUR OBJECT IN SAME SCOPE WHERE YOU CREATE THE OBJECT
              
              {
    "someobj": {
        "somefield" : "somedata",
        "anotherfield": "anotherdata",
        "anotherobj" : [
            {
                "somefield":"somedata"
            },
            {
                "somefield":"somedata"
            },
            ...
            ...
            ...
            {
                "somefield":"somedata"
            },
            {
                "somefield":"somedata"
            }
        ]
    }
}
              

Save a field to an object specifying the type of field:

              # CREATE A BLOCK FROM HTML STRING
- register_set: '<ul>
                    <li class="float">12.85</li>
                    <li class="int">158</li>
                </ul>'
- to_block
- find:
    path: ul
    do:
    # CREATE OBJECT WITH NAME `item`
    - object_new: item
    - find:
        path: .float
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SAVE VALUE OF THE REGISTER TO `somefloat` FIELD OF THE `item` OBJECT WITH SPECIFYING FIELD TYPE AS `float`
        - object_field_set:
            object: item
            field: somefloat
            type: float

    - find:
        path: .int
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SAVE VALUE OF THE REGISTER TO `someint` FIELD OF THE `item` OBJECT WITH SPECIFYING FIELD TYPE AS `int`
        - object_field_set:
            object: item
            field: someint
            type: int

    # SAVE OBJECT `item`
    - object_save:
        name: item
              
              {
    "item": {
        "somefloat": 12.85,
        "someint": 158
    }
}
              

Setting field value with joining old values:

              # CREATE BLOCK FROM HTML STRING
- register_set: '<ul class="int">
                    <li>125</li>
                    <li>158</li>
                </ul>
                <ul class="float">
                    <li>12.5</li>
                    <li>15.8</li>
                </ul>
                <ul class="default">
                    <li>sometext</li>
                    <li>15.8</li>
                </ul>'
- to_block
- find:
    path: ul.int
    do:
    # CREATE OBJECT WITH NAME `item`
    - object_new: item
    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `int` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: multint
            type: int
            joinby: "*" # MULTIPLY

    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `int` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: divint
            type: int
            joinby: "/" # DIVIDE

    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `int` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: subint
            type: int
            joinby: "-" # SUBSTRACT

    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `int` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: sumint
            type: int
            joinby: "+" # ADD

    # SAVE OBJECT `item`
    - object_save:
        name: item
              
              {
    "item": {
        "divint": 0,
        "multint": 19750,
        "subint": -33,
        "sumint": 283
    }
}
              
              # CREATE BLOCK FROM HTML STRING
- register_set: '<ul class="int">
                    <li>125</li>
                    <li>158</li>
                </ul>
                <ul class="float">
                    <li>12.5</li>
                    <li>15.8</li>
                </ul>
                <ul class="default">
                    <li>sometext</li>
                    <li>15.8</li>
                </ul>'
- to_block
- find:
    path: ul.float
    do:
    # CREATE OBJECT WITH NAME `item`
    - object_new: item
    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `float` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: multfloat
            type: float
            joinby: "*" # УМНОЖЕНИЕ
            precision: 2

    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `float` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: divfloat
            type: float
            joinby: "/" # ДЕЛЕНИЕ
            precision: 3

    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `float` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: subfloat
            type: float
            joinby: "-" # ВЫЧИТАНИЕ
            precision: 4

    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING FIELD TYPE AS `float` AND OPERATION FOR JOINING
        - object_field_set:
            object: item
            field: sumfloat
            type: float
            joinby: "+" # СЛОЖЕНИЕ
            precision: 5

    # SAVE OBJECT `item`
    - object_save:
        name: item
              
              {
    "item": {
        "divfloat": 0.7911392405063291,
        "multfloat": 197.5,
        "subfloat": -3.3000000000000007,
        "sumfloat": 28.3
    }
}
              
              # CREATE BLOCK FROM HTML STRING
- register_set: '<ul class="int">
                    <li>125</li>
                    <li>158</li>
                </ul>
                <ul class="float">
                    <li>12.5</li>
                    <li>15.8</li>
                </ul>
                <ul class="default">
                    <li>sometext</li>
                    <li>15.8</li>
                </ul>'
- to_block
- find:
    path: ul.default
    do:
    # CREATE OBJECT WITH NAME `item`
    - object_new: item
    - find:
        path: li
        do:
        # PARSE TEXT TO THE REGISTER
        - parse

        # SETTING OBJECT FIELD WITH SPECIFYING DELIMITER FOR JOINING
        - object_field_set:
            object: item
            field: default
            joinby: "*" # DELIMITER

    # SAVE OBJECT `item`
    - object_save:
        name: item
              
              {
    "item": {
        "default": "sometext*15.8"
    }
}
              

In the next chapter, we'll review the methods for working with the DOM structure.