Mikhail Sisin Co-founder of cloud-based web scraping and data extraction platform Diggernaut. Over 10 years of experience in data extraction, ETL, AI, and ML.

Scraping fashion retail data for machine learning purposes from Bloomingdales

4 min read

Scraping fashion retail data for machine learning purposes from Bloomingdales

Bloomingdale’s is a multi-brand store chain, founded in April 1872. At the moment, the network is owned by Macy’s, Inc. Using the parser below, you can scrape a lot of fashion retail data, including prices and images the bloomingdales.com online store. This data can be used for brand research, computer vision and any other machine learning problematic.

Approx number of goods: 350000
Approx number of page requests: 350000
Recommended subscription plan: Medium

PLEASE NOTE! The number of requests can exceed the number of products, because data about variations, images, etc. can be scraped from other resources and will require additional requests. Also part of the product data can be delivered using XHR requests, which also increases the total number of required page requests.

How to use the web scraper to extract data about goods and prices from bloomingdales.com

To use the web scraper for Bloomingdale’s store website, you must have an account with our Diggernaut service. You can just simply follow this comprehensive guide:

  1. Go through this registration link to open free account with Diggernaut
  2. After registering and confirming the email address, you will need to log in to your account
  3. Create a project with any name and description, if you do not know how to do it, please refer to our documentation
  4. Switch to the created project and create a digger with any name, if you do not know how to do it, please refer to our documentation
  5. Copy the following digger configuration to the clipboard and paste it into the digger you created, if you do not know how to do it, refer to our documentation
  6. PLEASE NOTE! Basic proxy servers may not work with this site and you may need to use your own proxy servers. You will need to specify proxy server to the specific location in the digger configuration as commented. If you feel confused about this item, please contact us using the support system or using our online chat, we will be glad to help you.
  7. Switch the mode of the digger from Debug to Active, if you do not know how to do it, please refer to our documentation
  8. Run your digger and wait until the completion, if you do not know how to do it, please refer to our documentation
  9. Download the scraped dataset in the format you need, if you do not know how to do it, please refer to our documentation

You can also setup a schedule for running your scraper and collect data regularly.

Scraping configuration for the digger

---
config:
    debug: 2
    agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
    proxy: #USE YOUR OWN PROXY LIST HERE, READ DOCUMENTATION ON HOW TO USE IT OR CONTACT US
do:
- walk:
    to: https://www.bloomingdales.com/index
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
    do:
    - find: 
        path: '#globalFlyouts a'
        do: 
        - pool_clear: main
        - parse:
            attr: href
            filter:
                - \?id=(\d+)
        - variable_set: pur
        - variable_set: 
            field: first
            value: 1
        - if:
            match: (\d)
            do:
            - register_set: https://www.bloomingdales.com/api/navigation/categories/facet?categoryId=<%pur%>&facet=false&pageIndex=1&bcomNavPPP=undefine
            - link_add:
                pool: main
            - walk:
                to: links
                pool: main
                headers:
                    accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                    accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
                    cache-control: no-cache
                do:
                - find: 
                    path: productids
                    do: 
                    - parse
                    - if:
                        match: (\d)
                        do:
                        - register_set: https://www.bloomingdales.com/shop/product/?ID=<%register%>&CategoryID=<%pur%>
                        - link_add:
                            pool: sub
                - find: 
                    path: productcount
                    do: 
                    - parse
                    - if:
                        match: (\d)
                        do:
                        - variable_set: count
                        - variable_get: first
                        - if:
                            match: (\d+)
                            do:
                            - variable_clear: first
                            - eval:
                                routine: js
                                body: (function () {var pages = []; for (var i=2; i*90 <= <%count%>; i++) {pages.push(i)}; return pages.join(",");})();
                            - to_block
                            - split:
                                context: text
                                delimiter: ","
                            - find: 
                                path: .splitted 
                                do: 
                                - parse
                                - register_set: https://www.bloomingdales.com/api/navigation/categories/facet?categoryId=<%pur%>&facet=false&pageIndex=<%register%>&bcomNavPPP=undefine
                                - link_add:
                                    pool: main
- walk:
    to: links
    headers:
        accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
        accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
        cache-control: no-cache
    pool: sub
    do:
    - proxy_switch
    - cookie_reset
    - variable_clear: allli
    - variable_clear: descr
    - variable_clear: n
    - object_new: product
    - find: 
        in: doc
        path: head 
        do: 
        - eval:
            routine: js
            body: '(function (){var d = new Date(); return d.toISOString()})();'
        - object_field_set:
            object: product
            field: date
        - static_get: url
        - filter:
            args:
             - (.+\?[idID]+=\d+)\&
        - object_field_set:
            object: product
            field: url
        - register_set: Bloomingdale
        - object_field_set:
            object: product
            field: brand
    - find: 
        path: '#productId' 
        do: 
        - parse:
            attr: value
        - if:
            match: (\d)
            do:
            - object_field_set:
                object: product
                field: sku
    - find: 
        path: '#brandNameLink' 
        do: 
        - parse
        - space_dedupe
        - trim
        - object_field_set:
            object: product
            field: brand
    - find: 
        path: '#productName, #productTitle' 
        do: 
        - variable_get: n
        - if:
            match: (\d)
            else:
            - parse
            - space_dedupe
            - trim
            - object_field_set:
                object: product
                field: name
            - variable_set: 
                field: n
                value: 1
    - find: 
        path: .selectedFOB
        do: 
        - parse
        - space_dedupe
        - trim
        - normalize:
            routine: lower
        - object_field_set:
            object: product
            field: category
            joinby: "|"
    - find: 
        path: 'script#pdp_data' 
        do: 
        - parse
        - normalize:
            routine: json2xml
        - to_block
        - find: 
            path: colorwayadditionalimages > *, colorwayprimaryimages > *, additionalimages, imagesource
            do: 
            - parse
            - split:
                context: text
                delimiter: ','
            - find: 
                path: .splitted 
                do: 
                - parse
                - if:
                    match: (\S)
                    do:
                    - register_set: https://images.bloomingdales.com/is/image/BLM/products/<%register%>
                    - object_field_set:
                        object: product
                        field: images
                        joinby: "|"
        - find: 
            path: colorfamily > * 
            do: 
            - parse
            - if:
                match: (\S)
                do:
                - object_field_set:
                    object: product
                    field: variations
                    joinby: "|"
        - find: 
            path: product > seokeywords
            slice: 0:-2
            do: 
            - parse
            - space_dedupe
            - trim
            - normalize:
                routine: lower
            - if:
                match: (\S)
                do:
                - object_field_set:
                    object: product
                    field: category
                    joinby: "|"
        - find: 
            path: longdescription 
            do: 
            - parse
            - space_dedupe
            - trim
            - if:
                match: (\S)
                do:
                - object_field_set:
                    object: product
                    field: description
        - find: 
            path: product > price 
            do: 
            - parse
            - space_dedupe
            - trim
            - if:
                match: (\d)
                do:
                - object_field_set:
                    object: product
                    field: price
                    type: float
                - register_set: USD
                - object_field_set:
                    object: product
                    field: currency
    - object_save:
        name: product

Sample of scraped data

Below is a sample of a dataset with several products in JSON format (so you can easily review it and see data structure). The dataset can be downloaded as CSV, XLSX, XML, or any other text format using the templates.

[{
    "product": {
        "brand": "Michael Aram",
        "category": "home|#homegoals",
        "currency": "USD",
        "date": "2017-12-07T21:43:20.868Z",
        "description": "In the designer's own words, the Molten collection is distinguished by \"streamlined, timeless shapes... objects which reverberate with the skill of their maker and yet do not fit into a traditional interpretation of craft. The pieces possess a soulfulness and organic energy only possible through the handmade process.\"",
        "images": "https://images.bloomingdales.com/is/image/BLM/products/5/optimized/8722225_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/5/optimized/8722225_fpx.tif",
        "name": "Michael Aram Molten 5-Piece Place Setting",
        "price": 80,
        "sku": "1197985",
        "url": "https://www.bloomingdales.com/shop/product/michael-aram-molten-5-piece-place-setting?ID=1197985",
        "variations": "Silver"
    }
}
,{
    "product": {
        "brand": "Lagostina",
        "category": "home|#homegoals",
        "currency": "USD",
        "date": "2017-12-07T21:43:24.007Z",
        "description": "Showcasing Lagostina's core values of impeccable Italian craftsmanship, technical innovation and elegant design, this ultrastrong grill pan triple-wall construction and a sturdy grooved surface for perfect grilling and searing.",
        "images": "https://images.bloomingdales.com/is/image/BLM/products/6/optimized/8694046_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/6/optimized/8694046_fpx.tif",
        "name": "Lagostina Accademia Bistecchiera 11\" Grill Pan",
        "price": 180,
        "sku": "1205514",
        "url": "https://www.bloomingdales.com/shop/product/lagostina-accademia-bistecchiera-11-grill-pan?ID=1205514"
    }
}
,{
    "product": {
        "brand": "Iittala",
        "category": "home|#homegoals",
        "currency": "USD",
        "date": "2017-12-07T21:43:26.117Z",
        "description": "Designed by Kaj Franck for Iittala, the Kartio carafe is a perfect balance of pure material and simple geometric form. Stripped of the superfluous, it is clean and timeless.",
        "images": "https://images.bloomingdales.com/is/image/BLM/products/5/optimized/1232615_fpx.tif|https://images.bloomingdales.com/is/image/BLM/products/5/optimized/1232615_fpx.tif",
        "name": "Iittala Kartio Carafe/Pitcher, 1 quart",
        "price": 100,
        "sku": "1239359",
        "url": "https://www.bloomingdales.com/shop/product/iittala-kartio-carafe-pitcher-1-quart?ID=1239359",
        "variations": "White"
    }
}]
Mikhail Sisin Co-founder of cloud-based web scraping and data extraction platform Diggernaut. Over 10 years of experience in data extraction, ETL, AI, and ML.

Leave a Reply

Your email address will not be published. Required fields are marked *