Free ecommerce scraper to extract product information from Asos store

Asos – the famous English fashion clothing and shoes online store, designed primarily for young people. The store sells more than 850 brands. This free ecommerce scraper will help you collect all information about products sold from the asos.com website.

Updated 07.09.2019 due to changes on asos website

Approx number of goods: 90000
Approx number of page requests: 100000
Recommended subscription plan: Small

PLEASE NOTE! The number of requests can exceed the number of products, because data about variations, images, etc. can be scraped from other resources and will require additional requests. Also part of the product data can be delivered using XHR requests, which also increases the total number of required page requests.

How to use the web scraper to extract data about goods and prices from asos.com

To use the ecommerce scraper for Asos store’s website, you must have an account with our Diggernaut service. You can just simply follow this comprehensive guide:

  1. Go through this registration link to open free account with Diggernaut
  2. After registering and confirming the email address, you will need to log in to your account
  3. Create a project with any name and description, if you do not know how to do it, please refer to our documentation
  4. Switch to the created project and create a digger with any name, if you do not know how to do it, please refer to our documentation
  5. Copy the following digger configuration to the clipboard and paste it into the digger you created, if you do not know how to do it, refer to our documentation
  6. Switch the mode of the digger from Debug to Active, if you do not know how to do it, please refer to our documentation
  7. Run your digger and wait until the completion, if you do not know how to do it, please refer to our documentation
  8. Download the scraped dataset in the format you need, if you do not know how to do it, please refer to our documentation

You can also setup a schedule for running your scraper and collect data regularly.

Scraping configuration for the digger

---
config:
    debug: 2
    agent: Firefox
do:
- link_add:
    url:
    - http://us.asos.com/women/a-to-z-of-brands/cat/?cid=1340
    - http://us.asos.com/men/a-to-z-of-brands/cat/?cid=1361
- variable_set:
    field: rip
    value: "yes"
- walk:
    to: links
    repeat_in_pool: 
    do:
    - find:
        path: body
        do:
        - parse
        - if:
            match: Access Denied
            do:
            - variable_set:
                field: rip
                value: "yes"
            - cookie_reset
            - proxy_switch
            else:
            - variable_set:
                field: rip
                value: "no"
            - pool_clear: main
            - find:
                path: ol[data-auto-id="brand-link"]>li>a
                do:
                - parse:
                    attr: href
                - link_add:
                    pool: main
            - variable_set:
                field: rip2
                value: "yes"
            - walk: 
                to: links
                repeat_in_pool: 
                pool: main
                do:
                - find:
                    path: body
                    do:
                    - parse
                    - if:
                        match: Access Denied
                        do:
                        - variable_set:
                            field: rip2
                            value: "yes"
                        - cookie_reset
                        - proxy_switch
                        else:
                        - variable_set:
                            field: rip2
                            value: "no"
                        - find:
                            path: a[data-auto-id="loadMoreProducts"]
                            do:
                            - parse:
                                attr: href
                            - normalize:
                                routine: url
                            - link_add:
                                pool: main
                        - find:
                            path: article[data-auto-id="productTile"]>a
                            do:
                            - parse:
                                attr: href
                            - link_add:
                                pool: sub
                        - cookie_reset
                        - proxy_switch
- variable_set:
    field: rip
    value: "yes"
- walk:
    to: links
    repeat_in_pool: 
    pool: sub
    do:
    - find:
        path: body
        do:
        - parse
        - if:
            match: Access Denied
            do:
            - variable_set:
                field: rip
                value: "yes"
            - cookie_reset
            - proxy_switch
            else:
            - variable_set:
                field: rip
                value: "no"
            - variable_clear: isP
            - variable_clear: allli
            - variable_clear: sdescr
            - variable_clear: li
            - variable_clear: json
            - variable_clear: id
            - find:
                path: script:matches(Pages/FullProduct)
                do:
                - variable_set:
                    field: isP
                    value: 1
                - parse:
                    filter:
                        - view\(\s*(.+),
                - normalize:
                    routine: replace_substring
                    args:
                        \\\': ''
                - normalize:
                    routine: unescape_html
                - variable_set: json
            - variable_get: isP
            - if:
                match: (1)
                do:
                - object_new: product
                - find: 
                    path: head
                    in: doc
                    do: 
                    - eval:
                        routine: js
                        body: '(function (){var d = new Date(); return d.toISOString()})();'
                    - object_field_set:
                        object: product
                        field: date
                    - static_get: url
                    - object_field_set:
                        object: product
                        field: url
                - find:
                    path: '#chrome-breadcrumb li'
                    slice: 1:-2
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - normalize:
                        routine: replace_matched
                        args:
                            A\s*To\s*Z\s*Of\s*Brands: ''
                    - if:
                        match: (\S)
                        do:
                        - object_field_set:
                            object: product
                            field: category
                            joinby: "|"
                - find:
                    path: .product-code > span
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - if:
                        match: (\S)
                        do:
                        - object_field_set:
                            object: product
                            field: sku
                - find:
                    path: meta[name="description"]
                    in: doc
                    do:
                    - parse:
                        attr: content
                    - space_dedupe
                    - trim
                    - if:
                        match: (\S)
                        do:
                        - object_field_set:
                            object: product
                            field: description
                - variable_get: json
                - normalize:
                    routine: replace_substring
                    args:
                        '\\\\': '\'
                - normalize:
                    routine: json2xml
                - to_block
                - find:
                    path: brandname
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - if:
                        match: (\S)
                        do:
                        - object_field_set:
                            object: product
                            field: brand
                - find:
                    path: body_safe > name
                    do:
                    - parse
                    - space_dedupe
                    - trim
                    - if:
                        match: (\S)
                        do:
                        - object_field_set:
                            object: product
                            field: name
                - find:
                    path: images
                    do:
                    - find:
                        path: colour
                        do:
                        - parse
                        - space_dedupe
                        - trim
                        - if:
                            match: (\S)
                            do:
                            - object_field_set:
                                object: product
                                field: variations
                                joinby: "|"
                    - find:
                        path: url
                        do:
                        - parse
                        - space_dedupe
                        - trim
                        - if:
                            match: (\S)
                            do:
                            - normalize:
                                routine: replace_substring
                                args:
                                    \s*$: ?scl=1
                            - object_field_set:
                                object: product
                                field: images
                                joinby: "|"
                - find:
                    path: price > current
                    do:
                    - parse:
                        filter:
                            - (\d+\.?\d*)
                    - if:
                        match: (\d)
                        do:
                        - object_field_set:
                            object: product
                            field: price
                            type: float
                        - register_set: USD
                        - object_field_set:
                            object: product
                            field: currency
                - object_save:
                    name: product
            - cookie_reset
            - proxy_switch

Sample of Asos scraped data

Below is a sample of a dataset with several products in JSON format (so you can easily review it and see data structure). The dataset can be downloaded as CSV, XLSX, XML, or any other text format using the templates.

[{
    "product": {
        "brand": "A-Gold-E",
        "category": "Women|A Gold E",
        "currency": "USD",
        "date": "2017-12-06T19:57:37.564Z",
        "description": "Shop A-GOLD-E Cigarette Low Waist Straight Leg Jean at ASOS. Discover fashion online.",
        "images": "http://images.asos-media.com/products/a-gold-e-cigarette-low-waist-straight-leg-jean/8450280-1-blue?scl=1|http://images.asos-media.com/products/a-gold-e-cigarette-low-waist-straight-leg-jean/8450280-2?scl=1|http://images.asos-media.com/products/a-gold-e-cigarette-low-waist-straight-leg-jean/8450280-3?scl=1|http://images.asos-media.com/products/a-gold-e-cigarette-low-waist-straight-leg-jean/8450280-4?scl=1",
        "name": "A-GOLD-E Cigarette Low Waist Straight Leg Jean",
        "price": 348,
        "sku": "1122820",
        "url": "http://us.asos.com/a-gold-e/a-gold-e-cigarette-low-waist-straight-leg-jean/prd/8450280?clr=blue&cid=20852",
        "variations": "Blue"
    }
}
,{
    "product": {
        "brand": "A-Gold-E",
        "category": "Women|A Gold E",
        "currency": "USD",
        "date": "2017-12-06T19:57:38.752Z",
        "description": "Shop A-GOLD-E 90s Mid Rise Loose Fit Jean at ASOS. Discover fashion online.",
        "images": "http://images.asos-media.com/products/a-gold-e-90s-mid-rise-loose-fit-jean/8450283-1-blue?scl=1|http://images.asos-media.com/products/a-gold-e-90s-mid-rise-loose-fit-jean/8450283-2?scl=1|http://images.asos-media.com/products/a-gold-e-90s-mid-rise-loose-fit-jean/8450283-3?scl=1|http://images.asos-media.com/products/a-gold-e-90s-mid-rise-loose-fit-jean/8450283-4?scl=1",
        "name": "A-GOLD-E 90s Mid Rise Loose Fit Jean",
        "price": 348,
        "sku": "1122848",
        "url": "http://us.asos.com/a-gold-e/a-gold-e-90s-mid-rise-loose-fit-jean/prd/8450283?clr=blue&cid=20852",
        "variations": "Blue"
    }
}
,{
    "product": {
        "brand": "A-Gold-E",
        "category": "Women|A Gold E",
        "currency": "USD",
        "date": "2017-12-06T19:57:39.241Z",
        "description": "Shop A-GOLD-E Jamie High Rise Mom Jean at ASOS. Discover fashion online.",
        "images": "http://images.asos-media.com/products/a-gold-e-jamie-high-rise-mom-jean/8450298-1-blue?scl=1|http://images.asos-media.com/products/a-gold-e-jamie-high-rise-mom-jean/8450298-2?scl=1|http://images.asos-media.com/products/a-gold-e-jamie-high-rise-mom-jean/8450298-3?scl=1|http://images.asos-media.com/products/a-gold-e-jamie-high-rise-mom-jean/8450298-4?scl=1",
        "name": "A-GOLD-E Jamie High Rise Mom Jean",
        "price": 348,
        "sku": "1122850",
        "url": "http://us.asos.com/a-gold-e/a-gold-e-jamie-high-rise-mom-jean/prd/8450298?clr=blue&cid=20852",
        "variations": "Blue"
    }
}
,{
    "product": {
        "brand": "A-Gold-E",
        "category": "Women|A Gold E",
        "currency": "USD",
        "date": "2017-12-06T19:57:39.689Z",
        "description": "Shop A-GOLD-E Sophie High Rise Crop Skinny Jean at ASOS. Discover fashion online.",
        "images": "http://images.asos-media.com/products/a-gold-e-sophie-high-rise-crop-skinny-jean/8450276-1-blue?scl=1|http://images.asos-media.com/products/a-gold-e-sophie-high-rise-crop-skinny-jean/8450276-2?scl=1|http://images.asos-media.com/products/a-gold-e-sophie-high-rise-crop-skinny-jean/8450276-3?scl=1|http://images.asos-media.com/products/a-gold-e-sophie-high-rise-crop-skinny-jean/8450276-4?scl=1",
        "name": "A-GOLD-E Sophie High Rise Crop Skinny Jean",
        "price": 215,
        "sku": "1122810",
        "url": "http://us.asos.com/a-gold-e/a-gold-e-sophie-high-rise-crop-skinny-jean/prd/8450276?clr=blue&cid=20852",
        "variations": "Blue"
    }
}]
Mikhail Sisin: Co-founder of cloud-based web scraping and data extraction platform Diggernaut. Over 10 years of experience in data extraction, ETL, AI, and ML.
Related Post