Extract data from iCal? It couldn’t be easier.<\/p>\n
Today we write a script for scraping various resources which uses files in an iCal format to send event data.<\/p>\n
Apple invented this format, and now many websites let you export the calendar events in this format. In this case, you do not need to scrape the website and parse the HTML. You only need to get and parse a file in iCal format. It makes the whole process much more manageable.<\/p>\n
Diggernaut.com<\/a> natively support this format and automatically converts it to XML. So we can work with iCal data as with a regular HTML page.<\/p>\n Let’s see how it works by extracting data from Science Fiction Conventions calendar, found by me on the icalshare.com website. Let’s start to write the config by defining some basic settings. First, we need to set a digger to the debug level 2. This is the only way we can see the source code of the converted file, and we need to check it so we could write the navigation instructions for walking to blocks with data we need to extract and collect.<\/p>\n Calendar file we are going to use is: https:\/\/www.google.com\/calendar\/ical\/lirleni%40gmail.com\/public\/basic.ics<\/a><\/p>\n So, our config starts with:<\/p>\n In this code we set Debug level 2, configure digger to use Firefox as browser name and get iCal file. Now we need to login to our account at Diggernaut.com<\/a>, select existing or create a new project and then create new digger, and save config we wrote to the digger.<\/p>\n Make sure that digger is set to the Debug mode (in the Status column you should see Debug). If it’s not so, you need to switch digger to debug mode using the selector in the Status column. Then we need to start the digger and wait until it finishes the run. When it’s done, we need to check logs by clicking on the “Log” button.<\/p>\n As you can see, page structure consists of We are not going to pick all these fields, let’s get only summary, description, start date\/time, end date\/time and location. It’s very easy to do: first, we walk to the event block, create a data object, then we walk to the fields blocks, parse data and save it to the object fields and finally save the data object.<\/p>\n Lets put our config to the digger and run it. Once it’s done, let’s jump to the Data section and make sure that data we have scraped is in a good state. You should see there something like:<\/p>\n If data is good, let’s switch our digger to the Active mode, as in Debug mode you cannot download data, all you can do in Debug mode is to review a limited set of data. Let’s start digger again and wait for completion. Then go to the Data section again and download data in the format we need. The sample in XLSX format you can download here<\/a>.<\/p>\n---\nconfig:\n debug: 2\n agent: Firefox\ndo:\n- walk:\n to: https:\/\/www.google.com\/calendar\/ical\/lirleni%40gmail.com\/public\/basic.ics\n do:\n - stop<\/code><\/pre>\n<html>\n\n<head><\/head>\n\n<body>\n <body_safe>\n <event>\n <alarmtime>0s<\/alarmtime>\n <class>PUBLIC<\/class>\n <created>2009-03-07 20:15:36 +0000 UTC<\/created>\n <description>Gaming Convention<\/description>\n <end>2011-01-31 00:00:00 +0000 UTC<\/end>\n <id>2a994c3e3b80f5af6e8fa178a3af45d4<\/id>\n <importedid>b82f342c-0b54-11de-b762-000d936372a6<\/importedid>\n <location>Champaign IL (USA)<\/location>\n <modified>2011-02-20 23:29:49 +0000 UTC<\/modified>\n <rrule><\/rrule>\n <sequence>4<\/sequence>\n <start>2011-01-28 00:00:00 +0000 UTC<\/start>\n <status>CONFIRMED<\/status>\n <summary>WinterWar 38<\/summary>\n <wholedauyevent>true<\/wholedauyevent>\n <\/event>\n <event>\n <alarmtime>0s<\/alarmtime>\n <class><\/class>\n <created>2009-01-10 17:33:11 +0000 UTC<\/created>\n <description>Gaming convention<\/description>\n <end>2011-08-08 00:00:00 +0000 UTC<\/end>\n <id>5c8f16d772ede097822e73a0c2e51c6c<\/id>\n <importedid>356F0F0C-FE52-47A8-AEAB-8E78F57D4F52<\/importedid>\n <location>Indianapolis IN<\/location>\n <modified>2011-02-20 23:29:48 +0000 UTC<\/modified>\n <rrule><\/rrule>\n <sequence>10<\/sequence>\n <start>2011-08-04 00:00:00 +0000 UTC<\/start>\n <status>CONFIRMED<\/status>\n <summary>GenCon<\/summary>\n <wholedauyevent>true<\/wholedauyevent>\n <\/event> ...\n<\/code><\/pre>\n<event>\n <alarmtime>0s<\/alarmtime>\n <class>PUBLIC<\/class>\n <created>2009-03-07 20:15:36 +0000 UTC<\/created>\n <description>Gaming Convention<\/description>\n <end>2011-01-31 00:00:00 +0000 UTC<\/end>\n <id>2a994c3e3b80f5af6e8fa178a3af45d4<\/id>\n <importedid>b82f342c-0b54-11de-b762-000d936372a6<\/importedid>\n <location>Champaign IL (USA)<\/location>\n <modified>2011-02-20 23:29:49 +0000 UTC<\/modified>\n <rrule><\/rrule>\n <sequence>4<\/sequence>\n <start>2011-01-28 00:00:00 +0000 UTC<\/start>\n <status>CONFIRMED<\/status>\n <summary>WinterWar 38<\/summary>\n <wholedauyevent>true<\/wholedauyevent>\n<\/event>\n<\/code><\/pre>\n---\nconfig:\n agent: Firefox\ndo:\n- walk:\n to: https:\/\/www.google.com\/calendar\/ical\/lirleni%40gmail.com\/public\/basic.ics\n do:\n - find:\n path: event\n do:\n - object_new: event\n - find:\n path: summary\n do:\n - parse\n - normalize:\n routine: replace_substring\n args:\n \\\\: ''\n - object_field_set:\n object: event\n field: summary\n - find:\n path: description\n do:\n - parse\n - normalize:\n routine: replace_substring\n args:\n \\\\: ''\n - object_field_set:\n object: event\n field: description\n - find:\n path: location\n do:\n - parse\n - normalize:\n routine: replace_substring\n args:\n \\\\: ''\n - object_field_set:\n object: event\n field: location\n - find:\n path: start\n do:\n - parse:\n filter: (\\d{4}-\\d{2}-\\d{2}\\s+\\d{2}:\\d{2}:\\d{2})\n - object_field_set:\n object: event\n field: start_date\n - find:\n path: end\n do:\n - parse:\n filter: (\\d{4}-\\d{2}-\\d{2}\\s+\\d{2}:\\d{2}:\\d{2})\n - object_field_set:\n object: event\n field: end_date\n - object_save:\n name: event<\/code><\/pre>\nItem #1 \nstart_date 2011-01-28 00:00:00\nsummary WinterWar 38\ndescription Gaming Convention\nend_date 2011-01-31 00:00:00\nlocation Champaign IL (USA)\nItem #2 \nstart_date 2011-08-04 00:00:00\nsummary GenCon\ndescription Gaming convention\nend_date 2011-08-08 00:00:00\nlocation Indianapolis IN\nItem #3 \nstart_date 2011-03-11 00:00:00\nsummary Madicon 20\ndescription Science Fiction, with a large proportion of Gaming\nend_date 2011-03-14 00:00:00\nlocation Harrisonburg VA, USA\n<\/code><\/pre>\n