Monday, April 7, 2025

Kicking Horse and Red Mountain Resorts Added

Phew! These two took a fair bit of work to get running. Unlike all of the previous resorts scraping these two required the use of additional software as their content is dynamically generated. The page must be prerendered before being able to scrape the relevant data from them. This posed quite a challenge and I struggled for quite some time trying to get this to work on the pi zero. Unfortunately the hardware doesn't seem capable of performing the pre-rendering so I had to outsource it to my PC. The way this works is using Splash on a more power system to remotely pre-render the dynamic content and then scraping the results using the traditional method I had been previously using. This will have to be the way it is for any future resorts that use dynamically generated content. I have it setup to work on my local network for now but in the future I plan to have a dedicated server hosting this software so that all Snow Scrapers will be able to contact it in order to request the generated html necessary to scrape the sites. I will update the source code to include the new hills I've added once they have shut down for the season. I have one more left on my list to complete before that happens, Whitewater. Unfortunately I have not been able to use this method to successfully scrape SilverStar's website. If anyone has any ideas id be open to try them but so far I'm at an impasse. 

No comments:

Post a Comment

Gemini AI Generated Snow Scraper Promo

 I was fooling around with the latest Gemini model and came up with this short promo video for the Snow Scraper. Its not perfect, definitely...