There was an old project I took as a freelancer before. It was about scraping the web pages of The Law Society of Hong Kong. They have put up a long comprehensive list of registered members.
Literally listing out right there inviting just anyone to scrape, all 10,288 of them.
The page structure were in simple table format in plain HTML, no Javascript to run before getting any useful value.
My way
For folks reading out there who’re probably also an engineer, at this momemnt you must’ve formulated various ways to scrape the page from HTML. You may have a preference towards a certain framework or language that you’ve used or heard about before.
So did I, back then the Python web crawling / scraping framework Scrapy were perfect for the job. (by the way Scrapy is one brillient framework… I scraped my university’s timetable and built “UST Break-in”)
It was done overnight and I earned some money for some better dinners for a week. Everybody was happy.
The Layman way
Later I came to learn from a guy, the COO Angus in EventXtra. Briefly speaking he is the “brute-force it” sort of guy who simply broke the lock in a room escape game (which didn’t go down well).
As Angus, he’d Google it and eventually land on a site called ‘import.io’.
Angus simply plug the links in and TaDa! All data from the site is automatically scraped by import.io and instantly available to Angus in spreadsheet formats.
Really this felt like cheating when I learned about import.io from Angus.
Angus could have earned better dinners for the week in a much shorter time.
Reflection
Angus and me have different mindsets; he is an operation guy with a couple years of hard-earned experiece in founding EventXtra whilst I am strictly from several years of engineering background.
As an engineer my ego wasn’t hurt, instead I was intrigued to employ what Angus’d have employed in many engineering problems I am facing daily.
Building a program to automate and perform a computationally expensive task is one thing that all software engineers are more than prepared for. But doing things the layman way and achieve virtually the same result in lower cost is much much better.
Of course not everything has an easy way out, still I think it doesn’t hurt if we could just google it before jumping to the solution. This way we could potentially have saved lots of time and money.
Angus is a layman.
Angus googles stuff and get things done at lower cost.
Angus is smart.
Be like Angus.