Arch Linux

Basu

Member

From: Cornell University

Registered: 2006-12-15

Posts: 296

Website

Web searcher/scraper program

I started college this fall and I've got a job with the director of research. Some of my job is going to involve looking through the websites of other colleges for certain information as needed. I completed my first assignment by hand using google to search each college's website. But there are over 30 of them, and I don't want to search them one at a time. I want to write a program that will look through the websites all at once and give me a neat table of search results (say the top 3-5 for each site). It's ok if it's just an interface to Google's engine, I only want the results in a format that I can easily look them up to check if they are what I want and then copy/paste into a spreadsheet. Could you guys recommend what sort of tools / programming languages/ APIs I could use to make something like this?
For starters I'm going to look the Google API and see if I can automate searches via a python script. Any suggestions would be welcome. I don't mind if I have to teach myself a fair bit (that's what I'm in college for).

---

The Bytebaker -- Computer science is not a science and it's not about computers
Check out my open source software at Github