Automatise data export from website

orschiro · 2013-05-21 14:33:15

Hello,

I stuck with an issue regarding data export from scopus.com. I have 61 keywords for which I want to export (as csv) number of documents by year, country, document type, subject area. Doing that by hand would be extremely tedious. Thus my question if this can be automatised. Unfortunately scopus does not provide an API to access the data directly.

So, the process to be automatised would be:

1. Open http://www.scopus.com/home.url
2. Enter the keyword in "search for"
3. Click on "search"
4. Click on "Analyze results"
5. Click on tab "year" and "export". Save the file
6. Click on tab "country" and "export". Save the file
6. Click on tab "document type" and "export". Save the file
6. Click on "subject area" and "export". Save the file

Do programs exist that help me retrieving the data?

Thanks!

aesiris · 2013-05-21 15:59:47

You could achieve your goal with bash and wget with some combination of --save-cookies --post-data.

But be aware that scopus is run by Elsevier, whom many view as the most evil of the publishing companies. Think twice before doing something that may annoy them.

Stebalien · 2013-05-21 22:25:35

Unfortunately, they use Flash so you can't write a program to do this easily. I would try iMacros (a Firefox extension).

orschiro · 2013-05-22 13:55:37

Thanks for your suggestions. I will have a more in-depth look at iMacros but so far it failed to finally export the data.

Arch Linux

#1 2013-05-21 14:33:15

Automatise data export from website

#2 2013-05-21 15:59:47

Re: Automatise data export from website

#3 2013-05-21 22:25:35

Re: Automatise data export from website

#4 2013-05-22 13:55:37

Re: Automatise data export from website

Board footer