Using curl for uploading file, need help.

lolilolicon · 2009-05-18 09:34:04

trying to upload a file to sendspace.com, I used this

curl -L -H Expect: -F file_0=@"$1" -F xml=yes  -F terms=1 http://www.sendspace.com/

similarly:

curl -F userfile=@"$1" -L  http://up3ah1.uploading.com/upload.php

Some options may be unnecessary, but I'm sure there's some option necessary that I missed out.

I'm pretty sure this got the file uploaded, since it took longer when uploading a bigger file.

But I did not get the resulting link, which makes all the uploading meaningless.

Last edited by lolilolicon (2009-05-19 05:04:12)

Procyon · 2009-05-19 11:14:05

You skipped some forms, you have to submit them in the right order and with the data you got from the original page.

This worked for me, I am not sure if some of it is redundant:

touch cookie.jar
curl -c cookie.jar 'http://www.sendspace.com' | grep -i -E '(form|input)'
curl --trace-ascii log.txt -b cookie.jar -L -H 'Expect: ' -A 'mozilla/4.0' -e 'http://www.sendspace.com' -F 'MAX_FILE_SIZE=314572800' -F 'UPLOAD_IDENTIFIER=1991058540.1242730800.91740358.23.0' -F 'DESTINATION_DIR=1' -F 'js_enabled=0' -F 'signature=2ddbfde48449acff69103ae8cfa00695' -F 'file_0=@file.txt' -F 'desc0=text' -F 'terms=1' -F 'upload1280=Upload%20File' 'http://fs09u.sendspace.com/processupload.html'

The upload1280 is for the submit button. I'm not sure if it's necessary. The name, upload1280, is from the name attribute at the start of the form tag, the submit input form didn't have a name attribute.

lolilolicon · 2009-05-19 13:36:00

Procyon, you're always there to help, really nice of you ^^
So, I modified yours to this ugly code. Just to make sure all the form value are valid.
This piece of code is fat, I believe I've done something stupid. Would you modify it so that it gets rid of all the unnecessary blah blahs.
Since you are the script man, always got good idea and clean code!

#/bin/bash

myfile="$1"

touch cookie.jar
curl -c cookie.jar 'http://www.sendspace.com' | grep -i -E '(multipart|<input)' | sed '2,$s/.*name="\([^"]*\)".* value="\([^"]*\)".*/\1=\2/' | sed 's/.*action="\([^"]*\)".*name="\([^"]*\)" id.*/\1\n\2/' | head -7 | sed '2s/$/=Upload%20File/' > args.tmp

var_args=$( echo -n $( sed '1,2d' args.tmp | sed 's/^/ -F "/;s/$/"/' ) )
var_upload=$( sed '2!d' args.tmp )
var_url=$( sed '1!d' args.tmp )

final_cmd=$( echo curl --trace-ascii log.txt -b cookie.jar -L -H "'"Expect: "'" -A "'"mozilla/4.0"'" -e "'"http://www.sendspace.com"'" $var_args  -F file_0=@"'""$myfile""'" -F "'"desc0=blah"'" -F "'"terms=1"'" -F "'"$var_upload"'" $var_url )

del_url=$( eval $final_cmd | grep http://www.sendspace.com/delete | cut -d\" -f 8 )
dl_url=$( echo $del_url | sed 's/\(.*\)\/delete\/\([^\/]*\).*/\1\/file\/\2/' )

echo $dl_url
echo $del_url

Last edited by lolilolicon (2009-05-19 13:57:58)

Procyon · 2009-05-19 14:39:00

I tried it a few more times and it seems a lot more tolerant, cookies and unique signature might not be needed.

Could you try the same values I used:

file="$1"

curl -F 'MAX_FILE_SIZE=314572800' -F 'UPLOAD_IDENTIFIER=1991058540.1242730800.91740358.23.0' -F 'DESTINATION_DIR=1' -F 'js_enabled=0' -F 'signature=2ddbfde48449acff69103ae8cfa00695' -F "file_0=@$file" -F 'desc0=text' -F 'terms=1' 'http://fs09u.sendspace.com/processupload.html'

About getting data from html, I am still looking into html2 from xml2 in AUR (I mentioned it before right), it may make things easier but in a script it's quite complicated.

EDIT: BTW, the above code is still working for me after some hours and I uploaded several files with it.

Last edited by Procyon (2009-05-19 14:40:53)

lolilolicon · 2009-05-19 18:31:48

Heh, maybe they just used some trick to check wether these parameters match one another, say, they don't mind whether 'UPLOAD_IDENTIFIER' is 1 or 2, as long as 'signature' returns "true"...

I'm recently trying to write simple scripts to upload files to filehosting sites. I find some sites are really simple, but some adds some identifier code or prefix some parameter with a complex ever-changing variable. Some use javascripts which I've no clue how to deal with, others use completely flash, looks fancy but can't enjoy... Complex ><

Procyon · 2009-05-20 00:07:08

Sendspace is nicely light weight. Though speed/slots suffer. How is mediafire, it has a basic uploader too.

As an experiment I tried to submit a 320MB file by increasing MAX_FILE_SIZE, but after it finished uploading nothing happened (returned a blank page I think)

Is the script working now? Maybe it is a good idea to get a new signature each time, but at least it can be simplified now without the submit input and cookies. And I realize html2 isn't very convenient if you want to share the script. There's nothing wrong with many sed lines. You should get rid of the eval and all those quotes though (output the receive website to a new temporary file). And I don't see how the variables at the end get the right value, they should be separate greps, right?

lolilolicon · 2009-05-20 03:47:55

Can you load this page? I got this link after uploading but I can't load it...
http://www.sendspace.com/file/kzl661

The last dl_rul del_url are PAGE urls for downloading and deleting.
(They don't have to be separate greps, they're tweens)
Maybe this will get me the direct download url of the file?

<input type="hidden" name="files_params" value="f=09a386f93737adf5603648414ae8617c&l=PTImIgZ6KYKpYP4kVig0vaZw6aF7">

Can't try it out myself...network here is broke.

***
Why I used eval was this won't work for me... Why, I don't clearly get it..

myfile="$1"
### Fetch all the parameters needed to fill in the forms, and write them to file args.tmp
curl 'http://www.sendspace.com' | grep -i -E '(multipart|<input)' | sed '2,$s/.*name="\([^"]*\)".* value="\([^"]*\)".*/\1=\2/' | sed 's/.*action="\([^"]*\)".*name="\([^"]*\)" id.*/\1\n\2/' | head -7 | sed '2s/$/=Upload%20File/' > args.tmp

var_args=$( echo -n $( sed '1,2d' args.tmp | sed 's/^/ -F "/;s/$/"/' ) )
var_upload=$( sed '2!d' args.tmp )
var_url=$( sed '1!d' args.tmp )

### use echo to see what these var_*s get
echo '######'
echo $var_args
echo $var_upload
echo $var_url
echo '######'

### This is the final command. Which needs fix.
curl $var_args  -F file_0=@"$myfile" -F 'desc0=blah' -F 'terms=1' -F $var_upload $var_url

The question is the last line, I got all the var_*s right, and if I manually replace them with the values they get, like this:

curl -F "MAX_FILE_SIZE=314572800" -F "UPLOAD_IDENTIFIER=1981194826.1242788992.7CCF6314.15.0" -F "DESTINATION_DIR=13" -F "js_enabled=0" -F "signature=c65708a8b4cdff1a71e0c0a467094e72"  -F file_0=@file.txt -F 'desc0=blah'  -F 'terms=1' -F upload4989=Upload%20File http://fs02u.sendspace.com/processupload.html

and it works.
So bash didn't interpret those var_*s as expected... I think, for example "-F"s are just escaped so they don't get passed to curl as an option.
that eval line looks really ugly. Gotta fix it! And how can I do that?

Last edited by lolilolicon (2009-05-20 03:52:56)

lolilolicon · 2009-05-20 04:03:23

So I did it this way

echo $var_args  -F file_0=@\"$myfile\" -F \'desc0=blah\' -F \'terms=1\' $var_url | xargs curl

So now I get it like this:

#/bin/bash
#returns download/delete PAGE url
#filesize limit:: 300MB

myfile="$1"
args_tmp=/tmp/args.tmp
curl -s 'http://www.sendspace.com' | grep -i -E '(multipart|<input)' | sed '2,$s/.*name="\([^"]*\)".* value="\([^"]*\)".*/\1=\2/' | sed 's/.*action="\([^"]*\)".*name="\([^"]*\)" id.*/\1\n\2/' | head -7 | sed '2s/$/=Upload%20File/' > $args_tmp

var_args=$( echo -n $( sed '1,2d' $args_tmp | sed 's/^/ -F "/;s/$/"/' ) )
var_upload=$( sed '2!d' $args_tmp )
var_url=$( sed '1!d' $args_tmp )

echo $var_args  -F file_0=@\"$myfile\" -F \'desc0=blah\' -F \'terms=1\' $var_url | xargs curl -s | grep http://www.sendspace.com/delete | cut -d\" -f 8 | sed 'p' | sed '1s/\(.*\)\/delete\/\([^\/]*\).*/\1\/file\/\2/'

example:

$ bash sendspace.sh advert.gif 
http://www.sendspace.com/file/e8ef0b
http://www.sendspace.com/delete/e8ef0b/9276e07ea551917927e052b4a22e6358

Last edited by lolilolicon (2009-05-20 04:23:10)

lolilolicon · 2009-05-20 08:30:44

So...what's missing here?

#!/bin/bash

my_file="$1"
my_id=$( curl -s http://gigasize.com/ | grep 'UPLOAD_IDENTIFIER' | cut -d\" -f 6 )

echo $my_id

curl -L -F UPLOAD_IDENTIFIER="$my_id" -F i_file=@"$my_file" -F 'titlel=blah' -F 'description=blah' -F 'numupfiles=1'  -F 'accept=1' "http://gigasize.com/up.php?sid="$my_id"&description="

there file is uploaded, pretty for sure.
But curl returns nothing when upload is done.

Last edited by lolilolicon (2009-05-20 08:31:08)

Procyon · 2009-05-20 10:11:35

Good to see it's working.

Can you load this page? I got this link after uploading but I can't load it...
http://www.sendspace.com/file/kzl661

Yeah it works, the advert.gif too. Probably sendspace acting up.

So bash didn't interpret those var_*s as expected... I think, for example "-F"s are just escaped so they don't get passed to curl as an option.

I think the " inside the arguments is a problem too. E.g. this won't work: touch file.txt; var=$(sed 's/.*/"&"/' <<< file.txt); stat $var

http://gigasize.com/

Gigasize doesn't seem to in firefox without js either.

lolilolicon · 2009-05-20 10:19:07

Gigasize doesn't seem to in firefox without js either.

So... how to handle this? Experiencing same problem with megashare.com

Procyon · 2009-05-20 12:17:40

lolilolicon wrote:

Gigasize doesn't seem to in firefox without js either.
So... how to handle this? Experiencing same problem with megashare.com

I have no idea, can javascript work completely beyond POST and such? Maybe it just adds some information to the form destination URL (because it ends in '=' ). I looked at some of the js files, but nothing really stands out to me.

lolilolicon · 2009-05-20 12:37:24

I googled a bit. curl cannot handle js, what javascripts do needs to be translated to curl or so.
"Perl with the www lib can handle java" somebody talked about this...well, I don't know perl.
Is there a tiny tool that would help curl a bit for handling js? That'd be nice.

Another quesion :

oops, it works here. Hah no problem then.

Last edited by lolilolicon (2009-05-20 12:38:33)

Procyon · 2009-05-20 13:18:30

Don't post NSFW.

There is a function in lib.js that is called after pressing the submit button:

function __store_endUpload(fileid, pass) {
if (__store_updater) {
__store_updater.stop();
}
Element.hide($('progresscontainer'));
Element.show($('uploadcompleted'));
document.location = '/index.php?f=' + fileid + '&pass=' + pass;
}

Called by:

function __store_updateProgress(pb ,ps, fn ,pt, req) {
...
var mes = req.responseText;
...
var data = mes.substring(5).split(' ');
...
__store_endUpload(data[1], data[2]);
...

Where 'req' is 'Request' by 'new Ajax.Request('upload_progress.php?start=1');'

No idea what that means though.

EDIT:

what I am suspecting is that the file is uploaded fine, but you need to get the page with the pass and fileid to see the result.

Last edited by Procyon (2009-05-20 13:21:06)

lolilolicon · 2009-05-20 14:17:02

document.location = '/index.php?f=' + fileid + '&pass=' + pass;

The result that curl could not return looks like:
Post two files.

http://gigasize.com/index.php?f=3198322 … 268c4dae96--

And I post one file a bit later.

http://www.gigasize.com/index.php?f=319 … 86802278a2--

One thing I noticed was in the field "f=", the value increased by one when 2 files posted at once. And it increased only a little after a while. Nothing useful though.

Another thing I noticed is that the length of the "pass=" field, it's the length of a md5sum.

Last edited by lolilolicon (2009-05-20 14:22:07)

Arch Linux

#1 2009-05-18 09:34:04

Using curl for uploading file, need help.

#2 2009-05-19 11:14:05

Re: Using curl for uploading file, need help.

#3 2009-05-19 13:36:00

Re: Using curl for uploading file, need help.

#4 2009-05-19 14:39:00

Re: Using curl for uploading file, need help.

#5 2009-05-19 18:31:48

Re: Using curl for uploading file, need help.

#6 2009-05-20 00:07:08

Re: Using curl for uploading file, need help.

#7 2009-05-20 03:47:55

Re: Using curl for uploading file, need help.

#8 2009-05-20 04:03:23

Re: Using curl for uploading file, need help.

#9 2009-05-20 08:30:44

Re: Using curl for uploading file, need help.

#10 2009-05-20 10:11:35

Re: Using curl for uploading file, need help.

#11 2009-05-20 10:19:07

Re: Using curl for uploading file, need help.

#12 2009-05-20 12:17:40

Re: Using curl for uploading file, need help.

#13 2009-05-20 12:37:24

Re: Using curl for uploading file, need help.

#14 2009-05-20 13:18:30

Re: Using curl for uploading file, need help.

#15 2009-05-20 14:17:02

Re: Using curl for uploading file, need help.

Board footer