[Slackbuilds-users] sbog ping: invalid download urls in .info files

[rundstutzen at gmx.de]

On Fri, 2 Mar 2018 13:49:55 -0500
B Watson <yalhcru at gmail.com> wrote:

> On 3/2/18, rundstutzen at gmx.de <rundstutzen at gmx.de> wrote:
> > i didn't even know that sbosrcarch does exist (even though i have
> > seen slackware.uk before). so the functionality i programmed does
> > already exist.  
> 
> ...or: git clone git://urchlay.naptime.net/sbostuff.git
> 
> It's in perl, and not what I'd call beautiful code, so you might want
> to put on your goggles before looking at it :)

thank you, this will be an interesting read. i don't care about code
quality - i care more about knowledge that usually lies within the
code. you know probably more about the sbo stuff than i do.
> 
> You might at least look through your logs and make a list of the
> servers that don't do HEAD requests, and add some code that logs
> "this server doesn't support HEAD requests", so the user knows it's
> the server's fault.
> 

please read my starter post, there is a pastebin link to the output of
"sbog ping". i print all errors to stdout, easy to parse. errors
caused by HEAD request failures are only a few. "forbidden" errors are
about 10 or so. can be easily found with grep.

> 
> You're right. How much overhead depends on how the server/proxy is
> configured, but it seems to be 64K bytes per request usually. I
> decided that wasn't a problem, partly because the guy who hosts the
> archive said so (I wrote the script, he runs it and lets people
> download the files it collects).

there used to be times where starting several downloads simultaneously
from the same server was frowned upon. i don't know if this has changed.

> 
> Make the client pool smart enough to serialize requests to the same
> server, and do them in parallel only when they're for different
> servers? That seems like it'd be worth doing even with regular HEAD
> requests like you use now.

i did a lot of "curl -LIv" testing on download urls - and *many* of them
have redirects. no way to tell the real download server by the
url found in the .info file. but i will find a way. right now i do some
runs, printing a lot of data and collect statistics - so i know what i
am actually dealing with.

saying that, i have no problem sending 5 or 10 HEAD request to the same
server simultaneously. GET is another story.