Proxy problems with Harvester?

[asmecher]

Hi Fred,

Glad to hear it's debugged, if not resolved yet. We'll be working on the platform code that OJS, OCS, and Harvester all use, and one of the things we'll be paying attention to is better error logging. In the meantime, good luck with the Inquisition...

Regards,
Alec Smecher
Public Knowledge Project Team

[fredriley]

Alec

I've written a long justification to our sysadmins for opening port 3128 to allow Harvester to work. Before I send it off, I'd be grateful if you could briefly answer 3 quick Qs which I'm sure they'll ask:

1. Does Harvester have to use port 3128, or can it use any HTTP port? (I have tried setting ports 80 and 8080 in the config.inc.php proxy section but no dice.)
2. Is the traffic on the port both outgoing and incoming?
3. Are any other ports used by Harvester? Other than 80 for normal web traffic, that is.

Cheers

Fred

[asmecher]

Hi Fred,

1. Harvester can be configured to use a proxy on any port.
2. Harvester uses standard protocol for HTTP proxy requests, i.e. all indexing requests are initiated by Harvester. No incoming connections need be accepted.
3. No, nothing else is used.

Regards,
Alec Smecher
Public Knowledge Project Team

[fredriley]

Hi Fred,

1. Harvester can be configured to use a proxy on any port.

Including port 80? So if I were to use the following in config.inc.php Harvester should work?

; The HTTP proxy configuration to use
http_host = 128.243.220.20
http_port = 80

I did give that a bash but no luck. I also tried just specifying the proxy host without a port, also no good. By default, what port do Harvester requests go out on? Questions, questions...

Cheers

Fred

[asmecher]

Hi Fred,

If this is a machine that you can run a web browser on (even a text-mode one like Lynx), I'd suggest hammering out your proxy settings using that, as it'll be quicker to test with and less of an unknown app than Harvester. Once you've got the proxy settings figured out for a browser, use the same settings for Harvester.

OAI requests use HTTP as the underlying protocol, so the technology is exactly the same.

Regards,
Alec Smecher
Public Knowledge Project Team

[fredriley]

Hi Fred,

If this is a machine that you can run a web browser on (even a text-mode one like Lynx), I'd suggest hammering out your proxy settings using that, as it'll be quicker to test with and less of an unknown app than Harvester. Once you've got the proxy settings figured out for a browser, use the same settings for Harvester.

OAI requests use HTTP as the underlying protocol, so the technology is exactly the same.

Regards,
Alec Smecher
Public Knowledge Project Team

Hmm. That's got me scratching my head again. The machine on which I've installed Harvester is our university's main multi-user Unix system, which runs Lynx, Firefox (in X Windows), and even Mosaic (a blast from the past!). What you're saying is that if they work then Harvester should work, as Harvester can also use port 80. In which case I'm absolutely stumped as to why it won't work. This is getting messy with all the recent tweaks - I'll try installing it clean on another account on the same machine.

Cheers

Fred

[asmecher]

Hi Fred,

What proxy settings are the various browsers using? Are they connecting directly via port 80 (e.g. a direct connection to the remote server), or is there a proxy configuration that accesses the proxy using port 80?

Regards,
Alec Smecher
Public Knowledge Project Team

[fredriley]

Hi Fred,

What proxy settings are the various browsers using? Are they connecting directly via port 80 (e.g. a direct connection to the remote server), or is there a proxy configuration that accesses the proxy using port 80?

I've looked in the connection settings of Firefox on the Unix system, and it uses the proxy config scrip at http://wwwcache.nottingham.ac.uk/proxy.pac, the same as the copy of Firefox on my local machine.

Having failed completely in getting Harvester to run on the multi-user system, I installed it on a Windows PC in our office running Apache, mySQL and PHP (courtesy of the excellent XAMPP package) with all root permissions. The installation went fine, then after I edited the config.inc.php to add:

http_host = 128.243.220.20
http_port = 3128

to the [proxy] section, it harvested records fine. Woo-hoo! I thought, at last I've got it working on a server which others can access. Sadly, my enthusiasm was dampened when I found that, although anyone inside our nottingham.ac.uk domain could access it at http://128.243.175.211/harvester2/, nobody outside the domain could get to it - if you give it go it'll time out, and neither can 128.243.175.211 be pinged. Almost certainly our sodding sysadmins have blocked incoming HTTP to anything other than 'approved' IP addresses, so I've sent them a sharp email asking them to 'approve' the test machine.

Put simply, I think that all this trouble has been caused by some mysterious settings on our university web server which I'm not privy to. It frustrates the almighty hell out of me, and means that I'll probably have to try to install harvester on my personal machine at home in order to demo it to work colleagues.

Has the pkp.sfu.ca server been up and down these last couple of days? It's been unreachable for much of today, and a fair bit of yesterday.

Cheers

Fred

[asmecher]

Hi Fred,

Sorry to hear it's been a frustrating experience, but at least we've confirmed that proxy support for the Harvester does work. You might try downloading that proxy configuration file and looking through it by hand -- but make sure you fetch it from the server itself, e.g. from a SSH session using "wget", as it's probably dynamically generated. (For example, I can download it from here, and it recommends using a direct connection to the Internet; someone on an internal subnet will get different contents if they fetch the same file, depending on where they're coming from.)

We've had some power issues on campus that caused some downtime for pkp.sfu.ca; hopefully it'll be smooth sailing going forward.

Regards,
Alec Smecher
Public Knowledge Project Team