OAI_PMH v2.0 - Flow Control

[robackja]

Does the PKP harvester support OAI protocol's flow control http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm#FlowControl?

Archives such as http://arxiv.org use HTTP 503 Retry-After replies to limit the speed at which harvester can harvest.

I personally think it is nothing but annoying and stupid, but none-the-less it is there.

What seems to be happening is after the first ListRecords request is made, the webserver returns 503, with a specified number of seconds to wait before requesting again. PKP just seems to stop there, only harvesting the first set of records returned from ListRecords.

I am currently doing a research assistantship at the University of Arizona and I can certainly provide help in adding this to PKP. Regardless, I will have to add it to our local copy of PKP, as we want to be able to harvest sites like http://arxiv.org.

[kevin]

Currently the PKP Harvester does not support flow control. This is a feature we will consider adding in a future version, when we next have time to work on improvements to the harvester.

If you do implement this yourself and make the code available, we'd be happy to merge it in with ours.