Does the PKP harvester support OAI protocol's flow control http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm#FlowControl
Archives such as http://arxiv.org
use HTTP 503 Retry-After replies to limit the speed at which harvester can harvest.
I personally think it is nothing but annoying and stupid, but none-the-less it is there.
What seems to be happening is after the first ListRecords request is made, the webserver returns 503, with a specified number of seconds to wait before requesting again. PKP just seems to stop there, only harvesting the first set of records returned from ListRecords.
I am currently doing a research assistantship at the University of Arizona and I can certainly provide help in adding this to PKP. Regardless, I will have to add it to our local copy of PKP, as we want to be able to harvest sites like http://arxiv.org