We are moving to Git Issues for bug tracking in future releases. During transition, content will be in both tools. If you'd like to file a new bug, please create an issue.

Bug 8043 - memory exhausted on mEDRA/datacite plugin
memory exhausted on mEDRA/datacite plugin
Status: RESOLVED INVALID
Product: OJS
Classification: Unclassified
Component: Plug-ins
2.4.2
All All
: P3 normal
Assigned To: PKP Support
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-11-27 07:56 PST by Giuseppe Digilio
Modified: 2012-12-03 09:45 PST (History)
4 users (show)

See Also:
Version Reported In:
Also Affects:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Giuseppe Digilio 2012-11-27 07:56:51 PST
Hi,
I've obtained an out of php memory while selecting "Export all unregistered issues, articles and galleys" option in mEDRA and datacite plugin.

I tried to apply this  patch 
http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=7967
but this didn't solve the problem.

I have 256M on memory limit in php.ini, I could increase it further, but maybe there's some bug, I thought.
Comment 1 Alec Smecher 2012-11-27 08:59:33 PST
Giuseppe, try also patching bug #8031. If you could post the error message too (including the file and line number), that would help.
Comment 2 Giuseppe Digilio 2012-11-28 01:29:47 PST
I tried to apply that patch too, but the problem there is yet.

this is the error I obtain :

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 24 bytes) in /var/www-ojs/lib/pkp/classes/core/PKPApplication.inc.php on line 388
Comment 3 Alec Smecher 2012-11-28 09:17:50 PST
Giuseppe, if you split the group of articles in half, do you still get an errorm message? If so, can you narrow in on a particular article that causes the problem, or does it only seem to affect large exports?
Comment 4 Giuseppe Digilio 2012-12-03 01:02:01 PST
With a smaller list, i don't obtain the error, i think it only seem to affect large exports
Comment 5 Alec Smecher 2012-12-03 08:22:51 PST
Thanks for the details, Giuseppe; that sounds like a legitimate out-of-memory error. The more data you export, the more memory will be used putting them together. The only solution I can recommend is either to increase your memory limit or export smaller batches.
Comment 6 Andrea Marchitelli 2012-12-03 08:36:36 PST
Hi Alec,
it seems that the paging bug fixed in #7967 still affects the function namend 
» Export all unregistered issues, articles and galleys
while it was resolved in » Export specific articles

So at the moment we can not export smaller batches.
Comment 7 Alec Smecher 2012-12-03 08:39:45 PST
Andrea, did you apply the patch in bug #7967 against both copies of DOIExportPlugin.inc.php? Depending on your OS, there may be two separate copies of this file and the patch needs to be applied to both. (This distribution quirk has been fixed elsewhere but remains for OJS 2.4.1.)
Comment 8 Andrea Marchitelli 2012-12-03 08:48:49 PST
(In reply to comment #7)
> Andrea, did you apply the patch in bug #7967 against both copies of
> DOIExportPlugin.inc.php? 

Yes, we did.

In 
/manager/importexport/plugin/DataciteExportPlugin/articles
and
/manager/importexport/plugin/MedraExportPlugin/articles

I can see paginated lists that I can not see in 
/manager/importexport/plugin/DataciteExportPlugin/all
and
/manager/importexport/plugin/MedraExportPlugin/all
where I see only never-ending lists without any button at the bottom
Comment 9 Alec Smecher 2012-12-03 08:49:40 PST
Thanks for the details, Andrea; I'll pass this along to the group working on this plugin.
Comment 10 jerico 2012-12-03 09:24:47 PST
The "Export all unregistered objects" feature is meant for continuous registration of new publication objects. Users usually wish to be able to select all those objects with one click without having to page through results and register object pages separately. Therefore it is on purpose that paging has not been implemented for this view.

The list of "unregistered objects" is not meant for the initial registration of a large journal. If you try to do so you'll need very large junks of memory and you will probably get further problems when trying to register such a large amount of objects all at once with DataCite or mEDRA.

It is recommended to do the initial registration of a journal via the other lists and then use the "unregistered objects" list only for later updates.

That said, it would be entirely possible to implement paging for the list of unregistered objects although I'm not convinced this is really what everybody wants. Unfortunately I do not currently have the resources to do so.

BTW: I made an in-depth analysis of OJS memory consumption recently in another context (Lucene search). I found out that in the case of large internal result sets as necessary for export or re-indexation of whole journals, the currently used PHP MySQL-extension together with ADOdb are the main culprits of huge memory leaks. As long as OJS has to rely on these tools I don't see a way around the memory issues as closing intermediate MySQL result sets does not have the effect of releasing internally acquired memory, too.
Comment 11 jerico 2012-12-03 09:29:30 PST
One more comment: Do you know that you can partially automate initial export via the command line export tool? Both, registering and exporting objects via command line is possible. A small script accessing your database may help you to avoid manual work when registering large amounts of publication objects.
Comment 12 Andrea Marchitelli 2012-12-03 09:31:23 PST
No, I didn't know, thanks! 
Is there any documentation about semi-automatic exports?
Comment 13 jerico 2012-12-03 09:42:08 PST
Hm, there may be better documentation. The only thing I know of is

php tools/importExport.php will give you general usage information.

And 

php tools/importExport.php DataciteExportPlugin usage

specific information for DataCite.

Probably Alec can tell you more.
Comment 14 Alec Smecher 2012-12-03 09:45:56 PST
There is some general documentation in docs/IMPORTEXPORT, though nothing there specific to mEDRA/datacite.