OJS OCS OMP OHS

You are viewing the PKP Support Forum | PKP Home Wiki



Unable to export issues

Are you responsible for making OJS work -- installing, upgrading, migrating or troubleshooting? Do you think you've found a bug? Post in this forum.

Moderators: jmacgreg, btbell, michael, bdgregg, barbarah, asmecher

Forum rules
What to do if you have a technical problem with OJS:

1. Search the forum. You can do this from the Advanced Search Page or from our Google Custom Search, which will search the entire PKP site. If you are encountering an error, we especially recommend searching the forum for said error.

2. Check the FAQ to see if your question or error has already been resolved.

3. Post a question, but please, only after trying the above two solutions. If it's a workflow or usability question you should probably post to the OJS Editorial Support and Discussion subforum; if you have a development question, try the OJS Development subforum.

Unable to export issues

Postby geoffg » Thu Apr 11, 2013 9:56 am

I'm running into memory allocation errors when trying to export a particular issue from a journal in OJS. We set php.ini memory limits and time limits very high (2GB and 20 minutes) and restarted apache after some initial errors. But now, we get this error when trying to export an issue to xml:

<b>Warning</b>: DOMDocument::saveXML() [<a href='function.DOMDocument-saveXML'>function.DOMDocument-saveXML</a>]: Memory allocation failed in <b>/srv/www/secure/htdocs/wnan/classes/xml/XMLCustomWriter.inc.php</b> on line <b>109</b><br />

We're running OJS 2.2. The server is loaded with RAM. (8GB) The reason we want to export is so we can merge this journal with other journals running on a different instance of OJS 2.3.7. Maybe there's a simpler way to export an entire journal?

Any ideas of how to get our data exported/imported?

Thanks,
Geoff
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Thu Apr 11, 2013 10:54 am

Hi Geoff,

First off, if you haven't disabled charset_normalization in your config.inc.php, try again with that option disabled. (It's safe to leave turned off.)

The XML code we use unfortunately has to span PHP4 and PHP5 support, so it's not the most efficient in its operations. Different versions of PHP also handle memory use very differently. Long story short, we have occasional reports of heavy memory usage and I'm not surprised one of those situations is happening here -- all the galley files are written into the XML file in base64 encoding, and those may well be large files.

Often there will be a different php.ini file for command-line PHP and you could try configuring that file to disable memory limits, then using the command line tool "tools/importExport.php" to export your data. Alternately, if you'd rather not base64-encode all that binary data, I can suggest a few changes in the code to avoid that process.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Fri Apr 12, 2013 4:13 pm

Thank you, Alec.

I've turned off charset_normalization, but still running into memory allocation problems. I bet it's large galley files trying to be written to the xml. If you have ideas to avoid that, that would be great. But if they're not written to the xml export file, will there be a way to get them imported/connected upon reimporting to a different instance of ojs?
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Fri Apr 12, 2013 4:21 pm

Hi Geoff,

Have a look in plugins/importexport/native/NativeExportDom.inc.php for lines containing the "base64_encode" function. That's where the file contents get encoded and embedded for the various kinds of files that the XML export supports (issue covers, article files, etc).

The node structure for those nodes looks like:
Code: Select all
<file>
    <embed filename="..." encoding="base64" mime_type="...">
        (Base64 contents here)
    </embed>
</file>
An alternative way that the import process accepts files is via HTTP or local file paths (the latter only available for command-line imports). Those look like...
Code: Select all
<file>
    <href src="..." mime_type="..." />
</file>
It would be fairly simple to adapt the XML generation code in NativeExportDom.inc.php to generate patterns of the second format, rather than the first.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Thu Apr 18, 2013 1:45 pm

Thanks Alec. I was able to export xml after altering the code so it's not embedding base64 data.

To import, I first created a new journal, then ran the import command from the command line. It ran for a few hours and seemed to work. It copied a ton of files to the new file repository for the journal. But when I look at the newly created journal from the website, there still aren't any issues. (browsing by issue, author, etc. just says "This journal has not published any issues.")

Any ideas?

Thanks for your help.

Geoff
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Thu Apr 18, 2013 1:51 pm

Hi Geoff,

If you look in the Editor's "Future Issues" list, do you see them there? If so, you may just need to hit the "Publish" button. Did you receive a list of successful imports after the import process finished?

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Thu Apr 18, 2013 2:23 pm

No, there aren't any issues in the Editor's "Future Issues." It reads "No Issues"

I didn't receive a list of successful imports. But there were a few errors reported. They didn't seem like they would cause the entire process to fail. I thought they were probably just related to isolated issues:
- I got "Error: Bad annotation destination" once.
- I got about a dozen errors that read "The title for an issue was missing." (But there are a lot more issues in this import than that.)
- and I got 2 errors about articles "in a locale ("es_ES") that this journal does not support."

But I expected the bulk of the issues to import okay. For the most part it ran without any output, but seemed to copy files over successfully. However, here's a strange thing, while it was running I could browse the articles directory and see that files were being copied. I even opened a file or 2. But now that the script has finished, all of the files are gone. The folders are still there. (It created a lot of numbered folders in the "articles" directory.) But the pdf files are gone.

Maybe I need to do a more extensive journal setup when I created a new journal? I just created a title and that's about it. Any thoughts?

Thanks,
Geoff
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Thu Apr 18, 2013 2:50 pm

Hi Geoff,

The import tool will attempt to clean up after itself when you have an import failure. I'd suggest spending some time trying to figure out those few error messages. First, since it sounds like some article content has Spanish-language metadata, enable Spanish in your journal configuration.

The "bad annotation destination" message can be ignored -- I believe those come from the text extraction tools OJS uses to index PDF files, and won't cause a failure.

The missing issue titles message may need some correction in your XML exports. These will arise when your journal specifies that issue titles are required, but there is no title specified; look at the <issue ...> nodes for cases like that and either correct it in the XML prior to import or look at those issues in your source system.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Wed Apr 24, 2013 3:25 pm

Thanks, Alec. I was able to enable Spanish and also add titles to issues without titles. I'm not seeing those errors anymore. Now I'm getting a new error when I try to run the command line tool to import issues. The error doesn't show any line numbers or other details, just:

<h1>DB Error: Column 'email' cannot be null</h1>ojs2: DB Error: Column 'email' cannot be null

Looking at the database, users table, there aren't any users without an email address. Any ideas on how to get around this error and import our issues?

Thanks,
Geoff
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Wed Apr 24, 2013 3:43 pm

Hi Geoff,

Check your XML for author nodes that don't contain email nodes; the email element is required. (Validating the XML against the DTD with any good XML editor should turn this up.)

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Fri Apr 26, 2013 10:33 am

Okay, I found quite a few authors without emails and fixed that problem...
But I'm still having issues when trying to import:

- A lot of files, but not all, have an incorrect url. For example: the export xml might show the file path to be something like /var/ojs_files/ojs/files/journals/1/articles/3781//3781-6784-2-SP.tif

and that causes an error when trying to import. The file does exist, but the correct path (in this example) would be /var/ojs_files/ojs/files/journals/1/articles/3781/supp/3781-6784-2-SP.tif (added "supp")
(As as reminder, we're not inserting base64 data for files, instead you suggested I modify the code to include a link to the file.)

- The other issue is an error that causes the import script to stop abruptly with "<h1>DB Error: Column 'first_name' cannot be null</h1>ojs2: DB Error: Column 'first_name' cannot be null." I've verified that all author nodes include a "firstname" node. There don't seem to be any "first_name" nodes at all. (with an underscore)

Thanks for your help with all of this.
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Fri Apr 26, 2013 10:51 am

Hi geoffg,

How are you getting the filenames for inclusion in the export XML?

The reason for the firstname/first_name distinction is that the XML expects nodes called firstname and the database stores them in the first_name column. This shouldn't be causing problems.

Do you have empty firstname nodes, i.e. <firstname/> or <firstname></firstname>? Those would result in error messages.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Fri Apr 26, 2013 12:35 pm

I'm getting the filenames using the getFilePath() method. Here's an example of how I modified the export code:

/* --- Galley file --- */
$fileNode = &XMLCustomWriter::createElement($doc, 'file');
XMLCustomWriter::appendChild($root, $fileNode);
$embedNode = &XMLCustomWriter::createChildWithText($doc, $fileNode, 'href', '');
$articleFile = &$articleFileDao->getArticleFile($galley->getFileId());
if (!$articleFile) return $articleFile; // Stupidity check
XMLCustomWriter::setAttribute($embedNode, 'src', $articleFile->getFilePath());
XMLCustomWriter::setAttribute($embedNode, 'mime_type', $articleFile->getFileType());

On the firstname issue, I can see some empty nodes (<firstname/>), so I'll fix those.

Any ideas about how to correct the file urls?
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Re: Unable to export issues

Postby asmecher » Fri Apr 26, 2013 1:11 pm

Hi Geoff,

And these incorrect paths are being generated by your installation of OJS 2.2.0, correct?

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 9212
Joined: Wed Aug 10, 2005 12:56 pm

Re: Unable to export issues

Postby geoffg » Fri Apr 26, 2013 1:16 pm

Yes. The resulting xml that is exported seems to have many file urls that are incorrect.
geoffg
 
Posts: 15
Joined: Tue Jul 10, 2012 4:08 pm

Next

Return to OJS Technical Support

Who is online

Users browsing this forum: Bing [Bot], Yahoo [Bot] and 3 guests