HOW-TO: Import and Export Data Using OJS

[jmacgreg]

Expanded documentation on importing and exporting data into OJS is now available, culled from previous forum posts, READMEs and developer documentation:

http://pkp.sfu.ca/files/docs/importexport/index.html (HTML)

The Table of Contents is as follows:

Preface
Introduction
Articles and Issues XML Plugin
-Importing Articles and Issues
-Exporting Articles and Issues From the Web
Using the Web-based Expedited Submission Process
Users XML Plugin
-Importing Users
-Exporting Users From the Web
CrossRef XML Export Plugin
-Exporting Crossref Information From the Web
Erudit Article Export Plugin
-Exporting Erudit Information From the Web
PubMed XML Export Plugin
-Exporting Pubmed Information From the Web
Using the Plugins from the Command Line
Appendix A. Example Articles and Issues Import file
Appendix B. Example Users Import XML File

If you have any further tips, tricks or suggestions, please post them here. This is an evolving document, and comments are welcome.

Cheers,
James

[jmacgreg]

A collection of relevant links to this article will be kept and updated in this post:

Import error with function getFilePath:
User has problem with exported XML filesize, and posts a fix.
viewtopic.php?f=8&t=3086&p=11165#p11165

[dlove]

I was just wondering if anyone has developed a user-friendly form (web or desktop) that allows a non-technical user to enter in back issues. I'm looking for a way for one of our users to enter in a large amount of back issues in both French and English. The form would either push the metadata directly into the Db or create an XML document that could then be imported using the OJS Articles & Issues XML Plugin. If no form exists are there any plans to develop a plugin?

Thanks.

[jmacgreg]

Hi dlove,

I don't know of any other easier-to-use import plugins than the ones that currently ship with OJS. Whiel writing XML files may seem to be a large barrier to entry at first, I think that once you give it a try you may find it easier than anticipated. This can be of great help especially when you are dealing with a large amount of back issues -- you can reuse code (ie., not fill in a web form over and over again) and even string multiple issues together in one file.

Cheers,
James

[dlove]

Hi James, thanks for the response. I think for us it boils down to our users wanting to add back issues without having to look at or create the XML. I agree it's not difficult to create the XML file but we have had requests for a user-friendly interface. We might end up creating a '1st step' interface that allows the user to enter in the information which then generates the XML file. Once that step is done the XML file can then be edited directly if needed before it's uploaded. Thanks again.

Darren

[asmecher]

Hi Darren,

We've had a fair amount of interest from a number of sources in making a plugin to allow for one-step, easy submission of articles with the goal of quickly building back-issues; however, none of this has yet paid off in the form of code. If you do have any resources to put into this, we'd be happy to help.

Regards,
Alec Smecher
Public Knowledge Project Team

[dantonucci]

We are looking to implement OJS. We are working on establishing work flows and estimating time for initial start-up tasks. We are looking at loading 40 years of back issues. Has anyone had experience using the XML coding to import a large number of issues and can you estimate the amount of time it takes to get the data in OJS? We will be using some staff and grad students to upload the back issues.

Dana

[mnason]

Our experience at UNB in this field is with batch uploaded issues. However, our journals are marked up with XML before being imported/converted to OJS's native.dtd. This initial markup usually takes a good while but it spans more than just metadata.

If one were to record metadata for, and upload, a backrun of 40 issues, depending on the issue, this could take a good chunk of time. Metadata would probably take something like a handful of minutes per article to copy over the relevant metadata to native.dtd at the issue level. If the issues are large this could take a while. I'm assuming it's PDF too... so those would have to be split up and actually available in web-accessible folders. Each article in native.dtd has to link to the individual files.

Anyway, the upload process itself (from command line) is pretty speedy. You tell the php program to go, you wait anywhere from 20-90 seconds (depending on file size) and then the issue is up in OJS.

[alally]

A quick question. I have tried to upload a test file from my desktop. The xml and pdf files are in a folder in My Documents [C:\Documents and Settings\person\My Documents\OJS test]; I included this path in the xml document, but I still get a "Missing a galley file" message. My question is: does OJS expect that the items will be on a server? And, more importantly, does OJS assume that the files will be on the same server as OJS?

Thanks.

[jmacgreg]

Hi Alally,

Yes, if you choose to point to external files they have to be on a public server -- the XML file is essentially telling your OJS server to grab files from a public machine. The files don't have to be on the same server as OJS; furthermore, if they are, you still have to point to the web directory to grab them (for example, www.example.com/file.xml instead of /www/html_docs/file.xml).

Cheers,
James

[asmecher]

Hi all,

FYI, there are two ways to import XML files -- the command line tool (tools/importExport.php) and the web-based tool. If you're using the command line tool, it's possible to use local files on the same machine. If you're using the web-based tool, you'll need to specify full URLs to external files. This is done for security reasons. See docs/IMPORTEXPORT for details.

Regards,
Alec Smecher
Public Knowledge Project Team

[Flying_Monkey]

Hi people,

I am currently migrating my journal to OJS. I am not a programmer but I know the basics and can alter code according to instructions. I created my own theme and stylesheets with no difficulties. I was starting to really enjoy the process.

But then I came to importing our past issues. I have been really, really struggling with the Import-Export functions. I've read the guide (above) and I have created XML files from the sample tempate provided. That seemed fine. But now I have a problem. How do you actually get the data in the XML file onto the site?

I have tried uploading my XML file, and I get the following message:

"One or more errors occurred during import. Please check to ensure that the format of the import file correctly matches the specification. Specific details of the import errors are listed below.

* This plugin does not support the supplied root node "". Please ensure that the file is correctly formed and try again."

What? The file is exactly based on the sample provided! Can someone please, please and fairly urgently tell me what I need to do because I am going crazy here....

Can I say for the future, to any programmers here, that there is an urgent need for a simpler uploading solution - as more people start using OJS who are even less geeky then me, and particularly for those who created online journals previously and now want to migrate several volumes of issues to an OJS site. OJS seems fantastic if you are starting from scratch, rather more daunting otherwise.

Thanks

[jmacgreg]

Hi Flying_Monkey,

Have you been able to validate your XML file? It's worth validating it to catch any errors. If it validates, please feel free to email me a copy of it to look over (to jmacgreg at gmail dot com). Please also include your OJS version number.

On a general note -- the XML import function, while tremendously useful for users who have a large amount of back issues to import, is also considered an advanced tool. We are in the process of updating the importexport documentation, but due to the vast difference of import scenarios we have to support, I don't foresee the process itself becoming any simpler in the near future.

Cheers,
James

[Flying_Monkey]

I think it's fine now, James - I have spent some time validating the XML files, and I think I know exactly what I am doing with that aspect at least! There are now other problems however, but those are questions for a different thread...

Thanks,

David.

[mgscano]

Hey David,
I was having the same error. Find out it was caused by a non-breaking space code left in the abstract, i.e. the infamous  . Once removed everything went smooth (except for the CDATA trick in emails which simply does not work and generates the empty issue in the database.

About the XML validation, I use the native.dtd on the site for validation, but sounds quite weird most of the fields DO NOT validate, even using the sample.xml from that same folder.

The import plugin needs documentation, for sure. I'm not a professional programmer, but I've always been skilled enuff to make ANY CMS work the way it was meant to work. OJS is a real deal, indeed.

bye,
Marco