Problems Importing with Users XML Plugin

[tshore]

I spent last night importing 1000+ users into my new OCS site - 50 users at a time!!! Very painful. The site is running OCS 2.3.5 and the users were exported - using the same plugin - from a 2.3.3 site. My exact problem is described in this OJS forum post:

http://pkp.sfu.ca/support/forum/viewtopic.php?f=2&t=2943&start=15#p19352

which I don't think was ever resolved.

It seemed odd to me that this guy's import was cutting out after exactly 10 records and mine was stopping after 50. I suspect that this error might have something to do with the # of items displayed on a page:

Code: Select all

; Number of items to display per page; overridable on a per-conference basis
items_per_page = 25

I really only suspected this because they were both such round numbers and I had this variable set to 50. Changing this variable didn't help, however.

Even if this is probably not the case, I'm hoping that the developers can offer a solution so that no one else will have to endure what Charles and I went through in order to get users imported into our new sites.

Thank you.

[JasonNugent]

Hi tshore,

Did you try doing this from the command line? You can use the import/export plugin to import users:

Code: Select all

php tools/importExport.php UserImportExportPlugin usage


this bypasses the per-page restriction.

Regards,
Jason

[tshore]

No. Is this documented somewhere?

Regardless, you're going to file this as a bug, correct? Pretty frustrating that someone had made a forum post about this error 3 years ago, and there was never any followup.

[tshore]

Ok, so I'm setting up another conference and I find myself back at this same unresolved thread. Tonight I have to import about 1200 users. After first having to track down a bug on the export end of things, now I'm faced - again - with a bug while importing.

Can somebody please file this as a bug, at the very least...

[asmecher]

Hi tshore,

Are you seeing the same error message that was reported on the other thread ("DB error:column 'first_name' cannot be null")? If so, have you tried validating your XML before importing it?
I'm not sure yet, but I suspect that items_per_page is a red herring (or if not, that you're experiencing a different problem than the fellow on the other thread).

Regards,
Alec Smecher
Public Knowledge Project Team

[tshore]

Yes, still get the same problem of only being able to import 50 records at a time.

I ended up doing it from the command line for this conference (based on Jason's suggestion above). That turned out to also be painful and a long, drawn out procedure.

I didn't validate my XML, but, considering it was generated by the OCS export user plugin, I didn't think that this would be necessary. Also, the fact that I can reliably import 50 records at a time seems to indicate that that XML is just fine.

I will provided details of the problem with the command line import and the solution/hack I came up with to get past it.

[tshore]

Ok, here's the command that I used:

Code: Select all

php tools/importExport.php UserImportExportPlugin import users-5.xml 2013

The current directory was the main OCS directory and I uploaded the XML file to that directory. 2013 is the path (schedConfPath).

However, I was still getting an error:

Code: Select all

An unknown scheduled conference path "2013" was specified.

This is because, in order to lookup the scheduled conference, the function getSchedConfByPath (in classes/schedConf/SchedConfDAO.inc.php) needed to know the conferenceId. It looks like it should be able to manage without it, but this was not the case. I assume this was because it was being called from the command line.

So, to get it to work, I temporarily hacked line 214 in tools/importExport.php by supplying the conferenceId=1 to getSchedConfByPath.

Code: Select all

-      $schedConf =& $schedConfDao->getSchedConfByPath($schedConfPath);
+      $schedConf =& $schedConfDao->getSchedConfByPath($schedConfPath, 1);


I suppose that in order for the import users to work from the command line, maybe the solution would be to accept the conference ID as another parameter.

[asmecher]

Hi tshore,

Can you confirm specifically that you're getting the error message I quoted above?

Regards,
Alec Smecher
Public Knowledge Project Team

[tshore]

Yes, it was the exact same error. Although I can't double-check because everythings already imported.

Code: Select all

DB error:column 'first_name' cannot be null

[asmecher]

Hi tshore,

I can't duplicate this error with OCS 2.3.5, the current stable branch, or the current unstable branch. I've tested with 60-user imports and all the users come in without trouble.

Without being able to duplicate the error it's difficult to help debug, but my suspicion is that you're trying to import users that don't have a first name. Technically OCS requires a first name to be specified. (There are work-arounds that can be used to trick OCS into accepting users without first names, and if your source conference has used those, it may be that it's generating a user export without first names and the target conference is rejecting it.)

This doesn't explain why importing users in smaller batches works while performing the whole import doesn't. Is it possible that the process you used somehow avoided importing a record without a first name specified? Did you try the entire import in one file when using the command-line based import?

Note that I've filed and entry for the inability to specify a scheduled conference path on the command line as you noted above. There's a patch at http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=8082.

Regards,
Alec Smecher
Public Knowledge Project Team

[tshore]

No users without a first name in the file.

Yes, entire file worked fine using the command line interface.

Perhaps you can try a larger import file? I'll try to attach the file I used last night.

Didn't work: Sorry, the board attachment quota has been reached.

Perhaps it will let me send it to you in a PM.

[asmecher]

Hi tshore,

Unfortunately I'm still not able to duplicate this problem. I started with a fresh OCS 2.3.5 installation, created a new conference and scheduled conference, applied the patch for bug #8082, and tried first the command-line import:

Code: Select all

$ php tools/importExport.php UserImportExportPlugin import /tmp/users-5.xml import test_sched_conf

This had the expected result:

Code: Select all

The following users were successfully imported into the system:

...followed by a list of 1187 usernames.

Checking the XML you sent for the number of users it includes:

Code: Select all

xmllint --format /tmp/users-5.xml | grep "<user>" | wc -l
1187

...and the database to see how many users it contains:

Code: Select all

mysql> select count(*) from users;
+----------+
| count(*) |
+----------+
|     1187 |
+----------+

(Both include the "admin" user, which is the administrator.)

Next, I restored my installation to its fresh state, and tried the same process via a web import. Importing many users that way results in some very long pages, but again, everything behaved as expected.

There are still many variables at play here -- your PHP version (mine is 5.3.3), your MySQL version (mine is 5.1.50), the upgrade and maintenance history of your OCS install, etc. If any of this process differs substantially from what you've been doing, please let me know.

If you're able to duplicate the error message, try turning on the "debug" option in config.inc.php just before you do it. This will result in any requests to OCS also including database statements, which may be relevant. (Note that you'll only want to turn this on briefly.)

Regards,
Alec Smecher
Public Knowledge Project Team

[NoS]

Hi there,

I'm facing the exact same problem, but with OJS 2.3.8. (but I think it's better to add this to this thread than to start a new one in the ojs-section.)

Server:
OS platform Linux
PHP version 5.3.3
Database driver mysql
Database server version 5.1.61
Update history: 2.3.6, 2.3.7, 2.3.8
No Hacks at all.

Code: Select all

DB error:column 'first_name' cannot be null

when I try to import more than 50 users with user-import-plugin (UI, I have no access to the server or config file). No problems occur when I split the users in numerous files.

Please help!

Regards,
Nora

[asmecher]

Hi Nora,

Have you validated the XML document? Are you sure that all users have a specified first name (which is a required field)?

Regards,
Alec Smecher
Public Knowledge Project Team

[NoS]

Hi Alec,

all the users were generated via OJS 2.3.8 GUI. I guess the XML is fine, then, including first names, that I checked scrolling through the list.

Regards,
Nora