Cannot get the Lucene plugin working

[springday]

Dear search indexers,

I'm struggling with the Lucene plugin for a few weeks and am now nearly gone mad. OK, I've been warned by the README file that getting remote server mode running is a tricky thing. My situation is aggravated by the fact that we had to book a SolR server package that we don't have configuration admin access to by ourselves (we can access a bit of admin diagnosis tools, though). I've sent the server setup instructions to the admin and have been informed that they were applied (well, that was actually not one step, but they should be so by now). I've been particularly ensured that the DataImportHandler is configured the way it was described in the README. However, when running the rebuildSearchIndex.php script from /tools, I don't get anything indexed in Lucene.

Here's the process the way I understand it:

1. The Lucene plugin builds the XML streams to be POST-ed to the SolR server and it seems to do that correctly (I've put one sample XML-post that I got out of the process with the help of echo [url src=http://www.reinhardt-journals.de/public/Sample_SolR_POST_Doc.xml]here[/url]).
2. The POST is sent to the correct URL of the DataImportHandler.
3. The SolR server returns HTTP code 200.
4. However, the $result variable assigned in the line

   Code: Select all

   $result = $this->_makeRequest($url, $articleXml, 'POST');

   is empty (echo gives empty string, var_dump() lists an empty, non-null object)
5. The number of indexed files, as counted by $numIndexed, is always 0, which is also output to the command shell: ... 0 articles indexed.
6. The troubleshooting part in the README says I should check the luke record. The luke records says the same as $numIndexed:
   

   Code: Select all

   <lst name="index">
  <int name="numDocs">0</int>
  <int name="maxDoc">0</int>
  <int name="numTerms">0</int>
  ...
</lst>

I can get some statistics about the DataImportHandler's performance. They look like this:

Status : IDLE
Documents Processed : 0
Requests made to DataSource : 0
Rows Fetched : 2
Documents Deleted : 0
Documents Skipped : 0
Total Documents Processed : 0
Total Requests made to DataSource : 0
Total Rows Fetched : 0
Total Documents Deleted : 0
Total Documents Skipped : 0
handlerStart : 1353407720923
requests : 156
errors : 0
timeouts : 0
totalTime : 141659
avgTimePerRequest : 908.0705
avgRequestsPerSecond : 1.9848118E-4

I can also see statistics of the update requests that are triggered by every run of rebuildSearchIndex to clear the index (I guess):

commits : 18
autocommits : 0
optimizes : 0
rollbacks : 155
expungeDeletes : 0
docsPending : 0
adds : 0
deletesById : 0
deletesByQuery : 0
errors : 0
cumulative_adds : 0
cumulative_deletesById : 0
cumulative_deletesByQuery : 18
cumulative_errors : 0

Why does the SolR server send a code 200 but does not index the articles?

Best regards,
Kai Weber

[jerico]

Hello Kai,

as the README says It's almost impossible to implement a non-default server configuration without dedicated developers on your side who know how to configure and debug Solr in general and the DataImportHandler extension in particular. I'll try to help you remotely but my resources are limited in this case. If you feel that you need custom support, I can of course help you under an individual service contract. (Please contact me at jerico.dev@gmail.com if you're interested...)

But first let's see whether it's possible to fix this remotely!

It looks as if the DataImportHandler (DIH) produced an error when trying to import your files. The rollback statistics may be an indicator of this. The fact that you don't see errors in the statistics is not necessarily an indicator as DIH may ignore certain types of errors. It therefore would be important that we get access to the full console output of the solr process to find out what's going on - especially the output being generated while trying to build the index. In the default (embedded) configuration this output is being written to files/lucene/solr-java.log. This output will most probably indicate the cause of the error. Have a look at the default start.sh script for an example how to configure logging.

Common reasons why DIH would produce errors are:
* The custom DIH import script (dih-ojs.xml) has not been installed on the remote server.
* DIH/Solr may not be correctly configured to find the DIH import script.
* The import script and/or Solr configuration has not been adapted to your individual environment (e.g. index location, dictionary location, log output, etc.).
* The DIH request handler has not been correctly configured.
* The Solr endpoint is not correctly configured in your plugin configuration.
* The Solr endpoint may not implement the right nomenclature or transport protocol.
* Any type of index access permission or index corruption problem.

I can tell you more once I see the console output.

Kind regards,

Florian

[springday]

Hi Florian,

thanks for your list of possible errors. I have now received a reply from the provider of our SolR server provider. They say they have analyzed the dih-ojs.xml and found a problem with the postProcessArticle function (starting from line 294). They say this function cannot be correctly imported. The transformations that this function is supposed to do cannot be run on the server side. (They seem to imply that those could or should be done on the client side?)

This reply from them sounds different from all the points you mentioned. Do you have any idea how we could circumvent this problem?

Best regards,
Kai Weber

[jerico]

Hello Kai,

sorry I had forgotten to subscribe to the topic so I didn't get notifications for this thread. The error seems to enter into the category "The import script and/or Solr configuration has not been adapted to your individual environment (e.g. index location, dictionary location, log output, etc.)." I'm afraid that without more specific information (logs!) and without looking at your specific server setup I'll not be able to help you.

Florian

[springday]

Hi Florian,

as I said I have no admin access to the external Lucene server either. I have requested logs from the server provider but they have never sent anything to me. The webserver of our OJS installation doesn't have Java, so I can't run an embedded mode SolR server. I'll try to get logs once again. The provider's SolR specalist has asked me if the postProcessArticle function could be disabled. Can it?

In case we can't get any further with that, I'll see if I get some funds for getting individual service from you...

Best regards,
Kai

[jerico]

Hi!

The postProcessArticle() function is needed to store several fields containing data collected across publication objects, e.g. certain kinds of full-text search. So it wouldn't work without that method. I'm sure, though, that there will be a way to get solr working for you. We've tried it out under many different circumstances (Oracle Java and OpenJDK, different versions of Solr and Linux and even on Windows) so I'm quite confident that with some minor adaptation there'll be a solution for your problem, too.

Kind regards,

Florian