OJS OCS OMP OHS

You are viewing the PKP Support Forum | PKP Home Wiki



Cannot get the Lucene plugin working

Are you responsible for making OJS work -- installing, upgrading, migrating or troubleshooting? Do you think you've found a bug? Post in this forum.

Moderators: jmacgreg, btbell, michael, bdgregg, barbarah, asmecher

Forum rules
What to do if you have a technical problem with OJS:

1. Search the forum. You can do this from the Advanced Search Page or from our Google Custom Search, which will search the entire PKP site. If you are encountering an error, we especially recommend searching the forum for said error.

2. Check the FAQ to see if your question or error has already been resolved.

3. Post a question, but please, only after trying the above two solutions. If it's a workflow or usability question you should probably post to the OJS Editorial Support and Discussion subforum; if you have a development question, try the OJS Development subforum.

Cannot get the Lucene plugin working

Postby springday » Thu Nov 29, 2012 6:55 am

Dear search indexers,

I'm struggling with the Lucene plugin for a few weeks and am now nearly gone mad. OK, I've been warned by the README file that getting remote server mode running is a tricky thing. My situation is aggravated by the fact that we had to book a SolR server package that we don't have configuration admin access to by ourselves (we can access a bit of admin diagnosis tools, though). I've sent the server setup instructions to the admin and have been informed that they were applied (well, that was actually not one step, but they should be so by now). I've been particularly ensured that the DataImportHandler is configured the way it was described in the README. However, when running the rebuildSearchIndex.php script from /tools, I don't get anything indexed in Lucene.

Here's the process the way I understand it:
  1. The Lucene plugin builds the XML streams to be POST-ed to the SolR server and it seems to do that correctly (I've put one sample XML-post that I got out of the process with the help of echo [url src=http://www.reinhardt-journals.de/public/Sample_SolR_POST_Doc.xml]here[/url]).
  2. The POST is sent to the correct URL of the DataImportHandler.
  3. The SolR server returns HTTP code 200.
  4. However, the $result variable assigned in the line
    Code: Select all
    $result = $this->_makeRequest($url, $articleXml, 'POST');
    is empty (echo gives empty string, var_dump() lists an empty, non-null object)
  5. The number of indexed files, as counted by $numIndexed, is always 0, which is also output to the command shell: ... 0 articles indexed.
  6. The troubleshooting part in the README says I should check the luke record. The luke records says the same as $numIndexed:
    Code: Select all
    <lst name="index">
      <int name="numDocs">0</int>
      <int name="maxDoc">0</int>
      <int name="numTerms">0</int>
      ...
    </lst>

I can get some statistics about the DataImportHandler's performance. They look like this:
Status : IDLE
Documents Processed : 0
Requests made to DataSource : 0
Rows Fetched : 2
Documents Deleted : 0
Documents Skipped : 0
Total Documents Processed : 0
Total Requests made to DataSource : 0
Total Rows Fetched : 0
Total Documents Deleted : 0
Total Documents Skipped : 0
handlerStart : 1353407720923
requests : 156
errors : 0
timeouts : 0
totalTime : 141659
avgTimePerRequest : 908.0705
avgRequestsPerSecond : 1.9848118E-4


I can also see statistics of the update requests that are triggered by every run of rebuildSearchIndex to clear the index (I guess):
commits : 18
autocommits : 0
optimizes : 0
rollbacks : 155
expungeDeletes : 0
docsPending : 0
adds : 0
deletesById : 0
deletesByQuery : 0
errors : 0
cumulative_adds : 0
cumulative_deletesById : 0
cumulative_deletesByQuery : 18
cumulative_errors : 0


Why does the SolR server send a code 200 but does not index the articles?

Best regards,
Kai Weber
springday
 
Posts: 111
Joined: Wed Jul 25, 2012 2:56 pm
Location: Munich, Germany

Re: Cannot get the Lucene plugin working

Postby jerico » Thu Nov 29, 2012 11:07 am

Hello Kai,

as the README says It's almost impossible to implement a non-default server configuration without dedicated developers on your side who know how to configure and debug Solr in general and the DataImportHandler extension in particular. I'll try to help you remotely but my resources are limited in this case. If you feel that you need custom support, I can of course help you under an individual service contract. (Please contact me at jerico.dev@gmail.com if you're interested...)

But first let's see whether it's possible to fix this remotely!

It looks as if the DataImportHandler (DIH) produced an error when trying to import your files. The rollback statistics may be an indicator of this. The fact that you don't see errors in the statistics is not necessarily an indicator as DIH may ignore certain types of errors. It therefore would be important that we get access to the full console output of the solr process to find out what's going on - especially the output being generated while trying to build the index. In the default (embedded) configuration this output is being written to files/lucene/solr-java.log. This output will most probably indicate the cause of the error. Have a look at the default start.sh script for an example how to configure logging.

Common reasons why DIH would produce errors are:
* The custom DIH import script (dih-ojs.xml) has not been installed on the remote server.
* DIH/Solr may not be correctly configured to find the DIH import script.
* The import script and/or Solr configuration has not been adapted to your individual environment (e.g. index location, dictionary location, log output, etc.).
* The DIH request handler has not been correctly configured.
* The Solr endpoint is not correctly configured in your plugin configuration.
* The Solr endpoint may not implement the right nomenclature or transport protocol.
* Any type of index access permission or index corruption problem.

I can tell you more once I see the console output. :-)

Kind regards,

Florian
jerico
 
Posts: 94
Joined: Sat May 16, 2009 2:45 pm

Re: Cannot get the Lucene plugin working

Postby springday » Tue Dec 11, 2012 12:56 am

Hi Florian,

thanks for your list of possible errors. I have now received a reply from the provider of our SolR server provider. They say they have analyzed the dih-ojs.xml and found a problem with the postProcessArticle function (starting from line 294). They say this function cannot be correctly imported. The transformations that this function is supposed to do cannot be run on the server side. (They seem to imply that those could or should be done on the client side?)

This reply from them sounds different from all the points you mentioned. Do you have any idea how we could circumvent this problem?

Best regards,
Kai Weber
springday
 
Posts: 111
Joined: Wed Jul 25, 2012 2:56 pm
Location: Munich, Germany

Re: Cannot get the Lucene plugin working

Postby jerico » Mon Dec 17, 2012 5:47 am

Hello Kai,

sorry I had forgotten to subscribe to the topic so I didn't get notifications for this thread. The error seems to enter into the category "The import script and/or Solr configuration has not been adapted to your individual environment (e.g. index location, dictionary location, log output, etc.)." I'm afraid that without more specific information (logs!) and without looking at your specific server setup I'll not be able to help you.

Florian
jerico
 
Posts: 94
Joined: Sat May 16, 2009 2:45 pm

Re: Cannot get the Lucene plugin working

Postby springday » Mon Dec 17, 2012 9:24 am

Hi Florian,

as I said I have no admin access to the external Lucene server either. I have requested logs from the server provider but they have never sent anything to me. The webserver of our OJS installation doesn't have Java, so I can't run an embedded mode SolR server. I'll try to get logs once again. The provider's SolR specalist has asked me if the postProcessArticle function could be disabled. Can it?

In case we can't get any further with that, I'll see if I get some funds for getting individual service from you...

Best regards,
Kai
springday
 
Posts: 111
Joined: Wed Jul 25, 2012 2:56 pm
Location: Munich, Germany

Re: Cannot get the Lucene plugin working

Postby jerico » Mon Dec 17, 2012 12:14 pm

Hi!

The postProcessArticle() function is needed to store several fields containing data collected across publication objects, e.g. certain kinds of full-text search. So it wouldn't work without that method. I'm sure, though, that there will be a way to get solr working for you. We've tried it out under many different circumstances (Oracle Java and OpenJDK, different versions of Solr and Linux and even on Windows) so I'm quite confident that with some minor adaptation there'll be a solution for your problem, too. :-)

Kind regards,

Florian
jerico
 
Posts: 94
Joined: Sat May 16, 2009 2:45 pm

Re: Cannot get the Lucene plugin working

Postby amitg » Wed Nov 27, 2013 7:39 am

Hi All,

I have installed OJS successfully and also integerate lucence plugin but my problem is that only Refine your search is coming while searching or facet but showing no result on main page. Can you please suggest me where i m wrong.

Thanks in advance.

Babul
Attachments
screen1.png
screen1.png (75.95 KiB) Viewed 614 times
amitg
 
Posts: 8
Joined: Wed Nov 27, 2013 7:23 am

Re: Cannot get the Lucene plugin working

Postby swing » Wed Nov 27, 2013 8:42 am

Hi Babul,

I assume that you have installed exactly the jetty 6.1.26 and solr 3.6.2 version in the embedded mode (i.e. on the same server in the OJS folder plugins/generic/lucene/embedded/), that you followed all the installation steps from the plug-in README file, that everything worked well (solr server is running and there are no error messages in the solr-java.log file) and that you saw something like this by rebuilding the search index:
...
# LucenePlugin: Indexing "you journal name" ... 7 articles indexed
...

Maybe then one more thing to try: to delete the OJS cache, either with:
#>rm -f cache/*.php or via administrator tools within OJS.

Else, I will have to ask the ojs-solr guru Florian Grandel :-)

Best!
Bozana Bokan
swing
 
Posts: 142
Joined: Tue Oct 09, 2007 2:59 am

Re: Cannot get the Lucene plugin working

Postby amitg » Wed Nov 27, 2013 8:34 pm

Hi Bozana Bokan,

Yes i have installed jetty-6.1.26 and apache-solr-3.6.2 version i have run the command "rm -f cache/*.php" and rebuild the index but no luck same problem. Could you please guide me where i m wrong.


Thanks,
Babul
amitg
 
Posts: 8
Joined: Wed Nov 27, 2013 7:23 am

Re: Cannot get the Lucene plugin working

Postby amitg » Thu Nov 28, 2013 4:51 am

Attachments
screen4.png
screen4.png (21.46 KiB) Viewed 558 times
amitg
 
Posts: 8
Joined: Wed Nov 27, 2013 7:23 am

Re: Cannot get the Lucene plugin working

Postby swing » Thu Nov 28, 2013 5:51 am

Hi Babul,

I think we need a magic wand for this :-P

Could you please then try to call this search:
http://localhost:8983/solr/ojs/search?d ... nst_id:%22test-inst%22&fq=journal_id:%22test-inst-2%22&start=0&rows=25&sort=score+desc&hl=on&hl.fl=&facet=on&facet.field=authors_facet&facet.range=publicationDate_dt&facet.range.start=NOW/YEAR-50YEARS&facet.range.end=NOW&facet.range.gap=%2B1YEAR&facet.range.other=all

It is looking for an author "amit" -- thus, I assume there is a publication with such an author in your system.
Please change the bold strings above (test-inst) with you installation ID, that you inserted in the plug-in settings.

How does the response then look like?

Thanks!
Bozana
Last edited by swing on Thu Nov 28, 2013 6:24 am, edited 1 time in total.
swing
 
Posts: 142
Joined: Tue Oct 09, 2007 2:59 am

Re: Cannot get the Lucene plugin working

Postby swing » Thu Nov 28, 2013 5:55 am

maybe the search once again:
Code: Select all
http://localhost:8983/solr/ojs/search?defType=edismax&qf=authors_txt&mm=1&q=amit&spellcheck=on&fq=inst_id:%22test-inst%22&fq=journal_id:%22test-inst-2%22&start=0&rows=25&sort=score+desc&hl=on&hl.fl=&facet=on&facet.field=authors_facet&facet.range=publicationDate_dt&facet.range.start=NOW/YEAR-50YEARS&facet.range.end=NOW&facet.range.gap=%2B1YEAR&facet.range.other=all
swing
 
Posts: 142
Joined: Tue Oct 09, 2007 2:59 am

Re: Cannot get the Lucene plugin working

Postby amitg » Thu Nov 28, 2013 7:40 am

Hi,

Thanks i will try and get back to you.



Regards,
Babul
amitg
 
Posts: 8
Joined: Wed Nov 27, 2013 7:23 am

Re: Cannot get the Lucene plugin working

Postby amitg » Thu Nov 28, 2013 7:47 am

Hi,

Is there any proper documentation is available for this. If yes can you please provide me.



Thanks,
Babul
amitg
 
Posts: 8
Joined: Wed Nov 27, 2013 7:23 am

Re: Cannot get the Lucene plugin working

Postby amitg » Thu Nov 28, 2013 9:11 am

Hi,

Please check the screenshot
Attachments
sc.png
sc.png (105.96 KiB) Viewed 517 times
amitg
 
Posts: 8
Joined: Wed Nov 27, 2013 7:23 am

Next

Return to OJS Technical Support

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 2 guests