No search result in farsi !

[eftekharb]

I set up my journal in farsi and english . When I use search for english words there is no problem but for farsi words there is no result.
character settings in config.inc.php file is:

locale = en_US
client_charset = utf-8
connection_charset = utf8
database_charset = utf8
charset_normalization = On

My sql database default character set is also utf-8
I use Windows 2003 server

Would you please tell me how I can fix this problem?

[msaghaei]

Hi eftekharb
If possible please place the url to your site, so everyone can test the search.

[eftekharb]

Thank you for your reply.
URL of the site is
http://smj.ajums.ac.ir/

I can not search farsi words !

[msaghaei]

Hi

I searched your site. Farsi search returns nothing! Very strange! is it because of Windows, IIS or mysql version? are your using mysql?
Seems that you have ojs 2.2.3
I think we should wait for PKP support team response. Meanwhile you may like to provide further info (IIS and mysql version)

Thank you

[jmacgreg]

Hi folks,

I've tried some limited troubleshooting, and concur that the Farsi isn't being searched. Mahmoud, have you encountered this problem with your journals before? Eftekharb, did you add the Farsi translation at the same time as the English abstract information as well -- that is, did you publish in English and Farsi simultaneously? As an initial step, could you try rebuilding your search index by running, from the command line:

Code: Select all

php tools/rebuildSearchIndex.php

Please let us know if that produces any further results.

Cheers,
James

[jmacgreg]

One quick follow-up: this thread has some points to check when trying to search against non-Latin locales:

viewtopic.php?f=8&t=4283

Cheers,
James

[msaghaei]

Hi James

Several issues addressed at the same time.
First. Farsi is searchable in our installation at
http://journals.mui.ac.ir
The only problem for Farsi search seems to be Full Text search which may need a separate thread. Some helper applications particularly pdftotext (some versions of it) may not reliably extract Farsi text within pdf files. This seems to depend on how the pdf file has been created. Those created with OpenOffice or Acrobat Distiller seems to have no problem. Jaws PDF Creator on the other hand may produce pdf files with Farsi text unextractable. Older versions of pdftotext as is available in xpdf3.01 and below may not convert Farsi pdf files appropriately. New Poppler version of pdftotext which is available in Poppler 0.10.xx and above seems to be more productive. Therefore the problem of Farsi full-text search seems to be unrelated to OJS itself. it is the problem of helper apps or quality of pdf file. Currently I am trying to upgrade our system, hence upgrading the pdftotext too. This way I hope some Farsi indexing problem will be solved.

Second. regarding the eftekharb installation at
http://smj.ajums.ac.ir/
using non-full text fields search returns nothing. This may not be related to helper apps, since the search caries out within the metadata of the article which are in separate table.

Third. The problem reported in
http://pkp.sfu.ca/support/forum/viewtopic.php?f=8&t=4283
seems unrelated to Farsi search. I checked some Farsi string from eftekharb and concluded that it has genuine space character (ascii 32) between words and no white space character within words.

Thank you

[jmacgreg]

HI Mahmoud,

Thanks for your detailed reply! I used to use PDF creation tools a while ago, and I do recall that their quality and their capacity can vary widely. One tool that my old institution used in their PDF creation workflow was ABBYY Reader -- http://www.abbyy.com/. It's definitely not free, but I do believe they had success running old PDFs through it to generate full text for indexing. If this sounds like it might be something that could help you, I can ask them to provide more detail.

Regarding your other points: that's a good observation that this is an issue with metadata not being searchable. I had included that other thread not to point out the possibility of non-valid spaces being the culprit; rather, to point out some useful checkpoints for eftekharb to check against:

Code: Select all

   1. check that your browser is submitting forms in UTF-8 (although this usually isn't a problem)
   2. check that your config.inc.php settings are using utf8 as the connection_charset
   3. your database collaiton is set to UTF-8 (utf8_general_ci or utf8_unicode_ci) collation
   4. all of your tables are set to the same collation as the database
   5. all of your columns in the tables are likewise set to the same collation
   6. check the article_search_keyword_list table and perhaps search the keyword_text column to see if the string you're looking for is there

My apologies for not making it clear why I included that link. Since non-full-text search is working for you (correct?) but not for eftekharb, my inclination is that eftekharb should check through that list first, and then get back to this thread with results.

Cheers,
James