Strange Database Output after "set names 'utf8';"

[sesch]

Hi all,

It seems that, after migrating our OJS 2.2.4 to another server, I had a charset problem with our journals at http://online-journals.org. I have been advised to run a

set names 'utf8';

on the used database. Which I have done, and now it's even worse. The navigation, for example, looks like this:

##navigation.home## ##navigation.about## ##navigation.login##

Thank god, I have done this on a copy of the original database. We have reactivated the original one in the config.inc.php, but the problem persists (Yes, I have cleared the template cache). Does anyone have an idea where we could find the error?

Thanks a lot!
Sebastian

PS.: By the way, the charset problem persists, too. If anyone had an idea...

[JasonNugent]

Hi Sebastian,

The hashed strings you are seeing (##navigation.home##, etc) are untranslated locale keys. They get displayed when OJS can't find a key for them in the current locale. Did anything change with your locale settings when you migrated?

Regards,
Jason

[sesch]

Hi Jason,

thanks for your answer! You mean the locale setting in the config.inc.php? No, I don't think so. It's still en_US, UTF8, and the XML file with the keys is there in the locals directory.
And the problem didn't occur after the migration, but after I had run the set names 'utf8'. Short version:
1. We migrated the OJS, charset problem, still visible notably here: http://online-journals.org/i-jet/issue/view/120 (interesting: the issue published after migration has a correct special character (http://online-journals.org/i-jet/issue/view/135 ,in Érico))
2. This afternoon I have done the set names thing in PhpMyAdmin, and now we have the untranslated keys
3. We have connected OJS to the DB backup, but the problem is still there.

Where else could I look?

Thank you,
Sebastian

[JasonNugent]

Hi Sebastian,

I'm beginning to wonder if the database character set conversion munged a setting that controls the supported locales for your journal. Can you browse your database via a tool like phpMyAdmin? In the journal_settings table there should be a record that has a setting_name of 'supportedLocales'. The setting_value in that record will be a serialized array that should look something like:

Code: Select all

a:1:{i:0;s:5:"en_US";}


if you only have one supported locale. I'd be curious to know what that says now. If it's garbled, you may want to see about resetting the languages in Site Administration -> Languages. The locale keys themselves are loaded from files on disk, not from a database, so they wouldn't be affected by the database command. If they used to work, but now do not, it's probably not a permission problem on the files, but you may want to also verify that the web server can read the files in locale/ and also in lib/pkp/locale/ (and all sub directories).

Good luck,
Jason

[sesch]

Hi Jason,

Code: Select all

a:1:{i:0;s:5:"en_US";}


indeed, we have 6 journals on this OJS, and I found this entry 8 times in the said table (which is a little strange because we have also installed de_DE as second locale, but in the DB is only en_US).
What do you mean by

you may want to see about resetting the languages in Site Administration -> Languages

I have clicked the "Reload Language" button, that's it? That doesn't have any effect.

Another thing: I cannot choose the primary locale: http://schreiter.fr/journals/screenshot2.gif Could this be an indication?

We do have the locale files in locale/en_US/locale.xml, and the server is able to read them. But we do not have lib/pkp/locale/. What is supposed to be in these directories?

Thanks a lot,
Sebastian

[JasonNugent]

Hi Sebastian,

Actually, since you're running 2.2.4 you won't have a lib/pkp/locale directory. That came later with 2.3. The form chooser for primary locale will show as it does for you since the values put in the list are also going to be locale keys.

I am beginning to believe that you might be encountering a bug that came later in OJS, with locales. One of the things that we fixed in the current release was a clash between a class that OJS defines and a class that comes with newer versions of PHP. (The Locale class name). Let me see if I can dig a reference to the bug.

Regards,
Jason

[JasonNugent]

Hi again,

This is the relevant bugzilla entry:

http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=5240

You might want to ask your new host what version of PHP they are using. If it's PHP 5.3, would it be possible to use a 5.2 version instead? The alternative might be to upgrade your OJS install, but that's probably a lot of work. There is a comment (#8) on that bugzilla report that may also be helpful, as well as a few other relevant links to forum pages.

Regards,
Jason

[sesch]

Hi Jason,

indeed, we use PHP 5.3.5. EDIT: That leaves still the question why the journals have been runnnig almost fine for 3 weeks now, and why the bug has occurred only after my "set names" operation.

Would it be helpful and possible to install a fresh 2.3.7 and to connect it to the old DB? Or has the DB changed since the new version?

Thanks,
Sebastian.

[JasonNugent]

Hi Sebastian,

Yes, the database between 2.3.7 and 2.2.4 is very different. You wouldn't be able to use the new code with the old database.

I'm not sure why PHP 5.3 has been working for you -- was it always 5.3? Maybe something changed on the server recently? OJS does some caching. Maybe the set names command triggered a reload. Anything at all in your server's error log? Is a rollback to a PHP 5.2 version a possibility?

Regards,
Jason

[sesch]

Hi Jason,

a rollback to PHP 5.2 seems not to be wanted. But anyway, we have made a new install of 2.2.4 in another directory, connected it to the old database, removed it to the usual directory. Now it works again. Without knowing what was wrong and with PHP 5.3.

We'll now try to upgrade to 2.3.7.

But the origin of all these troubles is still left: the wrong special characters. Do you have any idea what to check?

Sebastian.

[asmecher]

Hi Sebastian,

When running a mysql dump/restore, I've found that character sets can get garbled unless I specify --default-character-set utf8 on the mysqldump command line. If that's the case, you may have to run a database dump through a tool like iconv to get the data back in shape and then reload it. Be careful not to mix content that's broken with new data that's correctly encoded! Mixed encodings are extremely difficult to disentangle.

Regards,
Alec Smecher
Public Knowledge Project Team

[sesch]

Dear Alec,

after all this time, thanks for your help regarding this issue! We haven't found a good solution. It seems that the characters have been broken from the beginning, but our former OJS installation showed them correctly. That seems strange, but it's the only explanation. Currently we have somebody who revises the DB to correct characters semi-automatically, he has already done this before.

Thanks again!
Sebastian