Encoding problem after upgrading to 2.3.7

[Ajakss]

Hi,

I have upgraded system to 2.3.7 version from 2.3.3-3 but now I have problem with encoding polish characters. I think the parameters in config are correct and in the database I see that encoding is set as utf8_general_ci.

; Default locale
locale = en_US

; Client output/input character set
client_charset = utf-8

; Database connection character set
connection_charset = utf-8

; Database storage character set
database_charset = utf-8

; Enable character normalization to utf-8 (recommended)
charset_normalization = On

I would be very appreciated for your help. After reading many threads devoted encoding I do not have any idea how to fix this problem.

Kind regards,

Adam

[asmecher]

Hi Adam,

Try turning charset_normalization off; that's the best setting to use. If that doesn't work, please describe the problem you're having in more detail.

Regards,
Alec Smecher
Public Knowledge Project Team

[Ajakss]

Hi Alec,

I have turned charset normalization off but no effect. I have upgraded to 2.3.7 version from 2.3.3-3 because the last installation was failed, i.e., I have got blank page and no messages why. For a long time the OJS worked very well (http://www.ujk.edu.pl/infotezy/ojs/index.php/infotezy).
First, I have copied files from new code to directory, and then I have upgraded the database via the web. Any errors but the some characters were replaced by ?
I have enabled the debug options but these messages are quite unclear

(mysql): SET NAMES 'utf-8'
________________________________________
1115: Unknown character set: 'utf-8'

ADOConnection._Execute(SET NAMES 'utf-8') % line 845, file: adodb.inc.php
ADOConnection.Execute(SET NAMES ?, Array[1]) % line 602, file: adodb-mysql.inc.php
ADODB_mysql.SetCharSet(utf-8) % line 157, file: DBConnection.inc.php
DBConnection.connect() % line 120, file: DBConnection.inc.php
DBConnection.initConn() % line 77, file: DBConnection.inc.php

and

ojs2 has produced an error Message: WARNING: Cannot modify header information - headers already sent by (output started at.../ojs/lib/pkp/classes/core/PKPApplication.inc.php:368) In file: .../ojs/lib/pkp/classes/template/PKPTemplateManager.inc.php At line: 295 Stacktrace: File: (unknown) line (unknown) Function: PKPApplication->errorHandler(2, "Cannot modify header information - headers already sent by (output started at .../ojs/lib/pkp/classes/core/PKPApplication.inc.php:368)", ".../ojs/lib/pkp/classes/template/PKPTemplateManager.inc.php", 295, Array(7)) File: .../ojs/lib/pkp/classes/template/PKPTemplateManager.inc.php line 295 Function: header("Content-Type: text/html; charset=utf-8") File: .../ojs/pages/index/IndexHandler.inc.php line 78 Function: PKPTemplateManager->display("index/journal.tpl")

BTW, the list of errors is quite long

Kind regards,

Adam

[asmecher]

Hi Adam,

Did you do a database dump/restore? Sometimes mysqldump generates the output in latin1 even though the contents may be UTF8; upon reloading, your character encodings will be garbled. They may even look OK if you're querying the database directly (e.g. through a command-line mysql session) because your terminal emulator is catching and interpreting them before they get to your eyes.

The best place to start is in the database. Find a piece of content that's got an accent, e.g. run this in MySQL:

Code: Select all

SELECT length(setting_value), setting_value FROM article_settings WHERE setting_name = 'title' AND setting_value LIKE '%some snippet of text from an article title containing accents%';

Verify that the length is what it should be. If you see garbled accents, or if the length isn't the same as the number of characters in the title, then you'll have to correct the data in your database. If that all appears to match, then we can go back to looking for a solution in OJS.

Regards,
Alec Smecher
Public Knowledge Project Team

[JasonNugent]

Hey Adam,

Try using 'utf8' instead of 'utf-8' for your database_charset parameter in config.inc.php. The MySQL SET NAMES command is failing because utf-8 is not a valid character set -- it's utf8, without the dash. That would explain the headers error you're also seeing since that ADODB error is putting content out to your browser before the other stuff works its magic. If that content shows up first, then OJS doesn't have a chance to set the charset header correctly and your characters will be garbled.

You can leave the other config options, like client_charset, at utf-8 since that is correct for those.

Cheers,
Jason

[Ajakss]

Hi Alec and Jason,

thank you very much for your replies. Unfortunately, the change to utf8 did not solve my problem but the error is not displayed.
I am not sure if I do correctly the MySQL query.

SELECT length( setting_value ) , setting_value FROM article_settings WHERE setting_name = 'title' AND setting_value LIKE '%między%' LIMIT 0 , 30

I have got two records.

length(setting_value) setting_value
50 Dziennikarstwo radiowe: między misją a komercją
60 Udział dziennikarzy w reklamach. Między etyką a praktyką

In the first record the original length is 42 characters and 47 with spaces, in the second example 49 and 56 with spaces. Is does it mean that encoding is not in utf?

Kind regards,

Adam

[asmecher]

Hi Adam,

That's 1 extra byte for each accented character, meaning that you have the encodings garbled in your database, unfortunately. I suspect if you tinker with your terminal emulator's character set configuration you'll get a higher-fidelity result on the terminal; alternately, try using another database front-end like phpMyAdmin. I've heard that you can correct this fairly handily by running mysqldump into myconv and back into mysql again to reload.

Regards,
Alec Smecher
Public Knowledge Project Team

[Ajakss]

Hi Alec,

thanks a lot for your commitment. Doing this search I have used phpmyadmin but the view was normal.
I do not have experience of using mysqldump and I would be very appreciated if you could describe these method more specifically.
Thanks in advance for your help.

Kind regards,

Adam

[asmecher]

Hi Adam,

mysqldump is a command-line tool to generate a database dump from the command line. Everything you do with mysql/mysqldump can also be done via phpMyAdmin, so if you're more comfortable using that tool, that's OK too.

In any case, I'd suggest downloading a database dump (a.k.a. backup) from phpMyAdmin and working with it using a good programmer's text editor. That should show you what the content is like more clearly than browsing it via the web interface.

Regards,
Alec Smecher
Public Knowledge Project Team

[Ajakss]

Hi Alec,

thank you for reply. My provider advised me to add the line

Code: Select all

mysql_set_charset("UTF8", $db);

in OJS settings because the different encoding is defined on the client side. But I do not know where the file is located in order to check these settings. I would be appreciated for your help.

Kind regards,

Adam

[asmecher]

Hi Adam,

That's equivalent to the connection_charset setting in config.inc.php, which I think you've already said is set to UTF8. (Both implementations run a "SET NAMES" statement in MySQL.)

Regards,
Alec Smecher
Public Knowledge Project Team

[Ajakss]

Hi Alec,

it works perfectly, even the invisible pages after upgrade became available
Many thanks for your help.

Kind regards,

Adam