OJS OCS OMP OHS

You are viewing the PKP Support Forum | PKP Home Wiki



Problems with characters not showing up correctly

Are you responsible for making OJS work -- installing, upgrading, migrating or troubleshooting? Do you think you've found a bug? Post in this forum.

Moderators: jmacgreg, btbell, michael, bdgregg, barbarah, asmecher

Forum rules
What to do if you have a technical problem with OJS:

1. Search the forum. You can do this from the Advanced Search Page or from our Google Custom Search, which will search the entire PKP site. If you are encountering an error, we especially recommend searching the forum for said error.

2. Check the FAQ to see if your question or error has already been resolved.

3. Post a question, but please, only after trying the above two solutions. If it's a workflow or usability question you should probably post to the OJS Editorial Support and Discussion subforum; if you have a development question, try the OJS Development subforum.

Re: Problems with characters not showing up correctly

Postby soj » Tue Aug 05, 2008 11:59 am

I believe I have fixed my own problem. In phpMy Admin I looked through all linked table blocks individually and found a few instances of utf8_general_ci leftover in fields taking varchar or text, in the "access_keys" and email_templates" tables, to be exact. I changed these over to utf8_unicode_ci, and that appears to solved the issue.

Thanks anyway, soj
soj
 
Posts: 151
Joined: Fri Oct 28, 2005 1:53 pm
Location: Norman OK USA

Re: Problems with characters not showing up correctly

Postby mj » Tue Aug 12, 2008 9:01 am

Hi soj,

Sorry we weren't able to reply in time to address your question, but I'm glad to hear you were able to solve the problem!

Best regards,
MJ
mj
Site Admin
 
Posts: 304
Joined: Fri Mar 26, 2004 9:32 am
Location: Toronto, Canada

Re: Problems with characters not showing up correctly

Postby soj » Tue Sep 02, 2008 10:13 am

Well, this problem doesn't want to go away for me. I got the error

DB Error: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (latin1_swedish_ci,COERCIBLE) for operation '='

again when attempting to upload a new galley file. I looked through all my tables again and everything is utf8_unicode_ci. I ran a search of the database and no matches came up for latin1_swedish_ci. The only thing I can find that might be a problem is in the access keys row collation which reads as utf8_general_ci, and for which phpMyAdmin does not provide any means to edit. Any Ideas would be much appreciated!

soj
soj
 
Posts: 151
Joined: Fri Oct 28, 2005 1:53 pm
Location: Norman OK USA

Re: Problems with characters not showing up correctly

Postby mj » Tue Sep 02, 2008 11:20 am

Hi soj,

The "illegal mix of collations" error usually means you are trying to use two incompatible encodings within your database. In your case, "DB Error: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (latin1_swedish_ci,COERCIBLE) for operation '='" indicates that you are trying to mix utf8_unicode and latin1_swedish -- so somewhere in your database you have a column, table, etc. that is using latin1_swedish. I would take another look at your encodings and double-check the tables that are being accessed when this error appears.

MJ
mj
Site Admin
 
Posts: 304
Joined: Fri Mar 26, 2004 9:32 am
Location: Toronto, Canada

Re: Problems with characters not showing up correctly

Postby soj » Tue Sep 02, 2008 12:06 pm

Thanks MJ, I don't know exactly which tables are accessed with uploading a galley file. I have now looked through all the tables via phpMyAdmin three times today and found no latin1_swedish_ci value in any tables under any collation columns. I have run a search on all the tables for latin1 and latin1_swedish_ci. No matches in any tables. I don't know how else to look or where else to look. For what it's worth, this was not an issue at all until after the upgrade. At this point we cannot publish anything.

Any other thoughts?
Thanks in advance , soj
soj
 
Posts: 151
Joined: Fri Oct 28, 2005 1:53 pm
Location: Norman OK USA

Re: Problems with characters not showing up correctly

Postby soj » Tue Sep 02, 2008 1:32 pm

One more oddity that might narrow things down a bit: I can upload images and PDFs to the Galley, but not HTML files. Those give me the "DB Error: Illegal mix of collations (utf8_unicode_ci,IMPLICIT) and (latin1_swedish_ci,COERCIBLE) for operation '='"

soj
soj
 
Posts: 151
Joined: Fri Oct 28, 2005 1:53 pm
Location: Norman OK USA

Re: Problems with characters not showing up correctly

Postby asmecher » Tue Sep 02, 2008 1:35 pm

Hi Soj,

Did your database connection configuration (in config.inc.php) change when you upgraded? In particular, check your database connection character set configuration.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8345
Joined: Wed Aug 10, 2005 12:56 pm

Re: Problems with characters not showing up correctly

Postby soj » Tue Sep 02, 2008 2:22 pm

It's currently turned off, but it's been a while now since I upgraded... let me ck the backup. Yup, the original was set to utf8. I will try changing the current file to reflect the same.
soj
 
Posts: 151
Joined: Fri Oct 28, 2005 1:53 pm
Location: Norman OK USA

Re: Problems with characters not showing up correctly

Postby soj » Tue Sep 02, 2008 2:27 pm

It's been a merry little chase, but now looks solved, thanks! The localization settings were the problem, apparently.

soj
soj
 
Posts: 151
Joined: Fri Oct 28, 2005 1:53 pm
Location: Norman OK USA

Re: Problems with characters not showing up correctly

Postby josipkp » Tue Jan 27, 2009 6:28 am

I need help with a character setting problem: not showing up correctly.

I made a new OJS installation to receive an OJS 2.1.1 database.
Created a new database:
drop database ojsXXX;
create database ojsXXX character set 'utf8';
grant all on ojsXXX.* to ojsXXX@localhost identified by 'mypassword';

Localization Settings in config.inc.php:
locale = pt_BR
client_charset = utf-8
connection_charset = utf8
database_charset = utf8
charset_normalization = On

After OJS install, import the database and run "php tools/upgrade.php upgrade" and the version changed to 2.2.2.
No correct visualization:
Relação entre atividade turística, apropriação do território e patrimônio: uma contribuição para o planejamento sustentável do turismo na Bahia.

Try to convert the character set:
ALTER DATABASE ojsXXX DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;
and
ALTER TABLE <tableName> CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
for all 77 tables listed in mySQL

No difference: continues to showing up incorrectly.
If I try to edit metadados, I received the message:
Notice: unserialize() [function.unserialize]: Error at offset 197 of 1660 bytes in /var/www/turismo/classes/db/DAO.inc.php on line 288
but I can change the fields and correct the character set to show up correctly.

Anyone could to propose a solution for the entire database?

Thank you in advance,
Josi Perez
josipkp
 
Posts: 61
Joined: Fri Jun 27, 2008 8:51 am

Re: Problems with characters not showing up correctly

Postby mj » Tue Jan 27, 2009 11:44 am

Hi Josi,

You seem to be doing everything correctly as far as I can tell. Are you certain that the data in your original database is valid UTF-8, ie. that the characters aren't scrambled before getting into the new OJS database? OJS can clean incoming data in other (valid) encodings, but looking at the string you provide, it appears that the source content may already be corrupted.

If you have a link to the original journal or can provide some access details to me via PM or email, I'd be happy to take a look at your installation and see what's going on.

MJ
mj
Site Admin
 
Posts: 304
Joined: Fri Mar 26, 2004 9:32 am
Location: Toronto, Canada

Re: Problems with characters not showing up correctly

Postby mgscano » Tue Apr 28, 2009 3:48 am

After trying to alter all the latin_swedish crap out of the database via the conventional ways, i.e. creating a php function to do that recursively etcetera, I found out there's a much faster and effective way to fix the collation problem.

Seems long, but it's very easy and quick to do.

You just export your database via phpMyAdmin, then DROP it. Just drop all the tables from within the db if your host does not allow you to run the previuos command.

Go to the commands tab and change your entire database collation to utf8_unicode_ci.

Now, open the .sql file with a text editor like wordpad, emacs, gedit .... (NOT Microsoft Word :-)) and substitute the string latin1 with utf8. Save the file.

You should now be able to import your database from phpMyAdmin, and you'll see all the tables have the correct values.

my 0.02
mgscano
 
Posts: 72
Joined: Tue Jan 27, 2009 3:22 am
Location: Brescia, Italy

Re: Problems with characters not showing up correctly

Postby mj » Wed Apr 29, 2009 5:56 am

Hi Marco,

Regarding the process you describe, I've used the same approach many times and it works well, but there are a few points people might want to keep in mind when using it:

  1. Make sure when exporting from phpMyAdmin that your browser is receiving data (rather, that the server is sending pages) in UTF-8; sometimes it can default to Latin-1.
  2. Make sure your text editor can read and write UTF-8; as I recall, most versions of notepad and wordpad can't handle UTF-8, though more recent ones and most 3rd-party editors do.
  3. Again, when importing, make sure your browser is using UTF-8, otherwise it'll re-encode your nice UTF-8 text before saving it to the database.
  4. This approach only works on reasonable-sized database dumps, eg. less than a few MB, since many text editors can't handle opening files larger than that, although this varies with system and available memory. If you have a larger set of data, you may want to break the SQL into several files and process them one at a time.
Lastly, thanks for your contributions to the support forums -- this kind of community support is something we'd like to encourage, so please keep it up! :-)

MJ
mj
Site Admin
 
Posts: 304
Joined: Fri Mar 26, 2004 9:32 am
Location: Toronto, Canada

Re: Problems with characters not showing up correctly

Postby Kholodny » Fri Apr 19, 2013 6:59 am

Hello, Everyone!

I am trying to set up the multilanguage journal (russian/ukrainian/english) and installed OJS 2.4.2.0 from Softaculos in Cpanel.
I'm facing with the problem of wrong displaying of two cyrillic symbols entered anywhere in the forms or lines with names/titles etc.
This is the result of filling the form Focus and Scope with russian alphabet + extra ukrainian symbols lacking in russian:

АБВГДЕЁЖЗ�?КЛМОПРСТУФХЦЧЩШЬЪЫЭЮЯ ЇЄІ
абвгдеёжзиклмопрстуфхцчщ�?ьъыэюя їєі


All attempts to enter symbol "И" (U+0418: CYRILLIC CAPITAL LETTER I) and 'ш' (U+0448: CYRILLIC SMALL LETTER SHA) resulted in appearing �? (square and question mark). I tried to fix it using the 'recipe' descibed in this topic (the databases were in utf8_general_ci, I have changed the collation to utf8_unicode_ci), but without success. These symbols in other strings originated from ukrainian and russian locales appear normally. There was no problems with latin characters with diacritics from czech language (I've tried them just for example). I have not found any solution in the forum for that problem. If there any possible suggestions for the matter?

Thanks!


[Offtop: Just to add, if the problems have similar source - not all the information filled in to the forms (in setup row) in russian or ukrainian appear then e.g. in About section. If open the appropriate Steps again - the forms in non-english are empty.]


OJS configuration settings from config.inc.php.

general
installed On
base_url http://cryo.org.ua/journal
registry_dir registry
session_cookie_name OJSSID
session_lifetime 30
scheduled_tasks Off
date_format_trunc %m-%d
date_format_short %Y-%m-%d
date_format_long %B %e, %Y
datetime_format_short %Y-%m-%d %I:%M %p
datetime_format_long %B %e, %Y - %I:%M %p
time_format %I:%M %p
disable_path_info Off
allow_url_fopen Off
restful_urls Off
enable_cdn On
citation_checking_max_processes 3
show_upgrade_warning On
database
driver mysql
host localhost
username
password
name
persistent Off
debug Off
cache
object_cache none
memcache_hostname localhost
memcache_port 11211
web_cache Off
web_cache_hours 1
i18n
locale en_US
client_charset utf-8
connection_charset utf-8
database_charset utf-8
charset_normalization On
files
files_dir /home/khv/ojsdata
public_files_dir public
umask 18
finfo
mime_database_path /etc/magic.mime
security
force_ssl Off
force_login_ssl Off
session_check_ip On
encryption sha1
allowed_html <a href|target> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <b> <i> <u> <img src|alt> <sup> <sub> <br> <p>
email
enable_attachments On
time_between_emails 3600
max_recipients 10
require_validation Off
validation_timeout 14
search
min_word_length 3
results_per_keyword 500
result_cache_hours 1
oai
oai On
repository_id ojs.cryo.org.ua
oai_max_records 100
interface
items_per_page 25
page_links 10
captcha
captcha Off
captcha_on_register On
captcha_on_comments On
captcha_on_mailinglist On
font_location /usr/share/fonts/truetype/freefont/FreeSerif.ttf
recaptcha Off
cli
perl /usr/bin/perl
tar /bin/tar
xslt_command
proxy
debug
show_stats Off
show_stacktrace Off
display_errors Off
deprecation_warnings Off
Server Information
Basic operating system and server software versions. Click on Extended PHP Information to view extended details of this server's PHP configuration.

OS platform Linux
PHP version 5.2.17
Apache version Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.17
Database driver mysql
Database server version 5.0.96-community-log[/quote]
Kholodny
 
Posts: 10
Joined: Tue Apr 16, 2013 9:40 am

Re: Problems with characters not showing up correctly

Postby asmecher » Fri Apr 19, 2013 8:25 am

Hi Kholodny,

Try turning off the "charset_normalization" setting. (This will speed up your installation and is safe to leave disabled; it was useful before UTF-8 was so widely supported but is now a little redundant and occasionally trips over valid UTF8.)

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8345
Joined: Wed Aug 10, 2005 12:56 pm

PreviousNext

Return to OJS Technical Support

Who is online

Users browsing this forum: No registered users and 5 guests