OJS OCS OMP OHS

You are viewing the PKP Support Forum | PKP Home Wiki



Charset issue on mails with OJS 2.1.2 installation

Are you responsible for making OJS work -- installing, upgrading, migrating or troubleshooting? Do you think you've found a bug? Post in this forum.

Moderators: jmacgreg, btbell, michael, bdgregg, barbarah, asmecher

Forum rules
What to do if you have a technical problem with OJS:

1. Search the forum. You can do this from the Advanced Search Page or from our Google Custom Search, which will search the entire PKP site. If you are encountering an error, we especially recommend searching the forum for said error.

2. Check the FAQ to see if your question or error has already been resolved.

3. Post a question, but please, only after trying the above two solutions. If it's a workflow or usability question you should probably post to the OJS Editorial Support and Discussion subforum; if you have a development question, try the OJS Development subforum.

Charset issue on mails with OJS 2.1.2 installation

Postby mbria » Thu Oct 25, 2007 4:49 am

Dear all,

Recently I noticed and new issue with my OJS 2.1.1 installation.

Looks like my OJS is unable to encode ANY mails in the right way. Here you have a trimmed example of a mail sent to an author (in Spanish):

=?UTF-8?B?b23DrWE6IHJlcGVuc2FuZG8=?=
Content-Type: text/plain; charset="utf-8"
X-Mailer: Open Journal Systems v2
X-Originating-IP: 80.24.189.197
From: Marc =?UTF-8?B?w43DsWlndWV6IFJ1ZWRh?= <mark.bria@uab.es>
> Message-Id: <20071012114338.6BE7278C63D@myserver.uab.cat>
> Date: Fri, 12 Oct 2007 13:43:38 +0200 (CEST)
>
> Dear Víctor Doe,
> Una vez recibidas las evaluaciones de su artículo, el Comité Editorial de
> Athenea ha decidido asumir el dictamen de l@s evaluadores y en consecuencia,
> aceptar su artículo tras alguna modificación.
> Cordialmente,
>
> Comité editorial


I reviewed my locales, mysql databases (data, table, db and connection encodings), php.ini and so on and I'm unable to notice what is happening. :-(

I dig into the forum and the only reference I found about this issue is:
viewtopic.php?f=8&t=1352&p=4175&hilit=mail+charset#p4175

But looking into the diff file seams that this patch is just for mail comments and in my OJS it happens for EVERY mail sent by the platform. May I apply the patch? Is this a real OJS issue solved in the new release or is just my server's configuration?

Any suggestion about how to find what is happening is appreciated. Let me know if I more information is needed to clarify the issue.

Thanks a lot in advance,

m.
mbria
 
Posts: 294
Joined: Wed Dec 14, 2005 4:15 am

Re: Charset issue on mails with OJS 2.1.2 installation

Postby asmecher » Thu Oct 25, 2007 10:57 am

Hi mbria,

I've seen sporadic problems with UTF8 data getting double-encoded, all of which have been corrected in CVS with one or two minor patches, but I've never seen a bug that affects all emails on the system. Has your installation recently been updated or modified? Has your host recently changed something on the system, e.g. upgrading PHP or MySQL?

ps. I'm assuming that the version number in the title should read 2.1.1 -- there is no OJS v2.1.2.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8597
Joined: Wed Aug 10, 2005 12:56 pm

Re: Charset issue on mails with OJS 2.1.2 installation

Postby mbria » Thu Nov 08, 2007 1:26 pm

Hi Alec,

Thanks for your answer and sorry for my delay.

I've seen sporadic problems with UTF8 data getting double-encoded, all of which have been corrected in CVS with one or two minor patches, but I've never seen a bug that affects all emails on the system.


I will ask my users to be full sure... but they reported this issue for every OJS generated mail.
Are you suggesting a CVS install instead of a clean 2.1.1?
I would prefer patching my stable platform... but if there is no choice I can try with the CVS version.
BTW, before the CVS test... do you know what patches need to be applied?

Has your installation recently been updated or modified? Has your host recently changed something on the system, e.g. upgrading PHP or MySQL?


Sure. Sorry if I missed to say that we moved from our old hosting system to our own brand new housed server.

Databases were dumped but data looks great in the DB (any appreciable charset issue there).

I full installed it all, so here you have a brief summary of software versions (let me know if more info is needed):

    Apache 2.2.3
    PHP Version 5.2.1
    MySQL 5.0.38
    Postfix 2.3.8 (with Postfix to Sendmail compatibility interface)

About locales?
Code: Select all
marc@myserver:~$ locale
LANG=en_US.UTF-8
LC_CTYPE="es_ES.UTF-8"
LC_NUMERIC="es_ES.UTF-8"
LC_TIME="es_ES.UTF-8"
LC_COLLATE="es_ES.UTF-8"
LC_MONETARY="es_ES.UTF-8"
LC_MESSAGES="es_ES.UTF-8"
LC_PAPER="es_ES.UTF-8"
LC_NAME="es_ES.UTF-8"
LC_ADDRESS="es_ES.UTF-8"
LC_TELEPHONE="es_ES.UTF-8"
LC_MEASUREMENT="es_ES.UTF-8"
LC_IDENTIFICATION="es_ES.UTF-8"
LC_ALL=es_ES.UTF-8


As you notice, everything is es_ES.UTF-8 except LANG that is in English... mainly to get untranslated errors and facilitate administration tasks.

ps. I'm assuming that the version number in the title should read 2.1.1 -- there is no OJS v2.1.2.

No, I'm working on OJS v2.1.2 right now... so please, be patient. I will release it soon.
Just kidding, obviously was a typo error. :-P

Thanks a lot for your help,

m.
mbria
 
Posts: 294
Joined: Wed Dec 14, 2005 4:15 am

Re: Charset issue on mails with OJS 2.1.2 installation

Postby asmecher » Thu Nov 08, 2007 5:20 pm

Hi mbria,

I suspect that the database dump and move resulted in a change of character set encodings. Did your character set connection parameters (i.e. connection_charset) in config.inc.php change when you moved the install? It's sometimes necessary to specify a character set manually on the mysqldump command line. Even if the data appears to be fine in the database, it may be corrupted -- for example, if UTF-8 data is inserted in a Latin1 database, but you're using a UTF-8 terminal to fetch information out of it, it'll appear to be fine even though it's not.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8597
Joined: Wed Aug 10, 2005 12:56 pm

Re: Charset issue on mails with OJS 2.1.2 installation

Postby ramon » Tue Nov 20, 2007 10:26 am

Hello all,

This problem happens to us everytime. Since MySQL is used in many servers by many applications, in different charsets, you MUST set the default charset to UTF-8 when dumping, via SHELL, which is most secure.
ramon
 
Posts: 929
Joined: Wed Oct 15, 2003 6:15 am
Location: Brasí­lia/DF - Brasil

Re: Charset issue on mails with OJS 2.1.2 installation

Postby mbria » Wed Nov 21, 2007 4:45 am

Thanks you both for your answer... and sorry again for my silence. A new project is taking all my time :-(

Even if the data appears to be fine in the database, it may be corrupted -- for example, if UTF-8 data is inserted in a Latin1 database, but you're using a UTF-8 terminal to fetch information out of it, it'll appear to be fine even though it's not.

Yes. I remember original DB in "Latin_Swedish" so as you both suggest it could still be the problem, but:

1) How could I be sure this is the issue.
2) How could I fix it?

About 1:
Right now I'm not able to redump the DB, because the original hosting is death... so my question is:
¿Is there any way to know if this DB charset issue is the one that is happening here?

I checked my new DB with phpMyAdmin and reports:

    MySQL charset: UTF-8 Unicode (utf8)
    MySQL connection collation: UTF-8 Unicode (utf8)
    OJS database: utf8_general_ci
    Every OJS tables: utf8_general_ci
    email_templates_default_data.BODY: utf8_general_ci

Although phpMyAdmin default unicode charset, fortunately OJS systemInfo shows:

    locale: es_ES
    client_charset: utf-8
    connection_charset: utf8
    database_charset: utf8

Did your character set connection parameters (i.e. connection_charset) in config.inc.php change when you moved the install?

No it didn't. :-(

About 2:
I'm not very happy fixing problems without been sure about the origin, but if you are right, probably updating (by hand or due retailed OJS' install sql queries) every mail template with clean data will fix it... What do you think? Any further suggestion?

Thanks a lot for your help,

m.
mbria
 
Posts: 294
Joined: Wed Dec 14, 2005 4:15 am

Re: Charset issue on mails with OJS 2.1.2 installation

Postby asmecher » Wed Nov 21, 2007 5:49 am

Hi mbria,

This is a tricky one, and probably doesn't only affect emails, i.e. all special characters in the database might be affected. When you go into the Journal Manager's Prepared Emails interface and look at an email template, are special characters properly displayed there? Are special characters in article metadata (e.g. abstracts and titles) displayed properly? Or are outgoing emails the only problem you can identify?

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8597
Joined: Wed Aug 10, 2005 12:56 pm

Re: Charset issue on mails with OJS 2.1.2 installation

Postby mbria » Wed Nov 21, 2007 8:38 am

This is a tricky one, and probably doesn't only affect emails, i.e. all special characters in the database might be affected.

I just hope you to be wrong. :-)

When you go into the Journal Manager's Prepared Emails interface and look at an email template, are special characters properly displayed there?

Do you mean at: /manager/emails ?
Everything looks nice there. All mail templates are perfectly encoded in English, Spanish and Portuguese.

Are special characters in article metadata (e.g. abstracts and titles) displayed properly?

No problem there. All perfect too.

Or are outgoing emails the only problem you can identify?

Exactly... so I first thought was a general postfix problem, but everything goes well with all my other php applications in this same server (Drupal, Joomla, Moodle...) so I turn back to OJS and it's configuration.

Sorry a lot for the annoyance or asking and reasking, but I arrived to this moment where I need to recognize I have no idea about where else I can check.

Best regards,

m.
mbria
 
Posts: 294
Joined: Wed Dec 14, 2005 4:15 am

Re: Charset issue on mails with OJS 2.1.2 installation

Postby asmecher » Wed Nov 21, 2007 10:31 am

Hi mbria,

Hmm; if it's just the emails, it's probably not a database charset problem. Could you cause one of these messages to be sent to me personally (direct from your OJS installation, not forwarded)? Maybe I can figure out what's happening that way.

Are your Drupal, Joomla and Moodle messages being sent UTF-8? You should be able to check by viewing the message sources.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8597
Joined: Wed Aug 10, 2005 12:56 pm

Re: Charset issue on mails with OJS 2.1.2 installation

Postby ramon » Thu Nov 29, 2007 7:06 am

Hello all,

E-mails are really a big problem.
This issue is a recurring one in a few reports of support we've had, and no one has been able to really figure out what happens or actually fix this.

I know Joomla! uses ISO (latin-1) for encoding.
ramon
 
Posts: 929
Joined: Wed Oct 15, 2003 6:15 am
Location: Brasí­lia/DF - Brasil

Re: Charset issue on mails with OJS 2.1.2 installation

Postby mbria » Thu Nov 29, 2007 1:51 pm

Thanks you both guys for your support.

E-mails are really a big problem.

Full agree on this. Some authors, to correct their articles, request a Word Document, because mails sent by OJS are "really difficult or impossible to read".

Hmm; if it's just the emails, it's probably not a database charset problem.

I think the same. :-)

Could you cause one of these messages to be sent to me personally (direct from your OJS installation, not forwarded)? Maybe I can figure out what's happening that way.

Alec, I tried but at the end didn't forward anything to you because (as Ramón says) it's difficult to figure out WHEN it happens... sometimes work, and others didn't.

I need to close some other urgent stuff (not related with OJS :-P), but as soon as I get some time I will review this issue in detail to be sure I'm able to forward you more one of those "conflictive mails".

BTW, ¿any clue about how to trace it all?

Any case, thanks a lot again four your help.
I admire your efforts as far as a classical answer to users is "I can not fix an problem that I could not reproduce".

Best regards,

m.
mbria
 
Posts: 294
Joined: Wed Dec 14, 2005 4:15 am

Re: Charset issue on mails with OJS 2.1.2 installation

Postby mj » Thu Nov 29, 2007 6:05 pm

Hi all,

I've been following this thread with interest as it relates to two character encoding issues that we've been tracking (http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=2265, and http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=3089); if I recall correctly, all emails should be sent out in the default character set, which for most installations is UTF-8. OJS 2.2 will include a new character encoding core that attempts to normalize everything to UTF-8, although I can't remember if it's applied to sent emails as well. Reproducing encoding issues is very difficult, and email doubly so -- however, if we can find a case, then I'm glad to re-open the bugzilla entries.

Regards,
mj
Site Admin
 
Posts: 304
Joined: Fri Mar 26, 2004 9:32 am
Location: Toronto, Canada

Re: Charset issue on mails with OJS 2.1.2 installation

Postby asmecher » Thu Nov 29, 2007 11:24 pm

Hi mbria,

If you're able to narrow down the situation in any way, that would help -- e.g. only certain email templates have problems, etc. Ramon, if you've had similar problems, any thoughts you have for the conditions under which the problem occurs would be great.

Regards,
Alec Smecher
Public Knowledge Project Team
asmecher
 
Posts: 8597
Joined: Wed Aug 10, 2005 12:56 pm

Re: Charset issue on mails with OJS 2.1.2 installation

Postby ramon » Fri Nov 30, 2007 12:22 pm

Hello Alec & MJ,

As I fas as I can remember, most problems were due to the fact that the client was not able to read UTF-8 mail.
Sometimes, the subject was correct but the message corrupted.
Most of these were caused by server misconfiguration (MySQL collation issues most of the time).

There was one situation where a journal reported unreadable e-mails, even when changing the charset in the mail client, which led me to believe a double encoding was happening. I assumed it was a server problem, but no further news as far as I can remember...

Most support for these journals have attempted to change the mail function, but did not report back success or failure...
ramon
 
Posts: 929
Joined: Wed Oct 15, 2003 6:15 am
Location: Brasí­lia/DF - Brasil


Return to OJS Technical Support

Who is online

Users browsing this forum: Bing [Bot] and 3 guests