Horror with utf8 and LAMP ( perl ) web application contiunued:
We need to take care of mysql connection, so it is ut8 – ready:
if (my $dbh = DBI->connect(“DBI:mysql:database=”.$name.’;host=’.$hostname, $user, $password,
{
RaiseError => $raise,
# AutoCommit => 1,
mysql_enable_utf8 => 1,
on_connect_do => [ "SET NAMES 'utf8'", "SET CHARACTER SET +'utf8'" ],
})) {$dbh->{‘mysql_enable_utf8′} = 1;
return $dbh; # DBI database handler
}
We also must make sure that our tables are unicode:
CREATE TABLE `foo` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
….) DEFAULT CHARSET=utf8
And remember, that utf8 chars may take more space than normal – so prepare longer varchars in tables and be prepared for problem with indexes, like this: Specified key was too long; max key length is 1000 bytes – see http://bugs.mysql.com/bug.php?id=4541
After wrestling with perl encoding, we need to make sure the pages of website we create are displayed in utf8.
This means we need to have proper header in pages, for example:
<!DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01 Transitional//EN” “http://www.w3.org/TR/html4/loose.dtd”>
<html>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=UTF-8″>
…
…
…
</head>
<body>
and/or:
Content-Type: text/html; charset=utf-8
Second step, is to set binmode on STDOUT (if we print our dynamically generated webpages)
binmode STDOUT, “:utf8″;
to get rid of
Wide character in print at …
warnings.
The data we get from outside (like STDIN) is usually binary data.
Command like uc, lc wants text strings.
Here is how to change binary data to text strings:
use Encode ;
$foo = Encode::decode_utf8($foo);
you can also convert from other encodings:
$data = decode(“iso-8859-2″, $data);
To check whether a string have utf flag turned on:
use Encode qw(is_utf8);
print is_utf8($foo) ? “utf8″ : “not utf8″
If you do not know what encoding is the data, use perl module:
use Encode::Guess;
my $enc = guess_encoding($data, qw/euc-jp shiftjis 7bit-jis/);
You need to tell explicitly which encodings are suspected, because
by default, it checks only ascii, utf8 and UTF-16/32 with BOM.
Encode::Guess->set_suspects(qw/euc-jp shiftjis 7bit-jis/);
But remember, that the guessing is not magic, and it likely to fail.
For example it may fail to recognize whether data are in is-8859-1 or iso-8859-2
Because:
The reason is that Encode::Guess guesses encoding by trial and error. It first splits $data into lines and tries to decode the line for each suspect. It keeps it going until all but one encoding is eliminated out of suspects list. ISO-8859 series is just too successful for most cases (because it fills almost all code points in \x00-\xff).
Making a correct utf8 web application in LAMP (Perl) is not easy. There are lots of dangerous traps along the way.
We need to take care of source code encoding, web forms data – inputed by user - encoding, mysql encoding, displaying data (writing) encoding, and perhaps also take care of data taken from disk files.
First, we need to take care of source code, if we want to write there something like
$var = “zażółć gęślą jaźń”
we need to use utf8
The use utf8 pragma tells the Perl parser to allow UTF-8 in the program text in the current lexical scope
Do not use this pragma for anything else than telling Perl that your script is written in UTF-8.
use utf8 is not a magical trick to fix all problems with utf8, just a beginning of the journey…
We must have a good code editor or IDE that understands utf8. It will be also nice to have possibility to open files with other other encodings and convert them to utf8. For unix/linux there is for example KDevelop, and many other tools. For windows, there are many editors too. See: http://en.wikipedia.org/wiki/Comparison_of_text_editors#Unicode_and_other_character_encodings and http://www.alanwood.net/unicode/utilities_editors.html
When using use utf8 remember to save code in utf8 encoding!
To be continued…
More nice people I met:
Gábor Szabó – no, not the hungarian guitarist, nor football player, but the guy who makes Padre and advertises perl so it does not die.
Three nice guys from Poland (okay, two, as I travelled with one): Maciej, Perl developer from booking.com, and the second I do not know much about 
And lots and lots more, but its no use to write about all
On the YAPC::EU this summer I met a few nice people.Here is a short list, just a few form many nice people:
I was honoured to meet Larry Wall, who told me to have “appropiate amount of fun” on my presentation about WTFs.
A few minutes earlier I met Damian Conway, whom I helped with aplying last-minute changes to conference schedule and putting in on conference room door, before I realised who he was.
Paul Fenwick dressed as Star Trek hero did presentation about klingon Perl programming. A bit earlier this day, Damian Conway did completely different presentation about klingon Perl programming, making with his voice sounds that soon made him thirsty.
to be continued…
I wrote about wondering how to make “login” to a dynamic website in Perl. The best solution advised by http://perldesignpatterns.com/?WebAuthentication was to make a temporary token: “cookie with an authorization token. Store the token in the database along with an expiration time separate of the cookie. The token should be random generated and completely seperate from the password but handed out when the password is validated. This is the best case;”, but it was overshot for now, so I settled up for this scheme:
Whan user registers, his password is stored as md5 digest in database. Salt is generated – string of eight random letters, numbers etc.I use Crypt::PasswdMD5 qw(unix_md5_crypt);
When user logs in, password is checked- crypted using crypted pass from database as salt:
if ( $cryptedpassword eq unix_md5_crypt($password, $cryptedpassword)) {
and if it is ok, cookie is stored with user ID and crypted password.
The cookie is then checked on every page, whether it contains the crypted password from database.
Well, this is my idea of doing it for now, already implemented, I feel a bit unease about that – what is the point of crypting password and storing it crypted, as it really matters whether the pass from cookie is equal to pass in database – it could be not crypted and it would work the same way.
The only advantage is that the password is not stored in cookie – but it is not needed, as just the digest is needed to pretend to be logged in.
What do you think?
I received feedback from conference attendees about my presentation at YAPC::EU. And it was very good
After the conference attendees are encouraged to express their opinions on presentations they attended and grade them. Then the aggregated grades and opinions are sent to speaker.
I was a bit nervous, but fortunately presentation about WTFish side of using Perl was graded between 7 and 10 (on a scale from 1 to 10), mostly 8-9.
There were also opinions – good and bad.
Bad: I could have been speaking louder and with better intonation and gestures. Slides also could be a bit better. Someone even wrote “Poor guy, it is hard to present in another language. I hope he isnt disheartened, his presentation is good”
Good opinions (quoted from the feedback I got):
- Covered well some common but not obvious traps in Perl for newbies. The material was good.
- Good title and subject – humourous way to present this sid of Perl. As a total beginner some parts I knew of and felt the same WTFness and it is nice to feel that it is as mad as I thought!
- This presentation pin pointed some of the things that regularly “bite” newbies, so it was most valuable. It had many examples, and I hope I can remember them all. [ note: there are slides available to download at this blog, so you don't have to remeber all - Lech ]
Poor guy, it is hard to present in another language. I hope he isnt disheartened, his presentation is good
Thanks for good opinions and also for bad – those taught me something about presentations. Next time it will be better
Lately I was thinking about implementing logging (as in login/password) into a website with Perl. I mean, that I want to have a dynamic website, where users can login using login and password and do some stuff.
After googling for “how to make login site” mostly PHP session solutions came up. When I added “perl” to that google query, half of results was about logging into someone’s website (wget, LWP, WWW::Mechanize etc) and not making a login feature in existing dynamic website. Search for “perl authentication” found more relevant stuff.
Some pages suggested using .htaccess. Bleh. Beside, I want to have user in a SQL database.
One result pointed to CGI::Session on cpan. Nice. CGI::Session::Auth – nice too There is also Apache::Session to be found on cpan.
There was also a hint, to find a perl web forum script and see how it is done there. We’ll, I did it with one forum script, and found that user login and crypted password is stored in a cookie. Nice.
There also books that talk about how to do web authentication: Perl for Web Site Management or CGI programming with Perl
Here we have some design patterns: http://perldesignpatterns.com/?WebAuthentication and also: http://perldesignpatterns.com/?SessionPattern and http://perldesignpatterns.com/?PerlDesignPatterns . Even nicer
But I still look for ultimate “how to” with examples about authorizing users (register, login, remind password etc.) into some kind of dynamic website with Perl. Any hints?
Often, when I googled for information about Perl, that website came up in search results: http://www.rocketaware.com/perl/ .
It looks quite ugly, like but is great source of information about perl, especially for beginners. Take for example this page: http://www.rocketaware.com/perl/onepage.htm – it list all that cryptic “Short-hand variables.”, and also functions and operators.
This page does not only seem to be old, it is old. For example this page: http://www.rocketaware.com/perl/perlfaq4/Why_aren_t_my_random_numbers_ran.htm points to http://www.perl.com/CPAN/doc/FMTEYEWTK/random , where you can read (at least now) that:
Note that the old documentation, often advertised as www.perl.com/doc or www.perl.com/CPAN/doc, has been removed from CPAN as of November 2001 because it had not been updated for several years”.
The documentation in not-so-archaic distributions of perl points to this place instead: http://www.cpan.org/misc/olddoc/FMTEYEWTK.tgz where the old docs are stored. The rocketaware site seem to not be aware of those year 2001 changes, and in footer one can read: “Copyright 2000, Forrest J. Cavalier III.”
So to sum op, this site is:
- old
- ugly
- useful (at least if you don’t mind using perl as it was year 2000)
The page http://www.rocketaware.com/perl/onepage.htm alone makes this website worth recommendation.
