LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (http://www.linuxquestions.org/questions/slackware-14/)
-   -   Daemon runs perl gets locale warning (http://www.linuxquestions.org/questions/slackware-14/daemon-runs-perl-gets-locale-warning-4175441421/)

catkin 12-14-2012 04:09 AM

Daemon runs perl gets locale warning
 
When a daemon (actually Bacula 5.2.12 director) runs a perl script, it logs
Code:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
 LANGUAGE = (unset),
 LC_ALL = (unset),
 LANG = "en_GB-UTF8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

I have searched the Internet for a solution but have found only non-Slackware solutions like dpkg-reconfigure locales and non-daemon solutions like modifying shell initialisation files.

There's a perldoc about locales. Following that (quoted in blue below) ...

Perl must believe that the locale system is supported. If it does, perl -V:d_setlocale will say that the value for d_setlocale is define:
Code:

c@CW8:~$ perl -V:d_setlocale
d_setlocale='define';

If you want a Perl application to process and present your data according to a particular locale, the application code should include the use locale pragma (see The use locale pragma) where appropriate, and at least one of the following must be true:
1. The locale-determining environment variables (see ENVIRONMENT) must be correctly set up at the time the application is started, either by yourself or by whomever set up your system account; or
2. The application must set its own locale using the method described in The setlocale function.


According to the ENVIRONMENT text, LANG is a "catch all" and should be effective when LANGUAGE, LC_ALL are not set, so having only LANG set, as indicated by the warning message, should be OK.

/opt/bacula/scripts/make_catalog_backup.pl appears to do nothing with locale:
Code:

c@CW8:~$ grep --ignore-case locale /opt/bacula/scripts/make_catalog_backup.pl
[no output]

So, as best I can figure, the system is set up correctly but perl doesn't think so ... ?

This is on Slackware64 14.0 which has perl 5.16.1.

Incidentally, it would be nice to know where the reported LANG value comes from. I can not find it under /etc or in the Bacula directory (this Bacula instance is all under a single directory):
Code:

root@CW8:~# find /etc -type f -exec grep --files-with-matches 'en_GB.UTF-8' {} \;
/etc/profile.d/lang.sh
root@CW8:~# find /etc -iname '*locale*'
[no output]
root@CW8:~# find /opt/bacula -type f -exec grep --files-with-matches 'en_GB.UTF-8' {} \;
[no output]


Habitual 12-14-2012 09:00 AM

Maybe
Code:

/etc/profile.d/lang.sh
#!/bin/sh
...
export LANG=en_GB-UTF8

?

Log out|in per usual. :)

catkin 12-15-2012 12:42 AM

Thanks Habitual :)

Would that help?

Firstly it looks as if envar LANG is set to en_GB-UTF8 as shown by the warning messages.

Secondly when daemons are started from boot scripts they do not generally run shell initialisation files. In this case, rc.local includes
Code:

# Start Bacula
# Requires: MySQL and sendmail
rm -f /opt/bacula/working/*.pid
# Use Bacula's locally modified internal mechanism rather than its boot scripts
/opt/bacula/scripts/bacula start

and neither /opt/bacula/scripts/bacula nor the three individual control scripts it calls, call /etc/profile or /etc/profile.d/lang.sh directly.

I will add a debugging call to locale in rc.local to see what has already been set up when the boot scripts are run.

catkin 12-15-2012 02:28 AM

Update
 
This looks like an issue of Slackware locales and/or perl integration.

Experiments showed:
  1. Same locale output while rc.S, rc.local and the bacula boot/control script:
    Code:

    LANG=
    LC_CTYPE="POSIX"
    LC_NUMERIC="POSIX"
    LC_TIME="POSIX"
    LC_COLLATE="POSIX"
    LC_MONETARY="POSIX"
    LC_MESSAGES="POSIX"
    LC_PAPER="POSIX"
    LC_NAME="POSIX"
    LC_ADDRESS="POSIX"
    LC_TELEPHONE="POSIX"
    LC_MEASUREMENT="POSIX"
    LC_IDENTIFICATION="POSIX"
    LC_ALL=

  2. Changing LANG in /etc/profile.d/lang.sh from en_GB.UTF-8 to en_US.UTF-8 and rebooting fixed the Bacula/perl problem.
Presumably -- and academically now, as far as this problem is concerned -- the Bacula director daemon does simulate at least some of shell initialisation, including processing /etc/profile.d/lang.sh. AFAIK there is nothing about this in the Bacula Main Reference document where it simply says "... you must explicitly ensure that your locale is set properly. Typically this means that the LANG environment variable must end in .UTF-8. A full example is en US.UTF-8. The exact syntax may vary a bit from OS to OS, and exactly how you define it will also vary". The perl warning message mentioned en_GB-UTF8, not en_GB.UTF-8 as was set in /etc/profile.d/lang.sh -- strangely changed and contrary to the documentation.

I am now trying to reproduce the locale/perl problem outside Bacula, starting in class perl 101 ...

EDIT:

It turns out the strange format of the LANG value is what is causing the problem. First reproducing the problem at the command prompt:
Code:

c@CW8:~$ export LANG=en_GB-UTF8                             
c@CW8:~$ /opt/bacula/scripts/make_catalog_backup.pl MyCatalog
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_COLLATE = "C",
        LANG = "en_GB-UTF8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
c@CW8:~$ export LANG=en_GB.UTF-8
c@CW8:~$ /opt/bacula/scripts/make_catalog_backup.pl MyCatalog
[no output on screen, backup file created]


NonNonBa 12-15-2012 03:30 AM

Hi,

Quote:

Originally Posted by catkin (Post 4849708)
[...]A full example is en US.UTF-8. The exact syntax may vary a bit from OS to OS, and exactly how you define it will also vary". The perl warning message mentioned en_GB-UTF8, not en_GB.UTF-8 as was set in /etc/profile.d/lang.sh -- strangely changed and contrary to the documentation.

Under linux the correct syntax is "en_GB.utf8", so the locale as you set it can't be found (the Bacula's example show them as they are specified in *BSD). Take a look at locale -a, if you want to see the acknowledged locales. :)

catkin 12-15-2012 04:05 AM

Thanks NonNonBa :) that fixed it. Changing lang.sh to set LANG to the en_GB.utf8 listed by locale -a and re-starting the Bacula daemons then running the catalog backup job resulted in no perl warnings in the log.

A solution is always good but there are some nagging loose ends left over:
  • Why does the as-installed /etc/profile.d/lang.sh have the sample "en_US.UTF-8" instead of a valid value?
  • Why did en_GB.UTF-8 in lang.sh end up as en_GB-UTF8 when reported by perl when run by Bacula? I searched the Bacula source code for where this might be happening but found nothing. Perhaps it is done by non-Bacula library code.
  • Why did setting en_GB.UTF-8 at the command prompt fix the problem (reproduced by setting en_GB-UTF8)?
Perhaps these strings are not case sensitive and/or the . and dash characters are only for legibility. I could not find any documentation to support that. Whatever -- the safe option is to use the formats listed by locale -a.

NonNonBa 12-15-2012 06:14 AM

Quote:

Originally Posted by catkin (Post 4849742)
Why does the as-installed /etc/profile.d/lang.sh have the sample "en_US.UTF-8" instead of a valid value?

Good question... ah, indeed this syntax works as well:

Code:

$ ls /xyz
ls: impossible d'accéder à /xyz: Aucun fichier ou dossier de ce type
$ LANG=el_GR.UTF-8 ls /xyz
ls: cannot access /xyz: Δεν υπάρχει τέτοιο αρχείο ή κατάλογος

So, I've also learned something. :)

Quote:

Originally Posted by catkin (Post 4849742)
Why did en_GB.UTF-8 in lang.sh end up as en_GB-UTF8 when reported by perl when run by Bacula? I searched the Bacula source code for where this might be happening but found nothing. Perhaps it is done by non-Bacula library code.

The locales are POSIX, therefore managed by the glibc. I think the strange behaviour you encountered was caused because you've modified lang.sh without reloading it. Bacula inherits the environment from the shell it ran in, so to get the changes you have made in lang.sh, you need to source it:

Code:

$ vi /etc/profile.d/lang.sh # make your changes...
$ . /etc/profile.d/lang.sh # load your changes in the current shell (attached to *this* xterm/console)
$ bacula restart # restart the daemon with the new environment.


catkin 12-15-2012 08:04 AM

Quote:

Originally Posted by NonNonBa (Post 4849773)
The locales are POSIX, therefore managed by the glibc. I think the strange behaviour you encountered was caused because you've modified lang.sh without reloading it. Bacula inherits the environment from the shell it ran in, so to get the changes you have made in lang.sh, you need to source it:

Code:

$ vi /etc/profile.d/lang.sh # make your changes...
$ . /etc/profile.d/lang.sh # load your changes in the current shell (attached to *this* xterm/console)
$ bacula restart # restart the daemon with the new environment.


Bacula is normally started by rc.local, which is sourced by rc.S which is started by the init process under the control of inittab. rc.S begins with #!/bin/sh so, as I understand it, it runs as a non-logon, non-interactive, POSIX mode bash shell and, being non-logon and non-interactive, bash does not source the /etc/profile bash startup file and hence does not source the /etc/profile.d files including lang.sh.

If my understanding is correct, it is puzzling that the LANG setting in lang.sh is used to derive the LANG setting when Bacula executes a perl script. As above the locale output when the bacula boot/control script is run from rc.local from rc.S are all POSIX -- consistent with bash running in POSIX mode and no action taken to set a locale.

The only locale calls I found in the Bacula source code (this is from memory) were setlocale(LC_ALL, ""). According to the setlocale (3) man page, that call results in setting the program's locale according to the environment variables. That is consistent with the Bacula documentation which says it is the administrator's responsibility to set an appropriate LANG envar. Logging the locale output when the bacula boot/control script is run during boot is thus not enough. Tomorrow I will add logging of the locale envars mentioned in the setenv (3) man page -- LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC, LC_TIME and LANG.

NonNonBa 12-15-2012 08:39 AM

True. During the init the locale is not set, but nothing prevents you from setting the appropriate locale for the bacula script.

IMHO, the best way is to source /etc/profile.d/lang.sh in the bacula boot/control script.

The second way is to run the bacula script with this locale from your rc.local:

Code:

if [ -x /path/to/rc.bacula ]; then
  LANG=en_GB.utf8 /path/to/rc.bacula start
fi

When you prepend definitions to a command, they are inherited in the environment of this one (and only of this one, the locale of the current shell remains the same). The LANG value is used as default for every locale-related variables (except LC_ALL, which is meant to temporarily force the locale), so you don't need to define the others.

catkin 12-16-2012 09:13 AM

Thanks NonNonBa :) that would work well during boot.

Aiming for consistency and "least surprises" -- especially when using Bacula to recover a lost root file system -- I'm inclined to make the LANG value a Bacula configuration item. In this installation each of the three Bacula daemons is ultimately started during boot by its own control script in /opt/bacula/scripts. These can be run from the command prompt to start and stop the daemons individually. In case that is done the most robust solution would be to modify each script to source a LANG setting configuration file, say /opt/bacula/etc/lang.conf.

Such an arrangement would also suit doing a root file system recovery by doing a basic OS installation (in which /etc/profile.d/lang.sh would not need to be customised), recovering a non-Bacula backup of /opt/bacula and then using Bacula to recover the root file system.

As suspected, experimentation today showed LANG is not set during boot when the Bacula boot script is run. So the question of how /etc/profile.d/lang.sh influences the value of LANG set by Bacula for its perl scripts remains. I can't find any such thing in the Bacula source code which suggests it is done by one of the libraries installed with the OS -- and that is intriguing.

NonNonBa 12-18-2012 04:07 PM

Quote:

Originally Posted by catkin (Post 4850391)
As suspected, experimentation today showed LANG is not set during boot when the Bacula boot script is run. So the question of how /etc/profile.d/lang.sh influences the value of LANG set by Bacula for its perl scripts remains. I can't find any such thing in the Bacula source code which suggests it is done by one of the libraries installed with the OS -- and that is intriguing.

/etc/profile.d/lang.sh is sourced in /etc/profile, which is read during the initialization of the interactive login shells. That's why the LANG environment variable is not set during the init (and we don't want it, because it would mean localize every command ouputs of the init). On the bacula side, the locale is set by the setlocale() function call you've pointed out. If it finds some aknowledged filled locale-related variables in its environment (LANG + LC_*), it sets them. If LC_ALL is set, it enforces its value to every other variables, if LANG is sets, it sets its value to every unset variables (except LC_ALL, which has the precedence over it), if LANG is not set, it and all the unset variables are set to "C" (or to the "POSIX" synonym) locale every POSIX-compliant system must provide.

catkin 12-19-2012 05:05 AM

Thanks NonNonBa :)

That is a nice summary.

Further investigation suggests it is not Bacula that sets the LANG reported by the perl script and evidentially derived from /etc/profile.d/lang.sh. I am now investigating how/whether perl itself reads /etc/profile.d/lang.sh.

NonNonBa 12-19-2012 08:53 AM

AFAIK, perl doesn't care of your locale unless you use the dedicated POSIX module (see perldoc POSIX for the details). But even like this, the warning is issued only if the wrong locale is already set:

Code:

$ perl -we 'use strict;use POSIX; setlocale(LC_ALL, "xxx");my $l = localeconv();print "$l->{decimal_point}\n"'
.
$ LANG=en_GB-UTF8 perl -we 'use strict;use POSIX;print(setlocale(LC_CTYPE, "")."\n")'
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LC_COLLATE = "C",
        LANG = "en_GB-UTF8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

The first command proves a bad locale setting in a Perl goes silently to "C" (the French decimal separator is a comma).
So, I really don't know where this strange "en_GB-UTF8" bad locale comes from... :scratch:

catkin 12-21-2012 04:56 AM

Thank NonNonBa :) that's helpful extra information.

I've backed out all the debugging modifications and thought the problem symptoms were not reproducible but finally they re-appeared -- and then disappeared again. If I find out how to make the symptoms appear consistently, I will front-end the perl script with something to log the envars, to determine if this is in Bacula or Perl. AFAIK neither of them are likely candidates!

catkin 12-26-2012 02:31 AM

Update and closure
 
It was neither Bacula nor Perl; it was my own script that synchronises Bacula data to USB HDD :doh: :redface:.

The script stops the Bacula daemons during synchronisation. When it starts them again it passed the LANG envar as set in its own configuration file. The problem appeared when Bacula backed up its catalog after a USB HDD had been plugged in.

My script and the three Bacula daemon control scripts have now been modified to source locally created /opt/bacula/etc/lang.conf. lang.conf is functionally identical to /etc/profile.d/lang.sh so the LANG (and LC_COLLATE) envar setting is identical, regardless of whether the daemons are started via Bacula's daemon control scripts, directly or via my USB HDD Bacula synchronisation script.


All times are GMT -5. The time now is 09:20 PM.