Add "LC_ALL=C" to pktools

guanx · 04-20-2011, 03:52 AM

Hello Maintainers, Since pkgtools uses dd output, and the output text is not what pkgtools expected in some cases, it would be better to add "LC_ALL=C" to the beginning of the script.

bonixavier · 04-20-2011, 08:32 AM

Isn't C already set as LC_COLLATE by default to all shells?

guanx · 04-20-2011, 10:13 AM

Quote:

Originally Posted by bonixavier

Isn't C already set as LC_COLLATE by default to all shells?

The problem with dd is LC_MESSAGES.

wildwizard · 04-20-2011, 02:58 PM

Have you got the latest pkgtools?

Quote:

pkgtools-13.37-noarch-7
if [ ! "$LC_MESSAGES=C $TAR --version)" = "tar (GNU tar 1.13

guanx · 04-20-2011, 03:09 PM

Quote:

Originally Posted by wildwizard

Have you got the latest pkgtools?

Yes. But the problem is with dd (as mentioned above), not tar.

wildwizard · 04-21-2011, 04:39 AM

Quote:

Originally Posted by guanx

Yes. But the problem is with dd (as mentioned above), not tar.

Argh read the line wrong it tests tar with it set, I though it was testing it as well as tar.

Anyway some checking and I can't see how it would ever be a problem.

installpkg is the one that uses it and it only looks at the first line and only the data up to the + sign everything else is discarded.

eg LC_MESSAGES=ja_JP.UTF-8

Code:

260+0 ??????
260+0 ??????
133120 ??? (133 kB) ????????? 0.00577092 ?? 23.1 MB/?

Normal output

Code:

260+0 records in
260+0 records out
133120 bytes (133 kB) copied, 0.00428188 s, 31.1 MB/s

If you can give a locale where that first part of the first line is different could you actually tell us which one?

hello.freeman · 04-21-2011, 05:53 AM

Here's the shell code from 'installpkg' :

Code:

  cat $package | $packagecompression -dc | dd 2> $TMP/tmpsize$$ | $TAR tf - 1> $TMP/tmplist$$ 2> /dev/null
  TARERROR=$?
  if [ ! "$TARERROR" = "0" ]; then
    EXITSTATUS=1 # tar file corrupt
    if [ "$MODE" = "install" ]; then
      echo "Unable to install $package:  tar archive is corrupt (tar returned error code $TARERROR)"
    fi
    rm -f $TMP/tmplist$$ $TMP/tmpsize$$
    continue
  fi
  UNCOMPRESSED="$(expr $(cat $TMP/tmpsize$$ | head -n 1 | cut -f 1 -d +) / 2)"
  if [ $UNCOMPRESSED -lt 1024 ]; then
    UNCOMPRESSED="${UNCOMPRESSED}K"
  elif [ $UNCOMPRESSED -lt 10239 ]; then
    UNCOMPRESSED="$(expr $UNCOMPRESSED \* 1000 / 1024 | cut -b1).$(expr $UNCOMPRESSED \* 1000 / 1024 | cut -b2)M"
  else
    UNCOMPRESSED="$(expr $UNCOMPRESSED / 1024)M"
  fi
  rm -f $TMP/tmpsize$$

Here's my dd output :

Code:

bash$ LC_MESSAGES=zh_CN.UTF-8 dd if=/dev/zero of=/dev/null bs=1024 count=2
记录了2+0 的读入
记录了2+0 的写出
2048字节(2.0 kB)已复制，1.824e-05 秒，112 MB/秒
bash$ LC_MESSAGES=ja_JP.UTF-8 dd if=/dev/zero of=/dev/null bs=1024 count=2
2+0 records in
2+0 records out
2048 bytes (2.0 kB) copied, 1.78e-05 s, 115 MB/s
bash$ dd if=/dev/zero of=/dev/null bs=1024 count=2
2+0 records in
2+0 records out
2048 bytes (2.0 kB) copied, 1.64e-05 s, 125 MB/s
bash$

There's something wrong in 'installpkg' when LC_MESSAGES become zh_CN.UTF-8 :

Code:

UNCOMPRESSED="$(expr $(cat $TMP/tmpsize$$ | head -n 1 | cut -f 1 -d +) / 2)"

bash$ LC_MESSAGES=zh_CN.UTF-8 dd if=/dev/zero of=/dev/null bs=1024 count=2 2>&1 | head -n 1 | cut -f 1 -d +
记录了2
bash$

hello.freeman · 04-21-2011, 05:57 AM

BTW, the 'pkgtools' version is 13.37-noarch-7

hello.freeman · 04-21-2011, 06:01 AM

I don't think this is a bug.

Isn't C already set as LC_MESSAGES by default to all shells?

guanx · 04-21-2011, 06:17 AM

Quote:

Originally Posted by hello.freeman

I don't think this is a bug.

Isn't C already set as LC_MESSAGES by default to all shells?

Yes. But nearly nobody runs a system with all defaults. It'll be better to explicitly set locale to C, POSIX, or en_US.UTF-8 if UTF is necessary.
BTW, the LANGUAGE environment variable also affects message locale.

guanx · 04-21-2011, 06:18 AM

Quote:

Originally Posted by hello.freeman

Here's the shell code from 'installpkg' :
...
Here's my dd output :

Code:

bash$ LC_MESSAGES=zh_CN.UTF-8 dd if=/dev/zero of=/dev/null bs=1024 count=2
记录了2+0 的读入
记录了2+0 的写出
2048字节(2.0 kB)已复制，1.824e-05 秒，112 MB/秒
bash$ LC_MESSAGES=ja_JP.UTF-8 dd if=/dev/zero of=/dev/null bs=1024 count=2
2+0 records in
2+0 records out
2048 bytes (2.0 kB) copied, 1.78e-05 s, 115 MB/s
bash$ dd if=/dev/zero of=/dev/null bs=1024 count=2
2+0 records in
2+0 records out
2048 bytes (2.0 kB) copied, 1.64e-05 s, 125 MB/s
bash$

...

That's it. -- And similar problem in lg (Luganda).

Skaperen · 04-21-2011, 11:33 AM

All script writers are going to have to do settings like this for just about any program they capture output from. The complication will be the cases where they need to either have some programs just output to the user, or need to echo the captured output to the user.

Capturing output in a script has always been risky because some output formats change for other reasons, too, or portable scripts end up with different outputs on different systems. For decades, I've said we need programs specific for scripts, or at least special options to get script output somewhere. But even that doesn't handle the cases of output to be interpreted and shown to the user. The output probably needs a markup, isolating the informational parts from the display parts, along with the appropriate script tool to isolate them.

That, or switch to another scripting language that has built in methods to access data and perform functions otherwise giving to programs like dd. I'll suggest Python.

rg3 · 04-21-2011, 11:58 AM

I set LANG=C on all my slackbuild scripts for this reason. I've encountered a few cases where the "configure" script expects a given program output in this locale and wouldn't work or properly detect something in that case. I also change the locale in slackroll, which in turn calls installpkg, upgradepkg, etc. As mentioned above, this is a risk you have with every program that needs to parse the output of some other, and the program user may have changed the locale to their needs. The exact locale change I use in slackroll is pretty paranoid and conservative:

* Remove LC_COLLATE, LC_CTYPE, LC_MONETARY, LC_MESSAGES, LC_NUMERIC, LC_TIME, LC_ALL, LANGUAGE, LANG from the environment.
* Set LANG to "C".

Darth Vader · 04-21-2011, 11:59 AM

Quote:

Originally Posted by Skaperen

I'll suggest Python.

Come on!

Nothing is better (and faster) for package management than some programs written in C/C++ and linked full-static. They depend only on the Kernel and maybe Shell.

Slackware specifications for package management are very simple. If Slackware team is interested, I can write an equivalent installpkg / upgradepkg / removekg in a few weeks, using libarchive.

And the changes were minor in Slackware. For a full-static build, the static libraries would be needed since libarchive. So most likely, libarchive, XZ and libxml should be rebuilded with '--enable-static'.

gnashley · 04-21-2011, 12:41 PM

There's already a good C version called spkg (don't confuse it with a shell version of the same name).