[SOLVED] Diverses encoding of text files in mirrors Slackware-current

neymac · 10-30-2013, 12:12 PM

I notice that the files inside Slackware mirrors directories have different encoding for the text files there, is there any reason for this? Or it doesn't matters?

I did this check:

Code:

bash-4.2$ ls *.txt | file --mime-encoding *.txt
ChangeLog.txt: iso-8859-1
bash-4.2$ ls *.TXT | file --mime-encoding *.TXT
BOOTING.TXT:           us-ascii
CHANGES_AND_HINTS.TXT: us-ascii
COPYRIGHT.TXT:         us-ascii
CRYPTO_NOTICE.TXT:     us-ascii
FILELIST.TXT:          us-ascii
PACKAGES.TXT:          unknown-8bit
README_CRYPT.TXT:      us-ascii
README_LVM.TXT:        us-ascii
README_RAID.TXT:       iso-8859-1
README.TXT:            us-ascii
README_UEFI.TXT:       us-ascii
SPEAK_INSTALL.TXT:     us-ascii
SPEAKUP_DOCS.TXT:      us-ascii
UPGRADE.TXT:           us-ascii

Didier Spaier · 10-30-2013, 01:16 PM

Well, I don't know how it's actually possible to determine the encoding of these files as the characters in them are all part of US-ASCII, which is of course 8-bit and a subset of iso-8859-1 (and also a subset of UTF-8, by the way).

Anyhow, that doesn't matter at all, IMO.

neymac · 10-30-2013, 03:21 PM

@Didier Spaier: the command line "file" shows the file type, I did another check using the "file" command at Slackware's mirror folder, filtering text files:

Code:

bash-4.2$ file * | grep text
ANNOUNCE.14_1:          ASCII text
BOOTING.TXT:            ASCII text, with CRLF line terminators
ChangeLog.txt:          ISO-8859 text
CHANGES_AND_HINTS.TXT:  ASCII text, with CRLF line terminators
CHECKSUMS.md5:          ASCII text
COPYING:                Pascal source, ASCII text
COPYING3:               Pascal source, ASCII text
COPYRIGHT.TXT:          ASCII text, with CRLF line terminators
CRYPTO_NOTICE.TXT:      ASCII text, with CRLF line terminators
FILELIST.TXT:           ASCII text
GPG-KEY:                ASCII text
PACKAGES.TXT:           Non-ISO extended-ASCII text
README_CRYPT.TXT:       Pascal source, ASCII text, with CRLF line terminators
README.initrd:          ASCII text
README_LVM.TXT:         ASCII text, with CRLF line terminators
README_RAID.TXT:        ISO-8859 text, with CRLF line terminators
README.TXT:             ASCII text, with CRLF line terminators
README_UEFI.TXT:        ASCII text
RELEASE_NOTES:          ISO-8859 text
Slackware-HOWTO:        Pascal source, ASCII text, with CRLF line terminators
SPEAK_INSTALL.TXT:      assembler source, ASCII text, with CRLF line terminators
SPEAKUP_DOCS.TXT:       C source, ASCII text, with CRLF line terminators
UPGRADE.TXT:            ASCII text, with CRLF line terminators

Didier Spaier · 10-30-2013, 03:48 PM

Quote:

Originally Posted by neymac

@Didier Spaier: the command line "file" shows the file type, I did another check using The "file" command at Slackware's mirror folder, filtering text files:

That simply shows that algorithm used by "file" to determine the file type is not perfect, so you should not trust it blindly.

More specifically there is in general no way to determine a file's encoding with certainty so the algorithm should include a bit of heuristic and assumptions, that can succeed or fail.

For instance, we have man pages with various encodings, and all you can do sometimes is change value of GROFF_ENCODING before running "man" till the page be properly displayed