LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 01-05-2018, 02:45 PM   #1
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,453
Blog Entries: 15

Rep: Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449
gpg: do_plaintext(): wrote 1210414045 bytes but expected 822504068 bytes


Basic questions:
1) Is the above message in fact an "error"?
2) Why does it appear to be backwards? (i.e. Why is the first number it says it "wrote" larger than what it says it "expected"?
3) Has anyone seen this message before?

DETAILS:
We create an ascii armored (encrypted) file using GPG via a script we run via cron once a week (which has been running for years). The file is then uploaded to a bank partner.

It was reported by our cash team that the file uploaded this past weekend has a variance of some 1.3 Million records when they tried to reconcile it. It is normal to have some variance but this was extreme.

On reviewing the log I saw the following message which appears to have come from the actual encryption of the original file:

gpg: do_plaintext(): wrote 1210414045 bytes but expected
822504068 bytes

That appears to me to be an error but it didn't prevent the ascii armored (encrypted) file from being created and subsequently uploaded.

The original unencrypted file has nearly the same byte count (1210414056) as the first number (1210414045) in above message. A difference of 11 bytes.

The encrypted file it created was 331941767 bytes.

When I re-encrypt the original file the new encrypted file is:
331941796 bytes. (I did that a couple of times and got the same size.) I did NOT see the above message on my re-encryption runs.

There seems to be almost no information about this do_plaintext message. Most of what I found online is a repeat of the same question copied to various other sites. The discussion on that question seems to blithely ignore the actual error but instead goes into other commands (tar, time, etc...) in that user's command line (pipeline).

I reviewed the man page for gpg and found this tantalizing information:
Quote:
--exit-on-status-write-error
This option will cause write errors on the status FD to immedi-
ately terminate the process. That should in fact be the default
but it never worked this way and thus we need an option to
enable this, so that the change won’t break applications which
close their end of a status fd connected pipe too early. Using
this option along with --enable-progress-filter may be used to
cleanly cancel long running gpg operations.
That seems to explain why it didn't actually cause our script to error out. On adding the above flag to my encryption it created a slightly different size file of 331941792 bytes.

Looking back in our log I see we've occasionally had similar message in the past (most recently before this weekend back in October) but in those earlier runs the difference between first and second number was significantly smaller. This may have led to it appearing to be normal variance in records during reconciliation so no one questioned it.


Has adding the above flag caused the encryption to error out as desired and NOT created a file?

Answers to questions some will ask:
1) The original file size we encrypt is always extremely large so it is not the size that caused this issue. Also as noted I have run the encryption manually several times since yesterday so if size were an issue I'd expect one of those runs to have failed similarly yet none did.

2) We did not run out of space on the filesystem during the encryption. There is plenty of space.

3) The command line we used to do the encryption without the new flag was:
/usr/bin/gpg --always-trust --armor --recipient <RECIPIENT> -o <BASEFILENAME.dat.asc> --encrypt <BASEFILENAME.dat.txfr>
Where <BASEFILENAME.dat.asc> is the encrypted file and <BASEFILENAME.dat.txfr> is the original unencrypted file.

4) The command line with the new flag:
/usr/bin/gpg --exit-on-status-write-error --always-trust --armor --recipient <RECIPIENT BANK ID> -o <BASEFILENAME.dat.asc> --encrypt <BASEFILENAME.dat.txfr>

5) I can't decrypt the file created this past weekend because the recipient used the bank's publc key so only the bank can decrypt it with their private key.

6) Distro/version is RHEL6.9

7) gpg --version outputs:
gpg (GnuPG) 2.0.14
libgcrypt 1.4.5
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA
Cipher: 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH, CAMELLIA128,
CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Last edited by MensaWater; 01-05-2018 at 02:48 PM.
 
Old 01-06-2018, 05:48 PM   #2
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 19 MATE
Posts: 5,371
Blog Entries: 2

Rep: Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713Reputation: 1713
Have you tried posting on the gnupg-users mailing list?

https://lists.gnupg.org/mailman/listinfo/gnupg-users
 
Old 01-08-2018, 10:16 AM   #3
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,453
Blog Entries: 15

Original Poster
Rep: Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449
Quote:
Originally Posted by hydrurga View Post
Have you tried posting on the gnupg-users mailing list?

https://lists.gnupg.org/mailman/listinfo/gnupg-users
No. Mainly because all the links I found pointed there and as noted in my original post they all seem to ignore the error message itself in favor of guessing why it occurred rather than telling one what it actually means. I suspect were I to post there they'd simply refer me to the earlier posts as I see they did others that have posted there later.
 
Old 01-22-2018, 02:42 PM   #4
norobro
Member
 
Registered: Feb 2006
Distribution: Debian Sid
Posts: 791

Rep: Reputation: 329Reputation: 329Reputation: 329Reputation: 329
@MensaWater - I have never experienced this problem but I was wondering if you had resolved it.

Browsing the source code I noticed that, apparently (and curiously), on an VMS system any difference in those byte counts is ignored. https://git.gnupg.org/cgi-bin/gitweb...RANCH-1-4#l504
 
Old 01-22-2018, 03:29 PM   #5
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,453
Blog Entries: 15

Original Poster
Rep: Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449
That source code finding jives with what I posted from the man page. The issue is that the message, although ignored normally, appears to indicate a real issue with the encryption. i.e. The full original file does NOT get encrypted so the resulting file although encrypted doesn't contain everything intended from the original file. In our case this meant a loss of over one million records. Those records would then have to be manually reconciled with what should have been there.

What I did here was modify our script to add the "--exit-on-status-write-error" flag mentioned in the man page (see earlier post). This particular job only runs once every Sunday. In the two Sundays since I've added that flag I've not seen the message (or any other error indications). I don't know if this means we simply haven't encountered the byte mismatch issue (it was not happening every time even before I added the flag) or if it means that when gpg encountered the byte mismatch it simply retried behind the scenes - the flag name suggest it would exit rather than retry. Either way we've seen successful encryption of the entire file each of the past 2 Sundays.

To me (and apparently the man page author) it seems counter-intuitive for gpg NOT to error by default if there is an issue.
 
Old 02-05-2018, 08:19 AM   #6
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,453
Blog Entries: 15

Original Poster
Rep: Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449
Just as a follow up. The past 2 weekends when the job has run I've seen do-plaintext message again. In both cases the number of bytes shown as a difference was not very high (i.e. less than 50 bytes) unlike the one that caused me to start this thread. It appears adding the flag did not cause it to treat this as a write error and error out as I had hoped it would.

I'll need to put logic in my script to check for this specific message and re-encrypt if it appears.
 
Old 02-27-2018, 08:48 AM   #7
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,453
Blog Entries: 15

Original Poster
Rep: Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449Reputation: 1449
Another follow up. I did subscribe to and post to the GnuPG list.

I got a detailed suggestion of multiple things to try that mentioned using "--batch" option since it is in a script. Another less detailed suggestion also mentioned that. Most of my newer scripts I do use "--batch" but this script was one I inherited long ago

I added the "--batch" option and so far in the 3 weeks since doing that I've not seen the plaintext message recur.

As noted previously this seemed to occur in the past then quit without us noticing it had occurred so I can't swear adding "--batch" fixed it. However, it is a good idea to use "--batch" any time it is a script.
 
  


Reply

Tags
gpg


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Is it okay if my sector size is "512 bytes / 4096 bytes" Altiris Slackware 5 07-31-2015 03:19 AM
error "tar (child): /dev/st0: Wrote only 0 of 10240 bytes? hocheetiong Linux - Newbie 3 02-03-2008 05:25 PM
bytes in/out ilnli Programming 4 07-26-2007 02:44 AM
TX bytes vs. httpd bytes ovrload Linux - Networking 3 10-12-2005 04:19 PM
1 block = ? bytes captainstorm Linux - Newbie 2 07-08-2003 10:30 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration