LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-04-2016, 08:37 AM   #1
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Rep: Reputation: 44
Post And bash history dups still exist... a lot...


I want that some of you help me making a working solution for the Bash history duplicates that still exist and keep increasing.

There is even a bug report about it, but its solution has been postponed by the Bash team :( See http://savannah.gnu.org/support/?108530 (and make a sound there, if you have a Savannah account)

My history setup from .bashrc is:

Code:
HISTCONTROL=ignoreboth:erasedups
HISTFILESIZE=10000 # so large JUST to handle the big number of dups
HISTSIZE=10000     # so large JUST to handle the big number of dups
# append to the history file, don't overwrite it
shopt -s histappend
I usually open several sessions together, and this leads to many duplicate lines in the history.

I imagine a (not best, ok, does not matter) solution that:

- will clean all history duplicates, keeping the newest duplicate in its relative position

- do not need to affect the other open sessions, but need to guarantee that its work is not undone by these sessions being closed later

- has an easy way to do it for a running session without closing it (like running a command or script)

- anything else that I missed remembering now?

I have seen this problem being made/discussed/"almost solved" somewhere/sometime but could not find it now...

Last edited by dedec0; 09-04-2016 at 08:38 AM.
 
Old 09-04-2016, 09:11 AM   #2
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDo, tinycore, Q4OS,Manjaro
Posts: 4,003

Rep: Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853
Just out of curiosity and because it may impact any answers, could you provide some detail about:
What version of BASH are you using?
What version of which distribution are you running?
Have you upgraded to the latest available BASH version, or do you intend to do so soon?
 
Old 09-04-2016, 10:45 AM   #3
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
Quote:
Originally Posted by wpeckham View Post
Just out of curiosity and because it may impact any answers, could you provide some detail about:
What version of BASH are you using?
What version of which distribution are you running?
Have you upgraded to the latest available BASH version, or do you intend to do so soon?
Code:

$ bash --version
GNU bash, versão 4.1.5(1)-release (i486-pc-linux-gnu)

$ bash --version
GNU bash, versão 4.3.0(1)-release (i686-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
Licença GPLv3+: GNU GPL versão 3 ou posterior <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
I can update it whenever this problem is fixed, which I found out (from its official page) that it is not - I can even compile it from source, if at all needed. Otherwise I will keep the current version that (mostly) works.

What difference the distribution I use makes? I think it is not important for this thread - or I would have told it.

There are things that do not need and should not be always pushed to be the most of updated versions.

Last edited by dedec0; 09-04-2016 at 11:15 AM.
 
Old 09-04-2016, 10:47 AM   #4
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
Thumbs down

There are much more important things that I expected answers, in my first post...

(...)

Compiled it from source and installed now:

Code:
$ bash --version
GNU bash, versão 4.3.0(1)-release (i686-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
Licença GPLv3+: GNU GPL versão 3 ou posterior <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Last edited by dedec0; 09-04-2016 at 11:16 AM.
 
Old 09-05-2016, 09:52 AM   #5
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDo, tinycore, Q4OS,Manjaro
Posts: 4,003

Rep: Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853
Thank you, that helps.
Current release is version 4.3.3 available here: http://ftp.gnu.org/gnu/bash/
and the release candidate for version for 4.4 is available on the same page. I was not, at a quick glance, able to determine if the latest versions in fact address this issue.

Interestingly, I was not able to replicate the behavior on one of my CentOS systems. It may be possible that RH has a patch that has not made it into the source tree (unlikely), or simply that the kind of activity that causes your dups is not common on my system.
 
Old 09-05-2016, 10:15 AM   #6
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
I do not consider using the release candidates or the beta versions. I downloaded the source from the same page you pointed. =) Although I doubt that Bash would have some bad things these, but I prefer to wait for a real version. Further, even before I updated Bash, the major version is the same, so no big changes should be missing.

If the configuration I showed was not enough for you to replicate the behaviour I described, I may trace the steps more clearly. Would that be useful? Maybe some requirement for it to happen is that each paralell session uses a history line that existed before each one was started (so their order is changed in both sections to different positions) - and I do not know what should happen when one or both are closed, when history is updated and written (and with the config I use, try to avoid duplicates).

-> The wish for a program to clean the history after duplicates were inserted is still wanted, as a fail proof solution to have no more duplicates, never. I have seen it with Awk (in a place I could not find again), it seems to be easy. But I do not know if it would has the guarantees I pointed (relative order and newest dups kept).
 
Old 09-05-2016, 10:51 AM   #7
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDo, tinycore, Q4OS,Manjaro
Posts: 4,003

Rep: Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853
Well, you are running version 4.3.0?
You might want to examine the release notes for versions 4.3.1, 4.3.2, and 4.3.3 to see if this specific behavior was addressed in any of those releases.
If it has not, you may want to seek the same information for the 4.4 release candidates. That way you will know if 4.4 has your specific fix when it does come out of beta status and you can implement it on your platform.
 
Old 09-05-2016, 11:03 AM   #8
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
Arrow

This issue has not been addressed in the more current versions of Bash, nor will in the next version. In their bug tracking system (that we access with that Savannah link I gave in the first post), the problem has been postponed. It is confirmed: good. Postponed: who knows for how much time. So, I am trying two things in this thread:

1. To find an alternative solution with the use of some script|configuration|program, I imagine that some people around here are able to do it in Bash/Awk/others-similar.

2. Their bug tracking system considers how many people mention and says that a problem affects them. If some of us mark it there, say something about this I issue there (the one I linked to), the chances of it getting fixed are better.
 
Old 09-05-2016, 11:07 AM   #9
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
Red face

Quote:
Originally Posted by wpeckham View Post
Thank you, that helps.
Current release is version 4.3.3 available here: http://ftp.gnu.org/gnu/bash/
and the release candidate for version for 4.4 is available on the same page. I was not, at a quick glance, able to determine if the latest versions in fact address this issue.
You wrote it wrong. It is 4.3.30. Indeed, it is there. I did not choose it because I scrolled down until I did not see a normal version. First beta found, back up a few lines... there it is: 4.3.tar.gz - got that.

Tomorrow I will get 4.3.30 and install. But it does not address the issue!

My previous post is much more important than this one.

Last edited by dedec0; 09-05-2016 at 11:08 AM.
 
Old 09-05-2016, 11:22 AM   #10
norobro
Member
 
Registered: Feb 2006
Distribution: Debian Sid
Posts: 792

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Quote:
Originally Posted by dedec0 View Post
If the configuration I showed was not enough for you to replicate the behaviour I described, I may trace the steps more clearly. Would that be useful?
Please do. I was unable to replicate the behavior either.

I did find this: https://www.gnu.org/software/gawk/ma...y-Sorting.html

You could try setting a trap (man bash) to run that awk script on exit from a shell. However, that would not affect the history of a running session.
 
Old 09-05-2016, 11:31 AM   #11
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
Quote:
Originally Posted by norobro View Post
Please do. I was unable to replicate the behavior either.

I did find this: https://www.gnu.org/software/gawk/ma...y-Sorting.html

You could try setting a trap (man bash) to run that awk script on exit from a shell. However, that would not affect the history of a running session.
I will try to do it in the next few days, I cannot do it now.

History sorting? I did not open the page now, I will read it later, but changing the relative positions of history lines (except removing older duplicates) is not wanted.

Affect history of running session: I do not want this change, so that is not a problem. There is an option in Bash to make the history always reflect the same for all running sections (commands in one are seen in other without ending either).

Last edited by dedec0; 09-06-2016 at 09:20 AM.
 
Old 09-05-2016, 11:55 AM   #12
norobro
Member
 
Registered: Feb 2006
Distribution: Debian Sid
Posts: 792

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
"history sorting" is the title of the page. From paragraph 11.3.6:
Quote:
Suppose, however, you need to remove duplicate lines from a data file but that you want to preserve the order the lines are in.
In your link above Chet Ramey mentioned hist_erasedups. hist_erasedups.c is included with the readline library source code and could be set as a trap, also.
 
Old 09-06-2016, 09:20 AM   #13
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
Great, then. :) As I said, I will read and examine it when I have sometime for that.
 
Old 09-06-2016, 12:17 PM   #14
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Abingdon, VA
Distribution: Catalina
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
I thought "ignoreboth:erasedups" disallowed the recording of successive commands that are in fact, duplicates.
eg:
Code:
history
history
Yes, No, Maybe so?
 
Old 09-06-2016, 01:02 PM   #15
dedec0
Senior Member
 
Registered: May 2007
Posts: 1,254

Original Poster
Rep: Reputation: 44
The configuration means that, but it does not work in cases with parallel sessions. I will make clear steps to reproduce that.

They work by removing duplicates when we type them in a session, but they are not repeated when the session ends.

Last edited by dedec0; 09-06-2016 at 01:03 PM.
 
  


Reply

Tags
bash history


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Why does BASH history command have no history when run in script ? lump_of_lard Programming 4 01-30-2015 09:45 AM
Bash history samineru Linux - Software 1 07-14-2011 06:57 AM
Bash history delete command from bash itself ashishag Linux - Software 6 05-02-2010 03:39 AM
copying a lot of files with bash or terminal. rob-n Linux - Newbie 3 04-13-2004 06:01 PM
Bash History ukndoit Linux - Security 2 10-16-2003 09:02 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:13 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration