And bash history dups still exist... a lot...

dedec0 · 09-04-2016, 08:37 AM

I want that some of you help me making a working solution for the Bash history duplicates that still exist and keep increasing.

There is even a bug report about it, but its solution has been postponed by the Bash team :( See http://savannah.gnu.org/support/?108530 (and make a sound there, if you have a Savannah account)

My history setup from .bashrc is:

Code:

HISTCONTROL=ignoreboth:erasedups
HISTFILESIZE=10000 # so large JUST to handle the big number of dups
HISTSIZE=10000     # so large JUST to handle the big number of dups
# append to the history file, don't overwrite it
shopt -s histappend

I usually open several sessions together, and this leads to many duplicate lines in the history.

I imagine a (not best, ok, does not matter) solution that:

- will clean all history duplicates, keeping the newest duplicate in its relative position

- do not need to affect the other open sessions, but need to guarantee that its work is not undone by these sessions being closed later

- has an easy way to do it for a running session without closing it (like running a command or script)

- anything else that I missed remembering now?

I have seen this problem being made/discussed/"almost solved" somewhere/sometime but could not find it now...

wpeckham · 09-04-2016, 09:11 AM

Just out of curiosity and because it may impact any answers, could you provide some detail about:
What version of BASH are you using?
What version of which distribution are you running?
Have you upgraded to the latest available BASH version, or do you intend to do so soon?

dedec0 · 09-04-2016, 10:45 AM

Quote:

Originally Posted by wpeckham

Just out of curiosity and because it may impact any answers, could you provide some detail about:
What version of BASH are you using?
What version of which distribution are you running?
Have you upgraded to the latest available BASH version, or do you intend to do so soon?

Code:


$ bash --version
GNU bash, versão 4.1.5(1)-release (i486-pc-linux-gnu)

$ bash --version
GNU bash, versão 4.3.0(1)-release (i686-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
Licença GPLv3+: GNU GPL versão 3 ou posterior <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I can update it whenever this problem is fixed, which I found out (from its official page) that it is not - I can even compile it from source, if at all needed. Otherwise I will keep the current version that (mostly) works.

What difference the distribution I use makes? I think it is not important for this thread - or I would have told it.

There are things that do not need and should not be always pushed to be the most of updated versions.

dedec0 · 09-04-2016, 10:47 AM

There are much more important things that I expected answers, in my first post...

(...)

Compiled it from source and installed now:

Code:

$ bash --version
GNU bash, versão 4.3.0(1)-release (i686-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
Licença GPLv3+: GNU GPL versão 3 ou posterior <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

wpeckham · 09-05-2016, 09:52 AM

Thank you, that helps.
Current release is version 4.3.3 available here: http://ftp.gnu.org/gnu/bash/
and the release candidate for version for 4.4 is available on the same page. I was not, at a quick glance, able to determine if the latest versions in fact address this issue.

Interestingly, I was not able to replicate the behavior on one of my CentOS systems. It may be possible that RH has a patch that has not made it into the source tree (unlikely), or simply that the kind of activity that causes your dups is not common on my system.

dedec0 · 09-05-2016, 10:15 AM

I do not consider using the release candidates or the beta versions. I downloaded the source from the same page you pointed. =) Although I doubt that Bash would have some bad things these, but I prefer to wait for a real version. Further, even before I updated Bash, the major version is the same, so no big changes should be missing.

If the configuration I showed was not enough for you to replicate the behaviour I described, I may trace the steps more clearly. Would that be useful? Maybe some requirement for it to happen is that each paralell session uses a history line that existed before each one was started (so their order is changed in both sections to different positions) - and I do not know what should happen when one or both are closed, when history is updated and written (and with the config I use, try to avoid duplicates).

-> The wish for a program to clean the history after duplicates were inserted is still wanted, as a fail proof solution to have no more duplicates, never. I have seen it with Awk (in a place I could not find again), it seems to be easy. But I do not know if it would has the guarantees I pointed (relative order and newest dups kept).

wpeckham · 09-05-2016, 10:51 AM

Well, you are running version 4.3.0?
You might want to examine the release notes for versions 4.3.1, 4.3.2, and 4.3.3 to see if this specific behavior was addressed in any of those releases.
If it has not, you may want to seek the same information for the 4.4 release candidates. That way you will know if 4.4 has your specific fix when it does come out of beta status and you can implement it on your platform.

dedec0 · 09-05-2016, 11:03 AM

This issue has not been addressed in the more current versions of Bash, nor will in the next version. In their bug tracking system (that we access with that Savannah link I gave in the first post), the problem has been postponed. It is confirmed: good. Postponed: who knows for how much time. So, I am trying two things in this thread:

1. To find an alternative solution with the use of some script|configuration|program, I imagine that some people around here are able to do it in Bash/Awk/others-similar.

2. Their bug tracking system considers how many people mention and says that a problem affects them. If some of us mark it there, say something about this I issue there (the one I linked to), the chances of it getting fixed are better.

dedec0 · 09-05-2016, 11:07 AM

Quote:

Originally Posted by wpeckham

Thank you, that helps.
Current release is version 4.3.3 available here: http://ftp.gnu.org/gnu/bash/
and the release candidate for version for 4.4 is available on the same page. I was not, at a quick glance, able to determine if the latest versions in fact address this issue.

You wrote it wrong. It is 4.3.30. Indeed, it is there. I did not choose it because I scrolled down until I did not see a normal version. First beta found, back up a few lines... there it is: 4.3.tar.gz - got that.

Tomorrow I will get 4.3.30 and install. But it does not address the issue!

My previous post is much more important than this one.

norobro · 09-05-2016, 11:22 AM

Quote:

Originally Posted by dedec0

If the configuration I showed was not enough for you to replicate the behaviour I described, I may trace the steps more clearly. Would that be useful?

Please do. I was unable to replicate the behavior either.

I did find this: https://www.gnu.org/software/gawk/ma...y-Sorting.html

You could try setting a trap (man bash) to run that awk script on exit from a shell. However, that would not affect the history of a running session.

dedec0 · 09-05-2016, 11:31 AM

Quote:

Originally Posted by norobro

Please do. I was unable to replicate the behavior either.

I did find this: https://www.gnu.org/software/gawk/ma...y-Sorting.html

You could try setting a trap (man bash) to run that awk script on exit from a shell. However, that would not affect the history of a running session.

I will try to do it in the next few days, I cannot do it now.

History sorting? I did not open the page now, I will read it later, but changing the relative positions of history lines (except removing older duplicates) is not wanted.

Affect history of running session: I do not want this change, so that is not a problem. There is an option in Bash to make the history always reflect the same for all running sections (commands in one are seen in other without ending either).

norobro · 09-05-2016, 11:55 AM

"history sorting" is the title of the page. From paragraph 11.3.6:

Quote:

Suppose, however, you need to remove duplicate lines from a data file but that you want to preserve the order the lines are in.

In your link above Chet Ramey mentioned hist_erasedups. hist_erasedups.c is included with the readline library source code and could be set as a trap, also.

dedec0 · 09-06-2016, 09:20 AM

Great, then. :) As I said, I will read and examine it when I have sometime for that.

Habitual · 09-06-2016, 12:17 PM

I thought "ignoreboth:erasedups" disallowed the recording of successive commands that are in fact, duplicates.
eg:

Code:

history
history

Yes, No, Maybe so?

dedec0 · 09-06-2016, 01:02 PM

The configuration means that, but it does not work in cases with parallel sessions. I will make clear steps to reproduce that.

They work by removing duplicates when we type them in a session, but they are not repeated when the session ends.