Just annotations of little "how to's", so I know I can find how to do something I've already done when I need to do it again, in case I don't remember anymore, which is not unlikely. Hopefully they can be useful to others, but I can't guarantee that it will work, or that it won't even make things worse.
Keeping .bash_history without duplicate entries
I had the HISTCONTROL options set correctly for that for a while ("ignoreboth:erasedups" -- the first ignores duplicates and command lines starting with a space, and erasedups would "move" a command repeated exactly to the last entry when repeated), and yet it wasn't working, I had duplicates all over.
I had just devised a script-daemon that would do the same thing in a less-than-native manner, when I decided to search a little bit more about it before just "saving" it here. Then I found that the problem was that I had been using only "history -a" and no "history -w" on my PROMPT_COMMAND.
The first only appends the last entry to the history file, thus it preserves any repeated entries. While "-w" writes the entire file again.
Apparently it's still recommended to have "history -a" preceding or alongside "-w" in .bash_logout, according to this comment:
https://askubuntu.com/posts/1003030/revisions
Regardless of that, besides editing .bash_history, my script had it actually being stored on /dev/shm for the "live" session and a gzipped copy that's stored on disk. I think that's still interesting to have, even with the native configuration being correct, in order to reduce the wear on an SSD from writing and rewriting the file every time, for every command. Instead of only hourly or something. Ideally only if changed, just in case one isn't that constantly on terminal, so to avoid needless "fake" updates to disk/SSD.
The script with the commented/deactivated editing "sub-daemon":
To my surprise, the sed things were working as I hoped, without the string expansions replacing string names and so forth. Maybe it's not sufficiently commenting everything though, perhaps even asterisk is ideally/necessary to be escaped, if one were to do this kind of thing with sed, for some reason, which is not necessary, fortunately.
Maybe there's some problem waiting to happen with this daemon storing the history in shm and gzipping it every hour, I'm posting this kind of thing here mostly as a reference to myself, more than as recommendations people should trust and follow, as I'm often not that sure of what I'm doing.
Apparently history -w respects it being a symbolic link, instead of literally overwriting the file in a more complete manner, such as sed without "follow-symlinks" would do. I just hope they don't change that from the default behavior to one that needs to be made explicit, which I don't know if it actually can be done right now, preventively.
To clean the older history file I used:
"Sort -u" is faster but loses the original order, although as such it will also privilege the first/oldest entry, rather than the last. I guess using "tac" twice would privilege the last entry of a duplicate, but now it's too late for me.
Currently this combination of settingns and scripts seems to be somewhat "randomly" (likely not really randomly) erasing much of older history, and leaving me with coincidentally only about 500 entries, both times I checked for it, although at times it was normal, with about 4 times that number. I guess -w and -a may sort of "conflict" in a way, with -w overwriting the previous history based on the current session or something. Maybe it has to do with a command to also pause history temporarily that I have, but I don't know. The gz archive of history fortunately wasn't overwritten so far with a much smaller one, I guess it's worth adding some check in this regard before overwriting, until I find out what's going on exactly. (Added such check)
I had just devised a script-daemon that would do the same thing in a less-than-native manner, when I decided to search a little bit more about it before just "saving" it here. Then I found that the problem was that I had been using only "history -a" and no "history -w" on my PROMPT_COMMAND.
The first only appends the last entry to the history file, thus it preserves any repeated entries. While "-w" writes the entire file again.
Apparently it's still recommended to have "history -a" preceding or alongside "-w" in .bash_logout, according to this comment:
https://askubuntu.com/posts/1003030/revisions
Regardless of that, besides editing .bash_history, my script had it actually being stored on /dev/shm for the "live" session and a gzipped copy that's stored on disk. I think that's still interesting to have, even with the native configuration being correct, in order to reduce the wear on an SSD from writing and rewriting the file every time, for every command. Instead of only hourly or something. Ideally only if changed, just in case one isn't that constantly on terminal, so to avoid needless "fake" updates to disk/SSD.
The script with the commented/deactivated editing "sub-daemon":
Code:
#!/bin/bash
# requires $PROMPT_COMMAND to include "history -a"
# ACTUALLY NOT EVEN QUITE TOTALLY NEEDED WITH HISTORY -W
[[ -d /dev/shm/.history-clean ]] && echo locked && exit 0
mkdir /dev/shm/.history-clean
if [[ "$(realpath $HOME/.bash_history)" == "$HOME/.bash_history" ]] ; then
mv -t /dev/shm/ $HOME/.bash_history
ln -s /dev/shm/.bash_history $HOME/.bash_history
else
zcat $HOME/.clean-history.gz > /dev/shm/.bash_history
fi
#while true ; do
#
#inotifywait -e modify /dev/shm/.bash_history
#
#lastentry=$(tail -n 1 /dev/shm/.bash_history)
#
#lastentry=${lastentry//\?/\\\?}
#lastentry=${lastentry//\//\\\/}
#
#sed "/${lastentry}/d; $ a ${lastentry}" -i /dev/shm/.bash_history
#
#done &
while true ; do
sleep 1h
# NEED TO IMPLEMENT SOME CHECK FOR FILE CHANGES HERE
if [[ $(find "/dev/shm/.bash_history" -mmin -60) ]] && (( $(wc -l < /dev/shm/.bash_history) > $( zcat $HOME/.clean-history.gz | wc -l) )) ; then
cat /dev/shm/.bash_history | gzip -9 > $HOME/.clean-history.gz
fi
# "find" is checking whether it was modified, which is almost certainly the case to have happened
# only with changes, but maybe not, but that's perhaps all it can be done without an additional
# backup for comparison, or without zcatting the hard-disk one to compare, which may not be all that bad
done
To my surprise, the sed things were working as I hoped, without the string expansions replacing string names and so forth. Maybe it's not sufficiently commenting everything though, perhaps even asterisk is ideally/necessary to be escaped, if one were to do this kind of thing with sed, for some reason, which is not necessary, fortunately.
Maybe there's some problem waiting to happen with this daemon storing the history in shm and gzipping it every hour, I'm posting this kind of thing here mostly as a reference to myself, more than as recommendations people should trust and follow, as I'm often not that sure of what I'm doing.
Apparently history -w respects it being a symbolic link, instead of literally overwriting the file in a more complete manner, such as sed without "follow-symlinks" would do. I just hope they don't change that from the default behavior to one that needs to be made explicit, which I don't know if it actually can be done right now, preventively.
To clean the older history file I used:
Code:
while read his ; do grep -x "${his}" > /dev/null || echo "${his}" >> /dev/shm/history-clean ; done < .bash_history
Currently this combination of settingns and scripts seems to be somewhat "randomly" (likely not really randomly) erasing much of older history, and leaving me with coincidentally only about 500 entries, both times I checked for it, although at times it was normal, with about 4 times that number. I guess -w and -a may sort of "conflict" in a way, with -w overwriting the previous history based on the current session or something. Maybe it has to do with a command to also pause history temporarily that I have, but I don't know. The gz archive of history fortunately wasn't overwritten so far with a much smaller one, I guess it's worth adding some check in this regard before overwriting, until I find out what's going on exactly. (Added such check)
Total Comments 0
