🚀 go-pugleaf

On Fri, 26 Sep 2008, Icarus Sparry wrote:
> If you are going to number the lines and then do sorts on them 
> then you don't need to reverse the lines.

The numbering and sorting on it is to keep the original 
ordering. Reverse the lines it to make sure the last occurence 
of each duplication are kept.

> I suggest a
> 	tac | print_line_first_time_seen | head -n | tac 
> pipeline would be better. print_line_first_time_seen is 
> usually done as
>      awk '!seen[$0]++'

Thanks! I choose the sort method before because I believe it is 
more "general" in term of it does not depend on ability to put 
the whole file in memory (works for large files). Obviously it 
will be hardly the issue with history files. I tested your 
method and it seems to be a bit faster. I keep the original line 
commented nearby in-case someone one to use 1 GB history files. 
:-)

> I find the long variables names bad, It took several reads to 
> see that HISTTMPBASE and HISTFILEBASE were different.

Well, it was from mixed influences. I used to long file name in 
Pascal, but off course there it is camel-cased. The all capital 
influenced by common environment variable naming. I principally 
generally favor more explanatory variable name (since data 
structure is more important than algorithm in programming). In 
any case, I am changing the variable name to more camel case, 
hopefully it more understandable.

I don't see any code repetition can be saved by creating 
function, but I also trying comment more also to get better 
separation between code sections.

> if there are race conditions with multiple shells running at 
> the same time, but it certainly looks as if there are issues.

There is definitely one in reading the history file: it is 
possible to missed some files. But information is not lost and 
next session should got them. Also, there can be an issue with 
any process that open a file for extended time, but hard to get 
around it without mandatory locking and shell process should not 
do that, right?

So here is the improved code:


########### History management ###########
# SPAS 2008IO - http://www.arifsaha.com/
# need license? How about LGPL, ok?
#
# to be incorporated into or sourced from bash startup script, such as .bashrc

case $- in
   *i* )   INTERACTIVE=1 ;;
esac
ThisPID="$$"
HISTLogFile="/tmp/histmon.$(whoami).log"
HISTFileBase="$HOME/.bash_history"
HISTTmpBase="$HOME/.bash_histtmp"

if ! which tac ; then
   function tac() {
     tail -r "$@"
   }
fi

# define function to get the last time
# any activity in the tty of a process
if [[ -d /proc ]] ; then
   function timelastactiv() {
     date -r "/proc/$1/fd/255" +"%s"
   }
elif ( stat --help | grep "%y.*modification" ) >/dev/null 2>&1 ; then
   function timelastactiv() {
     stat -f "%y" /dev/$(ps -otty -p "$1" | tail -1)
   }
else
   function timelastactiv() {
     stat -f "%m" /dev/$(ps -otty -p "$1" | tail -1)
   }
fi

function routines() {
   history -a
   #echo -ne \\e[s\\e[1A$(date) history saved!\\e[u
   echo $ThisPID, $(date) history saved! >> $HISTLogFile
}

if [[ ! "$HISTFILESIZE" ]] ; then HISTFILESIZE=4096 ;fi

if [[ "$INTERACTIVE" ]] ; then
   trap routines SIGUSR1
   WaitIni=$((2 * 3600))
   WaitSmall=600
   LastSave=0
   # Routine below run a background loop that
   # will signal the shell to save history after some time.
   # Instead of fix periods, it try to do it in idle time,
   # to reduce duplication caused by vigorous work
   # using the same commands again and again.
   # However, the calculation of idle time required is shorter over time,
   # so it assure that the history saving signal will be sent after some time.
   ( trap "" SIGINT ; while WaitCur=$WaitIni ; while TOUCHED=$(timelastactiv "$ThisPID"); if [[ ! "$TOUCHED" ]] || [[ "$TOUCHED" -le "$LastSave" ]] ; then IDLE=1 ; else IDLE=$(( $(date +"%s") - $TOUCHED )); fi ; echo $ThisPID, $(date): $IDLE .. $WaitCur >> $HISTLogFile ; [[ $IDLE -lt $WaitCur ]] ; do WaitCur=$(( $WaitCur * 2 / 3 )) ; sleep $WaitSmall ; done ; kill -s SIGUSR1 $ThisPID ; do LastSave=$(date +'%s'); sleep $WaitSmall ; done & echo $! )
   echo -n Shell $ThisPID signalled by" "

   HISTTmpCollect=$(mktemp "$HISTTmpBase.XXXXXXXX")
   {
     # Routine below collects files being worked on
     # by other instance of this script:
     ls -rt "$HISTTmpBase".* | while read -r afile ; do echo got work file "$afile" >&2 ; cat "$afile" ; done

     # Routine below collects history files by previous sessions:
     ls -rt "$HISTFileBase" "$HISTFileBase".* | while read -r afile ; do
       # To avoid other session open & append the history file
       # just after being read and before deleted,
       # each file is renamed before read:
       HISTTmp=$(mktemp "$HISTTmpCollect.XXXXXXXX");
       if mv -f "$afile" "$HISTTmp" ; then echo got hist file "$afile" now "$HISTTmp" >&2 ; cat "$HISTTmp"
       else echo cannot mv "$afile" "$HISTTmp" >&2 ; cat "$afile" ; fi
     done
   } | tac | awk '!seen[$0]++' | head -n $HISTFILESIZE | tac > "$HISTTmpCollect"
   #  nl -ba -s: -nrz | tac | sort -t: -u -k2.1 | sort -t: -n -k1,1 | cut -d: -f2- | tail -n $HISTFILESIZE > "$HISTTmpCollect"
   # The phrase above will filter out duplicate,
   # keeping the order, keeping last occurrence of any duplicate.
   # The result saved to a new file, not original history file
   # to avoid other session open & append the original history file
   # just before it being overwritten.
   # Note that race condition can still occur in reading the history file:
   # it is possible a session started at the same time grab
   # a history file and rename it before it can be read
   # but just after collecting work file is done.
   # So some history may not get loaded.
   # But this is less big deal since the information is not
   # lost and next session will be able to read it.

   history -r "$HISTTmpCollect"
   HISTCollect=$(mktemp "$HISTFileBase.XXXXXXXX")
   if mv "$HISTTmpCollect" "$HISTCollect" ; then rm "$HISTTmpCollect".* ; fi

   # Actual history saving file is another new history file
   # instead of the new combined old history
   # to keep better ordering of history (i.e. newer history file
   # not to be put behind the old one):
   HISTNew=$(mktemp "$HISTFileBase.XXXXXXXX")
   if [[ -w "$HISTNew" ]] ; then HISTFILE="$HISTNew" ; fi
   echo load history from $(wc "$HISTCollect") resulting $(history | wc -l) lines, save to "$HISTFILE"
fi


-- 
                              (stephan paul) Arif Sahari Wibowo
    _____  _____  _____  _____
   /____  /____/ /____/ /____
  _____/ /      /    / _____/      http://www.arifsaha.com/

Disclaimer: IANAL, IANALP, IANAMD, IANAMP, IANAAP
my statements - if any - should be treated as such.

Updated version, now include sanity check: in case processing 
fails, not to remove older history files.


########### History management ###########
# SPAS 2008IO, 2008IR - http://www.arifsaha.com/
# need license? How about LGPL, ok?
#
# to be incorporated into or sourced from bash startup script, such as .bashrc

case $- in
   *i* )   INTERACTIVE=1 ;;
esac
ThisPID="$$"
HISTLogFile="/tmp/histmon.$(whoami).log"
HISTFileBase="$HOME/.bash_history"
HISTTmpBase="$HOME/.bash_histtmp"

# In some system 'which' never fails,
# requiring secondary check as below, duh!
if [[ ! -x $(which tac) ]] ; then
   echo define tag function >&2
   function tac() {
     tail -r "$@"
   }
fi

# define function to get the last time
# any activity in the tty of a process
if [[ -d /proc ]] ; then
   function timelastactiv() {
     date -r "/proc/$1/fd/255" +"%s"
   }
elif ( stat --help | grep "%y.*modification" ) >/dev/null 2>&1 ; then
   function timelastactiv() {
     stat -f "%y" /dev/$(ps -otty -p "$1" | tail -1)
   }
else
   function timelastactiv() {
     stat -f "%m" /dev/$(ps -otty -p "$1" | tail -1)
   }
fi

function routines() {
   history -a
   #echo -ne \\e[s\\e[1A$(date) history saved!\\e[u >&2
   echo $ThisPID, $(date) history saved! >> $HISTLogFile
}

if [[ ! "$HISTFILESIZE" ]] ; then HISTFILESIZE=4096 ;fi

if [[ "$INTERACTIVE" ]] ; then
   trap routines SIGUSR1
   WaitIni=$((2 * 3600))
   WaitSmall=600
   LastSave=0
   # Routine below run a background loop that
   # will signal the shell to save history after some time.
   # Instead of fix periods, it try to do it in idle time,
   # to reduce duplication caused by vigorous work
   # using the same commands again and again.
   # However, the calculation of idle time required is shorter over time,
   # so it assure that the history saving signal will be sent after some time.
   ( trap "" SIGINT ; while WaitCur=$WaitIni ; while TOUCHED=$(timelastactiv "$ThisPID"); if [[ ! "$TOUCHED" ]] || [[ "$TOUCHED" -le "$LastSave" ]] ; then IDLE=1 ; else IDLE=$(( $(date +"%s") - $TOUCHED )); fi ; echo $ThisPID, $(date): $IDLE .. $WaitCur >> $HISTLogFile ; [[ $IDLE -lt $WaitCur ]] ; do WaitCur=$(( $WaitCur * 2 / 3 )) ; sleep $WaitSmall ; done ; kill -s SIGUSR1 $ThisPID ; do LastSave=$(date +'%s'); sleep $WaitSmall ; done & echo -n $! >&2 )
   echo " "signalling shell $ThisPID >&2

   HISTTmpCollect=$(mktemp "$HISTTmpBase.XXXXXXXX")
   {
     # Routine below collects files being worked on
     # by other instance of this script:
     ls -rt "$HISTTmpBase".* 2>/dev/null | while read -r afile ; do echo got work file "$afile" >&2 ; cat "$afile" ; done

     # Routine below collects history files by previous sessions:
     ls -rt "$HISTFileBase" "$HISTFileBase".* 2>/dev/null | while read -r afile ; do
       # To avoid other session open & append the history file
       # just after being read and before deleted,
       # each file is renamed before read:
       HISTTmp=$(mktemp "$HISTTmpCollect.XXXXXXXX");
       if mv -f "$afile" "$HISTTmp" ; then echo got hist file "$afile" now "$HISTTmp" >&2 ; cat "$HISTTmp"
       else echo cannot mv "$afile" "$HISTTmp" >&2 ; cat "$afile" ; fi
     done
   } | tac | awk '!seen[$0]++' | head -n $HISTFILESIZE | tac > "$HISTTmpCollect"
   #  nl -ba -s: -nrz | tac | sort -t: -u -k2.1 | sort -t: -n -k1,1 | cut -d: -f2- | tail -n $HISTFILESIZE > "$HISTTmpCollect"
   # The phrase above will filter out duplicate,
   # keeping the order, keeping last occurrence of any duplicate.
   # The result saved to a new file, not original history file
   # to avoid other session open & append the original history file
   # just before it being overwritten.
   # Note that race condition can still occur in reading the history file:
   # it is possible a session started at the same time grab
   # a history file and rename it before it can be read
   # but just after collecting work file is done.
   # So some history may not get loaded.
   # But this is less big deal since the information is not
   # lost and next session will be able to read it.

   # Saity check whether the pipe was actually successful:
   # (alternatively bash' PIPESTATUS can be used,
   # just it will make it harder to port to other shell)
   if [[ -s "$HISTTmpCollect" ]] ; then
     history -r "$HISTTmpCollect"
     HISTCollect=$(mktemp "$HISTFileBase.XXXXXXXX")
     if mv "$HISTTmpCollect" "$HISTCollect" ; then rm "$HISTTmpCollect".* ; fi

     # Actual history saving file is another new history file
     # instead of the new combined old history
     # to keep better ordering of history (i.e. newer history file
     # not to be put behind the old one):
     HISTNew=$(mktemp "$HISTFileBase.XXXXXXXX")
     if [[ -w "$HISTNew" ]] ; then HISTFILE="$HISTNew" ; fi
   else
     rm "$HISTTmpCollect"
   fi

   # Move back any work file left over because of some failures:
   ls -rt "$HISTTmpCollect".* 2>/dev/null | while read -r afile ; do
     HISTTmp=$(mktemp "$HISTFileBase.XXXXXXXX");
     mv "$afile" "$HISTTmp"
     echo mv back "$afile" "$HISTTmp" >&2
   done
   echo load history from $(wc "$HISTCollect") resulting $(history | wc -l) lines, save to "$HISTFILE" >&2
fi


-- 
                              (stephan paul) Arif Sahari Wibowo
    _____  _____  _____  _____
   /____  /____/ /____/ /____
  _____/ /      /    / _____/      http://www.arifsaha.com/