Extract history from Korn shell

lucmove · 06-24-2009, 08:10 AM

I am not happy about the history file in binary format of the Korn shell.

I like to "collect" some of my command lines, many of them actually, and for a long time. I'm talking about years. That doesn't seem easy in Korn because the history file is not plain text so I can't edit it, and a lot of junk is piling up in it. By "junk" I mean lines that I don'twant to keep, like 'cat' or 'man'.

So I added these lines to my .profile:

fc -ln 1 9999 >> ~/khistory.txt

source ~/loghistory.sh > ~/khistory.txt

loghistory.sh contains a handful of sed and sort commands that gets rid of a lot of the junk. But apparently it is forbidden to run fc in the .profile file. I can't login whenever I do, the shell exits right away with signal 11. So I removed that 'fc -l' line from my .profile file and added it to the loghistory.sh script, but the shell still crashes.

I also tried this line in my .profile:

strings ~/.sh_history >> ~/khistory.txt

source ~/loghistory.sh

That doesn't crash, but the output is printed with an additional, random character in the beginning of many lines.

I can run 'fc -l' on the command line, but that's no good. I need to automate that. But how? How can I extract my ksh history as plain text?

TIA

tronayne · 06-24-2009, 10:56 AM

Quote:

But how? How can I extract my ksh history as plain text?

Uh, how about just using the history command (and maybe pipe it into cut or awk to strip off the numerics)? As you know, .sh_history requires some fooling around (say, with a simple C program -- you can read it with, oh, getc() or fread() or something that will read character data). The individual history lines are terminate with a line feed...

Set your HISTSIZE environment variable to something reasonable like, oh, I dunno, 1000 so you'll have enough to deal with and use history -500 or something to see the last 500 commands.

Here's histfile.c to get you started

Code:

#ident  "$Id$"

/*
 *      Copyright (C) 2000-2009 Thomas Ronayne
 *
 *      This program is free software; you can redistribute it and/or
 *      modify it under the terms of version 2 of the GNU General
 *      Public License as published by the Free Software Foundation.
 *
 *      This program is distributed in the hope that it will be useful,
 *      but WITHOUT ANY WARRANTY; without even the implied warranty of
 *      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 *      General Public License for more details.
 *
 *      You should have received a copy of the GNU General Public
 *      License along with this program; if not, write to the Free
 *      Software Foundation, Inc., 59 Temple Place - Suite 330, Boston,
 *      MA 02111-1307, USA.
 *
 *      Name:           $Source$
 *      Purpose:        display content of a user's .sh_history file
 *      Version:        $Revision$
 *      Modified:       $Date$
 *      Author:         T. N. Ronayne
 *      Date:           24 Jun 2009
 *      $Log$
*/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>

#ifndef TRUE
#       define  TRUE    1
#endif
#ifndef FALSE
#       define  FALSE   0
#endif

int     main    (int argc, char *argv [])
{
        int     c;                      /* general-purpose              */
        int     error = FALSE;          /* error flag                   */
        int     vopt = FALSE;           /* verbose option               */
        time_t  t0 = (time_t) 0;        /* start time                   */
        time_t  t1 = (time_t) 0;        /* finish time                  */
        FILE    *in;

        /*      process the command line arguments                      */
        while ((c = getopt (argc, argv, "?v")) != EOF) {
                switch (c) {
                case '?':
                        error = TRUE;
                        break;
                case 'v':
                        vopt = TRUE;
                        break;
                default:
                        (void) fprintf (stderr, "getopt() bug\n");
                        exit (EXIT_FAILURE);
                }
        }
        /*      any errors in the arguments, or a '?' entered...*/
        if (error) {
                (void) fprintf (stderr, "usage: %s [-v] argument...\n",
                    argv [0]);
                exit (EXIT_FAILURE);
        }
        /*      get a start time                                */
        if (time (&t0) < (time_t) 0)
                (void) fprintf (stderr,
                    "%s:\tcan't read system clock\n", argv [0]);
        /*      now process any arguments supplied...           */
        while (optind != argc) {
                (void) fprintf (stderr, "Processing %s...\n", argv [optind]);
                /*      open the input file             */
                if ((in = fopen (argv [optind], "r")) == (FILE *) NULL) {
                        (void) fprintf (stderr,
                            "%s:\tcan't open %s\n",
                            argv [0], argv [optind]);
                        exit (EXIT_FAILURE);
                }
                /*      scan it                         */
                while ((c = getc (in)) != EOF) {
                        /*
                         *      allow tab, new line and space
                        */
                        if (c == 9 || c == 10 || c == 32 ||
                           (c >= '!' && c <= '~'))
                                (void) putc (c, stdout);
                }
                /*      close the input file            */
                if (fclose (in))
                        (void) fprintf (stderr,
                            "%s:\tcan't close %s\n",
                            argv [0], argv [optind]);
                optind++;
        }
        /*      get a finish time                       */
        if (time (&t1) < (time_t) 0)
                (void) fprintf (stderr,
                    "%s:\tcan't read system clock\n", argv [0]);
        if (vopt)
                (void) fprintf (stderr,
                    "%s duration %g seconds\n",
                    argv [0], difftime (t1, t0));
        exit (EXIT_SUCCESS);
}

lucmove · 06-24-2009, 12:37 PM

The 'history' command doesn't work:

Code:

/usr/bin/ksh: /home/luc/loghistory.sh[4]: fc: no history (yet)

It's kind of you to provide me some C code, but it's not useful to me. I can only write a couple of scripting languages. If I'll have to go through the trouble of learning C just so I can enjoy the Korn shell, then I might as well stick to Bash...

bigearsbilly · 06-24-2009, 01:22 PM

what about script?

Code:

NAME
     script -- make typescript of terminal session

saves it all.

lucmove · 06-24-2009, 02:04 PM

Yeah, it saves WAY too much.

bigearsbilly · 06-24-2009, 02:48 PM

what ksh you using?
what platform?
the proper one?
I use pdksh and my history is in plaintext

what you could do is specify an EXIT trap in your profile.

Code:

trap command EXIT

which can be a function or script if you want.
this will invoke each time you leave a shell.

tronayne · 06-24-2009, 02:53 PM

Actually, you don't have to know anything about C -- save the source code file as histfile.c, then

Code:

make histfile
histfile /home/luc/.sh_history (that's the default Korn history file)

or

Code:

cc -o histfile histfile.c
histfile /home/luc/.sh_history

The program will produce the entire history file on the standard output; i.e., your terminal session. If you want to save that, simply

Code:

histfile /home/luc/.sh_history > whatever_file_you_want

You can then use, say, the uniq utility to extract unique command entries from that file (or in a pipe from running histfile) and, perhaps, pipe through the sort utility if you want a sorted list.

I wonder, though, if you're running Korn Shell as your login shell, why the history command does not work? It's not going to work if your login shell is BASH (because that's not Korn Shell and there won't be any history entries) perhaps... Korn Shell records every command in, by default, ~/.sh_history and you can read that with the Korn Shell built-in history command, or fc or the histfile program listed above. If you have a different history file defined (the HSITFILE environment variable), use that file name instead.

Anyway, best of luck with it.

bigearsbilly · 06-24-2009, 03:14 PM

ksh on solaris also has text history file.

?????

lucmove · 06-24-2009, 03:56 PM

@bigearsbilly, I am using mksh on Kubuntu Linux. I had too much trouble with key bindings with both ksh93 and pdksh. The key bindings are a lot better with mksh. But they all had binary history files. I checked. I deleted them and let korn recreate them in each case.

Your tip involving 'trap' isn't very clear, I'll have to read the friendly manual to get acquainted with the 'trap' command.

@tronayne Wow, that's a useful tool. Thanks!

But the output is given entirely in one single line. Can you modify it so it will break the lines? Or am I doing something wrong?

Korn is indeed my login shell and the history command is aliased to 'fc -l'. I think it's hard wired, I don't think I can change it. It works as advertised, but only interactively. It won't work in the .profile (.kshrc in my case) or an external script.

bigearsbilly · 06-24-2009, 04:25 PM

Quote:

Originally Posted by lucmove

Korn is indeed my login shell and the history command is aliased to 'fc -l'. I think it's hard wired, I don't think I can change it. It works as advertised, but only interactively. It won't work in the .profile (.kshrc in my case) or an external script.

you can change it, or remove it

Code:

unalias history

trap example:
try this:

Code:

$ ksh
$ blah()
> {
> echo bye bye
> }
$ trap blah EXIT
$ exit
bye bye

lucmove · 06-24-2009, 05:24 PM

@tronayne I just realized your program also prepends an extra character to some lines, just like the output of 'strings'. Except 'strings' outputs multiple lines, so it's a little better than your program.

@bigearsbilly I can unalias history, but then I try to use it and just get this:

/usr/bin/ksh: /home/luc/.kshrc[32]: history: not found

That figures. There is no 'history' executable in the file system, it's usually a Bash builtin, which Korn does not have. Well, it does, but it's a different resource.

tronayne · 06-25-2009, 07:15 AM

Actually, the Korn Shell built-in is hist; from the ksh manual page (along with others)

Code:

The  following  aliases are compiled into the shell but can be unset or redefined:
autoload='typeset -fu'
command='command  '
fc=hist
float='typeset -lE'
functions='typeset -f'
hash='alias -t --'
history='hist -l'
integer='typeset -li'
nameref='typeset -n'
nohup='nohup  '
r='hist -s'
redirect='command exec'
source='command .'
stop='kill -s STOP'
suspend='kill -s STOP $$'
times='{ { time;} 2>&1;}'
type='whence -v'

You can see what aliases are set by entering

Code:

alias

at a command prompt.

The hist built-in must be used as hist -l to get a listing of the shell history (otherwise it opens an ed-like editor).

These are, of course, "real" Korn Shell rules and regulations and your mileage may vary significantly if you're using an emulator or non-Korn port.

I'm interested: just what extra character(s) are prepended? Space character, tabs, something else? When I threw the program together I tested it against my own .sh_history file and got zero funky characters (all non alphanumeric or punctuation characters are ignored with the exception of tab, line feed and space, so I'm kind of wondering).

lucmove · 06-25-2009, 09:05 AM

I'm using mksh and it doesn't have 'hist'.

The extra characters are the same as the ones in 'strings': letters and numbers, sometimes a fairly common symbol, like = or @.