From ota at surfvi.com  Tue Sep  8 03:59:01 2009
From: ota at surfvi.com (Ted Anderson)
Date: Mon,  7 Sep 2009 20:59:01 -0400
Subject: History Href mechanism bug or misfeature
Message-ID: <20090907205901.xq2ow8gwckwo8cwg@my.surfvi.com>

[ This may be too arcane for this list; let me know if it is. ]

The history mechanism has two counters for each element: Hnum, which is the
event number, and Href, which is obscurely related.  As I understand it, an
element's Href is set to the current event when it is used in a history
substitution.  Then Href is used to purge old elements from the history list,
not Hnum.  This allows reused entries to be preserved without falling off the
back, at bit like an LRU list.  If someone could enlighten me further about
this, I'd appreciate it.

The problem I found is that when histdup=erase and a command is duplicated, it
resets the Href counters past where the duplicate was found (a "renumber" loop
just after hfree() in the erase clause of the enthist function).  This seems
like a bug, or maybe just a misfeature.  But perhaps I have overlooked some way
in which this is useful.

Here is the test case:
cat <<EOF > ~/src/tcsh-6.17.00/hist-events.csh
set histdup=erase
set histfile=/tmp/test.history
set history=5
: 1
: 2
: 3
: 2
: 4
: !-5
: : !7
: x !4
: x !7
history 99
!!
EOF
% ./tcsh -f -q -i < ~/src/tcsh-6.17.00/hist-events.csh
> : : 1
: : : 2
: x : 1
: x : 2
     4	20:52	: 1
     7	20:52	: 2
     9	20:52	: : 1
    10	20:52	: : : 2
    11	20:52	: x : 1
    12	20:52	: x : 2
    13	20:52	history 99
history 99
     9	20:52	: : 1
    10	20:52	: : : 2
    11	20:52	: x : 1
    12	20:52	: x : 2
    14	20:52	history 99
> exit

The bug is that events 4 & 7 are lost by the second "history 99" command due to
its being a duplicate of the previous command.  If I change that last line to
"history 999", then it isn't identical and preserves both old events.  Removing
the renumber loop fixes this problem.

Ted Anderson


From ota at surfvi.com  Sun Sep  6 21:17:32 2009
From: ota at surfvi.com (Ted Anderson)
Date: Sun,  6 Sep 2009 14:17:32 -0400
Subject: Inefficient history can make start and exit very slow
Message-ID: <20090906141732.5cs8c4ok4cw8ccc8@my.surfvi.com>

I recently increased my tcsh history size to 15000 and found it unacceptably
slow when starting the shell and when exiting.  A simple tcsh command requires
15 seconds of CPU time with my settings.

% echo 'echo $version;set | grep hist' | /usr/bin/time -f "%E real,%U user,%S
sys" ./tcsh
tcsh 6.17.00 (Astron) 2009-07-10 (i486-intel-linux) options
wide,nls,dl,al,kan,rh,color,filec
histdup	erase
history	(15000 %h  %D-%w %P  %R\n)
savehist	(15000 merge)
0:14.71 real,14.34 user,0.29 sys

Some research with gprof and initial experiments with using hash tables has
highlighted problems with histdump=erase and using merge in savehist.  However,
even after addressing these problems performance is still somewhat sluggish and
I think eliminating all O(n) behavior in enthist() is necessary.  A few simple
experiment with bash show no similar performance problems there.

Anyway, I am working on a patch and have three questions.

1. Has anyone else looked at this area and have prior work, ideas or suggestions
I should be building upon?

2. I noticed in v6.17 there is a new test function.  I'd very much like to
ensure my history changes don't break any user visible semantics.  I see
nothing about this testing infrastructure in the mailing list archives or in
README or similar files.  Can someone provide a reference to how to use this
mechanism that explains how to read the .at files and write new ones?  Are
there any special notes for testing tcsh?

3. I am using Ubuntu Hardy which comes with 6.14 (now 4+ years old), and the
latest Ubuntu and Debian have the same version.  I made my changes to the 6.14
source, but it looks like the 6.17 history code is quite similar.  Is there
some reason not to ask the Ubuntu/Debian folks to upgrade?  More forcefully, is
there a cogent and compelling reason that would make them want to upgrade?

Thanks,
Ted Anderson


From christos at zoulas.com  Tue Sep  8 13:51:30 2009
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 8 Sep 2009 06:51:30 -0400
Subject: Inefficient history can make start and exit very slow
In-Reply-To: <20090906141732.5cs8c4ok4cw8ccc8@my.surfvi.com>
	from Ted Anderson (Sep  6,  2:17pm)
Message-ID: <20090908105130.5E48C5654E@rebar.astron.com>

On Sep 6,  2:17pm, ota at surfvi.com (Ted Anderson) wrote:
-- Subject: Inefficient history can make start and exit very slow

| I recently increased my tcsh history size to 15000 and found it unacceptably
| slow when starting the shell and when exiting.  A simple tcsh command requires
| 15 seconds of CPU time with my settings.
| 
| % echo 'echo $version;set | grep hist' | /usr/bin/time -f "%E real,%U user,%S
| sys" ./tcsh
| tcsh 6.17.00 (Astron) 2009-07-10 (i486-intel-linux) options
| wide,nls,dl,al,kan,rh,color,filec
| histdup	erase
| history	(15000 %h  %D-%w %P  %R\n)
| savehist	(15000 merge)
| 0:14.71 real,14.34 user,0.29 sys
| 
| Some research with gprof and initial experiments with using hash tables has
| highlighted problems with histdump=erase and using merge in savehist.  However,
| even after addressing these problems performance is still somewhat sluggish and
| I think eliminating all O(n) behavior in enthist() is necessary.  A few simple
| experiment with bash show no similar performance problems there.
| 
| Anyway, I am working on a patch and have three questions.
| 
| 1. Has anyone else looked at this area and have prior work, ideas or suggestions
| I should be building upon?

No.

| 2. I noticed in v6.17 there is a new test function.  I'd very much like to
| ensure my history changes don't break any user visible semantics.  I see
| nothing about this testing infrastructure in the mailing list archives or in
| README or similar files.  Can someone provide a reference to how to use this
| mechanism that explains how to read the .at files and write new ones?  Are
| there any special notes for testing tcsh?

This is the standard gnu autotest suite.

| 3. I am using Ubuntu Hardy which comes with 6.14 (now 4+ years old), and the
| latest Ubuntu and Debian have the same version.  I made my changes to the 6.14
| source, but it looks like the 6.17 history code is quite similar.  Is there
| some reason not to ask the Ubuntu/Debian folks to upgrade?  More forcefully, is
| there a cogent and compelling reason that would make them want to upgrade?

Yes, many bugs. Just use the ls-F builtin and watch it wrap on many directories
for one. The list of changes is in a file called Fixes. There are more than
120 bug fixes between 6.14 and 6.17.

christos