Vim won't accept newline as enter

Question

I have a default installation of vim on linux, on a system with a vt52 terminal emulator and unicode capability.

Everything has been working fine until today when I moved my vt52 terminfo file from the temporary local user directory to the permanent system directory ... at first it seemed fine, but all the sudden, when I try to execute a colon command by pressing enter (either CTRL-J, or the key itself == decimal 10), vim just flashes the screen but doesn't execute the command or leave command entry mode. However, If I press CTRL-M to get a carriage return character (decimal 13), vim does accept the command.

I tried copying the file back to ~/.terminfo again, but it didn't fix the problem...

I thought this might be either a tty issue or a corrupted vimrc file, so... first I tried deleting the .viminfo and .vimrc files, and doing a stty sane before starting vim but neither helps. I just noticed that enter doesn't work in insert mode either, but CTRL-M does.

Then I checked from the bash shell, using CTRL-V, that when I press CTRL-M, it shows ^M and when I press CTRL-J, it just enters and shows nothing. So -- to double check -- I did a stty raw and cat - | hexdumpx, and sure enough pressing enter returns only 0x0a, and pressing CTRL-M returns only 0x0d ; so the keyboard driver is returning the proper character in raw mode, and I retested in sane mode which apparently maps both of them to 0x0A.

The termcap file I re-compiled with tic, and copied to both ~/.terminfo/v/vt52u where it worked fine before -- and over to /usr/share/terminfo/v/vt52u.

But I don't see anything in the termcap file which could possibly cause the problem.

vt52u|vt52 with UTF-8:\
    :am:eo:rs=\Ee\Eb0\Eco:is=\EE\Ee:\
    :nl=^j:sr=\EI:bl=^g:ta=^i:\
    :ho=\EH:cr=^m:le=\ED:nd=\EC:do=\EB:up=\EA:ta=^i:nw=^j^m:xn:\
    :cm=\EY%+ %+ :it#8:co#75:li#24:\
    :sc=\Ej:rc=\Ek:\
    :vi=\Ef:ve=\Ee:\
    :so=\Eb0\Ec3:se=\Eb0\Eco:mh=\Eb8\Eco:mr=\Ebo\Ec0:me=\Eb0\Eco:\
    :cl=\EH\EJ:cb=\Eo:cd=\EJ:ce=\EK:\
    :km:kb=^h:

EDIT: I've isolated the problem by experiment 5 listed below, as being caused by something either in or missing from the above termcap file, or in the tic compiler's conversion of it to terminfo.

So -- what termcap entry is missing or which existing one is causing the problem?

---------------------- Additional experiments as I try to figure it out -------------------

1: doing a :set term in vim reveals term=vt52u .... which is correct. So VIM should be using the above termcap file, but I don't know from where (eg: cached version, or not -- or corrupted.) and :version only reveals +termcap which says vim should be using a tic compiled termcap file which is what I've been trying to give it.

2: I recompiled ncurses-5.9, and re-installed it to make absolutely sure no corrupt files exist. Nothing changed even with: ./configure --prefix=/usr --without-cxx --without-cxx-binding --without-ada --without-manpages --without-progs --without-tests --with-build-cc=gcc --with-shared --without-debug --without-profile --without-gpm --without-dlsym --without-sysmouse --enable-sigwinch --enable-hashmap --enable-scroll-hints --build=i686-linux --host=arm-linux-gnueabi --without-pthread --enable-widec --with-fallbacks=vt52u --disable-big-core --enable-termcap --enable-getcap-cache

3: doing a :set termcap shows more keys defined than the termcap file has, which is bad... termcap only defined one key ... and that one should be ^H not ^?, so VIM's value doesn't match, but the other termcap values all match, since ^[ is the same as escape \E in the termcap file. So, I have proof that vim is definitely loading the proper termcap file because t_so 's value is unique to the vt52u. So -- it's not a corrupt termcap.... :( weird.

t_kb <BS>    ^?        <DecMouse>  ^[[ 
t_kd <Down>  ^@        <NetMouse>  ^[}

t_sr=^[I       t_bc=^[D        t_le=^[D        t_cd=^[J        t_ce=^[K        t_cl=^[H^[J
t_me=^[b0^[co  t_mr=^[b0^[co   t_ve=^[e        t_vi=^[f        t_nd=^[C        t_se=^[b0^[c0
t_ZH=^[bo^[c0  t_ZR=^[b0^[co   t_so=^b0^[c3    t_cm=^[Y%p1%' '%+%c%p2%' '%+%c

4: Recompiled vim 7.4, to remove all built in terminals, and to insure no corrupt files. Had no effect; did not fix default value of backspace key being wrong.

echo "Please Edit feature.h so that NO_BUILTIN_TERMCAPS is always #defined."
sleep 5
vim /src/feature.h

./configure --prefix=/usr/ --build=i686-linux --host=arm-linux-gnueabi --with-features=big --disable-darwin --disable-selinux --disable-xsmp --disable-xsmp-interact --disable-mzschemeinterp --disable-tclinterp --disable-netbeans --disable-sniff --disable-gui --disable-cscope --disable-workshop --enable-multibyte --disable-gtktest --disable-gpm --disable-sysmouse --disable-xim --enable-pythoninterp=dynamic --without-x --with-tlib=ncursesw vim_cv_toupper_broken="yes" vim_cv_terminfo="yes"  vim_cv_tty_group="world" vim_cv_tty_mode="0620" vim_cv_getcwd_broken="yes" vim_cv_stat_ignores_slash="yes" vim_cv_memmove_handles_overlap="yes"

echo "Please Edit src/Makefile such that STRIP=arm-linux-gnueabi-strip"
sleep 5
vim src/Makefile
make
make install

5: In bash, I changed the terminal type to generic "export TERM=VT52" rather than the unicode version, with color support which I compiled with tic. ** THE NEWLINE PROBLEM WENT AWAY WITH THE SACRIFICE OF COLOR COMMANDS, AND OTHER FEATURES OF VT52U ** I need the features, but apparently something about the termcap file I listed above is defective.

Answer 1

This was an ugly bug to track down by trial and error...

Vim appears to require keydown to be defined, and if not defined it assumes a default value of ^@ ; for an unknown reason, it then treats the enter key as if it were a key down, rather than newline. ^@ is logically the value for character code 0, null; which is the only character that would be found in an empty string in "C" which vim is written in and appears to trigger the issue/feature/bug.

But in any event in the termcap shown in the opening post, the kd symbol is not defined; and that's what is causing the problem.

Within vim, the problem can be solved by setting the termcap keydown variable as the same escape sequence which would normally exit insert mode, and then do letter j (down): :set t_kd=\Ej

The same default can always be added to the termcap entries, too; Since this is an input escape sequence and not a terminal output escape sequence, it doesn't conflict with the VT52's "save cursor" escape sequence defined in the termcap; eg: The VT52 will never see it since Linux tty's when in line entry mode do not echo escape sequences that come from the keyboard back to the terminal's output.

If a terminal doesn't have true arrow keys -- defining the arrow keys as escaped versions of vim's h,j,k,and l keys might be a reasonably compatible solution as long as it doesn't conflict with other programs which might want to use those input escape sequences for other things. I don't use emacs and other popular programs which might want those escape sequences for something, so if anyone else knows which (if any) programs would have problems with that solution, a comment would be appropriate.

In ncurses 5.9,there appears to be a bug in the terminfo compiler, so that on many installations (eg: slackware 14), 'tic' will not be able to compile terminfo source files -- but only termcap source files. If you want the termcap source code for an arbitrary terminal, you can run "infocmp -C fooTerminalName > fooTerminalName.tcap" to have the system generate a termcap source file for you that can be edited and recompiled with tic successfully.