Lines vs rows in the terminal

Question

There appears to be some concept of lines vs rows in terminal emulators, about which I'd like to know more.

Demonstration of what I mean by rows vs lines

The Python script below displays three lines of 'a' and waits, then with three lines of 'b'.

import sys, struct, fcntl, termios

write = sys.stdout.write
def clear_screen(): write('\x1b[2J')
def move_cursor(row, col): write('\x1b['+str(row)+';'+str(col)+'H')
def current_width(): #taken from blessings so this example doesn't have dependencies
    return struct.unpack('hhhh', fcntl.ioctl(sys.stdout.fileno(), termios.TIOCGWINSZ, '\000' * 8))[1]

clear_screen()
for c in 'ab':
    #clear_screen between loops changes this behavior
    width = current_width()
    move_cursor(5, 1)
    write(c*width+'\n')
    move_cursor(6, 1)
    write(c*width+'\n')
    move_cursor(7, 1)
    write(c*width+'\n')

    sys.stdout.flush()
    try: input() # pause and wait for ENTER in python 2 and 3
    except: pass

If you narrow the terminal window width by one character during this break, you see

That seems pretty reasonable - each line has been separately wrapped. When we hit enter again to print bs,

Everything works as expected. I've used absolute cursor positioning, and written to the same rows I wrote to previously - which of course doesn't overwrite all of the a's, because many of them are on other rows.

However, when we narrow the window by one more character, the wrapping works differently:

Why did the second and third rows of b wrap together, and why did last line of a's merge with the first line of b's? A hint of why is in the top visible row above - we see two a's because theyse two rows are still linked - of course if we move the window again, that one line will continue to wrap the same way. This seems to be happening even for lines which we replaced a whole row of.

It turns out that the rows that had wrapped before are now linked to their corresponding parent rows; it's more obvious that they belong to the same logical line once we widen the terminal a lot:

My question

Practically, my question is how to prevent or predict this rows-being-combined-into-lines. Clearing the whole screen eliminates the behavior, but it would be nice to do this only for individual lines that need it if possible so I can keep the caching by line that is significantly speeding up my application. Clearing to the end of a row unlinks that row from the row below it, but clearing to the beginning of a row does not unlink that row from the one above it.

I'm curious - what are these line things? Where can I read about them? Can I find out which rows are part of the same line?

I've observed this behavior with terminal.app and iterm, with and w/o tmux. I imagine source-diving into any of these would yield an answer even if there's no spec - but I imagine there's a spec somewhere!

---

Background: I'd like to make a terminal user interface that can predict the way terminal wrapping will occur if the user decreases the window width. I'm aware of things like fullscreen mode (tput smcup, or python -c 'print "\x1b[?1049h"', which are what ncurses uses) which would work for preventing line wrap, but don't want to use it here.

Edit: made it more clear that I understand the overwriting behavior of the script already and want an explanation of the wrapping behavior.

Answer 1

OK. Let's start with the causes for the behavior you are seeing:

I tested your code and noticed that it only happened when you resized the window. When the window was left alone, it would write out the a's, and upon pressing enter would over-write them with b's (I assume that's the intended behavior).

What appears to be happening is that when you resize the window partway through, the line indices change, so that on your next iteration, you can't trust the same coordinates when you call move_cursor().

Interestingly, when you resize the window, the word wrapping pushes the text before the cursor upwards. I assume this is part of the terminal emulator's code (since we almost always want to retain focus on the cursor and if the cursor is at the bottom of the screen, resizing might obscure it beyond the window's height if the word-wrapping pushed it downwards).

You'll notice that after a resize when you hit enter, only two lines of a's remain visible (and not all 3). Here's what appears to be happening:

First we begin with the initial output. (line numbers added for clarity)

1
2
3
4
5 aaaaaaaaaaaaaaa\n
6 aaaaaaaaaaaaaaa\n
7 aaaaaaaaaaaaaaa\n
8 

Note that there is a new line character at the end of each of these lines (which is why your cursor appears below the last despite your not having moved the cursor again)

When you shrink the window by one character, this happens:

1
2 aaaaaaaaaaaaaa
3 a\n
4 aaaaaaaaaaaaaa
5 a\n
6 aaaaaaaaaaaaaa
7 a\n
8

You'll notice what I mean by "pushing the text upwards"

Now when you hit enter and your loop reiterates, the cursor is sent to row 5 col 1 (as specified by your code) and is placed directly over the last a of the second line. When it starts writing b's it overwrites the last a of the second line with b's and the subsequent line as well.

1
2 aaaaaaaaaaaaaa
3 a\n
4 aaaaaaaaaaaaaa
5 bbbbbbbbbbbbbb\n
6 bbbbbbbbbbbbbb
7 bbbbbbbbbbbbbb\n
8

Importantly, this also overwrites the new-line character at the end of the second line of a's. This means that there is now no new-line dividing the second line of a's and the first line of b's, so when you expand the window: they appear as a single line.

1
2
3 
4
5 aaaaaaaaaaaaaaa\n
6 aaaaaaaaaaaaaabbbbbbbbbbbbbb\n
7 bbbbbbbbbbbbbbbbbbbbbbbbbbbb\n
8

I'm not totally sure why this second line of b's also gets put together but it appears to likely have something to do with the fact that the line of a's which the first one overwrites is now missing it's own new-line termination. However, that's just a guess.

The reason why you get two characters of line-wrap if you try to shrink the window by yet another character is because now you are shrinking two halves of the same line of text, which means that one pushes on the other, causing two characters instead of one at the end.

For example: in these test windows I've shown, the width begins at 15 characters, I then shrink it to 14 and print out the b's. There is still one line of a's which is 15 chars long, and now a line of 14 a's & 14 b's which is line-wrapped at 14 chars. The same (for some reason) is true of the last two rows of b's (they are one line of 28 chars, wrapped at 14). So when you shrink the window by one more character (down to 13): the first line of 15 a's now has two trailing characters (15 - 13 = 2); the next line of 28 chars now has to fit in a 13 character-wide window (28 / 13 = 2 R2), and the same applies to the last b's as well.

0 aaaaaaaaaaaaa
1 aa\n
2 aaaaaaaaaaaaa
3 abbbbbbbbbbbb
4 bb\n
5 bbbbbbbbbbbbb
6 bbbbbbbbbbbbb
7 bb\n
8

Why does it work like this?:

This sort of stuff is the difficulty you run into when you are trying to run your program within another program that has the power to reposition the text as it sees fit. In the event of a resize your indices become unreliable. Your terminal emulator is trying to handle the realignment for you and is pushing the text before your prompt (which is fixed at row 8) up and down in the scroll-back to ensure you can always see your active prompt.

Rows and columns are something defined by the terminal/terminal emulator and it is up to it to interpret their location accordingly. When the appropriate control sequences are given it is the terminal which interprets them accordingly for proper display.

Note that SOME terminals do behave differently and in an emulated terminal there is often a setting to change what sort of terminal it is emulating, which may also affect how certain escape sequences respond. This is why a UNIX environment usually has a setting or environment variable ($TERM) which tells it which type of terminal it is communicating with so that it knows what control sequences to send.

Most Terminals use standard ANSI compliant control sequences, or systems based on the DEC VT series of Hardware Terminals.

In the Terminal.app preferences under Preferences->Settings->Advanced you can actually see (or change) which type of Terminal is being emulated by your window in the drop-down menu next to "Declare terminal as:"

How to overcome this:

You might be able to mitigate this by storing the last known width and checking to see if there has been a change. In which case you can change your cursor logic to compensate for the changes.

Alternately you might consider using escape sequences designed for relative cursor movement (as opposed to absolute) to avoid accidentally overwriting previous lines after a resize. There is also the ability to save and restore specific cursor locations using only escape sequences.

Esc[<value>A  Up
Esc[<value>B  Down
Esc[<value>C  Forward
Esc[<value>D  Backward
Esc[s         Save Current Position
Esc[u         Restore Last Saved Position
Esc[K         Erase from cursor position to end of line

However you have no real guarantee that all Terminal emulators will deal with window resizes the same way (that's not really part of any terminal standard, AFAIK), or that it won't change in the future. If you are hoping to make a true terminal emulator, I suggest first getting your GUI window setup so that you can be in control of all resizing logic.

However if you want to run in a terminal-emulator window and deal with mitigating window resizes for a given command-line utility that you're writing. I'd suggest looking at the curses library for python. This is the sort of functionality used by all window-resize aware programs that I know of off the top of my head (vim, yum, irssi), and can deal with this sort of changes. Though I don't personally have any experience using it.

It's available for python via the curses module.

(and please, if you plan on redistributing your program, consider writing it in Python3. Do it for the children :D)

Resources:

These links might be helpful:

• ANSI Escape Sequences
• VT100 Escape Sequences

I hope that helps!

Answer 2

As 0x783czar pointed out, the key difference is whether an explicit newline was printed which caused the terminal to begin a new row, or there was an implicit overflow because there was no room left on the right to print the desired characters.

It's important to remember this at the end of each line for copy-pasting purposes (whether there'll be a newline character in the buffer or not), for triple-click highlight behavior in many terminals, and for rewrapping the contents when the window is resized (in those terminals that support it).

Applications running inside terminals hardly ever care about this difference, and they use the words "line" and "row" interchangeably. Hence, when we implemented rewrapping the contents on resize in gnome-terminal, we preferred the words "row" or "line" for one single visual line of the terminal, and the word "paragraph" for the contents between two adjacent newline characters. A paragraph wraps into multiple lines if it's wider than the terminal. (This is not by any means an official terminology, but IMO is quite reasonable and helps talk about these concepts.)