Define byte appearing in debug after a manually encoded far call

Question

I couldn't get MASM to accept a far call instruction written as call 0f000h:1260h, probably because of the issues brought up in this question.

Instead of messing around with cryptic MASM directives, I decided to manually encode it into my program using DB like so:

pushf                      ;IRET will be executed, push flags.
db 9ah,60h,12h,0,0f0h      ;call location f000:1260.
                           ;Location pointed to by int 1c (System timer tick)
                           ;BIOS defaults it to a dummy IRET

While tracing through the program with DEBUG.COM, I noticed that "DB FE" appears after the execution of the call instruction. However, this doesn't occur when executing int 1ch. What's the difference between these two methods of jumping to location f000:1260?

I assumed that DEBUG wasn't recognizing 0xfe (along with the following bytes) as a valid opcode. I dumped location f000:1260 in order to see which bytes were there.

Byte 0xfe is indeed present, along with some other bytes. I know that 0xcf by itself is the opcode for IRET (which is all that I was expecting to find), so what are these other bytes?

Here's the IVT entry for int 1ch at location 0000:0070.

UPDATE

As Michael Petch stated in his answer, the strange bytes make up a callback mechanism into DOSBox. I was curious to see what would happen if I tried to execute this callback in my main program.

Executing:

xor ah, ah                 ;select set video mode function
mov al, 13h                ;320x200 256 colors
db 0feh,38h,18h,00h        ;set video mode DOSBox callback.
                           ;Nothing pushed to stack.

Seems to be exactly the same as executing:

xor ah, ah                 ;select set video mode function
mov al, 13h                ;320x200 256 colors
int 10h                    ;set video mode.
                           ;Three registers pushed, FLAGS altered (by INT)
                           ;callback occurs, registers popped (by IRET)

The only difference being that int pushes FLAGS, CS, and IP, as well as clears IF and TF. The program is returned to an IRET (at location f000:1264) which undoes all of that.

"DB FE" still shows up in DEBUG. I think the callback is only triggered by the combination of 0xfe and 0x38. It first tries to execute 0xfe, which in this case is not part of a valid opcode and does nothing (0xfe is part of inc opcode when followed by valid bytes), then upon encountering the following 0x38 the callback occurs.

Answer 1

0xFE 0x38 is not a defined prefix and/or instruction on a real Intel x86 processor. In DOSBox the sequence 0xFE 0x38 is a special 4 byte instruction.The remaining two byte make up a 16-bit value that act as a DOSBox callback index number. In this case the index is 0x0013.

What this effectively does is calls across into DOSBox itself to perform the requested task. DOSBox will do whatever processing it needs, sets the registers inside the emulator and then returns. The next instruction is the IRET (0xCF). That will end the interrupt and continue processing instructions prior to the interrupt call.

I discovered this while looking at the DOSBox code. In particular the function CALLBACK_SetupExtra:

case CB_IRET:
    if (use_cb) {
        phys_writeb(physAddress+0x00,(Bit8u)0xFE);  //GRP 4
        phys_writeb(physAddress+0x01,(Bit8u)0x38);  //Extra Callback instruction
        phys_writew(physAddress+0x02,(Bit16u)callback);     //The immediate word
        physAddress+=4;
    }
    phys_writeb(physAddress+0x00,(Bit8u)0xCF);      //An IRET Instruction

This code sets up DOSBox callbacks where an IRET is required after the callback.