The default Messaging app in Android (tested against version 2.1) appears to put the soft keyboard in a special mode, where the return key is replaced with an "emoticon key". When you press it you get a selection of emoticons to insert into your message.

(source: futurenet.com)
How do I instruct the soft keyboard to appear in this mode in my app?
This is definitely an imeOption being set immediately on EditText. I don't know exactly which one it is, it doesn't seem to match any of the ones described here. You might be able to use Hierarchy Viewer to help you pin down what the exact options are.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With