Emacs how to bind command to all keys, with a few keys exempt?

Question

I want to make a minor-mode (foo-mode) which has its keymap (foo-mode-map), but when the user presses any key not in (foo-mode-map), the minor-mode should quit. How do I bind the turn-off-foo-mode all other keys?

EDIT: here is the solution I came up with based on the chosen answer. It accepts numeric input as well.

(defalias 'foo-electric-delete 'backward-kill-word)

(defun foo-mode-quit (&optional arg)
  (interactive)
  (let ((global-binding (lookup-key (current-global-map)
                                    (single-key-description last-input-event))))
    (unless (eq last-input-event ?\C-g)
      (push last-input-event unread-command-events))
    (unless (memq global-binding
                  '(negative-argument digit-argument))
      (foo-mode -1))))

(defvar foo-mode-map
  (let ((map (make-keymap)))
    (set-char-table-range (nth 1 map) t 'foo-mode-quit)
    (define-key map "-" 'negative-argument)
    (dolist (k (number-sequence ?0 ?9))
      (define-key map (char-to-string k) 'digit-argument))
    (define-key map [backspace] 'foo-electric-delete)
    map))

(define-minor-mode foo-mode
  "Toggle Foo mode.
     With no argument, this command toggles the mode.
     Non-null prefix argument turns on the mode.
     Null prefix argument turns off the mode.

     When Foo mode is enabled, the control delete key
     gobbles all preceding whitespace except the last.
     See the command \\[foo-electric-delete]."
  ;; The initial value.
  :init-value nil
  ;; The indicator for the mode line.
  :lighter " Foo"
  ;; The minor mode bindings.
  :keymap foo-mode-map
  :group 'foo)

Answer 1

I've included a full working example for creating a minor mode with the kind of behavior you want; the key is to use set-char-table-range on a keymap created by make-keymap which creates a dense keymap with a full char-table; using this on a sparse keymap created with make-sparse-keymap will not work.

(defalias 'foo-electric-delete 'backward-kill-word)

(defun foo-mode-quit (&optional arg)
  (interactive)
  (foo-mode -1))

(defvar foo-mode-map
  (let (map (make-keymap))
    (set-char-table-range (nth 1 map) t 'foo-mode-quit)
    (define-key map [backspace] 'foo-electric-delete)
    map))

(define-minor-mode foo-mode
  "Toggle Foo mode.
     With no argument, this command toggles the mode.
     Non-null prefix argument turns on the mode.
     Null prefix argument turns off the mode.

     When Foo mode is enabled, the control delete key
     gobbles all preceding whitespace except the last.
     See the command \\[foo-electric-delete]."
  ;; The initial value.
  :init-value nil
  ;; The indicator for the mode line.
  :lighter " Foo"
  ;; The minor mode bindings.
  :keymap foo-mode-map
  :group 'foo)

(defvar major-baz-mode-map '(keymap (t . major-baz-mode-default-function)))

Setting a default binding for a major mode map is easier and I include this example here, but as I noted above, this kind of spare keymap will not work for a minor mode:

(defvar major-baz-mode-map '(keymap (t . major-baz-mode-default-function)))

This is discussed in the documentation in Format of Keymaps where it says:

(t . binding)

This specifies a default key binding; any event not bound by other
elements of the keymap is given binding as its binding. Default
bindings allow a keymap to bind all possible event types without
having to enumerate all of them. A keymap that has a default binding
completely masks any lower-precedence keymap, except for events
explicitly bound to nil (see below).

Answer 2

One thing you have to decide is if, when the user presses another key, you should simply quit your mode, of if you should run the command bound to the key in question (like in incremental-search).

One way would be to use pre-command-hook to check if this-command is one of your commands, and turn off your mode if that would not be the case.