Visual Studio - Don't display all domain users in "Assign To" field

Question

If I load up TFS Web Access and go to Security > Users, I only see the 3 people I've added to my team. However, when I try to assign a task to someone in Web Access or in Visual Studio, it lists a bunch of users from the domain (not all users, looks like all IT people). Where does this come from? How can I change it... without exporting, editing and importing files via command line?

update: I found this line in the MSDN documentation:

Team Foundation \Team Foundation Valid Users

Members of this group have access to Team Foundation Server. This group automatically contains all users and groups that have been added anywhere within Team Foundation Server. You cannot modify the membership of this group.

I really don't understand... this is our own team's server, a separate install from the main dev team. I have no idea how these other 30 or 40 users got in this group. Major bonus <3 for any help on this. MikeR's answer will allow me to set administrators as the only assigness which will technically fix the issue, but I'd rather be able to use the groups as they were intended if possible.

Answer 1

The problem was that [TEAM FOUNDATION]\Valid Users included [TEAM FOUNDATION]\Team Foundation Administrators which included [BUILT IN]\Administrators

In the TFS Server Administration Console I selected Application Tier and clicked Group Membership. I then double-clicked on [TEAM FOUNDATION]\Team Foundation Administrators and removed [BUILT IN]\Administrators.

Now I only see my team and not all the SQL admins and engineers that were local admins on the server. All without any command line or addons.