"GSSException Defective token detected" - when trying to Authenticate to Tomcat running on Windows using Kerberos

Question

I am struggling to authenticate to a Java web container (I've tried both Tomcat and Jetty) when running on Windows 2012.

Every time I try the Negotiate auth scheme I get an error: org.ietf.jgss.GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)

Steps to reproduce

Start out by setting up a Windows Server 2012 or 2016 instance and install active directory domain services.

In my example, I created:

• NETBIOS Domain: NICKIS

• Dns domain: nickis.life

Create the kerberos subject user on Active Directory

IMPORTANT: MAKE SURE THAT THE FIRST NAME, LAST NAME AND FULL NAME ARE THE SAME!

The new user in my case is:

DN = CN=kerberos500,CN=Users,DC=nickis,DC=life

login+domain = [email protected]

NETBIOS\samAccountName = NICKIS\kerberos500

Run the setspn command from the Windows Active Directory Server

setspn -A HTTP/[email protected] kerberos500

Example output:

C:\Users\Administrator>setspn -A HTTP/nickis.life kerberos500
Checking domain DC=nickis,DC=life 
Registering ServicePrincipalNames for CN=kerberos500,CN=Users,DC=nickis,DC=life
        HTTP/kerberos500.nickis.life
Updated object

Run the ktpass command from the Windows Active Directory Server

ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/[email protected] -mapUser kerberos500 -mapOp set -pass XXXXpasswordforkerberos500userXXXX -crypto DES-CBC-MD5 -pType KRB5_NT_PRINCIPAL +DesOnly

Example output:

C:\Users\Administrator>ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/[email protected] -mapUser kerberos500 -mapOp set -pass xxxxxxxx -crypto DES-CBC-MD5 -pType KRB5_NT_PRINCIPAL +DesOnly
Targeting domain controller: WIN-OVV6VHBGIB8.nickis.life
Using legacy password setting method
Successfully mapped HTTP/kerberos500.nickis.life to kerberos500.
Key created.
Output keytab to c:\Users\Administrator\kerberos500.keytab:
Keytab version: 0x502
keysize 71 HTTP/[email protected] ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x3 (DES-CBC-MD5) keylength 8 (0xcd07200bea625d20)
Account kerberos500 has been set for DES-only encryption.

At this point, you will now have a keytab file:

c:\Users\Administrator\kerberos500.keytab

And a user principal:

HTTP/[email protected]

These are the 2 inputs that are needed to provide to the GSSApi to get single sign on with Kerberos.

So I deployed those inputs to my web container's kerberos security realm in the Hadoop security module.

Curl test I tried unsuccessfully to use curl to test it:

curl --negotiate -u : http://nickis.life:8080/my/webapp

Internet Explorer test I also tried using Internet Explorer. I added nickis.life domain to the Trusted Roles in Internet Explorer. Then I launch the site in internet explorer: http://nickis.life:8080

Either way, I get the error below:

org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
    at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:398) ~[hadoop-auth-2.7.1.jar:?]

...

Caused by: org.ietf.jgss.GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
    at sun.security.jgss.GSSHeader.<init>(Unknown Source) ~[?:1.8.0_131]
    at sun.security.jgss.GSSContextImpl.acceptSecContext(Unknown Source) ~[?:1.8.0_131]
    at sun.security.jgss.GSSContextImpl.acceptSecContext(Unknown Source) ~[?:1.8.0_131]
    at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:365) ~[hadoop-auth-2.7.1.jar:?]
    at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347) ~[hadoop-auth-2.7.1.jar:?]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
    at javax.security.auth.Subject.doAs(Unknown Source) ~[?:1.8.0_131]
    at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347) ~[hadoop-auth-2.7.1.jar:?]

I am stumped. NOTE: I have found several links around here and there but none of them were all inclusive on the steps that were followed like I have summed up here, and none of the solutions provided within worked for me.

• I am attempting kerberos login from a different machine on the domain than the server is running on
• I have tried all sorts of combinations of keytab generation variations none have worked.
• There is no duplicate SPN.
• I tried setting up the DNS in the domain server as an A record.
• I wonder maybe if there's some kerberos windows server setup steps and Microsoft employee verified that this should not be the case here: https://social.msdn.microsoft.com/Forums/sharepoint/en-US/db15ad96-e269-436e-952f-fe9dfb39da8a/setting-up-a-test-windows-server-active-directory-for-kerberos-testing?forum=winserverDS

Can anyone trace what I am screwing up here?

UPDATE:

• I have AD server with domain set to fusionis.life, and the AD server is WIN-OVV6VHBGIB8.fusionis.life
• I moved the tomcat server to another windows machine in the domain. DESKTOP-VTPBE99.fusionis.life
• I opened dnsmgmt.msc and added a "Forward Lookup Zone" with "kerberos500.nickis.life" with A HOST set to the IP of the DESKTOP-VTPBE99.fusionis.life box.
• I deleted the AD account, recreated it, then re-generated the keytab again as suggested in one of the answers on the ticket.

C:\Users\Administrator>ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/[email protected] -mapUser kerberos500 -mapOp set -pass xxxxxxxxx -crypto ALL -pType KRB5_NT_PRINCIPAL Targeting domain controller: WIN-OVV6VHBGIB8.fusionis.life Using legacy password setting method Successfully mapped HTTP/kerberos500.nickis.life to kerberos500. Key created. Key created. Key created. Key created. Key created. Output keytab to c:\Users\Administrator\kerberos500.keytab: Keytab version: 0x502 keysize 67 HTTP/[email protected] ptype 1 (KRB5_NT_PRINCIPAL) vno 4 etype 0x1 (DES-CBC-CRC) keylength 8 (0x04e30b9183ba8389) keysize 67 HTTP/[email protected] ptype 1 (KRB5_NT_PRINCIPAL) vno 4 etype 0x3 (DES-CBC-MD5) keylength 8 (0x04e30b9183ba8389) keysize 75 HTTP/[email protected] ptype 1 (KRB5_NT_PRINCIPAL) vno 4 etype 0x17 (RC4-HMAC) keylength 16 (0xe39a141de38abd8750bf9c0bf49fd1c5) keysize 91 HTTP/[email protected] ptype 1 (KRB5_NT_PRINCIPAL) vno 4 etype 0x12 (AES256-SHA1) keylength 32 (0xe368a1b060cfe4816f522c1c5f62ca07fe201ed96c6d018054dfbd5b86251892) keysize 75 HTTP/[email protected] ptype 1 (KRB5_NT_PRINCIPAL) vno 4 etype 0x11 (AES128-SHA1) keylength 16 (0x1b1a548fa2893a78c6f4c7f9c482b614)

• I saved the keytab updated file on the server, then updated the Service Principal to HTTP/[email protected]

• I logged into tomcat machine as a domain user, added http://kerberos500.nickis.life to the trusted sites, then navigated to http://kerberos500.nickis.life:8764

• I checked all combinations of the encryption check boxes in the kerberos500 AD "account" tab.

Now i'm getting a new error...

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)

UPDATE:

Resolved finally. I got this final error because I needed fusionis.life to be on the same host as nickis.life

Answer 1

The error “Defective token detected” likely means that an ntlm token was detected. That’s what the Negotiate mechanism uses inside popular web browsers if kerberos fails - when not instructed by the web server otherwise.  On windows operating systems, the IE web browser on it (and Firefox, if configured correctly) basically says, if you won’t do Kerberos, I’m going to send you an NTLM token.  And the server replies “no way” I don’t even know NTLM so I’m calling what you sent me defective.  Since you seem to be setting this up for the first time, you likely did not configure any fallback mechanism (such as NTLM) for when Kerberos fails, therefore, that error message. We solve this by understanding why Kerberos is failing.  I think I see the reason for the failure in your question, in two places, related to SPNs and Trusted Sites. Even if you resolve those two items, there is a third reason and fourth reason why it could continue to fail, related to encryption.

1. The spn for the HTTP service does not match the URL entered by the browser. These need to match, otherwise Kerberos will fail. To work, the browser should be using: http://kerberos500.nickis.life:8080, not http://nickis.life:8080. I say that based on what I saw in your keytab creation syntax. In that you’ve coded the SPN as such: HTTP/[email protected]. That’s why you need to use http://kerberos500.nickis.life:8080. The browser won’t know how to get to your web service when you tell it to go to http://nickis.life:8080. With that top URL, the browser assumes it needs to find a web service running on your Active Directorydomaincontroller (anything with just nickis.life is assumed to run on the Domain Controller). DCs should never run web servers for security reasons.
2. You’ll need to add http://kerberos500.nickis.life as a Trusted Site under IE settings. Alternatively, *.nickis.life would work as well. (You called it Trusted Roles, when it’s actually called Trusted Sites).
3. You are restricting the Kerberos encryption type to DES-CBC-MD5. Starting with Windows Server 2008 Active Directory R2, DES is disabled by default. DES is an outdated and generally insecure encryption type these days. Much better to use AES128 or even better, AES256. You can fix that by re-generating the keytab per my example below.
4. In the AD user account kerberos500, go to the Account tab, scroll to the bottom, and check all the boxes for DES, AES 128, and AES 256, and OK you’re way out of the dialog boxes. You must check these boxes even if you did everything right above, or else Kerberos authentication will still fail.

How to properly re-generate the keytab: You should not run the setspn -a command to add an SPN to an AD user whenever you are planning to make a keytab associated with that user account.  The reason why is because the keytab creation command adds the SPN to the user account as part of the command.  If your scenario doesn’t work after following my advice above, then you’ll need to remove the SPN via setspn -D like below:

setspn -D HTTP/[email protected] kerberos500

And the re-generate the keytab afterwards, my only change is that I told it to use all encryption types. The client and server will agree on the strongest common one during the authentication process. 

ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/[email protected] -mapUser kerberos500 -mapOp set -pass XXXXpasswordforkerberos500userXXXX -crypto ALL -pType KRB5_NT_PRINCIPAL

Then replace the old keytab with the new one. For additional in-depth information on keytabs, you can read more from my technical article on how to create Kerberos keytabs here: Kerberos Keytabs – Explained. I frequently go back and edit it based on questions I see here in this forum.

By the way, HTTP/kerberos500.nickis.life is a service principal, not a user principal as you wrote in your question. I only use web browsers to test Kerberos in HTTP scenarios like this one, I don’t use cURL.

I am positive if you diligently go through all four points I’ve highlighted above, you will resolve this problem.

EDIT1: This answer assumes you have an HTTP service running on a host with the fully-qualified domain name of kerberos500.nickis.life. If you don't have such a host with that name, my answer will slightly change. Please let me know if any.

EDIT2: To achieve the objective of authentication using the URL of http://nickis.life:8080, then you may keep on using the same keytab you already created.

On the AD account NICKIS\kerberos500, go to the Account tab, scroll to the bottom, and check the box for "Use Kerberos DES encryption types for this account".

Then enable DES encryption itself at the AD domain level via Group Policy. To do that, conduct the following:

1. Open up Group Policy Management Console (GPMC).
2. Edit Default Domain Policy GPO. (Its safer to create a new domain-level GPO instead and edit that, but that's up to you).
3. Navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Security Options > “Network Security: Configure Encryption types allowed for Kerberos” and check the both checkboxes for DES_CBC_MD5 and DES_CBC_MD5. IMPORTANT: In that same Group Policy, also ensure the checkboxes for RC4, AES128, and AES256 are also checked. Those encryption types won't be used for tickets to your web site, but they will for everything else in the domain. and OK you're way out of the dialog boxes and close the GPMC.
4. Run the 'gpupdate /force' command on both the DC server and the client.
5. Run 'klist purge' on the client to purge all Kerberos tickets.
6. In the web browser, clear the cache and delete all cookies.
7. Ensure that the DC server allows port 8080 TCP inbound.
8. Try it again.

Reference: Windows Configurations for Kerberos Supported Encryption Type

EDIT 3: Avoid running Kerberos KDC (the DC), client and server on the same machine. That is a classic recipe for getting the "Defective token error" even if you've done everything else right.

EDIT 4: (Final update which was verified by the OP): Looked at the new ktpass keytab creation output, and I saw this: Targeting domain controller: WIN-OVV6VHBGIB8.fusionis.life. Now, the defined SPN in the keytab is HTTP/kerberos500.nickis.life. The AD domain name is different from the SPN you defined, so this is not going to work unless you have some kind of trust setup between these domains. If you don't have a trust, you need to use an SPN of HTTP/kerberos500.fusionis.life instead.