How to query an on-premises Dynamics CRM from a Web App (Node/Express)

Question

Been banging my head against a few walls with this so hoping some CRM/Dynamics experts can give me a hand!

I'm trying to programatically obtain data out of our Dynamics CRM instance, using a single set of admin credentials within a Node powered Express app. This Express app is hosted on a separate server outside of our network where CRM is hosted. The app will then request, process and serve CRM data back to any logged in user who has access (controlled by roles/permissions within the app), meaning an end user only has to login into the Express app, and not have to also login via ADFS in order for the app to access the CRM instance.

Our CRM set up is an on premise server configured to be internet facing (IFD). This uses Active Directory Federation services. We have Web Application Proxy Servers running federation services on the perimeter of the network that communicate with ADFS servers on the internal network. ADFS authenticates users connecting from outside the network (from internet) against the on prem AD. Once authenticated the proxy allows users to connect through to CRM.

Our on prem active directory is synced with Azure AD as we have a hybrid deployment. Any O365 service (exchange online, sharepoint etc) uses Azure AD in the background. We synchronise the Active directory so we only have to manage users in one place.

The CRM has an endpoint, e.g. https://my.crm.endpoint and I have registered an app (called CRM App) in the Azure Portal, with the homepage set to the CRM endpoint https://my.crm.endpoint.

Question Is setting the app's Homepage to https://my.crm.endpoint enough to "link" it to our on premise CRM instance?

I've written a script (crm.js) that successfully requests an access token for my CRM App registered in Azure Portal, using it's App ID.

Example Token

eyJ0dWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzE5ZTk1...

Using the bearer token, I then attempt to get some contacts out of Dynamics via the the usual endpoint: https://my.crm.endpoint/api/data/v8.2/contacts?$select=fullname,contactid

This fails and I get a 401 Unauthorised error message.

Question Can anyone suggest what the problem could be? And/or provide details of how you can hook up a Web App (Express in my case) to make authenticated requests to a Dynamics CRM running on an on-premise server (IFD) that uses ADFS?

crm.js

let util = require('util');
let request = require("request");

let test = {
    username: '<[email protected]>',
    password: '<my_password>',
    app_id: '<app_id>',
    secret: '<secret>',
    authenticate_url: 'https://login.microsoftonline.com/<tenant_id>/oauth2/token',
    crm_url: 'https://<my.crm.endpoint>'
};
function CRM() { }

CRM.prototype.authenticate = function () {
    return new Promise((resolve, reject) => {
        let options = {
            method: 'POST',
            url: test.authenticate_url,
            formData: {
                grant_type: 'client_credentials',
                client_id: test.app_id,         // application id
                client_secret: test.secret,     // secret
                username: test.username,        // on premise windows login (admin)
                password: test.password,        // password
                resource: test.app_id           // application id
            }
        };

        // ALWAYS RETURNS AN ACCESS_TOKEN
        request(options, function (error, response, body) {
            console.log('AUTHENTICATE RESPONSE', body);
            resolve(body);
        });
    })
};

CRM.prototype.getContacts = function (token) {
    return new Promise((resolve, reject) => {

        let options = {
            method: 'GET',
            url: `${test.crm_url}/api/data/v8.2/contacts?$select=fullname,contactid`,
            headers: {
                'Authorization': `Bearer ${token}`,
                'Accept': 'application/json',
                'OData-MaxVersion': 4.0,
                'OData-Version': 4.0,
                'Content-Type': 'application/json; charset=utf-8'
            }
        };

        request(options, (error, response, body) => {
            console.log('getContacts', util.inspect(error), util.inspect(body));
            resolve(body);
        });

    });
};

let API = new CRM();    // instantiate the CRM object

API.authenticate()      // call authenticate function
    .then(response => {
        if (response) {

            let json = JSON.parse(response);
            let token = json.access_token;

            console.log('TOKEN', token);

            API.getContacts('token')
            .then(contacts => {
                // DO SOMETHING WITH THE CONTACTS
                console.log('CONTACTS', contacts);
            })
        }
    });


module.exports = CRM;

Error Response

HTTP Error 401 - Unauthorized: Access is denied

ADDITIONAL INFO

My current solution is based off these docs...

https://learn.microsoft.com/en-us/azure/active-directory/develop/active-directory-protocols-oauth-service-to-service

UPDATE

Following @andresm53's comment, I think I do need to authenticate against ADFS directly. I've found this blog post that describes generating a shared secret in ADFS that can be used with OAuth.

"Using this form of Client Authentication, you would POST your client identifier (as client_id) and your client secret (as client_secret) to the STS endpoint. Here is an example of such an HTTP POST (using Client Credentials Grant, added line breaks only for readability):"

resource=https%3a%2f%2fmy.crm.endpoint
&client_id=**2954b462-a5de-5af6-83bc-497cc20bddde ** ???????
&client_secret=56V0RnQ1COwhf4YbN9VSkECTKW9sOHsgIuTl1FV9
&grant_type=client_credentials

UPDATE 2

I have now created the Server Application in ADFS and am POSTing the above payload with the correct client_id and client_secret.

However, I get an Object moved message.

RESOLVED BODY: '<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to <a href="https://fs.our.domain.name/adfs/ls/?wa=wsignin1.0&amp;wtrealm=https%3a%2f%2fmy.crm.endpoint%2f&amp;wctx=http%253a%252f%252f2954b462-a5de-5af6-83bc-497cc20bddde%252f&amp;wct=2018-04-16T13%3a17%3a29Z&amp;wauth=urn%3afederation%3aauthentication%3awindows">here</a>.</h2>\r\n</body></html>\r\n'

QUESTION Can anyone please describe what I am doing wrong and what I should be doing in order to authenticate against ADFS/CRM correctly?

NB: When I'm in my browser and visit https://my.crm.endpoint, I get prompted to enter my username and password. Entering my creds works and I get access to CRM. Have noticed in the network tab that it's using NTLM to do this? Does this change what approach I need to take?

UPDATE 3

Please see new question here

Answer 1

We had this similar situation. Our Organization is OnPrem 8.2. It is accessible via VPN or from Home Network. If you look at the problem in very basic layman's way, our CRM cannot be reached from outside.

What we did was

1. We created WebAPI for Action from CRM.

2. We exposed this WebAPI via additional Port to outer world.

3. We added this WebAPI in IIS as a service.

4. But we made sure This WebAPI was accessible only via specific userName and Passoword which we created in our Web.config File.

5. In Background what we did was created Action.

6. Action in Turn will run Plugin and will Return Data as per asked i.e WebAPI url can be modified. For ex: .../acounts will return for Account entity Provided you had the logic built in your plugin.

   Please do not confuse this with Dynamics CRM OOB WebAPI. What I mean is that creating our own API and add this as a service in IIS with it's own Username and password.

I assume this will give you at-least some hint in which direction to look into.

Answer 2

So... I managed to get this off the ground by reverse engineering the browsers approach to authenticating :) No proxy or Azure nonsense!

I am now directly authenticating with our fs endpoint and parsing the resulting SAML response and using the cookie it provides... which works a treat.

NB: The code below was just knocked up in my Node scratch pad, so it's a mess. I might tidy it up and post a full write up at some point, but for now, if you use any of this code you will want to refactor appropriately ;)

let ADFS_USERNAME = '<YOUR_ADFS_USERNAME>'
let ADFS_PASSWORD = '<YOUR_ADFS_PASSWORD>'

let httpntlm = require('httpntlm')
let ntlm = httpntlm.ntlm
let lm = ntlm.create_LM_hashed_password(ADFS_PASSWORD)
let nt = ntlm.create_NT_hashed_password(ADFS_PASSWORD)
let cookieParser = require('set-cookie-parser')
let request = require('request')

let Entity = require('html-entities').AllHtmlEntities
let entities = new Entity()

let uri = 'https://<YOUR_ORGANISATIONS_DOMAIN>/adfs/ls/wia?wa=wsignin1.0&wtrealm=https%3a%2f%2f<YOUR_ORGANISATIONS_CRM_URL>%2f&wctx=rm%3d1%26id%3d1fdab91a-41e8-4100-8ddd-ee744be19abe%26ru%3d%252fdefault.aspx%26crmorgid%3d00000000-0000-0000-0000-000000000000&wct=2019-03-12T11%3a26%3a30Z&wauth=urn%3afederation%3aauthentication%3awindows&client-request-id=e737595a-8ac7-464f-9136-0180000000e1'
let apiUrl = 'https://<YOUR_ORGANISATIONS_CRM_URL>/api/data/v8.2/'
let crm = 'https://<YOUR_ORGANISATIONS_CRM_URL>'

let endpoints = {
  INCIDENTS: `${apiUrl}/incidents?$select=ticketnumber,incidentid,prioritycode,description`,
  CONTACTS: `${apiUrl}/contacts?$select=fullname,contactid`
}

httpntlm.get({
  url: uri,
  username: ADFS_USERNAME,
  lm_password: lm,
  nt_password: nt,
  workstation: '',
  domain: ''
}, function (err, res) {
  if (err) return err
  // this looks messy but is getting the SAML1.0 response ready to pass back as form data in the next request
  let reg = new RegExp('&lt;t:RequestSecurityTokenResponse([\\s\\S]*?)&lt;\/t:RequestSecurityTokenResponse>')
  let result = res.body.match(reg)
  let wresult = entities.decode(result[ 0 ])

  reg = new RegExp('name="wctx" value="([\\s\\S]*?)" /><noscript>')
  result = res.body.match(reg)

  let wctx = entities.decode(result[ 1 ])
  let payload = {
    wctx: wctx,
    wresult: wresult
  }
  getValidCookies(payload)
    .then(cookies => {

      getIncidents(cookies)
        .then(contacts => {
          console.log('GOT INCIDENTS', contacts)
        })
    })
})

getValidCookies = function (payload) {
  return new Promise((resolve, reject) => {

    let options = {
      method: 'POST',
      url: crm,
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded'
      },
      form: {
        'wa': 'wsignin1.0',
        'wresult': payload.wresult,
        'wctx': payload.wctx
      }
    }

    request(options, (error, response, body) => {
      let requiredCookies = []
      let cookies = cookieParser.parse(response)

      cookies.forEach(function (cookie) {
        if (cookie.name === 'MSISAuth' || cookie.name === 'MSISAuth1') {
          requiredCookies.push(`${cookie.name}=${cookie.value}`)
        }
      })
      resolve(requiredCookies)
    })

  })
}

getIncidents = function (cookies) {
  return new Promise((resolve, reject) => {

    let options = {
      method: 'GET',
      url: endpoints.INCIDENTS,
      headers: {
        'Cookie': cookies.join(';')
      }
    }

    request(options, (error, response, body) => {
      resolve(body)
    })

  })
}

getContacts = function (cookies) {
  return new Promise((resolve, reject) => {

    let options = {
      method: 'GET',
      url: endpoints.CONTACTS,
      headers: {
        'Cookie': cookies.join(';')
      }
    }

    request(options, (error, response, body) => {
      resolve(body)
    })

  })
}