Building enterprise app with Node/Express

Question

I'm trying to understand how to structure enterprise applciation with Node/Express/Mongo (actually using MEAN stack).

After reading 2 books and some googling (including similar StackOverflow questions), I couldn't find any good example of structuring large applications using Express. All sources I've read suggest to split application by following entities:

• routes
• controllers
• models

But the main problem I see with this structure is that controllers are like god objects, they knows about req, res objects, responsible for validation and have business logic included in.

At other side, routes seems to me like over-engineering because all they doing is mapping endpoints(paths) to controller methods.

I have Scala/Java background, so I have habit to separate all logic in 3 tiers - controller/service/dao.

For me following statements are ideal:

• Controllers are responsible only for interacting with WEB part, i.e. marshalling/unmarshalling, some simple validation (required, min, max, email regex etc);

• Service layer (which actually I missed in NodeJS/Express apps) is responsible only for business logic, some business validation. Service layer doesn't know anything about WEB part (i.e. they can be called from other place of application, not only from web context);

• Regarding to DAO layer is all clear for me. Mongoose models are actually DAO, so it most clear thing to me here.

I think examples I've seen are very simple, and they shows only concepts of Node/Express, but I want to look at some real world example, with much of the business logic/validation involved in.

EDIT:

Another thing isn't clear to me is absent of DTO objects. Consider this example:

const mongoose = require('mongoose'); const Article = mongoose.model('Article'); exports.create = function(req, res) {     // Create a new article object     const article = new Article(req.body);     // saving article and other code } 

There JSON object from req.body is passed as parameter for creating Mongo document. It smells bad for me. I would like to work with concrete classes, not with raw JSON

Thanks.

Answer 1

Controllers are God objects until you don't want them to be so...
   – you don't say zurfyx (╯°□°）╯︵ ┻━┻

Just interested in the solution? Jump onto the latest section "Result".

┬──┬◡ﾉ(° -°ﾉ)

Prior getting started with the answer, let me apologize for making this response way longer than the usual SO length. Controllers alone do nothing, it's all about the whole MVC pattern. So, I felt like it was relevant to go through all important details about Router <-> Controller <-> Service <-> Model, in order to show you how to achieve proper isolated controllers with minimum responsibilities.

Hypothetical case

Let's start with a small hypothetical case:

• I want to have an API that serves an user search through AJAX.
• I want to have an API that also serves the same user search through Socket.io.

Let's start with Express. That's easy peasy, isn't it?

routes.js

import * as userControllers from 'controllers/users'; router.get('/users/:username', userControllers.getUser); 

controllers/user.js

import User from '../models/User'; function getUser(req, res, next) {   const username = req.params.username;   if (username === '') {     return res.status(500).json({ error: 'Username can\'t be blank' });   }   try {     const user = await User.find({ username }).exec();     return res.status(200).json(user);   } catch (error) {     return res.status(500).json(error);   } } 

Now let's do the Socket.io part:

Since that's not a socket.io question, I'll skip the boilerplate.

import User from '../models/User'; socket.on('RequestUser', (data, ack) => {   const username = data.username;   if (username === '') {     ack ({ error: 'Username can\'t be blank' });   }   try {     const user = User.find({ username }).exec();     return ack(user);   } catch (error) {     return ack(error);   } }); 

Uhm, something smells here...

• if (username === ''). We had to write the controller validator twice. What if there were n controller validators? Would we have to keep two (or more) copies of each up to date?
• User.find({ username }) is repeated twice. That could possibly be a service.

We have just written two controllers that are attached to the exact definitions of Express and Socket.io respectively. They will most likely never break during their lifetime because both Express and Socket.io tend to have backwards compatibility. BUT, they are not reusable. Changing Express for Hapi? You will have to redo all your controllers.

Another bad smell that might not be so obvious...

The controller response is handcrafted. .json({ error: whatever })

APIs in RL are constantly changing. In the future you might want your response to be { err: whatever } or maybe something more complex (and useful) like: { error: whatever, status: 500 }

Let's get started (a possible solution)

I can't call it the solution because there is an endless amount of solutions out there. It is up to your creativity, and your needs. The following is a decent solution; I'm using it in a relatively large project and it seems to be working well, and it fixes everything I pointed out before.

I'll go Model -> Service -> Controller -> Router, to keep it interesting until the end.

Model

I won't go into details about the Model, because that's not the subject of the question.

You should be having a similar Mongoose Model structure as the following:

models/User/validate.js

export function validateUsername(username) {   return true; } 

You can read more about the appropriate structure for mongoose 4.x validators here.

models/User/index.js

import { validateUsername } from './validate';  const userSchema = new Schema({   username: {      type: String,      unique: true,     validate: [{ validator: validateUsername, msg: 'Invalid username' }],   }, }, { timestamps: true });  const User = mongoose.model('User', userSchema);  export default User; 

Just a basic User Schema with an username field and created updated mongoose-controlled fields.

The reason why I included the validate field here is for you to notice that you should be doing most model validation in here, not in the controller.

Mongoose Schema is the last step before reaching the database, unless someone queries MongoDB directly you will always rest assured that everyone goes through your model validations, which gives you more security than placing them on your controller. Not to say that unit testing validators as they are in the previous example is trivial.

Read more about this here and here.

Service

The service will act as the processor. Given acceptable parameters, it'll process them and return a value.

Most of the times (including this one), it'll make use of Mongoose Models and return a Promise (or a callback; but I would definitely use ES6 with Promises if you are not doing so already).

services/user.js

function getUser(username) {   return User.find({ username}).exec(); // Just as a mongoose reminder, .exec() on find                                 // returns a Promise instead of the standard callback. } 

At this point you might be wondering, no catch block? Nope, because we're going to do a cool trick later and we don't need a custom one for this case.

Other times, a trivial sync service will suffice. Make sure your sync service never includes I/O, otherwise you will be blocking the whole Node.js thread.

services/user.js

function isChucknorris(username) {   return ['Chuck Norris', 'Jon Skeet'].indexOf(username) !== -1; } 

Controller

We want to avoid duplicated controllers, so we'll only have a controller for each action.

controllers/user.js

export function getUser(username) { } 

How does this signature look like now? Pretty, right? Because we're only interested in the username parameter, we don't need to take useless stuff such as req, res, next.

Let's add in the missing validators and service:

controllers/user.js

import { getUser as getUserService } from '../services/user.js'  function getUser(username) {   if (username === '') {     throw new Error('Username can\'t be blank');   }   return getUserService(username); } 

Still looks neat, but... what about the throw new Error, won't that make my application crash? - Shh, wait. We're not done yet.

So at this point, our controller documentation would look sort of:

/**  * Get a user by username.  * @param username a string value that represents user's username.  * @returns A Promise, an exception or a value.  */ 

What's the "value" stated in the @returns? Remember that earlier we said that our services can be both sync or async (using Promise)? getUserService is async in this case, but isChucknorris service wouldn't, so it would simply return a value instead of a Promise.

Hopefully everyone will read the docs. Because they will need to treat some controllers different than others, and some of them will require a try-catch block.

Since we can't trust developers (this includes me) reading the docs before trying first, at this point we have to make a decision:

• Controllers to force a Promise return
• Service to always return a Promise

⬑ This will solve the inconsistent controller return (not the fact that we can omit our try-catch block).

IMO, I prefer the first option. Because controllers are the ones which will chain the most Promises most of the times.

return findUserByUsername          .then((user) => getChat(user))          .then((chat) => doSomethingElse(chat)) 

If we are using ES6 Promise we can alternatively make use of a nice property of Promise to do so: Promise can handle non-promises during their lifespan and still keep returning a Promise:

return promise          .then(() => nonPromise)          .then(() => // I can keep on with a Promise. 

If the only service we call doesn't use Promise, we can make one ourselves.

return Promise.resolve() // Initialize Promise for the first time.   .then(() => isChucknorris('someone')); 

Going back to our example it would result in:

... return Promise.resolve()   .then(() => getUserService(username)); 

We don't actually need Promise.resolve() in this case as getUserService already returns a Promise, but we want to be consistent.

If you are wondering about the catch block: we don't want to use it in our controller unless we want to do it a custom treatment. This way we can make use of the two already built-in communication channels (the exception for errors and return for success messages) to deliver our messages through individual channels.

Instead of ES6 Promise .then, we can make use of the newer ES2017 async / await (now official) in our controllers:

async function myController() {     const user = await findUserByUsername();     const chat = await getChat(user);     const somethingElse = doSomethingElse(chat);     return somethingElse; } 

Notice async in front of the function.

Router

Finally the router, yay!

So we haven't responded anything to the user yet, all we have is a controller that we know that it ALWAYS returns a Promise (hopefully with data). Oh!, and that can possibly throw an exception if throw new Error is called or some service Promise breaks.

The router will be the one that will, in an uniform way, control petitions and return data to clients, be it some existing data, null or undefined data or an error.

Router will be the ONLY one that will have multiple definitions. The number of which will depend on our interceptors. In the hypothetical case these were API (with Express) and Socket (with Socket.io).

Let's review what we have to do:

We want our router to convert (req, res, next) into (username). A naive version would be something like this:

router.get('users/:username', (req, res, next) => {   try {     const result = await getUser(req.params.username); // Remember: getUser is the controller.     return res.status(200).json(result);   } catch (error) {     return res.status(500).json(error);   } }); 

Although it would work well, that would result in a huge amount of code duplication if we copy-pasted this snippet in all our routes. So we have to make a better abstraction.

In this case, we can create a sort of fake router client that takes a promise and n parameters and does its routing and return tasks, just like it would do in each of the routes.

/**  * Handles controller execution and responds to user (API Express version).  * Web socket has a similar handler implementation.  * @param promise Controller Promise. I.e. getUser.  * @param params A function (req, res, next), all of which are optional  * that maps our desired controller parameters. I.e. (req) => [req.params.username, ...].  */ const controllerHandler = (promise, params) => async (req, res, next) => {   const boundParams = params ? params(req, res, next) : [];   try {     const result = await promise(...boundParams);     return res.json(result || { message: 'OK' });   } catch (error) {     return res.status(500).json(error);   } }; const c = controllerHandler; // Just a name shortener. 

If you are interested in knowing more about this trick, you can read about the full version of this in my other reply in React-Redux and Websockets with socket.io ("SocketClient.js" section).

How would your route look like with the controllerHandler?

router.get('users/:username', c(getUser, (req, res, next) => [req.params.username])); 

A clean one line, just like in the beginning.

Further optional steps

Controller Promises

It only applies to those who use ES6 Promises. ES2017 async / await version already looks good to me.

For some reason, I dislike having to use Promise.resolve() name to build the initialize Promise. It's just not a clear what's going on there.

I'd rather replace them for something more understandable:

const chain = Promise.resolve(); // Write this as an external imported variable or a global.  chain   .then(() => ...)   .then(() => ...) 

Now you know that chain marks the start of a chain of Promises. So does everyone who reads your code, or if not, they at least assume it's a chain a service functions.

Express error handler

Express does have a default error handler which you should be using to capture at least the most unexpected errors.

router.use((err, req, res, next) => {   // Expected errors always throw Error.   // Unexpected errors will either throw unexpected stuff or crash the application.   if (Object.prototype.isPrototypeOf.call(Error.prototype, err)) {     return res.status(err.status || 500).json({ error: err.message });   }    console.error('~~~ Unexpected error exception start ~~~');   console.error(req);   console.error(err);   console.error('~~~ Unexpected error exception end ~~~');     return res.status(500).json({ error: '⁽ƈ ͡ (ुŏ̥̥̥̥םŏ̥̥̥̥) ु' }); }); 

What's more, you should probably be using something like debug or winston instead of console.error, which are more professional ways to handle logs.

And that's is how we plug this into the controllerHandler:

  ...   } catch (error) {     return res.status(500) && next(error);   } 

We are simply redirecting any captured error to Express' error handler.

Error as ApiError

Error is considered the default class to encapsulate errors in when throwing an exception in Javascript. If you really only want to track your own controlled errors, I'd probably change the throw Error and the Express error handler from Error to ApiError, and you can even make it fit your needs better by adding it the status field.

export class ApiError {   constructor(message, status = 500) {     this.message = message;     this.status = status;   } } 

Additional information

Custom exceptions

You can throw any custom exception at any point by throw new Error('whatever') or by using new Promise((resolve, reject) => reject('whatever')). You just have to play with Promise.

ES6 ES2017

That's very opinionated point. IMO ES6 (or even ES2017, now having an official set of features) is the appropriate way to work on big projects based on Node.

If you aren't using it already, try looking at ES6 features and ES2017 and Babel transpiler.

Result

That's just the complete code (already shown before), with no comments or annotations. You can check everything regarding this code by scrolling up to the appropriate section.

router.js

const controllerHandler = (promise, params) => async (req, res, next) => {   const boundParams = params ? params(req, res, next) : [];   try {     const result = await promise(...boundParams);     return res.json(result || { message: 'OK' });   } catch (error) {     return res.status(500) && next(error);   } }; const c = controllerHandler;  router.get('/users/:username', c(getUser, (req, res, next) => [req.params.username])); 

controllers/user.js

import { serviceFunction } from service/user.js export async function getUser(username) {   const user = await findUserByUsername();   const chat = await getChat(user);   const somethingElse = doSomethingElse(chat);   return somethingElse; } 

services/user.js

import User from '../models/User'; export function getUser(username) {   return User.find({}).exec(); } 

models/User/index.js

import { validateUsername } from './validate';  const userSchema = new Schema({   username: {      type: String,      unique: true,     validate: [{ validator: validateUsername, msg: 'Invalid username' }],   }, }, { timestamps: true });  const User = mongoose.model('User', userSchema);  export default User; 

models/User/validate.js

export function validateUsername(username) {   return true; } 

Answer 2

Everyone has its own way of dividing the project into certain folders. the structure which I use is

• config
• logs
• routes
• controllers
• models
• services
• utils
• app.js/server.js/index.js (any name u prefer)

config folder contain configuration files like database connection settings for all phase of development like "production","development","testing"

example

'use strict' var dbsettings = {     "production": { //your test settings     },     "test": {      },     "development": {         "database": "be",         "username": "yourname",         "password": "yourpassword",         "host": "localhost",         "connectionLimit": 100     } } module.exports = dbsettings 

log folder contain your connection logs error logs for debugging

controller is for validating your req data and business logic

example

const service = require("../../service") const async = require("async") exports.techverify = (data, callback) => {      async.series([         (cb) => {             let searchObject = { accessToken: data.accessToken }             service.admin.get(searchObject, (err, result) => {                 if (err || result.length == 0) {                     callback(err, { message: "accessToken is invalid" })                 } else {                     delete data.accessToken                     service.tech.update(data, { verified: true }, (err, affe, res) => {                         if (!err)                             callback(err, { message: "verification done" })                         else                             callback(err, { message: "error occured" })                     })                 }             })         }     ]) } 

models is for defining your db schema

example mongoDb schema

'use strict' let mongoose = require('mongoose'); let schema = mongoose.Schema; let user = new schema({     accesstoken: { type: String },     firstname: { type: String },     lastname: { type: String },     email: { type: String, unique: true },     image: { type: String },     phoneNo: { type: String },     gender: { type: String },     deviceType: { type: String },     password: { type: String },     regAddress: { type: String },     pincode: { type: String },     fbId: { type: String, default: 0 },     created_at: { type: Date, default: Date.now },     updated_at: { type: Date, default: Date.now },     one_time_password: { type: String },     forgot_password_token: { type: String },     is_block: { type: Boolean, default: 0 },     skin_type: { type: String },     hair_length: { type: String },     hair_type: { type: String },     credits: { type: Number, default: 0 },     invite_code: { type: String },     refered_by: { type: String },     card_details: [{         card_type: { type: String },         card_no: { type: String },         card_cv_no: { type: String },         created_at: { type: Date }     }] }); module.exports = mongoose.model('user', user); 

services is for writing your data base query avoid writing queries in controller try to write query in this folder and call it in the controller

queries using mongoose

'use strict' const modelUser = require('../../models/user'); exports.insert = (data, callback) => {     console.log('mongo log for insert function', data)     new modelUser(data).save(callback) } exports.get = (data, callback) => {     console.log('mongo log for get function', data)     modelUser.find(data, callback) } exports.update = (data, updateData, callback) => {     console.log('mongo log for update function', data)     modelUser.update(data, updateData, callback); } exports.getWithProjection = (data, projection, callback) => {     console.log('mongo log for get function', data)     modelUser.find(data, projection, callback) } 

utils is for common utility function which is commonly used in your project maybe like encrypt,decrypt password etc

example

exports.checkPassword = (text, psypherText) => {     console.log("checkPassword executed")     console.log(text, psypherText)     return bcrypt.compareSync(text, psypherText) } exports.generateToken = (userEmail) => {     return jwt.sign({ unique: userEmail, timeStamp: Date.now }, config.keys.jsonwebtoken) }