Setting up Node Environments for a REST API - My Hair is Going Grey

Question

I am being very humble here coming to the community. I don't know jack about node environments yet, as well as am the only Node developer here. A one man team right now for building out my first Node REST API. (yea, really what we need are more people who already know a bit more Node than I to help this learning process along).

I've searched endlessly for hours on the web to try and do my due diligence..but frankly I'm just not finding much out there on setting up Node.js environments as well as setting up local vs. other environmental builds. First just starting with setting up a local build - baby steps is all I'm looking for... in a real world scenario. Most blog posts show the simplest development environment only at best. And so to make matters even worse, most the blog posts talk about website setup (browsify sh__ and stuff like that). I'm doing a REST API, not a website here.

I need some resources to start with which either I'm just not searching right for them for node or they just are that hard to find. I want really a walk through of how you'd set your environment variables, how you'd use gulp with those variables to run different environment situations, and how people are managing their persistent layer stuff like database connection strings, etc...which I assume is also automated in part by gulp tasks reading config files that contain the connection strings? I'm really not sure, there are not a lot of good examples out there..

Which is why I'm here to hopefully find some this route, by posting this here. (and hopefully comments on my current gulpfile...is it way f'd up or not even close to the right track or where the hell should I start for the build and environment conventions and tasks?)

What would be nice is to find a good posts that walk you through a day in the life of a node developer.

Where am I at Currently?

My API: Well I've build out a few Koa.js endpoints, I've got some controllers that take in the request, process it by using a gateway node modules I've guilt that have some CRUD persistence methods in what I call a gateway layer which calls underlying set of Model classes which mean the real CRUD calls to a real DB and some mocks for my tests, etc. Very basic at this point...all test driven via Mocha.js (BDD style).

My Day: So right now, I come into work, I have already setup my mocha.js test environment and I'm able to at least test drive development of my code and have tests run nicely on saves. I can watch my files and my tests run with mocha reports in the terminal. My app runs, all that.

My Problem: But I really haven't automated this much other than that via gulp. I wanna know more about setting up environment variables, what a developer would do when he/she's ready to build and push to git, yada yada and then ultimately how to switch environments, the motions of doing that when pushing to other servers, and all that via whatever configuration or gulp tasks that would facilitate this. I just need a good example of this kinda ecosystem. Any kind of example but something realistic would be so damn helpful.

So...Here's a glimps of my current gulpfile. Now there is build stuff in there for the babel transpiler (yes I'm using ES6 to ES5 code in my js), but I got that from some other code but really haven't figure out how all this will work yet. Really the stuff I setup is my test run and automation at this point. The build stuff I kinda disabled because a) I don't really know which files should be part of the build for a REST API outside your app.js, server.js and b) the build code I grabbed from some code I found transpiles js files via babel into a build folder but then I get issues with code collision/conflicts with my regular code. For example that build stuff below was pushing server.js into a build folder and when I tried to run the regular server.js in the root, it was colliding with the code in build/server.js so this is all f'd up and obviously I don't know what the hell I should be doing yet with builds.

'use strict';

require('babel/register');
require("harmonize")();

var gulp = require('gulp'),
    mocha = require('gulp-mocha'),
    gutil = require('gulp-util'),
    babel = require("gulp-babel"),
    server = require('gulp-develop-server'),
    del = require('del');


var config = {
    core: {
        src: '*.js',
        build: {
            src: 'build'
        }
    },
    server: {
        path: ['app.js']
    },
    bs: {
        proxy: 'http://localhost:3000'
    },
    test:{
        root: 'test/**',
        src: {
            bdd: {
                features:'test/bdd/features/**/*-spec.js',
                unit:'test/bdd/unit/**/*-test.js',
                integration: 'test/bdd/integration/**/*-integration.js'
            }
        },
        mocha: {
            reporter: 'spec'
        }
    }
};

gulp.task('default', ['babel','development'], function(){
    gulp.task("watch", function(){
        gulp.watch('**/*.js', ['babel'])
    });
});

gulp.task('development',['mocha-bdd-features', 'mocha-bdd-unit'], function() {
    gulp.watch('**/*.js', ['mocha-bdd-features', 'mocha-bdd-unit']);
});

gulp.task('babel', function () {
    return gulp.src('./*.js')
        .pipe(babel());
        //.pipe(gulp.dest('build'));
});

gulp.task('build', ['clean'], function () {
});

gulp.task('clean', del.bind(
    null, ['build/*'], {dot: true}
));

gulp.task('server:start', function() {
    server.listen( { path: config.server.path});
});

gulp.task('server:restart', function() {
    server.restart();
});

gulp.task('mocha-bdd-features', function() {
    process.env.PORT = 8001;
    return gulp.src([config.test.src.bdd.features], { read: false })
        .pipe(mocha({
            compilers: {
                js: babel
            },
            reporter: config.test.mocha.reporter,
            ui: 'bdd'
        }))
        .on('error', gutil.log);
});

gulp.task('mocha-bdd-unit', function() {
    process.env.PORT = 8002;
    return gulp.src([config.test.src.bdd.unit], { read: false })
        .pipe(mocha({
            compilers: {
                js: babel
            },
            reporter: config.test.mocha.reporter,
            ui: 'bdd'
        }))
        .on('error', gutil.log);
});

gulp.task('mocha-bdd-integration', function() {
    process.env.PORT = 8003;
    return gulp.src([config.test.src.bdd.integration], { read: false })
        .pipe(mocha({
            compilers: {
                js: babel
            },
            reporter: config.test.mocha.reporter,
            ui: 'bdd'
        }))
        .on('error', gutil.log);
});

gulp.on('err', function(e) {
    console.log(e.err.stack);
});

my package.json scripts thus far:

  "scripts": {
    "start": "gulp",
    "build": "gulp build", // I was running this for a while
    "release": "gulp build --release", //I haven't used this yet or know if I even will yet
// I definitely use these, I set these up myself
    "test-bdd-features": "gulp mocha-bdd-features",
    "test-bdd-unit": "gulp mocha-bdd-unit",
    "test-bdd-integration": "gulp mocha-bdd-integration"
  },

Basically as a developer the way I have my gulpfile right now is I type 'gulp' at the bash prompt (terminal from within Webstorm in my case) which runs the default gulp task, and then that first off a few more tasks at this point where a few things happen thus far:

1. it runs all my mocha tests on first run of gulp
2. watches all js files for mocha so that mocha tests run on save of any js code changes

Really that's all that's going on. I can run the node server ok and all that too. But it's very basic at this point. I've got the stupid app.listen() port hard coded in my dumb app.js file which obviously is...well hard coded!

And honestly I have different ports for my different test suites because I found I had to due to port conflicts but maybe that's just because of something weird I did, not sure if this is the norm to have diff ports for running your acceptance tests vs. unit but I'd assume so.

Ahhh!! Advil please!

So at this point I'm like pulling my hair out on how to get through this:

1. how to setup a real build for a REST API (ok you transpile js code via babel to where, and what and huh?)
2. the ecosystem for setting up tasks, configs, etc. for automating builds and pushing and running code in diff environments..what does that look like!
3. The pulling down of code in different environments and running those and what not on different server scenarios (dev, QA, stage, prod).

And because I know someone will respond with this for sure. Yes I understand "there are a million ways to do this". You're preaching to the choir. I get that but that's not the point here. I am asking for a lot of info here so feel free to pick and choose what to reply with to hopefully help get me in the right direction on builds or enviornments. I'm just asking for examples of ways period, or analysis of my gulp file, or just a place to start!...so I can understand generally the process people go through and what are typical task types run that do what in gulp, etc. and how environment variables are managed and what not typically in Node apps.

Answer 1

In my limited experience, I think the most beneficial thing I've learned so far with this build and config stuff ("sysadmin is why I drink") is to keep it as simple and composable as possible.

Config info

For starters, if this helps at all, I'm a big fan of keeping config information is a .json and referencing that throughout the project. For example:

// config.json
{
  "STAGING": "http://staging.yourdomain.com/api/",
  "PRODUCTION": "http://yourdomain.com/api/",
  "THE_BEST_PORT_EVER": 1337,
  "MY_DOGS_NAME": "Ginger"
}

This is that whole "magic string" avoidance thing, so you have just one place to reference all your pertinent environment information.

In your gulpfile, when you need to reference some URL, or some port, or just whatever, you just require it:

var config = require('config.json');

Test stuff

There's a 1,000 ways to do it... I'm not familiar with the current setup you have, but if it works it works. However, I will caveat that with, try to keep things as modular as possible (as in, even if the functions look like crap, at least they're all independent of each other... read: function composition.)

Building for different environments

I hate long, obfuscated gulpfiles. Currently, my team is working on various distinct web and mobile applications, and various iterations of static sites, all for one product. All of which need to be tracked in version control for various environments, and built and deployed to various environments. However, our build processes are relatively simple, I hope (though they could definitely be improved).

I like gulpfiles that look like this:

// all your require'd stuff

// this is helpful, 'cause you can do stuff like "gulp --production"
var argv = require('yargs').argv;

var paths = {
  sass: ['some/path/to/sass'],
  // various other paths
};

gulp.task('sass', compileSass);
gulp.task('html', html);
gulp.task('default', watch);
// etc.

// then, in pretty alphabetical order
function compileSass() {
  return gulp.src(paths.sass)
         .pipe(doSomeCoolStuff());
}

function html() {
  // etc.
}

function watch() {
  compileSass();
  html();

  if(argv.production) {
    someSpecificProductionFunction();
  }

  gulp.watch(paths.someAwesomePaths, ['someTask']);
}

I like gulpfiles that look like that, because they're compositional.

As far as building goes, I don't even worry about defining package.json scripts. Could you? Sure. I guess it depends on the project and matters of personal/team style, but for the projects we're doing, at the moment there's no benefit to add an additional layer of abstraction. Why run something like npm doSomething, which is just an alias for ./some-script, which is just an alias for, say, gulp --production?

Here's an excerpt from a README we have for an app:

Case 1

I want to run the app in my browser hitting my local server

Make sure your config-local.json matches config-example.json, but with your local server (something like http://your.ip.here:1337/api/). Make sure you have the most recent version of the server installed and running. The run ionic serve.

Case 2

I want to run the app in my browser hitting a different server.

Change the server address in your config-local.json and follow Case 1.

Case 3

I want to run the app on a device in a complete staging environment.

Run gulp --staging. Then run ionic run --device.

Case 4

I want to run the app on a device in a complete production environment.

Run gulp --production. Then run ionic run --device.

A lot of time is spent working on "how do I make this easier for the user?" But what about the question, "how do I make this easier for the developer?"

The fewer steps involved in a build process means that they're are fewer points of failure when trying to build.

That also means that your code needs to be as small as possible. 10 solutions might have the same outcome, but 9 of those solution might be crap.