What are the potential complexities / problems unique to a large NodeJS Application

Question

I come from a "traditional web application" background: think Java, .NET, PHP, ColdFusion, etc.

In evaluating NodeJS for use as the primary server-side technology for non-trivial applications, I'm wondering what complexities, problems, challenges a team of developers and operations folks might expect to face which are unique to NodeJS. In short, I'd like to reduce my unknowns. Some (not all) examples:

• How well does it lend itself to large-team development? What unique challenges exist, for Node, in a team of 20 or 50 or 200 developers?
• What unique challenges exist with respect to Database access? "Enterprisey" data access concerns are handled mostly trivially in Java (connection pools, security, etc via Spring). Is this the case with Node?
• Reporting-heavy applications often require Excel, PDF, even PNG export... How does Node fare in this type of application?
• Does Node present any unique challenges with respect to development-time debugging?
• What unique operations challenges exist? Everything from server restarts / hot code swap / load balancing to tools for monitoring and managing a production cluster.

And so forth. What lessons exist for developing, maintaining, and production-managing a 100+K LoC codebase, deployed across a farm of servers, touched by dozens of developers?

Answer 1

I'll try to answer some of your questions

How well does it lend itself to large-team development? What unique challenges exist, for Node, in a team of 20 or 50 or 200 developers?

• It does the same, as every other language does; nothing! Teams of programmers normally use versioning systems such as git, svn, mercurial etc. to deal with co-workers working on the same files

What unique challenges exist with respect to Database access? "Enterprisey" data access concerns are handled mostly trivially in Java (connection pools, security, etc via Spring). Is this the case with Node?

• Node is Database agnostic. You can use any database with node where drivers/wrappers exist for it (same as with PHP). There are many drivers/wrappers for relational Databases (MySQL, SQLite) and NoSQL (MongoDB)

Reporting-heavy applications often require Excel, PDF, even PNG export... How does Node fare in this type of application?

• Node can, as php and others do, access the normal shell. So if you handle an Image with php, you are using a wrapper for ImageMagick or GD. GD needs to be installed on the WebServer for it to work, because php sends the commands down to the command line. Same with node; find a wrapper (http://search.npmjs.org) and use the feature you desire.

Does Node present any unique challenges with respect to development-time debugging?

• Node is Javascript, so it doesn't compile, but does JIT. So a failure will only be detected as soon as it is executed. You'd want a local env for every developer next to your staging/dev machine and live servers so they can test the code prior to commiting it to the staging server

What unique operations challenges exist? Everything from server restarts / hot code swap / load balancing to tools for monitoring and managing a production cluster.

• No "Unique" challanges I am aware of. You'll deal with monitoring/heartbeats as with all other languages (yeah, there are languages that do this, like Erlang). You'll need a service like forever or supervisord to startup node on server restarts, but thats the same with Apache+PHP/Perl.
• (Node has a Cluster module which helps you handle multiple workers (for multi-core servers))

look at Git for managing your code. And choose the language based on what you want to be doing (high concurrency, scalability)

Answer 2

I will comment on the things for which I am qualified:

1. Connection Pooling to data sources. The standard Node HTTP(S) server tools implement pooling by default, and there are knobs that you can configure to improve or control performance. The community being very active, there are lots of other projects (like poolee) that implement either general-purpose or specialized connection pooling. You will need to look around. In fact, given your traditional Webdev background...

   1.1 Sidenote: Third-party libraries When developing with Node, you may find yourself using a lot of third-party libraries. depending on your particular background, this may seem strange. To understand why, consider .NET or Java: the standard libraries for those platforms are gargantuan. As a result, when using those platforms you might pick very few third-party tools or plugins to help you get your work done. Node, by comparison, has an intentionally narrow and strictly controlled standard library. All the necessary APIs to unify "the way things are written" in the interest of server performance are there, but no more.

   To help manage third-party libraries, the npm package manager was designed and included very early with node. It's known to be of high quality. The authors clearly anticipated a lot of third-party use.

2. Compute Power You mentioned image export. Javascript libraries for all of these things exist, so as far as "it can be done easily", it can. Keep in mind that if you have a compute-heavy task, Javascript may not be the most effective language to implement the core computation. The v8 engine allows you to write modules in C, if you need to, but forwarding a request to a specialized backend server is the thing that Node does extremely well.

3. Operations Challenges Node.js will not scale up to your hardware. If your server has multiple cores (which by now it most certainly does), you will need to run multiple server processes on the same physical hardware to achieve high utilization. This may make a different picture for operations: lots more processes than you would normally see, those processes groupable by physical or virtual machine.

   Breaking your server into processes is not a bad thing for reliability, by the way: One large traditional process will crash on an unhandled exception, taking down with it eight (or whatever) cores worth of active sessions. Multiple Node processes will still crash individually, but with a proportionally smaller impact. There's probably room to explore the point, "how many processes is too many?", taking into account both performance and monitorability. It may actually be worth it to use more Node processes per server than there are available cores.