Trouble in Node.js paradise: The mess that is npm
The preface
Let me begin by stating that I love building web apps with Node.js, and I think it’s one of the greatest things that’s happened in the web app space this decade. I have been using Node.js for various small projects the last nine months, so I think I have a reasonable grasp of the subject matter.
The mess
One of the most exiting things about Node.js is the intense vibrance of the community of developers around it and the strength of tools like npm and Github that makes it almost effortless to share your Node.js creations with the world.
However, this has also created a huge problem for developers like me – that of picking the right module to use. The number of modules on npm is growing at a staggering speed, and it is becoming increasingly difficult to shift the wheat from the chaff.
The example
A real-world example. I’ve been looking for a tool to help me serve up my app’s CSS and JavaScript in as small a package as possible to reduce load times. That means concatenation and minification, and preferably compression as well.
In my old Django-days, I’d have used django_compressor, and if I was new to the community, I’d have to look no further than this asset manager comparison page to see that it would be the right choice for me.
With Node.js, there are no such comforts. I’ve spent a couple of hours digging around on Github and npm (mainly by looking at modules that depend on UglifyJS), and after eliminating modules that were obviously unmaintaned/outdated/undocumented, I’ve managed to shorten my list to these options:
- ams
- app.js
- asereje
- assets-packager
- auton
- bastard
- beans
- browserify
- buddy
- buildr
- codesurgeon
- dryice
- express-asset
- folio
- hem
- inliner
- masher
- nap
- piler
- polymorph
- resmin
- smoosh
- snockets
- stitchup
- vivid-builder
- wepp
Yes, twenty-six possible modules to evaluate. I don’t have the entire list, but I’d expect it to be around a hundred modules.
The choice
This is an almost impossible choice. I’d have to spend a couple of days trying out each of these modules to figure out which one is the right fit for me. It’ll probably be faster for me to write my own little tool for it thand to educate myself of all these options.
And it’s not just this space that suffers from this problem. It’s the same for routing, templating, testing and similar things that are commonly used in Node apps.
The waste
Each of these modules have been crafted, built. Someone took the time to explore the problem space, figure out what the his requirements, build a reusable tool, document it (somewhat), make releases, publish them omn npm, fix bugs, etc, etc.
The above list probably represents more than two thousand man-hours of work, most of it wasted.
Of course, some duplication is necessary for the best ideas to float to the top, but contrasting this with my experiences from the Drupal community, this is absurd.
The solution
If you have been listening to NodeUp (which you probably should if you’re at all interested in Node.js), you would know that the npm developers are working to come up with metrics for packages, that will make it easier to gauge the quality of a module. This will definitely help with sorting out the mess, but I think we should also consider what we can do about the waste and duplication of effort.
I think this is mainly a question of culture. What makes the Drupal community so strong is how much collaboration is valued. Duplication of effort is frowned upon, responsible maintainership is encouraged.
There’s no easy way to accomplish this. I won’t suggest we restrict access to posting modules on npm or have a formal application system, like Drupal does, as I do not believe either of those would work with the Node.js community. We would need more formal structure for that and a much more cohesive community, and I do not think either of those are feasible (nor desirable) for the Node.js project.
The plea
Instead, I’d just like to plead with my fellow developers. Before you decide to roll your own, please see if there is an existing project you could help improve instead. Consider it a way of repaying for all the open source work you have benefited from. Paving the road for others, like others have paved it for you.
I know it’s fun to do it lone ranger style, to take on the beast and slay it all by yourself – but ask yourself this: Do you care enough about this problem space to maintain a module for the next year? Would you prefer your efforts being 100% of the experience for a dozen people – or 10% of the experience for thousands of people.
That is why I love open source. Knowing that each of the half a million Drupal installations out there have a small piece of my work in them. That the President of the United States relies on my work, in an infitessimally tiny way.
That is what I’d like to see more of in the Node.js community.