debuggable

Update: Isaac commented and explained why fuzzy version specifiers are here to stay. I'll be ok with it and will adapt my workflow accordingly.

NPM is the official node.js package manager. Unlike many package managers that came before, it is actually incredibly awesome, and has helped to create one of the most vibrant communities in the history of open source.

However, today I want to talk about a few aspects of npm that concern me. In particular I want to talk about stuff where I feel that NPM is making bad things easy, and good things hard.

NPM module versions are broken

Today, I tried to contribute to the forever module. The company I am helping had to patch their version of it because of a hard-to-reproduce bug in production and asked me to help submitting their fix upstream. Being the scientific type, I set out to write a test case against the forever version their patch is based on:

$ npm install forever@0.7.2

Fantastic, NPM lets me specify which version of forever I want to install. Now lets verify the installed version works:

$ ./node_modules/forever/bin/forever

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
TypeError: undefined is not a function
    at CALL_NON_FUNCTION_AS_CONSTRUCTOR (native)
    at Object. (/Users/Felix/Desktop/foo/node_modules/forever/lib/forever.js:43:23)
    ...

Oh no, what happened? Mind you, except for an unrelated patch, this version of forever is running perfectly fine in production.

Well, as it turns out, you have been lied to. There is no such thing as forever v0.7.2. At least not a single one. It depends on an implicit and unchangable second parameter: time.

Why is that? Well, it is because forever v0.7.2 depends on this:

And as it turns out, nconf has released newer versions matching this selector, featuring a different API.

You are doing it wrong

"Hah!", you might say. "That's why you should check your node_modules into git!".

I am sorry, but that is not helpful. While this will allow me to pin down the node modules used by my app exactly, it does not help me here. What I want to do is to reproduce this bug in a standalone copy of forever v0.7.2, then check if it exists in the latest version, and if so submit the test case and fix for it upstream.

However, I can't. Not without manually resolving all forever dependencies the way NPM resolved them when v0.7.2 was released. (The fact that forever is a bit of a spaceship when it comes to dependencies does not help either).

Discouraging Open Source

Speaking about Mikeal's article. I felt that something was wrong about checking your node_modules into git when reading it, but it is only now that I can point out what:

In the article, Mikeal argues that module authors should not try to exactly reference their dependency versions, so this way users would get more frequent updates of those dependencies and help test them.

However, he says doing so for your app is a good thing.

I disagree. To me, this approach discourages open source for two reasons:

a) Bug reports:

I currently maintain 44 NPM modules. It is very hard to keep up with that.

If you are asking me to support multiple versions of all my dependencies, I will have to stop helping people with bug reports for my modules.

When somebody reports a bug for a given version of my module, I want to know exactly what version he used. Figuring out when he installed my module to rule out dependency issues for every bug report is not an option for me.

b) Contributions

Ask yourself what is easier. Adding a quick patch to a node module you already track include in the git repo of your app, --or-- creating a fork of it, fixing the problem in the fork, pushing that fork on GitHub, changing your package.json to point to your fork, and submitting a pull request.

I know people cannot be forced to contribute back, nor should they be. But as things stand right now, checking in all node_modules of an app into git is the only sane option, as the version numbers in your package.json are essentially meaningless.

This means that contributing back to open source is made difficult by default, while keeping your patches to yourself is made easy. I would like this to be the other way arround.

Conclusion

I propose to gradually drop all support for fuzzy version specifiers from NPM.

To me, fuzzy version specifiers are entirely evil. They make things more complex. They force me to manually snapshot the packages I depend on for my apps. They prevent me from supporting and contributing to open source.

So rather than throwing more complexity at this problem, lets just remove this feature alltogether.

If you agree, please re-tweet this article or leave a comment.

--fg

You have written a node.js module lately? It has a test suite? Awesome! Time to get yourself a nerd badge of honor:

But hang on nerdy warrior, this precious award has to be earned. So go ahead and check out the sweetness that is Travis CI. Travis is an open source, free to use, continuous integration server. Initially it was just building ruby stuff, but these days it supports a ton of other languages, including node.js.

And luckily, getting travis to run your tests on every GitHub push is really easy as well:

Step 1: Go to Travis and login/connect with your GitHub account.

Step 2: Hover over your name on the top right, and select "Profile" from the dropdown.

Step 3: You should see all your GitHub projects. Flip the "Off" switch to "On" for a node.js project you want to use with travis.

Step 4: Add a .travis.yml file to your project with the following:

language: node_js
node_js:
  - 0.4
  - 0.6

Step 5: Make sure your package.json has something like this:

Step 6: Git push, and watch travis building your project on the home screen!

Step 7: Assuming your tests are passing, it is time to get your badge of honor. Adding it to your GitHub Readme.md is as simple as:

[![Build Status](https://secure.travis-ci.org/<GITHUB_USER>/<REPO_NAME>.png)](http://travis-ci.org/<GITHUB_USER>/<REPO_NAME>)

If you want to see an example of what this looks like, and you also happen to be in the market for some no-bullshit testing tools, check out my new libs:

• utest: The minimal unit testing library.
• urun: The minimal test runner.

That's it. And in case you are not excited enough yet, go and check out the Travis Docs to discover additional goodies like how to work with databases, etc.

--fg

Thanks to Isaac, npm is getting more and more awesome by the hour. One of the coolest recent additions (you need at least v1.0.26) is the ability to specify private git repositories urls as a dependency in your package.json files.

At transloadit, we are currently using the feature to move some of our infrastructure code into separate packages, allowing for those to be tested and developed in isolation making our core application easier to maintain and work on.

The syntax for referencing a git repository (and commit) is as follows:

This will include a private npm module called "private-repo" from GitHub. The url also contains an optional refspec (#v0.0.1) that tells npm which branch, commit, or in this case tag you want to have checked out.

Now of course this is not the only way to do private npm repositories, but it is much simpler than running your own registry, so I would recommend it to most people.

Before you head of to play with this, here is a final tip that may safe you some headaches. In all your private npm modules, add "private": true to your package.json. This will make sure npm will never let you accidentally publish your secret sauce to the official npm registry.

Happy hacking, --fg

PS: When deploying, don't forget that you need to authorize the servers ssh key for the GitHub repository you are depending on.

With now more than 3000 modules, there are huge gaps in the quality of things you find in the npm registry. But more often than not, it's easy to find a module that is really close to what you need, except if it wasn't for that one bug or missing feature.

Now depending on who is maintaining this module, you may get the problem resolved by simply opening a GitHub issue and waiting for a few days. However, open source doesn't really work without community, nor do you always want to be at the mercy of someone else. So a much better approach is to actually roll up your sleeves, and fix the problem yourself.

Here is the proper way to do this while using npm to manage your forked version of the module:

1. Fork the project on GitHub
2. Clone the fork to your machine
3. Fix the bug or add the feature you want
4. Push your commits up to your fork on GitHub
5. Open your fork on GitHub, and click on the latest commit you made
6. On the page of that commit, click on the "Downloads" button
7. Right click on the "Download .tar.gz" button inside the popup, and copy the link ("Copy Link Address" in Chrome)
8. Open up your package.json file, and replace the version number of the module with the url you just copied
9. Send a pull request upstream (Optional, but this way you will avoid having to maintain that patch of yours against newer versions of the module you forked)

Example: My new airbrake module uses a forked version of xmlbuilder. I submited my fix as a pull request, but it has not been merged yet. In order to pull in my changes via npm anyway, I simply pointed my package.json to the download url of my fork on GitHub like so:

Alright, let me know if this is helping your node.js adventures, or if you have an alternative workflow you are using. Otherwise, happy hacking!

--fg

PS: You should upgrade to the latest npm version first, some older versions had problems with handling url dependencies properly.

We apologize for the short notice, but if you are looking to put node.js in production, this full day node.js workshop we are organizing is where it's at!

The workshop is happening on Friday June 10, one day before nodecamp.eu. Space is limited to 15 people and expected to sell out quickly.

As a reader of our blog, you can get a 15% discount on the regular ticket by using the code 'debuggable'.

Should you attend?

This workshop will teach you everything you need in order to write and deploy powerful node.js applications. We'll try to cover a lot of ground, so if you are interested in any of the following, you should definitly attend:

• Setting up node.js on your local machine
• Understanding the module system
• Using npm for installing and upgrading modules
• Publishing your own npm modules
• Everything you need to know about http.Server
• Structuring your code using OOP in JavaScript
• Dealing with all the callbacks in a sane fashion
• Using the same code in node.js and the browser
• Building realtime apps with Socket.IO
• Using the express framework
• An overview over testing tools available for node.js
• Deploying node.js to Ec2 / Joyent

The first half of the day will be guided by slides (which will be made available afterwards), with the second half being a hands-on session where we will build a small node app from scratch.

About the instructor

This workshop will be led by Felix Geisendörfer (that's me). He is one of the earliest and most active contributors to node.js, author of over 20 npm modules and also running one of the biggest node.js applications in production over at transloadit.com.

In addition, Tim Koschützki who is also a co-founder at transloadit, will be available all day to help with individual questions and trouble shooting.

Questions & More Workshop

If you have additional questions or can't make it to this workshop, please head over to the workshop page which has information on other upcoming workshops and questions.

--fg