How to get all npm packages that match a particular keyword in JSON format?

Question

Not having any experience with couch and redis, this is becoming more than I can handle at this point.

The npm website allows you to search for packages by a keyword - https://npmjs.org/browse/keyword/awesome

However, it doen't provide any way of obtaining this information in json format - ideally, we could just do https://npmjs.org/browse/keyword/awesome.json but that is not the case :(

I know that the npm website is powered by couchdb and a local redis instance. The remote couchdb installation is http://registry.npmjs.org/ and powered by https://github.com/isaacs/npmjs.org

However, spending the day looking into this, I just cannot figure out how to get all packages of a particular keyword. Does anyone know how? Bonus points if you also explain the process that you went about finding out how to do it so I know for next time :)

Answer 1

Great question!

This will give you what you're looking for about a specific module:

npm view request

To get what you want for all modules you can hit the URL: https://registry.npmjs.org/-/all/

After pouring through these two files:

1. https://github.com/isaacs/npm/blob/master/lib/search.js
2. https://github.com/isaacs/npm-www/blob/master/models/browse.js

I came to the following conclusions:

1. I'm super surprised there's not a better way to do search without hitting couchdb directly.
2. The command-line NPM client does search inside of node.js by sorting and filtering through the full results of that /all/ search listed above.
3. The website doesn't even bother with real search as it pawns it to a search engine
4. The search by keyword thing you want won't get the same results as command-line NPM. It's really limited in scope to the keyword attribute, other search options may be available through (see search.js above)
5. The query is going to look really weird.

Try this: https://registry.npmjs.org/-/_view/byKeyword?startkey=["keyword"]&endkey=["keyword",{}]&group_level=3

Also, one quick note, this is the kind of question that would probably get answered in the node.js chat room or mailing list in about 4 seconds :)

Hope that helps.