Code School Review: The Magical Marvels of MongoDB gets you started with shell syntax

Code School’s got a new database path! Woohoo!

I was invited by Code School (and comped with a free month subscription) to try out The Magical Marvels of MongoDB. I’m not that experienced with backend data stuff, so I was excited to fill in a bit of my knowledge gap with this intro-level course.

code_school_banner

About Code School

As some readers already know, I’m a big fan of CodeSchool.com. Code School is great for both curious newbies (“is coding something I want to do with my life?!”) and for more experienced programmers who want to get a feel for a new technology without going through the hassle of setting up an environment, a new project, and then wading through a pile of documentation.

They’ve grown a lot in just the last year, so check ’em out! Especially if it’s been a while since you last visited.

About the MongoDB course

What you need: a modern web browser – and that’s it. I used Google Chrome and experienced no technical difficulties. All the videos and interactive lessons take place in-browser.

Time spent from start to finish: about 4 hours. The course is divided into 5 sections, with about 10 minutes of video instruction per section.

Topics covered: inserting and finding database documents, transforming record data, filtering documents by various search criteria, embedding vs. referencing, and aggregation.

Overall instruction quality: Excellent. The class’s instructor, Joel Taylor, speaks clearly and concisely. On-screen text is clear and readable, and the graphic design is pleasing and modern.

mongodb_which_route

Wins

The course is largely a tour through the syntax used in MongoDB’s shell. The course does a great job of showing the student how MongoDB structures its data and how to perform simple operations like adding, removing, and updating parts of database objects as well as filtering results by more sophisticated criteria.

The error checking is pretty good – it caught most of my bonehead mistakes and the hints were relative to the mistakes made. Occasionally, I ran into a problem where the error checking couldn’t tell me what I had done wrong, such as omitting the “$” before “$lte” in one case, but I never stayed stuck for long.

codeschool_mongodb_editor

I liked how the course never gave me ready-made code to work from. I have a vague memory of an earlier Code School course giving me existing code to add to, and I feel like I got a lot more value out of the repetition of typing the same “boilerplate” (ie: db.wands.find() ) each time I started another section.

The pacing was good: I especially enjoyed the multiple-choice format to section 4, which I felt deepened my understanding of embedding vs. referencing and also was a nice change of scenery from the usual coding challenges. Lessons never dragged on too long, so I could squeeze them in between other things I had going on.

I love the in-lesson shortcuts to the relevant video sections, the reviewable slides, and the suggestion near the end of Lesson 2 to look in Mongo’s own documentation for help with $mul. Since I usually work with documentation up on my second monitor, I appreciate the validation. :P

Shortcomings

As with many CodeSchool courses, the larger context is probably the thing I miss the most in this course. I loved the intro at the beginning: the conceptual rundown (“NoSQL”, collections, the diagrams), but I didn’t get a great feel for the whys of MongoDB. Why would a project be better served by MongoDB over MySQL? What are the advantages of MongoDB and tradeoffs?

The course is a great way to try out the technology, but for me, finding the syntax of a language or technology is usually the simplest step (thanks to documentation). My personal learning curve tends to steepen as a result of the quirks, the gotchas, and the… mindset (for lack of a better word) of a technology.

And, of course, the usual caveat applies: practicing a language or technology in the browser is a lot like reading foreign language words out of a book vs. attempting a conversation with a native speaker. If you were to actually make a project using MongoDB, you’d find there’s quite a bit of setup, interfacing with the rest of your project, and debugging to get through first. CodeSchool doesn’t cover this aspect of the experience, but it’s a pretty big part of using any new technology.

But hey, these aren’t really problems with the course, just things that an intro-level online class can’t be expected to deliver on in the first place. The fact that I’m missing them is probably a testament to Code School’s ability to get me interested enough in the technology to want to know more.

The bottom line?

Great course! I finished it and wanted more. My favorite parts were the sections that went beyond syntax and into best practices, like the embedding vs. referencing parts. Don’t expect to be able to speak fluently about MongoDB after just this one class, it’s too small in scope for that.

But I’m wishing there was more, so – good work, Code School!

mongodb_codeschool_course_complete

Deploying a MEAN stack app to Heroku

The time had come at last to deploy Chicken Breast Meals to an external  server so that it could be enjoyed by a larger audience. I chose Heroku because it’s a friendly “my-first-deployment” kind of technology that handles a lot of the nitty-gritty details that Amazon Web Services and others leave up to you. For a simple MEAN stack app deployment, Heroku has been sufficient for my needs so far.

heroku

However, despite Heroku’s fairly straightforwardness, I still encountered a number of problems along the way. This post is about all the steps I took to get my MEAN app from GitHub to Heroku.

For clarity’s sake, these are the technologies I used in the project and in my development environment:

  • AngularJS
  • MongoDB
  • Express server
  • node.js (and a bunch of npm packages)
  • Heroku
  • MongoLab on Heroku
  • GitHub
  • Windows 7 with msysgit bash (my environment)

And unlike Heroku’s tutorial, this tutorial assumes you already have a git repo on your hard drive and it’s already full of your project files.

Step 1: Open a Heroku Account and Add a New App to your Dashboard

Hopefully, Heroku’s site can walk you through this sufficiently well.

Once you have an account, add a new app via the dashboard. On the current version of the Heroku dashboard, adding a new app is done with the + button.

app-plus
Heroku’s “add new app” button is easy to miss.

Step 2: Get the Heroku Toolbelt

Heroku’s own site will tell you to do this, too. Go to https://toolbelt.heroku.com/ and install the toolbelt appropriate to your environment. The toolbelt allows you to use the heroku command from your shell.

Step 3: Enter your credentials

Heroku’s toolbelt site walks you through these steps, too, but just in case you’re following along here:

$ heroku login
Enter your Heroku credentials.
Email: myaddress@gmail.com
Password (typing will be hidden)
Authentication successful.

You may get a response like this:

Your Heroku account does not have a public ssh key uploaded.
Could not find an existing public key at ~/.ssh/id_rsa.pub
Would you like to generate one? [Yn] Y
Generating new SSH public key.
Uploading SSH public key /home/jim/.ssh/id_rsa.pub... done

If this happens, choose Y and continue.

Since you already made a new Heroku app in step 1 you should skip the “heroku create” step.

Step 4: Add your Heroku app as a remote to your existing git clone’d repo

If you’re like me and you already have your git repo as a folder on your hard drive, you don’t need to make a new repo, you just need to add Heroku as a remote for it.

Navigate to your app’s root folder with cd and then use heroku git:remote -a yourappnamehere to add your remote.

If you follow these steps on Heroku’s own site, it will suggest using git init here (which you shouldn’t do since you already have a repo set up) and it will fill in your chosen app name where mine says chickenbreastmeals.

These are the steps I used to add my Heroku app as a remote to my existing GitHub repo:

$ cd /your/project/location
$ heroku git:remote -a chickenbreastmeals

Step 5: Attempt to push to Heroku – Permission Denied!

Pushing your repo to Heroku is done with just one line:

$ git push heroku master

…But if you’re like I was originally, you’ll get a permission denied (publickey) error.

(If you don’t get this error, hooray – you’re probably good to go. Or you’re stuck on a new problem that I didn’t encounter. Good luck.)

$ git push heroku master
Permission denied (publickey).
fatal: Could not read from remote repository.

Oh, snap. I Googled the “git push heroku master permission denied (publickey)” error and landed on this helpful Stack Overflow question. The first reply suggested a series of steps starting with heroku keys: add ~/.ssh/id_rsa.pub

heroku keys:add ~/.ssh/id_rsa.pub // or just heroku keys:add and it will prompt you to pick one of your keys

Alas, in my case, this didn’t work. Here’s what I got:

Uploading SSH public key c:/Users/Mandi/.ssh/id_rsa.pub... failed! Could not upload SSH public key: key file 'c:/Users/Mandi/.ssh/id_rsa.pub' does not exist

Well, that’s just super: I didn’t have an id_rsa.pub file yet. I needed to generate a new set of SSH keys, as detailed in my next step.

Step 6: Generate SSH keys

Fortunately, GitHub has an excellent guide on generating ssh keys, which will get you most of the way there. I encountered some problems along the way, which I’ve explained in this section.

The first step in GitHub’s instructions failed for me, of course, since I had no SSH keys.

All I got was:

ls -al ~/.ssh
total 7
drwxr-xr-x 1 Mandi Administ 0 Nov 10 16:04 .
drwxr-xr-x 48 Mandi Administ 12288 Nov 10 16:04 ..
-rw-r--r-- 1 Mandi Administ 405 Nov 10 16:04 known_hosts

If you also have no SSH keys (files with names like id_dsa.pub, id_ecdsa.pub, id_rsa.pub, etc) you’ll need to move right along to GitHub’s second step and generate a new SSH key:

ssh-keygen -t rsa -C "your_email@example.com"
# Creates a new ssh key, using the provided email as a label
# Generating public/private rsa key pair.
# Enter file in which to save the key (/c/Users/you/.ssh/id_rsa): [Press enter]

Just press enter when it prompts for a file location – you want the default. You’ll enter a passphrase twice (remember what you type here!):

Enter passphrase (empty for no passphrase): [Type a passphrase]
# Enter same passphrase again: [Type passphrase again]

And then you’ll get something like this, telling you where your identification and public key were saved as well as your key fingerprint and a random ascii art image for your viewing pleasure.

Your identification has been saved in /c/Users/you/.ssh/id_rsa.
# Your public key has been saved in /c/Users/you/.ssh/id_rsa.pub.
# The key fingerprint is:
# 01:0f:f4:3b:ca:85:d6:17:a1:7d:f0:68:9d:f0:a2:db your_email@example.com

Then you start the SSH agent:

$ ssh-agent -s
SSH_AUTH_SOCK=/tmp/ssh-ALBBCxgfEl11/agent.7104; export SSH_AUTH_SOCK;
SSH_AGENT_PID=6672; export SSH_AGENT_PID;
echo Agent pid 6672;

And then you add your key to the SSH agent… and, if you’re like me, get a new failure message to investigate:

$ ssh-add ~/.ssh/id_rsa
Could not open a connection to your authentication agent.

Looks like my authentication agent was never started. Huh? The previous step failed silently, apparently.

Fortunately, there is another great Stack Overflow question about this “could not open a connection to your authentication agent” issue. However, the first couple of answers didn’t actually work for me!

This is the one that did:

eval $(ssh-agent)

Caveat: I’m on Windows 7 64-bit using msysgit bash, so your experience may differ from mine. Responses to this answer suggest the problem is not unique to the Windows environment.

Anyway, now that the authentication agent is running I can properly complete the ssh-add step:

$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /c/Users/Mandi/.ssh/id_rsa:
Identity added: /c/Users/Mandi/.ssh/id_rsa (/c/Users/Mandi/.ssh/id_rsa)

Phew! Onwards to the GitHub step.

Step 7: Add new key to GitHub account

Following GitHub guide to generating SSH keys still, the next step is to copy the contents of your id_rsa.pub file to your clipboard. This is easily done with clip, like so:

clip < ~/.ssh/id_rsa.pub
  1. Go to GitHub and click the “Settings” gear icon in the upper right.
  2. Click “Add SSH Key”
  3. Give your key a title (I named mine after my computer)
  4. Paste the contents of clipboard into the large field
  5. Click “Add Key” to save it

(GitHub has a visual guide to these steps here)

Step 8: SSH into GitHub

Okay, almost done. The next step (taken from GitHub’s own guide) is:

ssh -T git@github.com

You’ll get these warnings, but that’s okay:

The authenticity of host 'github.com (207.97.227.239)' can't be established.
# RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
# Are you sure you want to continue connecting (yes/no)?

Type “yes” and if everything goes okay, you’ll get:

Hi username! You've successfully authenticated, but GitHub does not provide shell access.

This step worked successfully for me, but if you get an error here GitHub has a guide for that too: Error: Permission Denied (publickey)

Step 9: Push to Heroku

Oh, yeah – I just remembered what I was trying to do before I went down the SSH error rabbithole: I was trying to push my GitHub repo to Heroku!

First, add that same .ssh file to with heroku:keys add

$ heroku keys:add ~/.ssh/id_rsa.pub
Uploading SSH public key c:/Users/Mandi/.ssh/id_rsa.pub... done

Phew, success! Now I was able to run heroku push.

$ git push heroku master
Warning: Permanently added the RSA host key for IP address '50.19.85.132' to the
 list of known hosts.
Initializing repository, done.
Counting objects: 801, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (705/705), done.
Writing objects: 100% (801/801), 994.30 KiB | 519.00 KiB/s, done.
Total 801 (delta 419), reused 0 (delta 0)

This message was followed by several screens depicting the installation of node and my project’s node packages. Heroku handles this setup automatically, and in my case, the installation processes went off without a hitch.

Step 10: Check out your app on Heroku – Application Error, hooray!

I’m glad I didn’t celebrate too early, because my Heroku app looks like this:

heroku-application-error

Application Error

An error occurred in the application and your page could not be served. Please try again in a few moments.

If you are the application owner, check your logs for details.

And no, it doesn’t go away in a few moments.

Step 11: Using MongoDB? Install MongoLab on your Heroku app

If you’ve ever tried to run your app locally while forgetting to fire up your MongoDB first, then you’ve probably seen your build process fail due to your database not being up and running.

There’s really no way to know that a not-running database is the cause of the application error screen, but I’ll spoil the surprise for you and tell you that in this case, that’s exactly what it was. If your Heroku-hosted MEAN app is using a MongoDB then you need to install an add-on called MongoLab.

Go to your app’s dashboard and click Get more addons

get-more-addons
If your Heroku-hosted MEAN stack app requires MongoDB, add MongoLab as a free add-on.

The addons page looks different every time I come in here, but the MongoLab icon hasn’t changed:

mongolab

Click the icon to learn more about MongoLab, including its pricing structure and features. You will have to enter a credit card number to enable MongoLabs, but sandbox (which is what you’re using here) will be free. (I think this is super annoying, BTW. If it’s free, it shouldn’t require a credit card to use. I’ve never actually been charged by Heroku or MongoLab.)

To install, head back over to your Command Line/Terminal window and enter:

$ heroku addons:add mongolab

You’ll get this sort of response:

Adding mongolab on chickenbreastmeals... done, v4 (free)
Welcome to MongoLab. Your new subscription is being created and will be available shortly. Please consult the MongoLab Add-on Admin UI to check on its progress.
Use `heroku addons:docs mongolab` to view documentation.

IMPORTANT SIDE NOTE: My server.js file is already configured to expect MONGOLAB_URI. I’ve provided my server.js code here in case you need to do the same to your server file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
'use strict';
 
var express = require('express');
var bodyparser = require('body-parser');
var mongoose = require('mongoose');
var http = require('http');
var app = express();
 
mongoose.connect(process.env.MONGOLAB_URI || 'mongodb://localhost/meals-development');
app.use(express.static(__dirname + '/build'));
 
app.use(bodyparser.json({limit:'50mb'}));
app.use(bodyparser.urlencoded({limit: '50mb', extended: true}));
 
require('./routes/admin-routes')(app);
 
var server = http.createServer(app);
 
var port = process.env.PORT || 3000;
app.listen(port, function() {
console.log("Listening on " + port);
});

From here, I attempted to view my app again. This time I got:

cannot-get

Le sigh. But this is progress – I don’t get an Application Error anymore, so the database installation made a difference. Checking the Chrome console, my Heroku app is generating this error:

Failed to load resource: the server responded with a status of 404 (Not Found)

Step 12: Giving Heroku access to my Build folder

I scratched my head a bit over this “cannot GET/” problem and Googled it, which led me to this Stack Overflow question, Heroku Cannot Get.

Just like the original asker, my .gitignore contained a line for my build folder, which meant Heroku had nothing to serve as it had no access to my “compiled” project.

I removed the build line from .gitignore, and pushed the updated .gitignore file and build/ folder to both GitHub and Heroku like so:

$ git push origin master
$ git push heroku master

Step 13: IT’S ALIVE!

At last, I see my app when I visit chickenbreastmeals.com. It’s lacking the database entries from my local development environment, so I’ll update this post once I get those in.

Hope this guide helped you deploy your MongoDB / AngularJS / Express / node.js app to Heroku! There’s only about a thousand things that can go wrong between point A and point Z, so if something in this guide doesn’t work from you it’s probably a difference in our environments or an error on my part – please leave a comment letting me know (and start Googling – good luck!).

Addendum

Did you use Gulp to build your app and automate some build processes? If so, your app probably doesn’t look so hot on Heroku right now. This is because Heroku doesn’t know which of your Gulp tasks needs to run after all your Node packages are installed. Let’s fix that!

Dev Dependencies

First off, it’s important to mention that if you installed any packages as a dev dependency (like you probably did with Gulp), Heroku will not include them in your build by default. This is because Heroku assumes you’re deploying a production build, and will run npm install –production, which ignores dev dependencies. There’s two ways to fix this:

1. In your app’s package.json, move Gulp and all related packages from the “devDependencies” list into the “dependencies” list. This is a pain and I do not recommend it.

2. Run the following terminal command to tell Heroku that it should use the standard npm install command:

heroku config:set NPM_CONFIG_PRODUCTION=false

Postinstall Scripts

With that taken care of, we need to tell Heroku what commands we want to run after all of our packages are downloaded and installed. Luckily Heroku has made this easy! Just add a “scripts” block to your package.json file, like so:

"scripts": {
 "start": "node server.js"
 "postinstall": "bower install && gulp build-libs && gulp build"
}

The “start” script tells Heroku how to start my server: run node with the file server.js. The “postinstall” script is actually three commands separated by &&, ran in sequence: bower installgulp build-libs, and gulp build. In my gulpfile.js, the build-libs task concatenates and minifies several libraries like Angular and Bootstrap. This task relies on those libraries being in the bower_components folder, which is why I run bower install first.

Troubleshooting

If any of the steps in this article don’t work, there’s a couple things you can try. The most helpful thing to know is that you can run common Linux shell commands on your Heroku container with heroku run. Like this:

heroku run ls -la

This is just like running ls -la on your own system, and will list all of the files in your Heroku deployment’s main directory. This is how I figured out that I need to run bower install: there was no bower_components folder in my deployment!

 

MongoDB cheat sheet

mongodbThis simple MongoDB tutorial is for you if:

  • you’re completely new to MongoDB and just want to do SOMETHING with it
  • you have MongoDB running but forgot the particulars of using the MongoDB shell
  • you want to look inside your db and confirm data’s actually getting written to it
  • you rebooted and lost your Mongo server and you can’t remember how you got it running in the first place
  • you don’t want to wade through the documentation again

Or, you’re just me in the future looking for where I wrote this down. You’re welcome, future self.

1. Install Mongo!

Install steps are better covered by Mongo itself: official MongoDB

2. Start Mongod

These steps differ by OS.

Mac / Linux

On my Mac machine, I can start mongod from anywhere because it’s in my $path (see this guide for steps on adding MongoDB to your $PATH).

Just use:

mongod

Successful connection looks something like:
Screen Shot 2015-04-26 at 11.59.03 AM

Windows

On my Windows machine, I have to navigate to Mongo’s installation folder to start mongod. Open Terminal (Mac) or Command Prompt (Windows). Navigate all the way into the bin folder. On my Windows machine, my mongo folder is here:

J:\mongo\mongodb\bin

Now use:

mongod

On Windows, I see a connection spam scroll by. Leave this window open and go to the next step.

mongod_started

Problems starting Mongodb?

If you get the “Unable to lock file: data/db/mongod.lock. Is a mongod instance already running?” problem, you probably have multiple instances of mongodb already running. This can happen as you switch projects, switch between user accounts on the same machine, etc.

To fix it, do this to list your computer’s processes and filter them to just mongo (this example is from when I had the problem on my Mac):

ps aux | grep mongo

On my machine, running that command revealed a couple instances of mongo already running (these were started by Jim using a separate account on the same computer). The third process in the list (the one owned by mjgrant) is the grep itself.

Screen Shot 2015-04-26 at 11.24.10 AM

Because my mongo instance was started by “root” (another Mac account, really), I had to be all dirty and use sudo to kill it by its process number (second column from the left).

sudo kill 61180

If you run the ps aux command again, you should see that there are now no instances of mongo running. If there are, just kill them using the same steps.

But what’s this? Trying to start mongo gives me this error now:

2015-04-26T11:30:11.114-0700 [initandlisten] couldn't open /data/db/memry_database.ns errno:13 Permission denied
2015-04-26T11:30:11.114-0700 [initandlisten] error couldn't open file /data/db/memry_database.ns terminating
2015-04-26T11:30:11.114-0700 [initandlisten] dbexit:

Rather annoyingly in our shared-computer situation, mongo’s knowledge of databases transcends user accounts. Navigating up to /data/db I can see all the databases on this computer. cbm_database is the one I’m trying to use, but mongo is choking on trying to access Jim’s memry_database.

Screen Shot 2015-04-26 at 11.35.10 AM

I check their permissions…

ls -la

Screen Shot 2015-04-26 at 11.37.15 AM

When asked why his databases belong to “root”, Jim says, “I probably did it wrong” :D Alas, we don’t know how we ended up with databases belonging to “root”, but Jim must have been using mongo as a root user, hence why he didn’t run into problems accessing databases owned by mjgrant.

Anyway… I used chown to assign ownership of these rogue root databases to my own account to unblock my work. (Standard disclaimer applies: use sudo with caution.)

sudo chown mjgrant memry_database.ns
sudo chown mjgrant memry_database.0

I run ls -la again and confirm that now I own all of the databases.
sudochown_sailormoon

Now you should be able to start MongoDB with…

mongod

And now you should see the connection data:

Screen Shot 2015-04-26 at 11.59.03 AM

3. Start the Mongo Shell

Open a new window (and navigate again to the bin folder if you’re on Windows).

mongo

This line starts up the Mongo shell.

(So to recap, mongod has to happen before mongo.)

On Mac:

Screen Shot 2015-04-26 at 12.09.22 PM

On Windows: 

mongo_shell

MongoDB shell version: x.x.x
connecting to: test

You can now start your localhost server.  (If you were blocked by Error: failed to connect to [localhost:27017] that should now be resolved.)

From here on out, commands you type into the command line will be mongo-specific.

3. Viewing your MongoDBs

Let’s say you want to see your databases:

show dbs

show dbs delivers a list of your databases. You should see something like this in your terminal window after you type it:

mongodb_see_dbs

 

On mine, the result is:

> show dbs
admin <empty>
local 0.078GB
meals-development 0.078GB

4. Using your Mongo DBs

These are your database names. Go inside them with “use”:

use meals-development

Once you’re “using” a database, though, the terminal doesn’t give much clue as to what to do next.

use_db

5. Viewing Collections

A collection is a group of MongoDB documents. Generally, they’re similar in purpose, but they don’t have to conform to one shared schema. You can see collections inside a db by typing:

show collections

As an example, inside my recipes-development example I have:

show_collections_mongo_db

meals
system.indexes

Ah hah, finally. Now I know the name of the collection that contains my recipe (meal) data.

6. Look inside a collection

We’re almost to the good part. To see inside the meals collection, type:

db.meals.find()

You should get a number of objects with ids, names, etc. Each object will start with something like: { “_id” : ObjectID<“544dabfba054…

That’s it!

This was just a short guide to my most commonly used MongoDB shell commands. When I’m setting up a new db, I use these steps to look inside my db and see if data is being saved the way I expect it to.

Helpful Links