Source Maps – Debug your Bundled, Minified Code

While developing a web app, it’s a good idea to bundle all your various JavaScript files and libraries into a single, minified file for production. Bundling makes your app faster by reducing download size and saving the browser from having to perform multitudes of HTTP requests just to render the page.

However, bundling comes at a cost. Debugging bundled JavaScript is often a nightmare. If your app has an error, the error in the console points at the point in the bundle file where the code failed, which may or may not bear any resemblance to your actual project files.

Which would you rather debug?

Which would you rather debug?

Wouldn’t it be great if the debugger was smart enough to point you at your original, pre-bundled code instead? Enter source maps, a way to preserve reference to your original file and code structure, even through bundling.

Before We Begin

This article uses the following tools, so make sure you have them first (or at least the basic concepts about them):

(PS: Grunt and Browserify can do source maps, too.)

How Source Maps Work

A source map is a composed of data that links pieces of your minified bundle back to the original, unbuilt files. Modern dev tools will automatically follow the map for any console.log or line queries you perform.

Showing The Problem

Let’s say I have a simple console.log in an angularjs controller someplace:

angular.module('myApp')
  .controller('CtrlMain', function ($scope) {
    console.log('hello');
  });

When we run the app and look in Chrome’s JavaScript console, it provides a very unhelpful link to where the log was executed:
Screenshot from 2014-11-27 17:07:20Oh, it’s in the one and only line in my bundled script? I never would have guessed.

To generate bundle.js, I’m using the following task in Gulp:

gulp.task('build-js', function () {
  gulp.src([jsDir + 'module.js',jsDir + '*.js'])
    .pipe(concat('bundle.js'))
    .pipe(ngAnnotate())
    .pipe(uglify())
    .pipe(gulp.dest('build'));
});

This concatenates all JavaScript files into a single file called bundle.js, creates dependency injection annotations so Angular doesn’t choke on the minified code, minifies the code using gulp-uglify, and finally writes it to a build directory.

Adding Source Maps To The Build Process

Here’s how to add source map creation to this build process, using gulp-sourcemaps:

gulp.task('build-js', function () {
  gulp.src([jsDir + 'module.js',jsDir + '*.js'])
    .pipe(sourcemaps.init())
    .pipe(concat('bundle.js'))
    .pipe(ngAnnotate())
    .pipe(uglify())
    .pipe(sourcemaps.write())
    .pipe(gulp.dest('build'));
});

Before we do anything to the JavaScript files, we call sourcemaps.init(). This tells gulp-sourcemaps to pay attention and generate our source maps while the build process occurs. sourcemaps.write() signals that we’re done processing the .js files and ready to write out the bundled file.

IMPORTANT NOTE: All plugins used between sourcemaps.init() and sourcemaps.write() must be supported specifically by gulp-sourcemaps. You can find a list of supported plugins here.

The Final Product

Now let’s build and run the app again and see what we get:

Screenshot from 2014-11-27 18:19:39

Much better

Now the console.log behaves as if we’re working with the clear, unbuilt versions of our files. We can click that link on the left to see exactly which line contains the log. Ahhhh.

 

pytest CLI tips, tricks, and additional settings

There are several Python unit testing frameworks, but pytest is both the most popular and the easiest to use. It requires no boilerplate, no imports, no API. Just name your files, classes, or methods starting with “test”, type py.test into the command line, and observe the results.

Simple, right? And we want it that way. Pytest allows our tests to be uncluttered and easy to read. But as far as the output of those tests goes… we can do better! This article will show you how to master the pytest CLI.

To start off, make sure you have pytest installed:

pip install pytest

For those of you new to pytest, note that the package is named “pytest”, while the command to run it is “py.test”. The dot remains for historical reasons: pytest used to be part of the “py” library.

If you already have pytest installed, double-check your version number:

py.test --version

This guide is written with pytest version 2.7.0 or greater in mind. If you have an older version, you can upgrade using pip:

pip install --upgrade pytest

All good? Let’s get started!

Useful Switches

-x : Stop After First Failure

When you’re trying to track down a bug, it can help to run all of your tests, so that you can draw conclusions based on what groups of tests are failing.

That said, sometimes you just want to work through failing tests one by one, or there are so many tests in your project that running them all every single time (not to mention sifting through all the output) would be too time-consuming. There’s a nice simple switch to take care of this:

py.test -x

This will stop pytest after the first failure is encountered, saving you oodles of time. Is one just not enough? How about after two failures? No problem!

py.test --maxfail=2

-k : Run Tests Matching Keywords

What if you know some group of tests is going to fail, but you just don’t want to think about them right now? You could go through your test files and comment them out… but there’s a much better way! Let’s say you only want to run tests with the word “log” in the names:

py.test -k “log”

Easy… but wait! This also ran the tests with “logout” in the names. Let’s say we don’t want those:

py.test -k “log and not logout”

That’s right! Instead of just a simple string match, you can feed -k a full expression to use.

Controlling Verbosity

Of course, if you want to run all your tests, but just make them run more quietly, you can tone down their output like this:

py.test -q

The opposite is increasing the output, which will put all the names of your tests in a list, along with their pass and fail states:

py.test -v

This can be useful if you’re logging your tests for someone else to read. This will only control the state output, however. For the rest, see the next section!

Changing Defaults

Changing Traceback Printing

When a test fails, pytest will attempt to do a traceback to give you some information on what went wrong. And when I say “some”, I mean a lot… often too much. What if you could shorten that output, so that you didn’t have to scroll for pages to see the results of all your tests? Well, try this:

py.test --tb=short

Using –tb will change pytest’s traceback formatting to the format you specify. If “short” is still not short enough, you can do it one better:

py.test --tb=line

Or if you’re looking for something a little more familiar, you can set pytest to use the Python standard traceback formatting, like so:

py.test --tb=native

Invoke pdb On Failure

Oftentimes, when debugging a problem, you’ll stick the following statement on the line before the test failed to set a pdb breakpoint:

# code here
import pdb; pdb.set_trace()
# more code here

But why go through that hassle, when you can just do this when you run pytest?

py.test -x --pdb

This will invoke pdb immediately upon failing a test. Notice that I combined it with -x to stop pytest after the first failure. This combination is quite useful, as you don’t want to have to go through pdb for every single failing test.

Watching Files

All of these tricks will help you save time and energy while running tests. But all of them have one thing in common: you still have to head back into the command line and type them out. Well, we can take care of that, too. Enter pytest-xdist. This handy little package (in addition to other things) can keep an eye on your project files and re-run your tests every time it detects something has changed. Just install and run with the -f option:

pip install pytest-xdist
py.test -f

This will immediately run all of your tests, then sit and wait for any file to be changed. Even better, if any tests failed in the previous run, it will re-run ONLY those tests until they pass.

Alternatively, you can use a package called pytest-watch:

pip install pytest-watch
ptw

The downside to pytest-watch is that it’s a separate executable, and therefore you can’t use any of the tips in the rest of this article. On the other hand, it has some neat options, like configuring actions to occur on test failure. If you’re on OSX, try this out to have your computer actually vocalize how your tests are doing:

ptw --onpass=”say passed” --onfail=”say failed”

Final Thoughts

These are some of my favorite pytest options, but it’s really only the beginning. pytest has many options for reporting, collection, and debugging, and that’s not even getting into the rich assortment of plugins and packages to extend its capabilities. You’re going to be spending a lot of time running tests, so take the time to poke around and find the configuration that works best for you. It’ll be worth the investment!

How to quit Vi (and more tips for getting started with Vi)

If you opened vi or Vim because you heard it’s what all the cool kids use, chances are your first question is “Aaaaah! How the heck do I get out of this bizarre thing?!”

Don’t panic! Press the Escape key, to make sure you’re in Command Mode. Type “:q“, without the quotation marks, and press Enter.

If you get an error message saying “No write since last change”, type “:q!” and press Enter. You should now be back in your normal command line interface.

Phew! That’s a relief, huh? Now that you’re safely out of vi, here are some reasons why you should go back in and learn how to use it.

Reasons to use Vi (or Vim)

Vi (and usually Vim) is on practically every Unix-like system in existence

Vimlogo.svgVi is specified in the POSIX standard, which means that practically any flavor of Unix or Linux will include it. Sitting down at a coworker’s computer? Vi is waiting for you. Using SSH to go into a completely foreign system with literally nothing but the OS installed? Oh, hi vi, nice to see you again!

A brief note on the difference between vi and Vim: vi was written back in 1976. Vim is a “sequel” of sorts to vi that added features like syntax highlighting, scripting, advanced regular expressions, a help system, and more. I’m using Vim, but my understanding is the basic command structure is shared by both programs.

It’s fast to use

Or it will be, once you’ve used it enough. You rarely have to take your hands off the keyboard to complete almost any text editing task you can think of. In addition, vi’s interface is optimized such that the most commonplace tasks are on the home row, and extremely quick to perform. An experienced vi user can be extraordinarily productive.

It’s what all the cool kids use

Don’t underestimate the benefits of peer pressure! Vi and Vim have a huge community of users who swear by them, some of whom have been using it for decades. Through countless iterations of hardware and software, vi has stood the test of time as the preferred text editor of programmers and sysadmins everywhere. That’s got to count for something, right?

I could go into how powerful and customizable the vi editor is, but I’m just starting my own exploration of it, and frankly, the most important thing is that you can perform basic tasks, so the editor stops feeling like an alien torture device. Deeper understanding can wait until you’re comfortable. So let’s move onto:

Vi / Vim Basics

Opening vi

To open vi, open a terminal window, type “vi“, and press Enter. To open a file with vi, type the filename afterwards, as in “vi filename.txt“. If that file does not exist in the local directory, vi will begin creating a new file with that name.

Type “i” to enter Insert Mode

Vi is a modal editor, which means that the commands you can use depend on which mode you are in. When you open vi, it defaults to the Command Mode. For people who are used to standard text editors, Insert Mode will be far more familiar. Press the “i” key to enter Insert Mode, then begin typing normally. You’ll see your text entered onto the screen.

Undo and Redo (Vim only)

When you’re starting out, it’s natural to make mistakes. To undo, first make sure you’re in Command Mode (by pressing the Escape key). Then tap the “u” key. You should see your last change undone. You can keep pressing “u” to back up through vi’s change history. To redo, press “Ctrl + r” (“r” by itself is the Replace command).

Navigation

h“, “j“, “k“, and “l” in Command Mode correspond to left, down, up, and right respectively. This will seem really weird at first, but using the keys right in the home row makes it very fast to navigate.

Note that you can use the regular arrow keys as well, but you should try your best to get used to the home row keys. Train that muscle memory: it’s worth it.

For faster navigation, use “Ctrl + u” and “Ctrl + d” to half-page up and down, respectively. There are many other ways to navigate a file as well, but these will serve you well enough for now.

Saving and Quitting

Once you’ve made some changes, type “:“. You’ll now notice your cursor is at the bottom of the terminal window. Press the “w” key so that the bottom of the window says “:w“. Press Enter, and you’ll get a message that your file was saved.

As mentioned above, you can now type “:q” to quit vi. If you tried to quit without saving, you’ll get an error. You can type “:wq” to save and quit at the same time, or “:q!” to quit without saving.

Searching

Type “/” in Command Mode to open a search prompt at the bottom of the window. Then type whatever word you want to search for and press Enter. To go to the next search result, press the “n” key. To go to the previous result, press “Shift + N“.

Basic Vi Cheat Sheet

Quit without saving

:q!

Save and quit

:wq

Enter insert mode

i

Move left / down / up / right

h  j  k  l

Beginning of word / end of word

b  e

Move to the end of the line

$

Page up / page down

ctrl + u   ctrl + d

Leave insert mode

:

Search (must be in Command Mode)

/

Next Steps

You now know enough about vi to use it for simple tasks. Once you’re comfortable with the above, try typing “vimtutor” at the command line for a nice tour of some of Vim’s intermediate features.

Vi and Vim have a vast array of commands and customizable elements, far more than any blog post can hope to cover. Once I’m a little more familiar with Vim myself, I’ll post a follow-up article with some good tips.

In the meantime, here are some helpful resources on Vi:

Gulp – Build Automation for Javascript

Tired of restarting your server manually, over and over again, whenever you change something? Straining under the labor of recompiling your Less / Sass dozens of times an hour? No more. Let a robot do it for you. A robot named Gulp.

Why Gulp?

First off, I should mention that there are several build automation tools written specifically for Node.js apps. Grunt is probably the most popular, but there’s also Jake and a couple others. I started with Grunt, but I’m liking Gulp more and more as I use it. It’s faster, more flexible, and just “feels” more intuitive and straightforward.

basedHonestly, though, which one you use isn’t important. All that’s important is that you use SOME KIND of build automator / task runner. Over the long run, it will save you hours of repetitive, frustrating, mindless drudgery. Sound good? Read on.

Installing Gulp

To use Gulp in your app, you must install it globally onto your system, as well as locally in your project directory. Use the following commands in terminal, while inside your project dir:

npm install --global gulp
npm install --save-dev gulp
touch gulpfile.js

Your First Gulp Task

Open the gulpfile.js you just made in a text editor and put in the following:
var gulp = require('gulp');
gulp.task('default', function() {
  console.log('If you can read this, gulp is working!');
});

Go back into your terminal, type gulp, and press Enter. You should see the following output:

[20:33:59] Using gulpfile ~/YourProjectFolderHere/gulpfile.js
[20:33:59] Starting 'default'...
If you can read this, gulp is working!
[20:33:59] Finished 'default' after 74 μs

So what happened here? We created a task called default in the gulpfile, which calls a function when the task is run. That function then performs a console log. Using the gulp command with nothing after it will run any task named default. It’s a good idea to always have a default task that does the important work of building and running your project. That way, anybody else who has to work with your project can just type gulp in the terminal and see it run without having to paw through your code.

Your First Useful Gulp Task

So that was fun, but not particularly worthwhile. Let’s do something useful with Gulp!

Let’s say you have a directory full of JavaScript files, a directory called ‘js’. All these files need to be copied over to a directory called ‘build’ before you can publish your app. No problem! Put this into your gulpfile:

var gulp = require('gulp');

gulp.task('copy-js', function () {
  gulp.src(['js/**/*.js'])
    .pipe(gulp.dest('build'));
});

gulp.task('default', ['copy-js']);

There’s a lot going on here, so I’ll explain bit-by-bit:

  • We created a new task called copy-js, which will do all our copying for us.
  • The first line inside that task, beginning with gulp.src, tells gulp where to look for the files we want to copy. That bunch of /s and *s we gave it is a pattern-matching string called a glob. Here’s how to interpret this glob:
    • The js/ part tells gulp to look inside the directory named ‘js’.
    • The **/ part tells gulp to look inside any subdirectories within the ‘js’ directory.
    • The *.js part tells gulp to find all files that end with the .js file extension.
  • On the next line, we chain a method onto the end of gulp.src… specifically, the .pipe() method. .pipe() takes the output of the previous method (i.e., the .js files we found) and lets us use it as input for another method, just like a unix pipe. This is extremely useful, as you’ll soon see.
  • .pipe() passes the files we found to gulp.dest(‘build’)gulp.dest() is used to save files to a particular location. Which location? Why, the one we told it: the ‘build’ directory.
  • Finally (and importantly!) we changed the default task. Instead of executing a function, default will now execute a list of sub-tasks. For now, we just want it to execute our copy-js task.

Now, if you type gulp into the terminal, any JavaScript files in the ‘js’ directory will be copied into the ‘build’ directory. Gulp will even create a ‘build’ directory for you if it can’t find one. How thoughtful!

Watch This

“This is all well and good,” you might be thinking, “but how does this actually save me time?” After all, you still have to keep typing gulp into the terminal every time you want this copy and paste to happen, right?

The answer is no, you don’t. Gulp can run tasks for you, automatically. Enter gulp.watch():

var gulp = require('gulp');

var jsDir = 'js/**/*.js';

gulp.task('copy-js', function () {
  gulp.src([jsDir])
    .pipe(gulp.dest('build'));
});

gulp.task('watch-js', function () {
  gulp.watch(jsDir, ['copy-js'])
    .on('change', function (event) {
      console.log('File ' + event.path + ' was ' + event.type);
    });
});

gulp.task('default', ['watch-js']);

Ok, so what happened here?

  • We made a new task called watch-js. When this task is executed, gulp.watch() will keep a close eye on the directory we tell it, watching for files inside to change. When they do, the tasks in the array we provide will be executed… in this case, the copy-js task.
    • To put it simply, whenever we change a .js file, it’ll be copied over automatically. How cool is that?
  • We chained .on() to the end of gulp.watch(). This lets us execute code when certain conditions are met. In this case, when a file changes, we execute a function. This function uses the event parameter to let us console log which file changed, and how it was changed (added, changed, deleted, etc.)
  • Also, we put the JavaScript directory glob into a separate var called jsDir, which we use in both the copy-js and watch-js task. That way, we can make sure it stays consistent.
  •  Finally, we change the default task to execute watch-js when it’s called. By the way, you’ll notice this is an array; we can comma-separate multiple sub-task names to be called here, if we choose.

Sweet! What Else?

Gulp can help you automate all kinds of development-related tasks, including but not limited to:

  • Linting
  • Unit / Integration Testing
  • Bundling / Concatenation
  • Minifying / Compression
  • CSS pre-processor compilation (i.e. Sass / Less)
  • Image resizing / processing
  • Asset versioning
  • Running shell commands

To learn more, check out Gulp’s documentation and browse their extensive, searchable list of plugins. To use a plugin, npm install it, require it at the top of your gulpfile as a variable, and then use it based on the plugin’s documentation. Like the following example does with gulp-sass:

var gulp = require('gulp');
var sass = require('gulp-sass');

gulp.task('default', function() {
  gulp.src('sass/*.scss')
    .pipe(sass())
    .pipe(gulp.dest('css'));
});

That should be enough to get you started. Happy gulping!

How to fork your own repo on Github

Forking one of your own Github repositories ought to be easy, right? After all, forking somebody else’s repo is as simple as clicking a single button! Surely you can just press that same button on your own repo?

fork

NOPE!

If you press the Fork button on your own repo, the page will refresh and… that’s it. No error message, no suggested course of action, nothing. It turns out forking your own repo on Github is impossible, but don’t worry: following the steps below will get you the next best thing.

1. Create a New Repo On Github

new_repo

First, go to github.com and create a new repository. This will contain our fork when we’re done. I’ll refer to this repo as “fork-repo”, with the original being “orig-repo”.

Make sure you don’t check the box for “Initialize this repository with a README”. You’ll see why in Step 4!

2. Clone the New Repo Locally

Next, make a local copy of the blank repo we just made. In Terminal, cd to the base directory you want to keep the fork in and then type the following:

git clone https://github.com/YourUsernameHere/fork-repo.git

3. Add an Upstream Remote

We’ll now add an upstream remote pointing at the original repo. This will allow us to pull files from the original repo, both now and in the future if we wish. Make sure you navigate to the directory you cloned the fork repo into first!

cd fork-repo
git remote add upstream https://github.com/YourUsernameHere/orig-repo.git

4. Pull From the Original Repo

Now we can pull all the files from our original repo into the fork, like so:

git pull upstream master

Your fork directory should now be identical to your original repo!

Note that if you made a README.md for the new repo (or added any other file) you may have some merge conflicts to resolve before you can go to the next step. Make the necessary changes to your files and commit to resolve the conflict.

5. Push!

You’re done! Well, locally at least. All that’s left is to push your new fork repo back up to Github:

git push origin master

Install NPM packages globally without sudo

To be an effective full-stack Javascript developer, you pretty much have to use npm (Node Packaged Modules)Npm-logo.svg

npm is the official package manager for Node.js. Using it allows you to easily install packages such as underscore, express, grunt, gulp, socket.io from the command line. (read more about npm on Wikipedia and npmjs.org).

Many of these packages need to be installed globally to be used, like so:

$ npm install -g grunt-cli

The -g denotes a global install.

But, in some environments, attempting to install globally without including sudo might give you this sort of error message:

npm ERR! Error: EACCES, open '/Users/YourNameHere/.npm/-/all/.cache.json'
 npm ERR! { [Error: EACCES, open '/Users/YourNameHere/.npm/-/all/.cache.json']
 npm ERR! errno: 3,
 npm ERR! code: 'EACCES',
 npm ERR! path: '/Users/YourNameHere/.npm/-/all/.cache.json' }
 npm ERR!
 npm ERR! Please try running this command again as root/Administrator.

Should you just do what the error message says and run this command as root using sudo? The short answer is: NO! Packages can run their own scripts, which makes installing them with sudo about as safe as shaving with a blowtorch.

There are many ways to solve this problem, but the easiest I’ve found is to simply change what folder npm installs packages into. Just use this terminal command:

npm config set prefix ~/npm

This tells npm to install packages into your home directory in an “npm” subfolder. It will automatically create this new subfolder the first time you globally install a package.

But we’re not done yet! You also need to modify your $PATH to include this new folder. To do this, open the config file of your shell in your favorite text editor. This is most likely “.bashrc” in your home directory, or “.zshrc” if you use Z shell. Once you have the file open, search for a line that looks like this:

export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"

At the end of this line, add:

 ":$HOME/npm/bin".

After I made the change, my path line looked like this:

export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:$HOME/npm/bin"

The $HOME part of that just tells the shell to look in your home directory for a folder called “npm”, and then a folder inside that called “bin”. Once you’ve done these two things, you should now be able to install packages globally in npm without using sudo. Enjoy the tremendous feeling of freedom this brings.