Author: John Au-Yeung

Web developer specializing in React, Vue, and front end development.

Node.js Best Practices — Testing and Quality

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Testing and Quality

Node.js is a popular runtime to write apps for. These apps are often production quality apps that are used by many people. To make maintaining them easier, we have to set some guidelines for people to follow.

In this article, we’ll look at testing and maintaining the quality of Node.js code.

Write API (Component) Tests at Least

We should write tests as soon as we have time to do it. Tests prevent regressions by checking that existing functionality is still working. That way, we don’t have to worry about our code changes breaking any critical functionality.

It’s easy to create tests for our app. We can use test frameworks like Jest and libraries like Superagent to test our APIs.

Also, we should make sure that code coverage is high so that we’re actually testing most of our code with our tests.

For instance, we can add tests easily to an Express app with Jest and Supertest by running:

npm i jest supertest

Then we write the following:

index.js :

const express = require('express');
const bodyParser = require('body-parser');
const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.get('/', (req, res, next) => {
  res.json({hello: 'hello'});
});

module.exports = app;

index.test.js :

const request = require('supertest');
const app = require('./index');

describe('hello test', () => {
  it('/ should return hello response', async () => {
    const res = await request(app)
      .get('/')
    expect(res.statusCode).toEqual(200)
    expect(res.body).toEqual({hello: 'hello'})
  })
})

In the code file, we added supertest to our index.test.js test file. Then we test our route by making a request to the / route and checking the response code and body.

The toEqual method checks for deep equality so that we can check for any value.

Include 3 Parts in Each Test Name

Each test name should include 3 parts so that people reading the test cases know what’s being tested. A test name should include what is being tested, what scenarios, and what’s the expected result.

Structure Tests by the AAA Pattern

AAA stands for Arrange, Act, and Assert. The first part of the test should set up the data for our tests. Then we actually run the code, and then assert that the returned result is what’s expected.

With these kinds of tests, we understand the main code just by looking at the tests.

Detect Code Issues with a Linter

We should use a code linter to check for basic quality issues with our code. Spacing, formatting, and syntax errors are within its domain. It also checks for common antipatterns to make sure that our code don’t have them. Linters with Node add-ons can also check for security issues with our code.

It’s easy to overlook them without a linter. So we may be running bad quality and code with security vulnerabilities in our code.

Avoid Global Test Fixtures and Seeds

Each test should have its own test fixtures and seeds so that they run independently without anything else. This is important because we need tests to be testing things without depending on anything external. It makes tests easy to add and debug.

It reduces lots of headaches with tests. They shouldn’t depend on any external dependencies.

Inspect for Vulnerable Dependencies

We can use npm audit or snyk.io to check for vulnerable dependencies so that we update those packages as quickly as possible.

With these automated tools, we can check for vulnerable packages without doing anything ourselves.

Tagging Tests

Tagging tests let us search for them easily. We can add the tags to the test names so that we can find them easily.

Check Test Coverage

Test coverage lets us check if tests are running enough parts of our code. Code coverage tools provide highlights to see which parts of the code have been run by tests and what hasn’t. Then we can look at what kinds of code that we need to run with our tests to increase test coverage of our code.

We may also want to fail the build if test coverage falls below a certain threshold.

Inspect for Outdated Packages

We can check for outdated packages with npm outdated and npm-check-updates to detect outdated packages. We can also run it in our CI pipeline to prevent the build from succeeding if we have outdated packages.

This way, we won’t be using outdated packages in our app.

Conclusion

Adding tests is easy with Node apps. It lets us check for regressions without much effort. When we divide our tests to test small pieces of the code, then adding tests takes little effort. This is especially easy with test frameworks like Jest and test HTTP clients like Superagent.

We should also check for vulnerable packages and update them as soon as possible. In addition, we should check our own code for security vulnerabilities and fix them as soon as possible.

Node.js Best Practices

Node.js Best Practices — Using Modern Features

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Using Modern Features

Labrador retriever puppy walking on green grass

Node.js is a popular runtime to write apps. These apps are often production quality apps that are used by many people. To make maintaining them easier, we’ve to set some guidelines for people to follow.

In this article, we’ll look at some modern JavaScript features that we should use to create code that’s clean and easy to maintain.

Prefer const over let. Ditch the var

var is an outdated keyword for creating variables that should never be used again. The scope is inconsistent unlike let and const . var is function scoped, so that it can be accessed from outside blocks and create potential issues with our code.

let and const are blocked scoped so they can’t be accessed outside a block. const prevents reassignment of the constant to another value.

For example, if we have the following code:

var callbacks = [];
(function() {
  for (var i = 0; i < 5; i++) {
    callbacks.push( function() { return i; } );
  }
})();

console.log(callbacks.map( function(cb) { return cb(); } ));

Then we’ll see [ 5, 5, 5, 5, 5 ] as the value of callbacks.map( function(cb) { return cb(); } ).

This is because i ‘s value isn’t passed into the callback until it reaches 5. Then we run each of them with the value 5. The code above is actually the same as:

var callbacks = [];
(function() {
  var i
  for (i = 0; i < 5; i++) {
    callbacks.push( function() { return i; } );
  }
})();
console.log(callbacks.map( function(cb) { return cb(); } ));

because of hoisting. Therefore, the value of i would be 5 when the callbacks are run.

let variables don’t host, so we won’t have the same issue. Therefore, we won’t have the same issue:

var callbacks = [];
(function() {
  for (let i = 0; i < 5; i++) {
    callbacks.push( function() { return i; } );
  }
})();
console.log(callbacks.map( function(cb) { return cb(); } ));

As we can see, var is a pain and it’s confusing, so we should never use it.

Require Modules First, Not Inside Functions

We should require modules on the top of each code file. This lets us easily tell which dependencies are required.

Require runs synchronously in Node.js. Therefore, if they’re called within a function, it may block other pieces of code from running at a more critical time.

If any required module or dependencies throw an error and crash the server, it’s better to find out earlier.

Require Modules by Folders, as Opposed to the Files Directly

We should require modules by folders instead of the files directly. This is because we don’t want to break the require expressions in our user’s apps if we change the folder structure of our module.

Therefore, the following is good:

require('./foo');

But the following is bad:

require('./bar/foo');

Use the === Operator

The strict equality operator === is better than the == operator because it doesn’t coerce the types of the variables before comparing them. With the === operator, both operands must have the same type for them to be equal.

This is good because it prevents lots of errors when comparing things. Using the == operator is bad because expressions like the following all return true:

null == undefined
false == '0'
0 == ''
0 == '0'

In many cases, it’s not what we want. There’re also many other strange edge cases that may cause errors in our app if we use the == operator. Therefore, we should use the === operator.

Use Async Await and Avoid Callbacks

Since Node 8 LTS, async and await is a feature in Node. Therefore, we should use it to chain promises whenever possible. It’s a great shorthand to for chaining promises.

The old callback APIs are slowly converted into promises API in cord Node modules like fs . Therefore, now we can use async and await in places other than our own code.

To handle errors in async and await , we can handle errors as follows:

(async ()=>{
  try {
    await Promise.reject('error')
  }
  catch(ex){
    console.log(ex);
  }
})();

In the code above, we catch the error with the catch block and log the value of ex , which should be 'error' from the Promise.reject .

Conclusion

New constructs in JavaScript are there because they’re good. They make code shorter and cleaner. They make the code pleasant to read and change. It just makes everyone happy to code in JavaScript. Old constructs like var should be eliminated from all code. === should be used instead of == .

Node.js Best Practices

Node.js Best Practices — Validation and Code Style

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Validation and Code Style

Node.js is a popular runtime to write apps for. These apps are often production quality apps that are used by many people. To make maintaining them easier, we’ve to set some guidelines for people to follow.

In this article, we’ll look at input validation and writing code that has a good style.

Validate Arguments Using a Dedicated Library

We should validate inputs in our app so that users can’t submit data that are invalid and cause data corruption.

For express apps, we should check that our endpoints are checking for valid inputs. We should check if we aren’t checking already.

For instance, we can use the express-validator package to make form validation easy. To install it, we run:

npm install --save express-validator

Then we can use it as follows:

const express = require('express');
const bodyParser = require('body-parser');
const { check, validationResult } = require('express-validator');

const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.post('/', [
  check('email').isEmail(),
], (req, res, next) => {
  const errors = validationResult(req);
  if (!errors.isEmpty()) {
    return res.status(422).json({ errors: errors.array() });
  }
  const { email } = req.body;
  res.send(email);
});

app.listen(3000, () => console.log('server started'));

In the code above, we have the check(‘email’).isEmail() to check if the email that’s submitted with the body is a valid email.

Then in the route handler, we send an error response back to the user if it’s not a valid email.

Use ESLint

JavaScript is a very forgiving language. This means it lets us do lots of things that people may not like. It’s also very easy to make mistakes with it with its dynamic types and permissive syntax.

Therefore, we should use linter like ESLint to check for possible code errors and fix bad style. It’s useful for identifying small issues like spacing but also checks for serious anti-patterns like throwing errors without classification.

It can also be used in conjunction with Prettier and Beautify to format code in an easy to read way,

Node.js Specific Plugins for ESLint

There’re specific ESLint plugins for Node like eslint-plugin-node , eslint-plugin-mocha , and eslint-plugin-node-security .

They check for Node specific errors and possible security issues in our code so that our apps won’t be susceptible to them.

Start a Codeblock’s Curly Braces on the Same Line

We should start curly braces on the same line for blocks. For instance, if we define a function, we write:

function foo() {
  // code block
}

instead of:

function foo()
{
  // code block
}

Separate Statements Properly

We should use semicolons to separate our statements and we should use line breaks properly to make sure that our code is easy to read. They both help eliminate regular syntax errors.

ESLint, Prettier, and Standardjs can automatically resolve these issues.

We shouldn’t let the JavaScript interpreter insert semicolons automatically. It’s an easy way to create unexpected errors.

Examples of properly separated states include:

function foo() {
  // code block
}

foo();

or:

const items = [1, 2, 3];
items.forEach(console.log);

The following example is a bad example that’ll throw an error:

const count = 1
(function foo() {

}())

We’ll get 1 is not a function since it’s trying to run 1 as a function. To avoid the error, we should put a semicolon after the 1 :

const count = 1;
(function foo() {

}())

Name Our Functions

We should name our functions so that we can debug them easily. Anonymous functions have no name, so it’s hard to trace them. Named functions are easier to understand what we’re looking at when checking a memory snapshot.

Use Naming Conventions for Variables, Constants, Functions, and Classes

The lower camel case is the convention for constants, variables, and function names. Upper camel case is used when naming classes. This will help us distinguish between plain variables and functions, and classes that require instantiation.

Also, we should use descriptive names, but keep them short.

JavaScript allows invoking a constructor without instantiating it, so it’s important to distinguish by the case so that we don’t confuse constructor functions with regular functions.

For example, we should write:

function Person(name){
  this.name = name;
}

function fooBar(){

}

for functions, and name variables as follows:

let fooBar = 1;

Conclusion

We should name things according to convention and lint our code to prevent bad styles and syntax errors.

Also, we should validate our inputs to prevent data corruption and other errors.

Node.js Best Practices

Node.js Best Practices — Error Handling and Logging

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Error Handling and Logging

selective focus photography of white dog

Node.js is a popular runtime to write apps in JavaScript. To make maintaining them easier, we’ve to set some guidelines for people to follow.

In this article, we’ll look at how to document APIs and gracefully exit processes.

Document API errors using GraphQL

We can build our API using GraphQL libraries. This provides us with a sandbox for querying data. It also provides strong typing and only returns what we want.

It also provides schema and comments so that we can document our APIs without adding more documentation.

To create a GraphQL API, we can use the graphql and express-graphql packages to create a GraphQL API. We install it by running:

npm i express-graphql graphql

Then we can create a simple GraphQL API by writing:

const express = require('express');
const bodyParser = require('body-parser');
const graphqlHTTP = require('express-graphql');
const { buildSchema } = require('graphql');
const app = express();

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const schema = buildSchema(`
  type Query {
    quoteOfTheDay: String
  }
`);

const root = {
  quoteOfTheDay: () => {
    return 'hello';
  },
};

app.use('/graphql', graphqlHTTP({
  schema: schema,
  rootValue: root,
  graphiql: true,
}));

app.listen(3000, () => console.log('server started'));

We should then be able to go to /graphql and test out our quotwOfTheDay query. Autocomplete should be available because of strong typing.

Exit the Process Gracefully When an Unknown Error Occurs

When an unknown error occurs, we should exit the process since we don’t know why it’s failing. A common practice is to restart the process with a process management tool like Forever or PM2.

It’s a bad idea to continue running an app in a faulty state.

Use a Mature Logger to Increase Error Visibility

To make errors easier to spot, we can use a logger like Winston, Bunyan, Log4js or Pino to log activities that happened in our app. For instance, we can use Winston with the express-winston package to add logging to an Express app as follows:

const express = require('express');
const bodyParser = require('body-parser');
const winston = require('winston');
const expressWinston = require('express-winston');

const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.use(expressWinston.logger({
  transports: [
    new winston.transports.Console()
  ],
  format: winston.format.combine(
    winston.format.colorize(),
    winston.format.json()
  ),
  meta: true,
  msg: "HTTP {{req.method}} {{req.url}}",
  expressFormat: true,
  colorize: false,
  ignoreRoute: function (req, res) { return false; }
}));

app.get('/', (req, res, next) => {
  res.send('hello')
});

app.get('/foo', (req, res, next) => {
  try {
    throw new Error('error')
    res.send('hello')
  } catch (err) {
    next(err)
  }
});

app.use((err, req, res, next) => {
  res.send('error occurred')
})
app.listen(3000, () => console.log('server started'));

We just added the logger straight into our app by using the expressWinston.logger function.

Test Error Flows Using Our Favorite Test Framework

Relying on manual testing is slow and error-prone. Therefore, we should add automated tests to our app. It lets us check for both positive and error scenarios by running code which runs in seconds.

There’re many test frameworks like Mocha, Chai, and Jest which can do this for us.

Discover Errors and Downtime Using APM Products

We can use downtime and performance monitoring products to monitor the status of our app.

For Express apps, we can add the express-status-monitor package to watch the status of our app, including CPU and memory usage, response time, request per second, and more.

We just have to install the package by running:

npm i `express-status-monitor`

Then we can use it by writing the following code:

const express = require('express');
const bodyParser = require('body-parser');

const app = express();
app.use(require('express-status-monitor')());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.get('/', (req, res, next) => {
  res.send('hello')
});

app.listen(3000, () => console.log('server started'));

We just put the package straight into our app with app.use(require(‘express-status-monitor’)()); .

Then when we go to the /status page, we’ll see all the performance and health metrics listed.

Catch Unhandled Promise Rejections and Error Events

Unhandled promise rejections should be caught. Therefore, we should always add a catch callback for regular promises or catch block catching rejected promises.

We should also subscribe to the process.unhandledRejection to handle error events. For example, if we try to access a file that doesn’t exist and fails, we should write something :

const fs = require('fs')
const stream = fs.createReadStream('does-not-exist')

process.on('unhandledRejection', (reason, promise) => {
  console.log(`Unhandled Rejection at: ${reason.stack || reason}`)
})

Then we handle the error raised from trying to access a file that’s not found without crashing the app.

To catch rejected promises errors, we write:

Promise.reject('error')
.catch(err=> console.log(err))

or:

(async()=>{
  try {
    await Promise.reject('error')
  }
  catch(ex){
    console.log(ex);
  }
})();

Conclusion

We should catch errors in our code and handle them gracefully. Also, we should log activities in our app with loggers and watch the health and performance of our app with a monitoring tool. Documentation of our app is also very important.

Angular Nodejs

How to Build a Job Queue With Node.js

Post author By John Au-Yeung
Post date August 12, 2020
No Comments on How to Build a Job Queue With Node.js

If you want to make an app that handles long-running tasks, you need a job queue running in the background. Otherwise, your user will be kept waiting for requests, and the server hosting your app may hang. That’s not a pleasant user experience for anyone. Node.js has libraries for building a job queue that run in the background without too much hassle.

Preparation

In this piece, we’ll build a YouTube video downloader that lets users enter a URL from YouTube. Our app will download the video to a local folder where it can be downloaded automatically from the UI once it’s done. The download progress will be displayed while it’s being downloaded. The user can’t download another video until the first one is finished. The way it works is that when a user enters a valid YouTube video URL, a database entry for the job will be recorded in the database. Then a background job will be created which will be downloaded in the background. The job’s progress will be reported back via Socket.io so it can be displayed to the user. Once the job is done, it’ll be marked as done in the database entry for the job. If it fails, it’ll be removed from the queue. The URL for the video will be sent back to the user, and then it will be downloaded automatically.

We’ll build a back end app with Express and a front end app with Angular. To do this, we use Express Generator. With the latest versions of Node.js, we can run npx express-generator after we make a folder for our back end app. This will generate the code files. Next, we need to install some packages. We do this by running npm i in our back end project folder’s root.

We’ll need to install some libraries in order to use the latest JavaScript features, build our queue, store our environment variables, and manipulate our database. We install these libraries by running npm i sequelize @babel/register babel-polyfill body-parser bull cors dotenv pg pg-hstore uuid ytdl-core. We’ll use PostgresSQL as our database, meaning we’ll need the pg and pg-hstore packages. We need theuuid package to generate UUIDs. ytld-core is the YouTube download library. babel-polyfill and @babel/register allow us to use the latest JavaScript features. We also need Sequelize CLI to create our models and allow us to run database migrations to change our database’s structure. To do this, we run npm i -g sequelize-cli.

Now, we need to create our database. First, we create an empty database with pgAdmin 3.x by connecting to our server and doubleclicking. Right-click the database item, then click New Database. pgAdmin 3.x is used because it’s much faster than 4.x and has more features.

Finally, we need to initialize our Sequelize code. We run npx sequelize-cli init in our back end app’s project folder to do this.

The Code

Now we can write some code.

Building the back end

In bin/www, we put:

#!/usr/bin/env node

/**
 * Module dependencies.
 */

const app = require('../app');
const debug = require('debug')('backend:server');
const http = require('http');
/**
 * Get port from environment and store in Express.
 */

const port = normalizePort(process.env.PORT || '3000');
app.set('port', port);

/**
 * Create HTTP server.
 */

const server = http.createServer(app);
const io = require('socket.io')(server, { origins: '*:*' });
global.io = io;

/**
 * Listen on provided port, on all network interfaces.
 */

server.listen(port);
server.on('error', onError);
server.on('listening', onListening);
io.on('connection', (socket) => {
  socket.emit('connected', { message: 'connected' });
});

/**
 * Normalize a port into a number, string, or false.
 */

function normalizePort(val) {
  const port = parseInt(val, 10);

if (isNaN(port)) {
    // named pipe
    return val;
  }

if (port >= 0) {
    // port number
    return port;
  }

return false;
}

/**
 * Event listener for HTTP server "error" event.
 */

function onError(error) {
  if (error.syscall !== 'listen') {
    throw error;
  }

const bind = typeof port === 'string'
    ? 'Pipe ' + port
    : 'Port ' + port;

// handle specific listen errors with friendly messages
  switch (error.code) {
    case 'EACCES':
      console.error(bind + ' requires elevated privileges');
      process.exit(1);
      break;
    case 'EADDRINUSE':
      console.error(bind + ' is already in use');
      process.exit(1);
      break;
    default:
      throw error;
  }
}

/**
 * Event listener for HTTP server "listening" event.
 */

function onListening() {
  const addr = server.address();
  const bind = typeof addr === 'string'
    ? 'pipe ' + addr
    : 'port ' + addr.port;
  debug('Listening on ' + bind);
}

This is the entry point of our app. We initialize Socket.io here to allow us to listen for messages from client-side. It will also set the socket object globally so that it can be used in other files.

Next, in the config folder, we rename config.json, which is generated when running npx sequelize-cli init to config.js and add the following:

require('dotenv').config();
const dbHost = process.env.DB_HOST;
const dbName = process.env.DB_NAME;
const dbUsername = process.env.DB_USERNAME;
const dbPassword = process.env.DB_PASSWORD;
const dbPort = process.env.DB_PORT || 5432;

module.exports = {
    development: {
        username: dbUsername,
        password: dbPassword,
        database: dbName,
        host: dbHost,
        port: dbPort,
        dialect: 'postgres'
    },
    test: {
        username: dbUsername,
        password: dbPassword,
        database: 'youtube_app_test',
        host: dbHost,
        port: dbPort,
        dialect: 'postgres'
    },
    production: {
        use_env_variable: 'DATABASE_URL',
        username: dbUsername,
        password: dbPassword,
        database: dbName,
        host: dbHost,
        port: dbPort,
        dialect: 'postgres'
    }
};

This allows us to use environment variables instead of hard coding database credentials to our database. Then we make a files folder in the root and put an empty .gitkeep file in it so it can be committed to Git.

Then, we make a database migration with Sequelize to build our database. We run:

npx sequelize-cli model:generate --name Job --attributes status:enum,url:string,fileLocation:string

to create a migration file and its corresponding model file. In the model file, which should be called job.js in the models folder, we put:

'use strict';
module.exports = (sequelize, DataTypes) => {
  const Job = sequelize.define('Job', {
    status: DataTypes.ENUM('started', 'cancelled', 'done'),
    url: DataTypes.STRING,
    fileLocation: DataTypes.STRING
  }, {});
  Job.associate = function(models) {
    // associations can be defined here
  };
  return Job;
};

and in index.js in the models folder, we put:

'use strict';

const fs = require('fs');
const path = require('path');
const Sequelize = require('sequelize');
const basename = path.basename(__filename);
const env = process.env.NODE_ENV || 'development';
const config = require(__dirname + '/../config/config.js')[env];
const db = {};

let sequelize;
if (config.use_env_variable) {
  sequelize = new Sequelize(process.env[config.use_env_variable], config);
} else {
  sequelize = new Sequelize(config.database, config.username, config.password, config);
}

fs
  .readdirSync(__dirname)
  .filter(file => {
    return (file.indexOf('.') !== 0) && (file !== basename) && (file.slice(-3) === '.js');
  })
  .forEach(file => {
    const model = sequelize['import'](path.join(__dirname, file));
    db[model.name] = model;
  });

Object.keys(db).forEach(modelName => {
  if (db[modelName].associate) {
    db[modelName].associate(db);
  }
});

db.sequelize = sequelize;
db.Sequelize = Sequelize;

module.exports = db;

The most import part is renaming config.json to config.js in const config = require(__dirname + ‘/../config/config.js’)[env];.

Next, we build our queue with the bull package. We create a folder called queue in the project root folder and add video.js. In that file, we put:

const Queue = require('bull');
const fs = require('fs');
const models = require('../models');
const ytdl = require('ytdl-core');
const uuidv1 = require('uuid/v1');
const util = require('util');

const createVideoQueue = () => {
    const videoQueue = new Queue('video transcoding', {
        redis: {
            port: process.env.REDIS_PORT,
            host: process.env.REDIS_URL
        }
    });

videoQueue.process(async (job, done) => {
        const data = job.data;
        try {
            job.progress(0);
            global.io.emit('progress', { progress: 0, jobId: data.id });
            const uuid = uuidv1();
            const fileLocation = `./files/${uuid}.mp4`;
            await new Promise((resolve) => {
                ytdl(data.url)
                    .on('progress', (length, downloaded, totallength) => {
                        const progress = (downloaded / totallength) * 100;
                        global.io.emit('progress', { progress, jobId: data.id });
                        if (progress >= 100) {
                            global.io.emit('videoDone', { fileLocation: `${uuid}.mp4`, jobId: data.id });
                            global.io.emit('progress', { progress: 100, jobId: data.id });
                        }
                    })
                    .pipe(fs.createWriteStream(fileLocation))
                    .on('finish', () => {
                        resolve();
                    })
            })
            await models.Job.update({
                status: 'done',
                fileLocation: `${uuid}.mp4`
            }, {
                    where: {
                        id: data.id
                    }
                })
            done();
        }
        catch (ex) {
            console.log(ex);
            job.moveToFailed();
        }
    });
    return videoQueue;
}

module.exports = { createVideoQueue };

Note that we passed in the socket object to send progress back to the client, and that we converted all the asynchronous code to promises so they can be called sequentially. We use ytdl to download YouTube videos. It has a progress event handler which reports progress of the download, which we send back to the client via Socket.io’s broadcast function. This sends messages to all the clients. We will filter out the irrelevant messages on the client side. Any failed jobs will be removed from the queue.

Next, we create our routes. In the routes folder, we add a new file called jobs.js and put:

const express = require('express');
const models = require('../models');
const path = require('path');
const router = express.Router();
const ytdl = require('ytdl-core');
const { createVideoQueue } = require('../queue/video');

router.post('/new', async (req, res) => {
  const url = req.body.url;
  try {
    const isValidUrl = ytdl.validateURL(url);
    if (!isValidUrl) {
      res.status(400);
      return res.send({ error: 'invalid URL' });
    }
    const job = await models.Job.create({
      url,
      status: 'started'
    })
    await createVideoQueue().add({ url, id: job.id });
    return res.send(job);
  }
  catch (ex) {
    console.log(ex);
    res.status(400);
    return res.send({ error: ex });
  }
});

router.get('/file/:fileName', (req, res) => {
  const fileName = req.params.fileName;
  const file = path.resolve(__dirname, `../files/${fileName}`);
  res.download(file);
})

module.exports = router;

We need a route to add new jobs and to download the generated files. We validate the URL submitted before creating the job to minimize errors. In this line:

await createVideoQueue(global.socket).add({ url, id: job.id });

we pass in the global.socket object we created when the client connects to this app in binwww . Note that we don’t wait for the job to be done before returning a response. This is why we need Socket.io, to communicate the results back to the client.

In app.js, we add the initialization code. We add the following code to the file:

require("@babel/register");
require("babel-polyfill");
require('dotenv').config();
const createError = require('http-errors');
const express = require('express');
const path = require('path');
const cookieParser = require('cookie-parser');
const logger = require('morgan');
const bodyParser = require('body-parser')
const cors = require('cors')
const indexRouter = require('./routes/index');
const usersRouter = require('./routes/users');
const jobsRouter = require('./routes/jobs');
const app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use(express.static(path.join(__dirname, 'files')));
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json())
app.use(cors())
app.use('/', indexRouter);
app.use('/users', usersRouter);
app.use('/jobs', jobsRouter);

// catch 404 and forward to error handler
app.use((req, res, next) => {
  next(createError(404));
});

// error handler
app.use((err, req, res, next) => {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get('env') === 'development' ? err : {};

// render the error page
  res.status(err.status || 500);
  res.render('error');
});

module.exports = app;

We add app.use(express.static(path.join(__dirname, ‘files’))); to expose the files folder that we created to the public, and we add:

const jobsRouter = require('./routes/jobs');

and

app.use('/jobs', jobsRouter);

so that clients can access the route we created.

Finally, we create an .env file and put the following:

REDIS_URL='localhost'
REDIS_PORT='6379'
DB_HOST='localhost'
DB_NAME='youtube_app_development'
DB_USERNAME='postgres'
DB_PASSWORD='postgres'

The bull package requires Redis, so we have to install it. To do so, we run the following in Ubuntu or related Linux distributions:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install redis-server
$ sudo systemctl enable redis-server.service
$ sudo service redis-server restart

The first two commands are run to update the package repository references and to update our Linux packages. We run sudo apt-get install redis-server to install Redis, and we run the fourth line to enable Redis on startup. If Redis is not started or needs restarting, we run sudo service redis-server restart.

Note—there is no recent Windows version of Redis, so Linux is required. Now we have everything needed to run the back end.

Building the UI

The back end is done and we can move on to building the UI. We build it with Angular and Angular Material. To get started, we install the Angular CLI by running npm i -g @angular/cli . Then we run ng new frontend in our top-level project folder to create the app. Be sure to choose to include routing and use SCSS for styling when prompted. After that, we run npm i @angular/cdk @angular/material file-saver socket.io-client. The first two packages are Angular Material packages. file-saver helps us download files, and socket.io-client allows us to connect to the back end to get download progress and file location.

In environment.ts, we put:

export const environment = {
  production: false,
  apiUrl: 'http://localhost:3000',
  socketIoUrl: 'http://localhost:3000'
};

Then we create our components and services.

We run ng g component homePage and ng g service video to create our code files.

In video.service.ts, we put:

import { Injectable } from '@angular/core"';
import { HttpClient, HttpHeaders } from '@angular/common/http';
import { environment } from 'src/environments/environment';

@Injectable({
  providedIn: 'root'
})
export class VideoService {

  constructor(
    private http: HttpClient
  ) { }

  addVideoToQueue(data) {
    return this.http.post(`${environment.apiUrl}/jobs/new`, data);
  }

  getVideo(videoUrl: string) {
    return this.http.get<Blob>(videoUrl, {
      headers: new HttpHeaders({
        'accept': 'application/octet-stream',
        'content-type': 'application/json'
      }),
      responseType: 'blob' as 'json'
    })
  }
}

to let our app make requests to add YouTube videos to the queue for download, and we call the getVideo to download. Note that we set the accept header to ‘application/octet-stream’ so that we can download video files.

Next in home-page.component.ts, we put:

import { Component, OnInit } from '@angular/core"';
import { VideoService } from '../video.service';
import { NgForm } from '@angular/forms';
import io from 'socket.io-client';
import { environment } from 'src/environments/environment';
import { saveAs } from 'file-saver';

@Component({
  selector: 'app-home-page',
  templateUrl: './home-page.component.html',
  styleUrls: ['./home-page.component.scss']
})
export class HomePageComponent implements OnInit {
  videoData: any = <any>{};
  progress: number = 0;
  fileLocation: string;
  downloaded: boolean = false;
  jobId: number;
  connected: boolean = false;
  socket;
  getVideoSub;

constructor(
    private videoService: VideoService
  ) { }

ngOnInit() {
      this.addConnectionHandlers();
  }

  addConnectionHandlers() {
    const manager = io.Manager(environment.socketIoUrl);
    manager.on('connect_error', () => {
      this.socket = io.connect(environment.socketIoUrl);
    });

      this.socket = io.connect(environment.socketIoUrl);
    this.socket.on('connect', (data) => {
      this.socket.on('connected', (msg) => {

});

      this.socket.on('progress', (msg) => {
        if (this.jobId != msg.jobId) {
          return;
        }
        this.progress = msg.progress;
        if (msg.progress == 100) {
          this.progress = 0;
        }
      });

      this.socket.on('videoDone', (msg) => {
        if (this.jobId != msg.jobId || this.downloaded) {
          return;
        }
        this.getVideoSub = this.videoService.getVideo(`${environment.apiUrl}/jobs/file/${msg.fileLocation}`)
          .subscribe(res => {
            if (!this.downloaded) {
              saveAs(res, `${msg.fileLocation}.mp4`);
              this.progress = 0;
              this.downloaded = true;
              this.getVideoSub.unsubscribe();
            }
          })
      });
    });
  }

  addVideoToQueue(videoForm: NgForm) {
    this.downloaded = false;
    if (videoForm.invalid) {
      return;
    }
    this.videoService.addVideoToQueue(this.videoData)
      .subscribe(res => {
        this.jobId = (res as any).id;
      }, err => {
        alert('Invalid URL');
      })
  }
}

This provides the logic for the UI to let the user enter their YouTube URLs, watch their video’s download progress, and download it when it’s done. Since we used socket.broadcast.emit in the back end, we have to filter it out in the front end. The back end returns the jobId for the download job, so we can filter out by jobId. We also need to add retry in case the back end app goes down with the setTimeout block in the connect_error handler. We check if the same file has been downloaded before with the this.downloaded flag so it won’t download again. Otherwise, it might try to download too many times, causing freezes and crashes.

In home-page.component.html, we put:

<div class="center">
    <h1>Download Video From YouTube</h1>
</div>
<div id='content'>
    <form #videoForm='ngForm' (ngSubmit)='addVideoToQueue(videoForm)'>
        <mat-form-field>
            <input matInput placeholder="YouTube URL" required #url='ngModel' name='url' [(ngModel)]='videoData.url'
                [disabled]='progress != 0'>
            <mat-error *ngIf="url.invalid && (url.dirty || url.touched)">
                <div *ngIf="url.errors.required">
                    URL is required.
                </div>
            </mat-error>
        </mat-form-field>
        <br>
        <button mat-raised-button type='submit'>Convert</button>
    </form>
    <br>
    <mat-card *ngIf='progress > 0'>
        Downloading: {{progress}}%
    </mat-card>
</div>

to let the user enter their YouTube URL and display progress. Note that we disabled input when a video is downloaded, so that users can’t keep entering new requests.

In home-page.component.scss, we put:

#content {
  width: 95vw;
  margin: 0 auto;
}

to add some padding to the form.

In app-routing.module.ts, we put:

import { NgModule } from '@angular/core"';
import { Routes, RouterModule } from '@angular/router';
import { HomePageComponent } from './home-page/home-page.component';

const routes: Routes = [
  { path: '', component: HomePageComponent }
];

@NgModule({
  imports: [RouterModule.forRoot(routes)],
  exports: [RouterModule]
})
export class AppRoutingModule { }

so that users can see our page.

In app.component.html, we put:

<router-outlet></router-outlet>

so that our page will be displayed. In app.module.ts, we put:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core"';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';
import {
  MatButtonModule,
  MatCheckboxModule,
  MatInputModule,
  MatMenuModule,
  MatSidenavModule,
  MatToolbarModule,
  MatTableModule,
  MatDialogModule,
  MAT_DIALOG_DEFAULT_OPTIONS,
  MatDatepickerModule,
  MatSelectModule,
  MatCardModule,
  MatFormFieldModule
} from @angular/material;
import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { HomePageComponent } from './home-page/home-page.component';
import { FormsModule } from '@angular/forms';
import { HttpClientModule } from '@angular/common/http';

@NgModule({
  declarations: [
    AppComponent,
    HomePageComponent
  ],
  imports: [
    BrowserModule,
    AppRoutingModule,
    MatButtonModule,
    BrowserAnimationsModule,
    MatButtonModule,
    MatCheckboxModule,
    MatFormFieldModule,
    MatInputModule,
    MatMenuModule,
    MatSidenavModule,
    MatToolbarModule,
    MatTableModule,
    FormsModule,
    HttpClientModule,
    MatDialogModule,
    MatDatepickerModule,
    MatSelectModule,
    MatCardModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

so that we can use Angular Material widgets in our app.

In styles.scss, we put:

/* You can add global styles to this file, and also import other style files */
@import "~@angular/material/prebuilt-themes/indigo-pink.css";
body {
  font-family: "Roboto", sans-serif;
  margin: 0;
}

form {
  mat-form-field {
    width: 95vw;
    margin: 0 auto;
  }
}

.center {
  text-align: center;
}

to include Material Design styles and add some padding to our forms and style for centering text.

In index.html, we put:

<!doctype html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <title>YouTube Download App</title>
  <base href="/">
  <link href="https://fonts.googleapis.com/css?family=Roboto&display=swap" rel="stylesheet">
  <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
</head>

<body>
  <app-root></app-root>
</body>

</html>

to include Material Icons and Roboto font.