Category: JavaScript

Node.js Best Practices — Using Modern Features

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Using Modern Features

Labrador retriever puppy walking on green grass

Node.js is a popular runtime to write apps. These apps are often production quality apps that are used by many people. To make maintaining them easier, we’ve to set some guidelines for people to follow.

In this article, we’ll look at some modern JavaScript features that we should use to create code that’s clean and easy to maintain.

Prefer const over let. Ditch the var

var is an outdated keyword for creating variables that should never be used again. The scope is inconsistent unlike let and const . var is function scoped, so that it can be accessed from outside blocks and create potential issues with our code.

let and const are blocked scoped so they can’t be accessed outside a block. const prevents reassignment of the constant to another value.

For example, if we have the following code:

var callbacks = [];
(function() {
  for (var i = 0; i < 5; i++) {
    callbacks.push( function() { return i; } );
  }
})();

console.log(callbacks.map( function(cb) { return cb(); } ));

Then we’ll see [ 5, 5, 5, 5, 5 ] as the value of callbacks.map( function(cb) { return cb(); } ).

This is because i ‘s value isn’t passed into the callback until it reaches 5. Then we run each of them with the value 5. The code above is actually the same as:

var callbacks = [];
(function() {
  var i
  for (i = 0; i < 5; i++) {
    callbacks.push( function() { return i; } );
  }
})();
console.log(callbacks.map( function(cb) { return cb(); } ));

because of hoisting. Therefore, the value of i would be 5 when the callbacks are run.

let variables don’t host, so we won’t have the same issue. Therefore, we won’t have the same issue:

var callbacks = [];
(function() {
  for (let i = 0; i < 5; i++) {
    callbacks.push( function() { return i; } );
  }
})();
console.log(callbacks.map( function(cb) { return cb(); } ));

As we can see, var is a pain and it’s confusing, so we should never use it.

Require Modules First, Not Inside Functions

We should require modules on the top of each code file. This lets us easily tell which dependencies are required.

Require runs synchronously in Node.js. Therefore, if they’re called within a function, it may block other pieces of code from running at a more critical time.

If any required module or dependencies throw an error and crash the server, it’s better to find out earlier.

Require Modules by Folders, as Opposed to the Files Directly

We should require modules by folders instead of the files directly. This is because we don’t want to break the require expressions in our user’s apps if we change the folder structure of our module.

Therefore, the following is good:

require('./foo');

But the following is bad:

require('./bar/foo');

Use the === Operator

The strict equality operator === is better than the == operator because it doesn’t coerce the types of the variables before comparing them. With the === operator, both operands must have the same type for them to be equal.

This is good because it prevents lots of errors when comparing things. Using the == operator is bad because expressions like the following all return true:

null == undefined
false == '0'
0 == ''
0 == '0'

In many cases, it’s not what we want. There’re also many other strange edge cases that may cause errors in our app if we use the == operator. Therefore, we should use the === operator.

Use Async Await and Avoid Callbacks

Since Node 8 LTS, async and await is a feature in Node. Therefore, we should use it to chain promises whenever possible. It’s a great shorthand to for chaining promises.

The old callback APIs are slowly converted into promises API in cord Node modules like fs . Therefore, now we can use async and await in places other than our own code.

To handle errors in async and await , we can handle errors as follows:

(async ()=>{
  try {
    await Promise.reject('error')
  }
  catch(ex){
    console.log(ex);
  }
})();

In the code above, we catch the error with the catch block and log the value of ex , which should be 'error' from the Promise.reject .

Conclusion

New constructs in JavaScript are there because they’re good. They make code shorter and cleaner. They make the code pleasant to read and change. It just makes everyone happy to code in JavaScript. Old constructs like var should be eliminated from all code. === should be used instead of == .

Node.js Best Practices

Node.js Best Practices — Validation and Code Style

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Validation and Code Style

Node.js is a popular runtime to write apps for. These apps are often production quality apps that are used by many people. To make maintaining them easier, we’ve to set some guidelines for people to follow.

In this article, we’ll look at input validation and writing code that has a good style.

Validate Arguments Using a Dedicated Library

We should validate inputs in our app so that users can’t submit data that are invalid and cause data corruption.

For express apps, we should check that our endpoints are checking for valid inputs. We should check if we aren’t checking already.

For instance, we can use the express-validator package to make form validation easy. To install it, we run:

npm install --save express-validator

Then we can use it as follows:

const express = require('express');
const bodyParser = require('body-parser');
const { check, validationResult } = require('express-validator');

const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.post('/', [
  check('email').isEmail(),
], (req, res, next) => {
  const errors = validationResult(req);
  if (!errors.isEmpty()) {
    return res.status(422).json({ errors: errors.array() });
  }
  const { email } = req.body;
  res.send(email);
});

app.listen(3000, () => console.log('server started'));

In the code above, we have the check(‘email’).isEmail() to check if the email that’s submitted with the body is a valid email.

Then in the route handler, we send an error response back to the user if it’s not a valid email.

Use ESLint

JavaScript is a very forgiving language. This means it lets us do lots of things that people may not like. It’s also very easy to make mistakes with it with its dynamic types and permissive syntax.

Therefore, we should use linter like ESLint to check for possible code errors and fix bad style. It’s useful for identifying small issues like spacing but also checks for serious anti-patterns like throwing errors without classification.

It can also be used in conjunction with Prettier and Beautify to format code in an easy to read way,

Node.js Specific Plugins for ESLint

There’re specific ESLint plugins for Node like eslint-plugin-node , eslint-plugin-mocha , and eslint-plugin-node-security .

They check for Node specific errors and possible security issues in our code so that our apps won’t be susceptible to them.

Start a Codeblock’s Curly Braces on the Same Line

We should start curly braces on the same line for blocks. For instance, if we define a function, we write:

function foo() {
  // code block
}

instead of:

function foo()
{
  // code block
}

Separate Statements Properly

We should use semicolons to separate our statements and we should use line breaks properly to make sure that our code is easy to read. They both help eliminate regular syntax errors.

ESLint, Prettier, and Standardjs can automatically resolve these issues.

We shouldn’t let the JavaScript interpreter insert semicolons automatically. It’s an easy way to create unexpected errors.

Examples of properly separated states include:

function foo() {
  // code block
}

foo();

or:

const items = [1, 2, 3];
items.forEach(console.log);

The following example is a bad example that’ll throw an error:

const count = 1
(function foo() {

}())

We’ll get 1 is not a function since it’s trying to run 1 as a function. To avoid the error, we should put a semicolon after the 1 :

const count = 1;
(function foo() {

}())

Name Our Functions

We should name our functions so that we can debug them easily. Anonymous functions have no name, so it’s hard to trace them. Named functions are easier to understand what we’re looking at when checking a memory snapshot.

Use Naming Conventions for Variables, Constants, Functions, and Classes

The lower camel case is the convention for constants, variables, and function names. Upper camel case is used when naming classes. This will help us distinguish between plain variables and functions, and classes that require instantiation.

Also, we should use descriptive names, but keep them short.

JavaScript allows invoking a constructor without instantiating it, so it’s important to distinguish by the case so that we don’t confuse constructor functions with regular functions.

For example, we should write:

function Person(name){
  this.name = name;
}

function fooBar(){

}

for functions, and name variables as follows:

let fooBar = 1;

Conclusion

We should name things according to convention and lint our code to prevent bad styles and syntax errors.

Also, we should validate our inputs to prevent data corruption and other errors.

Node.js Best Practices

Node.js Best Practices — Error Handling and Logging

Post author By John Au-Yeung
Post date August 13, 2020
No Comments on Node.js Best Practices — Error Handling and Logging

selective focus photography of white dog

Node.js is a popular runtime to write apps in JavaScript. To make maintaining them easier, we’ve to set some guidelines for people to follow.

In this article, we’ll look at how to document APIs and gracefully exit processes.

Document API errors using GraphQL

We can build our API using GraphQL libraries. This provides us with a sandbox for querying data. It also provides strong typing and only returns what we want.

It also provides schema and comments so that we can document our APIs without adding more documentation.

To create a GraphQL API, we can use the graphql and express-graphql packages to create a GraphQL API. We install it by running:

npm i express-graphql graphql

Then we can create a simple GraphQL API by writing:

const express = require('express');
const bodyParser = require('body-parser');
const graphqlHTTP = require('express-graphql');
const { buildSchema } = require('graphql');
const app = express();

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const schema = buildSchema(`
  type Query {
    quoteOfTheDay: String
  }
`);

const root = {
  quoteOfTheDay: () => {
    return 'hello';
  },
};

app.use('/graphql', graphqlHTTP({
  schema: schema,
  rootValue: root,
  graphiql: true,
}));

app.listen(3000, () => console.log('server started'));

We should then be able to go to /graphql and test out our quotwOfTheDay query. Autocomplete should be available because of strong typing.

Exit the Process Gracefully When an Unknown Error Occurs

When an unknown error occurs, we should exit the process since we don’t know why it’s failing. A common practice is to restart the process with a process management tool like Forever or PM2.

It’s a bad idea to continue running an app in a faulty state.

Use a Mature Logger to Increase Error Visibility

To make errors easier to spot, we can use a logger like Winston, Bunyan, Log4js or Pino to log activities that happened in our app. For instance, we can use Winston with the express-winston package to add logging to an Express app as follows:

const express = require('express');
const bodyParser = require('body-parser');
const winston = require('winston');
const expressWinston = require('express-winston');

const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.use(expressWinston.logger({
  transports: [
    new winston.transports.Console()
  ],
  format: winston.format.combine(
    winston.format.colorize(),
    winston.format.json()
  ),
  meta: true,
  msg: "HTTP {{req.method}} {{req.url}}",
  expressFormat: true,
  colorize: false,
  ignoreRoute: function (req, res) { return false; }
}));

app.get('/', (req, res, next) => {
  res.send('hello')
});

app.get('/foo', (req, res, next) => {
  try {
    throw new Error('error')
    res.send('hello')
  } catch (err) {
    next(err)
  }
});

app.use((err, req, res, next) => {
  res.send('error occurred')
})
app.listen(3000, () => console.log('server started'));

We just added the logger straight into our app by using the expressWinston.logger function.

Test Error Flows Using Our Favorite Test Framework

Relying on manual testing is slow and error-prone. Therefore, we should add automated tests to our app. It lets us check for both positive and error scenarios by running code which runs in seconds.

There’re many test frameworks like Mocha, Chai, and Jest which can do this for us.

Discover Errors and Downtime Using APM Products

We can use downtime and performance monitoring products to monitor the status of our app.

For Express apps, we can add the express-status-monitor package to watch the status of our app, including CPU and memory usage, response time, request per second, and more.

We just have to install the package by running:

npm i `express-status-monitor`

Then we can use it by writing the following code:

const express = require('express');
const bodyParser = require('body-parser');

const app = express();
app.use(require('express-status-monitor')());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.get('/', (req, res, next) => {
  res.send('hello')
});

app.listen(3000, () => console.log('server started'));

We just put the package straight into our app with app.use(require(‘express-status-monitor’)()); .

Then when we go to the /status page, we’ll see all the performance and health metrics listed.

Catch Unhandled Promise Rejections and Error Events

Unhandled promise rejections should be caught. Therefore, we should always add a catch callback for regular promises or catch block catching rejected promises.

We should also subscribe to the process.unhandledRejection to handle error events. For example, if we try to access a file that doesn’t exist and fails, we should write something :

const fs = require('fs')
const stream = fs.createReadStream('does-not-exist')

process.on('unhandledRejection', (reason, promise) => {
  console.log(`Unhandled Rejection at: ${reason.stack || reason}`)
})

Then we handle the error raised from trying to access a file that’s not found without crashing the app.

To catch rejected promises errors, we write:

Promise.reject('error')
.catch(err=> console.log(err))

or:

(async()=>{
  try {
    await Promise.reject('error')
  }
  catch(ex){
    console.log(ex);
  }
})();

Conclusion

We should catch errors in our code and handle them gracefully. Also, we should log activities in our app with loggers and watch the health and performance of our app with a monitoring tool. Documentation of our app is also very important.

Angular Nodejs

How to Build a Job Queue With Node.js

Post author By John Au-Yeung
Post date August 12, 2020
No Comments on How to Build a Job Queue With Node.js

If you want to make an app that handles long-running tasks, you need a job queue running in the background. Otherwise, your user will be kept waiting for requests, and the server hosting your app may hang. That’s not a pleasant user experience for anyone. Node.js has libraries for building a job queue that run in the background without too much hassle.

Preparation

In this piece, we’ll build a YouTube video downloader that lets users enter a URL from YouTube. Our app will download the video to a local folder where it can be downloaded automatically from the UI once it’s done. The download progress will be displayed while it’s being downloaded. The user can’t download another video until the first one is finished. The way it works is that when a user enters a valid YouTube video URL, a database entry for the job will be recorded in the database. Then a background job will be created which will be downloaded in the background. The job’s progress will be reported back via Socket.io so it can be displayed to the user. Once the job is done, it’ll be marked as done in the database entry for the job. If it fails, it’ll be removed from the queue. The URL for the video will be sent back to the user, and then it will be downloaded automatically.

We’ll build a back end app with Express and a front end app with Angular. To do this, we use Express Generator. With the latest versions of Node.js, we can run npx express-generator after we make a folder for our back end app. This will generate the code files. Next, we need to install some packages. We do this by running npm i in our back end project folder’s root.

We’ll need to install some libraries in order to use the latest JavaScript features, build our queue, store our environment variables, and manipulate our database. We install these libraries by running npm i sequelize @babel/register babel-polyfill body-parser bull cors dotenv pg pg-hstore uuid ytdl-core. We’ll use PostgresSQL as our database, meaning we’ll need the pg and pg-hstore packages. We need theuuid package to generate UUIDs. ytld-core is the YouTube download library. babel-polyfill and @babel/register allow us to use the latest JavaScript features. We also need Sequelize CLI to create our models and allow us to run database migrations to change our database’s structure. To do this, we run npm i -g sequelize-cli.

Now, we need to create our database. First, we create an empty database with pgAdmin 3.x by connecting to our server and doubleclicking. Right-click the database item, then click New Database. pgAdmin 3.x is used because it’s much faster than 4.x and has more features.

Finally, we need to initialize our Sequelize code. We run npx sequelize-cli init in our back end app’s project folder to do this.

The Code

Now we can write some code.

Building the back end

In bin/www, we put:

#!/usr/bin/env node

/**
 * Module dependencies.
 */

const app = require('../app');
const debug = require('debug')('backend:server');
const http = require('http');
/**
 * Get port from environment and store in Express.
 */

const port = normalizePort(process.env.PORT || '3000');
app.set('port', port);

/**
 * Create HTTP server.
 */

const server = http.createServer(app);
const io = require('socket.io')(server, { origins: '*:*' });
global.io = io;

/**
 * Listen on provided port, on all network interfaces.
 */

server.listen(port);
server.on('error', onError);
server.on('listening', onListening);
io.on('connection', (socket) => {
  socket.emit('connected', { message: 'connected' });
});

/**
 * Normalize a port into a number, string, or false.
 */

function normalizePort(val) {
  const port = parseInt(val, 10);

if (isNaN(port)) {
    // named pipe
    return val;
  }

if (port >= 0) {
    // port number
    return port;
  }

return false;
}

/**
 * Event listener for HTTP server "error" event.
 */

function onError(error) {
  if (error.syscall !== 'listen') {
    throw error;
  }

const bind = typeof port === 'string'
    ? 'Pipe ' + port
    : 'Port ' + port;

// handle specific listen errors with friendly messages
  switch (error.code) {
    case 'EACCES':
      console.error(bind + ' requires elevated privileges');
      process.exit(1);
      break;
    case 'EADDRINUSE':
      console.error(bind + ' is already in use');
      process.exit(1);
      break;
    default:
      throw error;
  }
}

/**
 * Event listener for HTTP server "listening" event.
 */

function onListening() {
  const addr = server.address();
  const bind = typeof addr === 'string'
    ? 'pipe ' + addr
    : 'port ' + addr.port;
  debug('Listening on ' + bind);
}

This is the entry point of our app. We initialize Socket.io here to allow us to listen for messages from client-side. It will also set the socket object globally so that it can be used in other files.

Next, in the config folder, we rename config.json, which is generated when running npx sequelize-cli init to config.js and add the following:

require('dotenv').config();
const dbHost = process.env.DB_HOST;
const dbName = process.env.DB_NAME;
const dbUsername = process.env.DB_USERNAME;
const dbPassword = process.env.DB_PASSWORD;
const dbPort = process.env.DB_PORT || 5432;

module.exports = {
    development: {
        username: dbUsername,
        password: dbPassword,
        database: dbName,
        host: dbHost,
        port: dbPort,
        dialect: 'postgres'
    },
    test: {
        username: dbUsername,
        password: dbPassword,
        database: 'youtube_app_test',
        host: dbHost,
        port: dbPort,
        dialect: 'postgres'
    },
    production: {
        use_env_variable: 'DATABASE_URL',
        username: dbUsername,
        password: dbPassword,
        database: dbName,
        host: dbHost,
        port: dbPort,
        dialect: 'postgres'
    }
};

This allows us to use environment variables instead of hard coding database credentials to our database. Then we make a files folder in the root and put an empty .gitkeep file in it so it can be committed to Git.

Then, we make a database migration with Sequelize to build our database. We run:

npx sequelize-cli model:generate --name Job --attributes status:enum,url:string,fileLocation:string

to create a migration file and its corresponding model file. In the model file, which should be called job.js in the models folder, we put:

'use strict';
module.exports = (sequelize, DataTypes) => {
  const Job = sequelize.define('Job', {
    status: DataTypes.ENUM('started', 'cancelled', 'done'),
    url: DataTypes.STRING,
    fileLocation: DataTypes.STRING
  }, {});
  Job.associate = function(models) {
    // associations can be defined here
  };
  return Job;
};

and in index.js in the models folder, we put:

'use strict';

const fs = require('fs');
const path = require('path');
const Sequelize = require('sequelize');
const basename = path.basename(__filename);
const env = process.env.NODE_ENV || 'development';
const config = require(__dirname + '/../config/config.js')[env];
const db = {};

let sequelize;
if (config.use_env_variable) {
  sequelize = new Sequelize(process.env[config.use_env_variable], config);
} else {
  sequelize = new Sequelize(config.database, config.username, config.password, config);
}

fs
  .readdirSync(__dirname)
  .filter(file => {
    return (file.indexOf('.') !== 0) && (file !== basename) && (file.slice(-3) === '.js');
  })
  .forEach(file => {
    const model = sequelize['import'](path.join(__dirname, file));
    db[model.name] = model;
  });

Object.keys(db).forEach(modelName => {
  if (db[modelName].associate) {
    db[modelName].associate(db);
  }
});

db.sequelize = sequelize;
db.Sequelize = Sequelize;

module.exports = db;

The most import part is renaming config.json to config.js in const config = require(__dirname + ‘/../config/config.js’)[env];.

Next, we build our queue with the bull package. We create a folder called queue in the project root folder and add video.js. In that file, we put:

const Queue = require('bull');
const fs = require('fs');
const models = require('../models');
const ytdl = require('ytdl-core');
const uuidv1 = require('uuid/v1');
const util = require('util');

const createVideoQueue = () => {
    const videoQueue = new Queue('video transcoding', {
        redis: {
            port: process.env.REDIS_PORT,
            host: process.env.REDIS_URL
        }
    });

videoQueue.process(async (job, done) => {
        const data = job.data;
        try {
            job.progress(0);
            global.io.emit('progress', { progress: 0, jobId: data.id });
            const uuid = uuidv1();
            const fileLocation = `./files/${uuid}.mp4`;
            await new Promise((resolve) => {
                ytdl(data.url)
                    .on('progress', (length, downloaded, totallength) => {
                        const progress = (downloaded / totallength) * 100;
                        global.io.emit('progress', { progress, jobId: data.id });
                        if (progress >= 100) {
                            global.io.emit('videoDone', { fileLocation: `${uuid}.mp4`, jobId: data.id });
                            global.io.emit('progress', { progress: 100, jobId: data.id });
                        }
                    })
                    .pipe(fs.createWriteStream(fileLocation))
                    .on('finish', () => {
                        resolve();
                    })
            })
            await models.Job.update({
                status: 'done',
                fileLocation: `${uuid}.mp4`
            }, {
                    where: {
                        id: data.id
                    }
                })
            done();
        }
        catch (ex) {
            console.log(ex);
            job.moveToFailed();
        }
    });
    return videoQueue;
}

module.exports = { createVideoQueue };

Note that we passed in the socket object to send progress back to the client, and that we converted all the asynchronous code to promises so they can be called sequentially. We use ytdl to download YouTube videos. It has a progress event handler which reports progress of the download, which we send back to the client via Socket.io’s broadcast function. This sends messages to all the clients. We will filter out the irrelevant messages on the client side. Any failed jobs will be removed from the queue.

Next, we create our routes. In the routes folder, we add a new file called jobs.js and put:

const express = require('express');
const models = require('../models');
const path = require('path');
const router = express.Router();
const ytdl = require('ytdl-core');
const { createVideoQueue } = require('../queue/video');

router.post('/new', async (req, res) => {
  const url = req.body.url;
  try {
    const isValidUrl = ytdl.validateURL(url);
    if (!isValidUrl) {
      res.status(400);
      return res.send({ error: 'invalid URL' });
    }
    const job = await models.Job.create({
      url,
      status: 'started'
    })
    await createVideoQueue().add({ url, id: job.id });
    return res.send(job);
  }
  catch (ex) {
    console.log(ex);
    res.status(400);
    return res.send({ error: ex });
  }
});

router.get('/file/:fileName', (req, res) => {
  const fileName = req.params.fileName;
  const file = path.resolve(__dirname, `../files/${fileName}`);
  res.download(file);
})

module.exports = router;

We need a route to add new jobs and to download the generated files. We validate the URL submitted before creating the job to minimize errors. In this line:

await createVideoQueue(global.socket).add({ url, id: job.id });

we pass in the global.socket object we created when the client connects to this app in binwww . Note that we don’t wait for the job to be done before returning a response. This is why we need Socket.io, to communicate the results back to the client.

In app.js, we add the initialization code. We add the following code to the file:

require("@babel/register");
require("babel-polyfill");
require('dotenv').config();
const createError = require('http-errors');
const express = require('express');
const path = require('path');
const cookieParser = require('cookie-parser');
const logger = require('morgan');
const bodyParser = require('body-parser')
const cors = require('cors')
const indexRouter = require('./routes/index');
const usersRouter = require('./routes/users');
const jobsRouter = require('./routes/jobs');
const app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use(express.static(path.join(__dirname, 'files')));
app.use(bodyParser.urlencoded({ extended: true }));
app.use(bodyParser.json())
app.use(cors())
app.use('/', indexRouter);
app.use('/users', usersRouter);
app.use('/jobs', jobsRouter);

// catch 404 and forward to error handler
app.use((req, res, next) => {
  next(createError(404));
});

// error handler
app.use((err, req, res, next) => {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get('env') === 'development' ? err : {};

// render the error page
  res.status(err.status || 500);
  res.render('error');
});

module.exports = app;

We add app.use(express.static(path.join(__dirname, ‘files’))); to expose the files folder that we created to the public, and we add:

const jobsRouter = require('./routes/jobs');

and

app.use('/jobs', jobsRouter);

so that clients can access the route we created.

Finally, we create an .env file and put the following:

REDIS_URL='localhost'
REDIS_PORT='6379'
DB_HOST='localhost'
DB_NAME='youtube_app_development'
DB_USERNAME='postgres'
DB_PASSWORD='postgres'

The bull package requires Redis, so we have to install it. To do so, we run the following in Ubuntu or related Linux distributions:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install redis-server
$ sudo systemctl enable redis-server.service
$ sudo service redis-server restart

The first two commands are run to update the package repository references and to update our Linux packages. We run sudo apt-get install redis-server to install Redis, and we run the fourth line to enable Redis on startup. If Redis is not started or needs restarting, we run sudo service redis-server restart.

Note—there is no recent Windows version of Redis, so Linux is required. Now we have everything needed to run the back end.

Building the UI

The back end is done and we can move on to building the UI. We build it with Angular and Angular Material. To get started, we install the Angular CLI by running npm i -g @angular/cli . Then we run ng new frontend in our top-level project folder to create the app. Be sure to choose to include routing and use SCSS for styling when prompted. After that, we run npm i @angular/cdk @angular/material file-saver socket.io-client. The first two packages are Angular Material packages. file-saver helps us download files, and socket.io-client allows us to connect to the back end to get download progress and file location.

In environment.ts, we put:

export const environment = {
  production: false,
  apiUrl: 'http://localhost:3000',
  socketIoUrl: 'http://localhost:3000'
};

Then we create our components and services.

We run ng g component homePage and ng g service video to create our code files.

In video.service.ts, we put:

import { Injectable } from '@angular/core"';
import { HttpClient, HttpHeaders } from '@angular/common/http';
import { environment } from 'src/environments/environment';

@Injectable({
  providedIn: 'root'
})
export class VideoService {

  constructor(
    private http: HttpClient
  ) { }

  addVideoToQueue(data) {
    return this.http.post(`${environment.apiUrl}/jobs/new`, data);
  }

  getVideo(videoUrl: string) {
    return this.http.get<Blob>(videoUrl, {
      headers: new HttpHeaders({
        'accept': 'application/octet-stream',
        'content-type': 'application/json'
      }),
      responseType: 'blob' as 'json'
    })
  }
}

to let our app make requests to add YouTube videos to the queue for download, and we call the getVideo to download. Note that we set the accept header to ‘application/octet-stream’ so that we can download video files.

Next in home-page.component.ts, we put:

import { Component, OnInit } from '@angular/core"';
import { VideoService } from '../video.service';
import { NgForm } from '@angular/forms';
import io from 'socket.io-client';
import { environment } from 'src/environments/environment';
import { saveAs } from 'file-saver';

@Component({
  selector: 'app-home-page',
  templateUrl: './home-page.component.html',
  styleUrls: ['./home-page.component.scss']
})
export class HomePageComponent implements OnInit {
  videoData: any = <any>{};
  progress: number = 0;
  fileLocation: string;
  downloaded: boolean = false;
  jobId: number;
  connected: boolean = false;
  socket;
  getVideoSub;

constructor(
    private videoService: VideoService
  ) { }

ngOnInit() {
      this.addConnectionHandlers();
  }

  addConnectionHandlers() {
    const manager = io.Manager(environment.socketIoUrl);
    manager.on('connect_error', () => {
      this.socket = io.connect(environment.socketIoUrl);
    });

      this.socket = io.connect(environment.socketIoUrl);
    this.socket.on('connect', (data) => {
      this.socket.on('connected', (msg) => {

});

      this.socket.on('progress', (msg) => {
        if (this.jobId != msg.jobId) {
          return;
        }
        this.progress = msg.progress;
        if (msg.progress == 100) {
          this.progress = 0;
        }
      });

      this.socket.on('videoDone', (msg) => {
        if (this.jobId != msg.jobId || this.downloaded) {
          return;
        }
        this.getVideoSub = this.videoService.getVideo(`${environment.apiUrl}/jobs/file/${msg.fileLocation}`)
          .subscribe(res => {
            if (!this.downloaded) {
              saveAs(res, `${msg.fileLocation}.mp4`);
              this.progress = 0;
              this.downloaded = true;
              this.getVideoSub.unsubscribe();
            }
          })
      });
    });
  }

  addVideoToQueue(videoForm: NgForm) {
    this.downloaded = false;
    if (videoForm.invalid) {
      return;
    }
    this.videoService.addVideoToQueue(this.videoData)
      .subscribe(res => {
        this.jobId = (res as any).id;
      }, err => {
        alert('Invalid URL');
      })
  }
}

This provides the logic for the UI to let the user enter their YouTube URLs, watch their video’s download progress, and download it when it’s done. Since we used socket.broadcast.emit in the back end, we have to filter it out in the front end. The back end returns the jobId for the download job, so we can filter out by jobId. We also need to add retry in case the back end app goes down with the setTimeout block in the connect_error handler. We check if the same file has been downloaded before with the this.downloaded flag so it won’t download again. Otherwise, it might try to download too many times, causing freezes and crashes.

In home-page.component.html, we put:

<div class="center">
    <h1>Download Video From YouTube</h1>
</div>
<div id='content'>
    <form #videoForm='ngForm' (ngSubmit)='addVideoToQueue(videoForm)'>
        <mat-form-field>
            <input matInput placeholder="YouTube URL" required #url='ngModel' name='url' [(ngModel)]='videoData.url'
                [disabled]='progress != 0'>
            <mat-error *ngIf="url.invalid && (url.dirty || url.touched)">
                <div *ngIf="url.errors.required">
                    URL is required.
                </div>
            </mat-error>
        </mat-form-field>
        <br>
        <button mat-raised-button type='submit'>Convert</button>
    </form>
    <br>
    <mat-card *ngIf='progress > 0'>
        Downloading: {{progress}}%
    </mat-card>
</div>

to let the user enter their YouTube URL and display progress. Note that we disabled input when a video is downloaded, so that users can’t keep entering new requests.

In home-page.component.scss, we put:

#content {
  width: 95vw;
  margin: 0 auto;
}

to add some padding to the form.

In app-routing.module.ts, we put:

import { NgModule } from '@angular/core"';
import { Routes, RouterModule } from '@angular/router';
import { HomePageComponent } from './home-page/home-page.component';

const routes: Routes = [
  { path: '', component: HomePageComponent }
];

@NgModule({
  imports: [RouterModule.forRoot(routes)],
  exports: [RouterModule]
})
export class AppRoutingModule { }

so that users can see our page.

In app.component.html, we put:

<router-outlet></router-outlet>

so that our page will be displayed. In app.module.ts, we put:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core"';
import { BrowserAnimationsModule } from '@angular/platform-browser/animations';
import {
  MatButtonModule,
  MatCheckboxModule,
  MatInputModule,
  MatMenuModule,
  MatSidenavModule,
  MatToolbarModule,
  MatTableModule,
  MatDialogModule,
  MAT_DIALOG_DEFAULT_OPTIONS,
  MatDatepickerModule,
  MatSelectModule,
  MatCardModule,
  MatFormFieldModule
} from @angular/material;
import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { HomePageComponent } from './home-page/home-page.component';
import { FormsModule } from '@angular/forms';
import { HttpClientModule } from '@angular/common/http';

@NgModule({
  declarations: [
    AppComponent,
    HomePageComponent
  ],
  imports: [
    BrowserModule,
    AppRoutingModule,
    MatButtonModule,
    BrowserAnimationsModule,
    MatButtonModule,
    MatCheckboxModule,
    MatFormFieldModule,
    MatInputModule,
    MatMenuModule,
    MatSidenavModule,
    MatToolbarModule,
    MatTableModule,
    FormsModule,
    HttpClientModule,
    MatDialogModule,
    MatDatepickerModule,
    MatSelectModule,
    MatCardModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

so that we can use Angular Material widgets in our app.

In styles.scss, we put:

/* You can add global styles to this file, and also import other style files */
@import "~@angular/material/prebuilt-themes/indigo-pink.css";
body {
  font-family: "Roboto", sans-serif;
  margin: 0;
}

form {
  mat-form-field {
    width: 95vw;
    margin: 0 auto;
  }
}

.center {
  text-align: center;
}

to include Material Design styles and add some padding to our forms and style for centering text.

In index.html, we put:

<!doctype html>
<html lang="en">

<head>
  <meta charset="utf-8">
  <title>YouTube Download App</title>
  <base href="/">
  <link href="https://fonts.googleapis.com/css?family=Roboto&display=swap" rel="stylesheet">
  <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
</head>

<body>
  <app-root></app-root>
</body>

</html>

to include Material Icons and Roboto font.

JavaScript Mistakes

Common JavaScript Mistakes to Avoid

Post author By John Au-Yeung
Post date August 12, 2020
No Comments on Common JavaScript Mistakes to Avoid

There’re some very common programming mistakes that we want to avoid to keep our code clean and readable.

In this article, we’ll look at them and how to avoid each one.

Too Many Things in One Function

One function should do one thing only, which is whatever is indicated on their own.

If there’s something else, then it should be moved to its own function. Having too many things in one function makes it hard to read and follow.

For example, if there’s a function that gets both the subtotal and the taxes, then they should be split into 2 separate functions since they’re different.

Commented-Out Code

Commented-out code are code that won’t run. Therefore, they shouldn’t be there after we’re done with working on that part of our code.

Once we’re done, we should remove the commented-out or uncomment them.

Either way, the final product shouldn’t have commented out code in it.

Variable Names That Aren’t Descriptive

Naming variables in ways that don’t convey their meaning frustrate readers and ourselves once we forgot about that code we wrote and came back to it.

Therefore, we should name variables with names that describe what they hold.

Instead of writing let x; , write something like let numApples so that we know that our variable will hold the number of apples.

Magic Numbers and String

We shouldn’t have magic numbers and strings. Those are values that appear in multiple and mean the same thing, but it’s not explained explicitly in the code.

For instance, if we have the following code:

for (let i = 0; i < 10; i++) {  
  ///...  
}

Then we don’t know what the 10 means. Instead, we should set it to a named constant so that we know what it means.

For instance, we can write:

const numApples = 10;  
for (let i = 0; i < numApples; i++) {  
  ///...  
}

Now we know that 10 actually means the number of apples.

Messy Code Formatting

Messy code formatting makes code hard to read, so we should clean it up with a linter or a code formatter.

There’re plenty of choices out there, so we can use something like Prettier or ESLint to do the tidying automatically.

Hard Coding Values

We should never hard code values into our code, especially if they’re secrets. Instead, we should keep them as environment variables and read all the values from there.

There’re many ways to do that. For the front end, Angular, React, and Vue, for example, all have places to keep variables to keep variables for different environments in different files.

For the back end, we can use something like the dotenv package to read environment variables from a .env file so that we don’t have to hard code them.

The .env file shouldn’t be checked in, so we don’t have secrets in our code repositories.

Photo by Jens Johnsson on Unsplash

Repetitive Code

The repetitive code is bad. If we change something that repeats, we have to change them in all the places that they’re repeated.

Instead, we should move the common parts into their own file so that they can be used in multiple places.

The do not repeat yourself (DRY) principle applies everywhere. If we’re copying and pasting and using them exactly as is, then we should move them to a shared location.

Not Backing Up Code

We should manage our code with Git so that we can have a local and remote repo for our code.

This way we can keep one copy in a remote location automatically. Also, we can revert bad code easily and fetch code from earlier commits easily.

We can’t do that without version control.

If we don’t have our code backed up, then we may lose everything if something goes wrong.

Complicated Code

Complex code should be simplified so that we can understand them more easily.

We should also break them up into smaller parts so that we can reuse stuff and also have individual parts that are easy to read and test.

Conclusion

These common mistakes can easily be avoided if we’re careful about what we’re doing.

Breaking stuff to small pieces is always good, as with keeping things simple and not repeating things.

Commented out should be gone, and variables and values that aren’t descriptive should also be replaced with something with more meaning.