The Web Dev

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Write Formatted JSON in Node.js

To print formatted JSON in Node apps, we can use the JSON.stringify method with some extra arguments.

For instance, we can write:

JSON.stringify(obj, null, 2)

to add 2 spaces for indentation.

Remove Object from Array with MongoDB

We can remove an object from an array with MongoDB by using the update method.

For instance, we can write:

db.collection.update(
  {'_id': ObjectId("5150a1199fac0e6910000002")},
  { $pull: { "items" : { id: 123 } } },
  false,
  true
);

We pass in the query, which is the ID of the document that has the array.

Then 2nd argument gets the item to update, which is what we want to remove from the items array.

$pull removes the item in the array.

The 3rd argument means we don’t want to do upsert.

The last argument means we process multiple documents.

console.log vs console.info in Node.js

We use the console.log method to log general messages.

console.info is specifically used for logging informational messages.

However, they pretty much do the same thing other than the name and their intended use.

Copy to Clipboard in Node.js

To copy something to the clipboard, we can run a shell command to copy something to the clipboard.

For instance, in Linux, we can write:

const exec = require('child_process').exec;

const getClipboard = (func) => {
  exec('/usr/bin/xclip -o -selection clipboard', (err, stdout, stderr) => {
    if (err || stderr) {
      throw new Error(stderr);
    }
    func(null, stdout);
  });
};

getClipboard((err, text) => {
  if (err) {
    console.error(err);
  }
  console.log(text);
});

We get the clipboard’s content and output it to standard out with the command.

Also, we can use the clipboardy package to read and write from the clipboard.

For instance, we can write:

const clipboardy = require('clipboardy');
clipboardy.writeSync('copy me');

to copy to the clipboard.

To paste from the clipboard, we can write:

const clipboardy = require('clipboardy');
`clipboardy.readSync();`

Coordinating Parallel Execution in Node.js

To coordinate parallel execution of code in a Noe app, we can use the async library.

For instance, we can write:

const async = require('async');
const fs = require('fs');
const A = (c) => { fs.readFile('file1', c) };
const B = (c) => { fs.readFile('file2', c) };
const C = (result) => {
  // get all files and use them
}

async.parallel([A, B], C);

We have 3 read file processes and we group the first 2 together so that we can run them together in parallel.

Then once they’re both done, we run function C .

Then we have full control of which ones to run in parallel and which one to run in series with the parallel method.

Require and Functions

If we want to call require to import a function.

Then we’ve to assign the function to module.exports .

For instance, if we have:

app/routes.js

module.exports = (app, passport) => {
  // ...
}

Then we can write:

require('./app/routes')(app, passport);

to call import the function and call it immediately.

async/await and ES6 yield with Generators

async and await are very closely related to generators.

They are just generators that always yield promises.

They’re compiled to generators with Babel.

async and await always use yield .

It’s used to unwrap the yielded values as promises and pass the resolved value to the async function.

We should use async and await for chaining promises since we can chain them as if the code is synchronous.

However, it can only return promises.

async and await is an abstraction built on top of generators to make working with them easier.

Get Data Out of a Node.js HTTP Get Request

We can make a GET request with the http module by using its get method.

For instance, we can write:

const http = require('http');

const options = {
  //...
};

http.get(options, (response) => {
  response.setEncoding('utf8')
  response.on('data', console.log)
  response.on('error', console.error)
})

response is a read stream, so we’ve to listen to the data event to get the data.

And we listen to the error event to get the error.

Conclusion

We can use the http module to make GET requests.

To remove an item from an array in a MongoDB document, we can use the update method with the $pull command.

The async module lets us deal with parallel and serial execution of functions.

Also, we can manipulate the clipboard by using shell commands or the 3rd party libraries.

async and await are abstraction on top of generators.

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Fix ‘ReferenceError: fetch is not defined’ Error

The Fetch API isn’t implemented in Node.

If we want to use it, we can use the node-fetch library.

To install it, we run:

npm i node-fetch --save

Then we can use it by writing:

const fetch = require("node-fetch");

There’s also the cross-fetch library.

We can install it by running:

npm install --save cross-fetch

Then we can write:

import fetch from 'cross-fetch';

fetch('https//example.com')
  .then(res => {
    if (res.status >= 400) {
      throw new Error("error");
    }
  })

Avoid Long Nesting of Asynchronous Functions in Node.js

We can reorganize our nested async functions by rewriting it as multiple functions.

For example, we can write:

http.createServer((req, res) => {
  getSomeData(client, (someData) => {
    getMoreData(client, function(moreData) => {
         //
      });
   });
});

to:

const moreDataParser = (moreData) => {
   // date parsing logic
};

const dataParser = (data) => {
  getMoreData(client, moreDataParser);
};

const callback = (req, res) => {
  getSomeData(client, dataParser);
};

http.createServer(callback);

We move the anonymous functions into their own named functions so we can call them one by one.

Use of the “./bin/www” in Express 4.x

./bin/www is the file with the entry point of the Express app.

In the start script, we have:

"scripts": {
  "start": "node ./bin/www",
}

to start the app.

Get and Set a Single Cookie with Node.js HTTP Server

If we’re using the http module to create a server, we can parse a cookie by splitting the string .

For instance, we can write:

const http = require('http');

const parseCookies = (cookieStr) => {
  const list = {};
  const cookies = cookieStr && cookieStr.split(';');
  for (const cookie of cookies) {
    const [key, value] = cookie.split('=');
    list[key.trim()] = decodeURI(value);
  });
  return list;
}

http.createServer((request, response) => {
  const cookies = parseCookies(request.headers.cookie);
  //...
}).listen(8888);

We get the cookie string with request.headers.cookie .

Then we pass that into the parseCookie function.

It splits the cookie string by the semicolon.

Then we loop through them to split the parts by the = sign.

The left side is the key and the right is the value.

To set a cookie, we can write:

const http = require('http');

`http.createServer((request, response) => {
  ` response.writeHead(200, {
    'Set-Cookie': 'foo=bar',
    'Content-Type': 'text/plain'
  });
  response.end('hellon');`
}).listen(8888);`

We use writeHead to set headers.

'Set-Cookie' sets the response cookie.

'Content-Type' sets the content type of the response.

Then we use response.end to return the response body.

Get Request Path with the Express req Object

We can use the req.originalUrl property to get the request path of the Express req object.

Copy Folder Recursively

We can copy a folder recursively with the node-fs-extra package.

Its copy method lets us do the copying recursively.

For instance, we can write:

fs.copy('/tmp/oldDir', '/tmp/newDir', (err) => {
  if (err) {
    console.error(err);
  } else {
    console.log("success");
  }
});

The argument is the old directory path.

The 2nd is the new directory path.

The last argument is the callback that’s run when the copy operation ends.

err has the error.

Return the Current Timestamp with Moment.js

We can return the current timestamp in Moment.js with a few methods.

We can write:

const timestamp = moment();

or:

const timestamp = moment().format();

or:

const timestamp = moment().unix();

or:

`const timestamp =` moment().valueOf();

to get the timestamp.

We can also convert it to a Date instance and call getTime :

const timestamp = moment().toDate().getTime();

Fix the ‘TypeError: path must be absolute or specify root to res.sendFile’ Error

This error can be fixed if we pass in an absolute path to res.sendFile .

To do that, we write:

res.sendFile('index.html', { root: __dirname });

or:

const path = require('path');
res.sendFile(path.join(__dirname, '/index.html'));

Either way, we get the absolute path which is required by res.sendFile .

Conclusion

We’ve to parse cookies manually if we use http to create our server.

Fetch API isn’t included with Node’s standard library.

res.sendFile only takes an absolute path.

We can rewrite nested callbacks by reorganizing them into their own functions.

Moment returns the current timestamp with a few functions.

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Converting a Buffer into a ReadableStream in Node.js

We can convert a buffer to a readable stream with the ReadableStreamBuffer constructor.

For instance, we can write:

const { ReadableStreamBuffer } = require('stream-buffers');
const readableStreamBuffer = new ReadableStreamBuffer({
  frequency: 10,
  chunkSize: 2048
});

readableStreamBuffer.put(aBuffer);

where aBuffer is the buffer we want to convert.

frequency is the frequency in which the chunks are pumped out.

The size can be changed in the constructor with the chunkSize property.

Scrape Web Pages in Real-Time with Node.js

We can use the cheerio library to scrape web pages in real-time.

To install it, we run:

npm install cheerio

Then we can use it by writing:

const cheerio = require('cheerio');
const $ = cheerio.load('<h1 class="title">Hello world</h1>');

$('h1.title').text('Hello James');
$('h1').addClass('welcome');

$.html();

We can get HTML’s content and manipulate it as we do with jQuery.

We can combine this with an HTTP client like Axios to get the HTML.

Then we can use Cheerio to parse its content.

For instance, we can write:

const axios = require('axios');
const cheerio = require('cheerio');

axios.get('https://example.com')
.then(({ data }) => {
  const $ = cheerio.load(data);
  const text = $('h1').text();
  console.log(text);
})

We make a GET request with Axios to a website.

Then we use cheerio to parse the data with cheerio.load .

Then we get the content of h1 with the text method.

Any selector can be used to get data.

Generate an MD5 file Hash in JavaScript

We can generate an MD5 hash with the crypto-js package.

To install it, we can run:

npm install crypto-js

Then we can write:

import MD5 from "crypto-js/md5";
const md5Hash = MD5("hello world");

to generate the hash.

Server-Side Browser Detection with Node.js

We can get the user-agent header to get the user agent string from the request.

To parse the string, we can use the ua-parser-js package.

To install it, we run:

npm install ua-parser-js

In our Express app, we can create our own middleware to check the user agent:

const UAParser = require('ua-parser-js');

const checkBrowser = (req, res, next) => {
  const parser = new UAParser();
  const ua = req.headers['user-agent'];
  const browserName = parser.setUA(ua).getBrowser().name;
  const fullBrowserVersion = parser.setUA(ua).getBrowser().version;

  console.log(browserName);
  console.log(fullBrowserVersion);
  next();
}

app.all(/*/, checkBrowser);

We get the user-agent header with req.headers[‘user-agent’] .

Then we use the UAParser constructor to parse the user agent string.

We call getBrowser to get the browser data.

name has the browser name and version has the version.

Best Way to Store Database Config in an Express App

We can store the config in a configuration file.

For instance, we can write:

const fs = require('fs');
const configPath = './config.json';
const configFile = fs.readFileSync(configPath, 'utf-8')
const parsedConfig = JSON.parse(configFile);
exports.storageConfig = parsedConfig;

We called readFileSync to read config.json .

Then we parse the data from the JSON string.

And we export the config in the last line.

Promisify Node’s child_process.exec and child_process.execFile Functions

We can convert the child_process exec and execFile methods with Bluebird.

For instance, we can write:

const util = require('util');
const exec = util.promisify(require('child_process').exec);

const listFiles = async () => {
  try {
    const { stdout, stderr } = await exec('ls');
    console.log('stdout:', stdout);
    console.log('stderr:', stderr);
  } catch (e) {
    console.error(e);
  }
}

listFiles();

We used the util library’s promisify method to convert the exec method to a promise.

Then we can call the promisified exec method with the ls command.

And then we get the full output with stdout and stderr .

stdout has the results. stderr has the error output.

Conclusion

We can convert a buffer into a readable stream.

We can scrape web pages with an HTTP client and cheerio.

crypto-js has the MD5 method to create an MD5 hash.

Methods from the child_process can be converted to promises.

We can parse the user agent string on the server-side and parse the browser version data.

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Best Way to Limit Concurrency when Using ES6’s Promise.all()

To limit concurrency when using Promise.all , we can use the es6-promise-pool package.

For instance, we can write:

const promiseProducer = function * () {
  for (let count = 1; count <= 5; count++) {
    yield delayValue(count, 1000)
  }
}

const promiseIterator = generatePromises();
const pool = new PromisePool(promiseIterator, 3);

pool.start()
  .then(function () {
    console.log('Complete')
  })

We get the values for the promises in the promiseProducer generator function.

Then we call the generator function so that we get an iterator.

We then pass that into the PromisePool constructor.

3 is the number of concurrent promises allowed.

Finally, we call pool.start to invoke the promises.

Remove All Files from Directory without Removing Directory in Node.js

We can remove all files in a directory without removing the directory by using the readdir method to read the directory’s contents.

Then we can loop through each file and call unlink on them.

For instance, we can write:

const fs = require('fs');
const path = require('path');

const directory = '/foo/bar';

fs.readdir(directory, (err, files) => {
  if (err){
    return console.log(err);
  }

  for (const file of files) {
    fs.unlink(path.join(directory, file), err => {
      if (err) {
        console.log(err);
      }
    });
  }
});

We call readdir to read the directory’s content. Then we use the for-of loop to loop through the items.

Then we call unlink to remove each file.

Redis and Node.js

We can access a Redis database by using the redis library.

To install it, we run:

npm install redis

Then in our Node app, we can use the Redis client by writing:

const redis = require("redis");
const client = redis.createClient();

client.on("error", (err) => {
  console.log(err);
});

client.set("key", "val", redis.print);
client.hset("hash key", "foo", "value 1", redis.print);
client.hset(["hash key", "bar", "value 2"], redis.print);
client.hkeys("hash key", (err, replies) => {
  replies.forEach((reply, i) => {
    console.log(i, reply);
  });
  client.quit();
});

We use the redis module and call createClient to create the client.

Then we attach a listener to listen to the 'error' event.

Then we call set to set the key and value.

We can use hset to set a hash key with their own value.

redis.print prints the values.

hkeys get the hash keys and print the index and results.

Parsing Query String in Node.js

We can parse the query string in Node.jhs with the url module.

For instance, if we’re using the http module to create our server, we can write:

const http = require('http');
const url = require('url');

const server = http.createServer((request, response) => {
  const queryData = url.parse(request.url, true).query;
  response.writeHead(200, { "Content-Type": "text/plain" });

  if (queryData.name) {
    response.end(queryData.name);
  } else {
    response.end("hello worldn");
  }
});

server.listen(8000);

We call url.parse to parse the query string that’s in the URL.

The request URL is stored in the request.url property.

The parsed result is assigned to the queryData constant.

Then we parse the name query parameter if it exists.

If make a request with query string ?name=joe , then queryData.name will be 'joe' .

Preventing SQL Injection in Node.js

To prevent SQL injection in Node apps, we should use a library that lets to make parameterized queries.

For instance, with the node-mysql-native library, we can write:

const userId = 5;
const query = connection.query('SELECT * FROM users WHERE id = ?', [userId], (err, results) => {
  //...
});

The ? indicates the parameter that we can pass in. The 2nd argument has the parameters.

The callback is called when the query is done.

results has the results.

Sanitization is done by the following rules:

• numbers are untouched
• booleans are converted to strings
• date objects are converted to YYYY-mm-dd:ii:ss strings
• buffers are converted to hex strings
• strings are escaped
• arrays are turned into lists
• nested arrays are turned into grouped lists
• objects are turned into key = 'val' pairs
• undefined or null are converted to NULL
• NaN and Infinity are left as is. They aren’t supported by MySQL and will error out if these are present.

Conclusion

We should use a library to make database queries to prevent SQL injection.

Also, we can use a library to limit the concurrency of parallel promises.

We can use Node’s Redis library to access Redis databases.

The url package can parse query strings.

ind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Storing passwords with Node.js and MongoDB

We can store passwords in a MongoDB document with the bcrypt library.

For instance. we can write:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const bcrypt = require('bcrypt');
const SALT_WORK_FACTOR = 10;

const UserSchema = new Schema({
  username: {
    type: String,
    required: true,
    index: {
      unique: true
    }
  },
  password: { type: String, required: true }
});

module.exports = mongoose.model('User', UserSchema);

We create the UserSchema with the Schema constructor.

It has the username and password string fields.

Then we can use the pre method to hash and salt the password before it’s saved.

To do that, we write:

UserSchema.pre('save', function(next) {
  const user = this;
  if (!user.isModified('password')){
    return next();
  }

  bcrypt.genSalt(SALT_WORK_FACTOR, (err, salt) => {
    if (err) {
       return next(err);
    }

    bcrypt.hash(user.password, salt, (err, hash) => {
      if (err) {
        return next(err);
      }
      user.password = hash;
      next();
    });
  });
});

We override the plain text password with the hash and salted one/

To do that, we used the genSalt method to create the salt.

Then we create the hash from the password.

We passed our generated salt into the hashmethod.

Then we set the password property to the hash .

And finally, we call next to continue with saving the user data.

Fix Protocol “https:” not supported. Expected “http:” Error

We can fix this error, which occurs when we try to make HTTPS requests with http .

We’ve to use https.get instead of http.get to make a request to a URL that starts with https .

Mongoose Connect Error Callback

We can pick up the error in the callback we pass into connect .

For example, we can write:

mongoose.connect('mongodb://localhost/db', (err) => {
  if (err) throw err;
});

err has the error object that’s set when there’s an error.

Run Some Code and Then Go Into Node REPL

We can use the repl module to start the REPL.

For instance, we can write:

const repl = require("repl");
const r = repl.start("node> ");

We call start the REPL with the command prompt of our choice.

Get the Node.js Version at Runtime

We can get the Node version at runtime with the process.version property.

To get more version information, we can use the process.versions property.

Then we get something like:

{
  node: '12.16.3',
  v8: '7.8.279.23-node.35',
  uv: '1.34.2',
  zlib: '1.2.11',
  brotli: '1.0.7',
  ares: '1.16.0',
  modules: '72',
  nghttp2: '1.40.0',
  napi: '5',
  llhttp: '2.0.4',
  http_parser: '2.9.3',
  openssl: '1.1.1g',
  cldr: '36.0',
  icu: '65.1',
  tz: '2019c',
  unicode: '12.1'
}

Unzip (Decompress) a NodeJS Request’s Module gzip Response Body

We can decompress a gzipped file by using the zlib library.

For instance, we can write:

const http = require("http"),
const zlib = require("zlib");

const getGzipped = (url, callback) => {
  const buffer = [];
  http.get(url, (res) => {
    const gunzip = zlib.createGunzip();
    res.pipe(gunzip);

    gunzip.on('data', (data) => {
      buffer.push(data.toString())
    })
    .on("end", () => {
      callback(null, buffer.join(""));
    })
    .on("error", (e) => {
      callback(e);
    })
  })
  .on('error', (e) => {
    callback(e);
  });
}

getGzipped(url, (err, data) => {
   console.log(data);
});

We created the getGzipped function by making the GET request to get a zipped file with the http.get method.

Then call createGunzip to create the gunzip object that we can pipe the res stream object to.

Then we can listen to the data event to get the data.

We call push on the buffer object to gather the data.

Then when the end event is emitted, we call the callback to send the data to the callback.

When an error event is emitted from the gunzip or http.get methods, we’ll call the callback to send the error.

Then we can use the function to get the error or the unzipped file’s data.

Conclusion

We can get the version with the process object.

To unzip files obtained from the http.get method, we can unzip it with zlib .

We’ve to hash and salt the password manually if we want to store it securely.

We can use the repl module to access the REPL.