Node.js Tips — Streams, Scraping, and Promisifying Functions

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Converting a Buffer into a ReadableStream in Node.js

We can convert a buffer to a readable stream with the ReadableStreamBuffer constructor.

For instance, we can write:

const { ReadableStreamBuffer } = require('stream-buffers');
const readableStreamBuffer = new ReadableStreamBuffer({
  frequency: 10,
  chunkSize: 2048
});

readableStreamBuffer.put(aBuffer);

where aBuffer is the buffer we want to convert.

frequency is the frequency in which the chunks are pumped out.

The size can be changed in the constructor with the chunkSize property.

Scrape Web Pages in Real-Time with Node.js

We can use the cheerio library to scrape web pages in real-time.

To install it, we run:

npm install cheerio

Then we can use it by writing:

const cheerio = require('cheerio');
const $ = cheerio.load('<h1 class="title">Hello world</h1>');

$('h1.title').text('Hello James');
$('h1').addClass('welcome');

$.html();

We can get HTML’s content and manipulate it as we do with jQuery.

We can combine this with an HTTP client like Axios to get the HTML.

Then we can use Cheerio to parse its content.

For instance, we can write:

const axios = require('axios');
const cheerio = require('cheerio');

axios.get('https://example.com')
.then(({ data }) => {
  const $ = cheerio.load(data);
  const text = $('h1').text();
  console.log(text);
})

We make a GET request with Axios to a website.

Then we use cheerio to parse the data with cheerio.load .

Then we get the content of h1 with the text method.

Any selector can be used to get data.

Generate an MD5 file Hash in JavaScript

We can generate an MD5 hash with the crypto-js package.

To install it, we can run:

npm install crypto-js

Then we can write:

import MD5 from "crypto-js/md5";
const md5Hash = MD5("hello world");

to generate the hash.

Server-Side Browser Detection with Node.js

We can get the user-agent header to get the user agent string from the request.

To parse the string, we can use the ua-parser-js package.

To install it, we run:

npm install ua-parser-js

In our Express app, we can create our own middleware to check the user agent:

const UAParser = require('ua-parser-js');

const checkBrowser = (req, res, next) => {
  const parser = new UAParser();
  const ua = req.headers['user-agent'];
  const browserName = parser.setUA(ua).getBrowser().name;
  const fullBrowserVersion = parser.setUA(ua).getBrowser().version;

  console.log(browserName);
  console.log(fullBrowserVersion);
  next();
}

app.all(/*/, checkBrowser);

We get the user-agent header with req.headers[‘user-agent’] .

Then we use the UAParser constructor to parse the user agent string.

We call getBrowser to get the browser data.

name has the browser name and version has the version.

Best Way to Store Database Config in an Express App

We can store the config in a configuration file.

For instance, we can write:

const fs = require('fs');
const configPath = './config.json';
const configFile = fs.readFileSync(configPath, 'utf-8')
const parsedConfig = JSON.parse(configFile);
exports.storageConfig = parsedConfig;

We called readFileSync to read config.json .

Then we parse the data from the JSON string.

And we export the config in the last line.

Promisify Node’s child_process.exec and child_process.execFile Functions

We can convert the child_process exec and execFile methods with Bluebird.

For instance, we can write:

const util = require('util');
const exec = util.promisify(require('child_process').exec);

const listFiles = async () => {
  try {
    const { stdout, stderr } = await exec('ls');
    console.log('stdout:', stdout);
    console.log('stderr:', stderr);
  } catch (e) {
    console.error(e);
  }
}

listFiles();

We used the util library’s promisify method to convert the exec method to a promise.

Then we can call the promisified exec method with the ls command.

And then we get the full output with stdout and stderr .

stdout has the results. stderr has the error output.

Conclusion

We can convert a buffer into a readable stream.

We can scrape web pages with an HTTP client and cheerio.

crypto-js has the MD5 method to create an MD5 hash.

Methods from the child_process can be converted to promises.

We can parse the user agent string on the server-side and parse the browser version data.