Categories
Node.js Tips

Node.js Tips — Redis, Query Strings, Concurrency of Promises, SQL Injections

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Best Way to Limit Concurrency when Using ES6’s Promise.all()

To limit concurrency when using Promise.all , we can use the es6-promise-pool package.

For instance, we can write:

const promiseProducer = function * () {
  for (let count = 1; count <= 5; count++) {
    yield delayValue(count, 1000)
  }
}

const promiseIterator = generatePromises();
const pool = new PromisePool(promiseIterator, 3);

pool.start()
  .then(function () {
    console.log('Complete')
  })

We get the values for the promises in the promiseProducer generator function.

Then we call the generator function so that we get an iterator.

We then pass that into the PromisePool constructor.

3 is the number of concurrent promises allowed.

Finally, we call pool.start to invoke the promises.

Remove All Files from Directory without Removing Directory in Node.js

We can remove all files in a directory without removing the directory by using the readdir method to read the directory’s contents.

Then we can loop through each file and call unlink on them.

For instance, we can write:

const fs = require('fs');
const path = require('path');

const directory = '/foo/bar';

fs.readdir(directory, (err, files) => {
  if (err){
    return console.log(err);
  }

  for (const file of files) {
    fs.unlink(path.join(directory, file), err => {
      if (err) {
        console.log(err);
      }
    });
  }
});

We call readdir to read the directory’s content. Then we use the for-of loop to loop through the items.

Then we call unlink to remove each file.

Redis and Node.js

We can access a Redis database by using the redis library.

To install it, we run:

npm install redis

Then in our Node app, we can use the Redis client by writing:

const redis = require("redis");
const client = redis.createClient();

client.on("error", (err) => {
  console.log(err);
});

client.set("key", "val", redis.print);
client.hset("hash key", "foo", "value 1", redis.print);
client.hset(["hash key", "bar", "value 2"], redis.print);
client.hkeys("hash key", (err, replies) => {
  replies.forEach((reply, i) => {
    console.log(i, reply);
  });
  client.quit();
});

We use the redis module and call createClient to create the client.

Then we attach a listener to listen to the 'error' event.

Then we call set to set the key and value.

We can use hset to set a hash key with their own value.

redis.print prints the values.

hkeys get the hash keys and print the index and results.

Parsing Query String in Node.js

We can parse the query string in Node.jhs with the url module.

For instance, if we’re using the http module to create our server, we can write:

const http = require('http');
const url = require('url');

const server = http.createServer((request, response) => {
  const queryData = url.parse(request.url, true).query;
  response.writeHead(200, { "Content-Type": "text/plain" });

  if (queryData.name) {
    response.end(queryData.name);
  } else {
    response.end("hello worldn");
  }
});

server.listen(8000);

We call url.parse to parse the query string that’s in the URL.

The request URL is stored in the request.url property.

The parsed result is assigned to the queryData constant.

Then we parse the name query parameter if it exists.

If make a request with query string ?name=joe , then queryData.name will be 'joe' .

Preventing SQL Injection in Node.js

To prevent SQL injection in Node apps, we should use a library that lets to make parameterized queries.

For instance, with the node-mysql-native library, we can write:

const userId = 5;
const query = connection.query('SELECT * FROM users WHERE id = ?', [userId], (err, results) => {
  //...
});

The ? indicates the parameter that we can pass in. The 2nd argument has the parameters.

The callback is called when the query is done.

results has the results.

Sanitization is done by the following rules:

  • numbers are untouched
  • booleans are converted to strings
  • date objects are converted to YYYY-mm-dd:ii:ss strings
  • buffers are converted to hex strings
  • strings are escaped
  • arrays are turned into lists
  • nested arrays are turned into grouped lists
  • objects are turned into key = 'val' pairs
  • undefined or null are converted to NULL
  • NaN and Infinity are left as is. They aren’t supported by MySQL and will error out if these are present.

Conclusion

We should use a library to make database queries to prevent SQL injection.

Also, we can use a library to limit the concurrency of parallel promises.

We can use Node’s Redis library to access Redis databases.

The url package can parse query strings.

Categories
Node.js Tips

Node.js Tips — Unzipping Files, Storing Passwords, and REPLs

ind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Storing passwords with Node.js and MongoDB

We can store passwords in a MongoDB document with the bcrypt library.

For instance. we can write:

const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const bcrypt = require('bcrypt');
const SALT_WORK_FACTOR = 10;

const UserSchema = new Schema({
  username: {
    type: String,
    required: true,
    index: {
      unique: true
    }
  },
  password: { type: String, required: true }
});

module.exports = mongoose.model('User', UserSchema);

We create the UserSchema with the Schema constructor.

It has the username and password string fields.

Then we can use the pre method to hash and salt the password before it’s saved.

To do that, we write:

UserSchema.pre('save', function(next) {
  const user = this;
  if (!user.isModified('password')){
    return next();
  }

  bcrypt.genSalt(SALT_WORK_FACTOR, (err, salt) => {
    if (err) {
       return next(err);
    }

    bcrypt.hash(user.password, salt, (err, hash) => {
      if (err) {
        return next(err);
      }
      user.password = hash;
      next();
    });
  });
});

We override the plain text password with the hash and salted one/

To do that, we used the genSalt method to create the salt.

Then we create the hash from the password.

We passed our generated salt into the hashmethod.

Then we set the password property to the hash .

And finally, we call next to continue with saving the user data.

Fix Protocol “https:” not supported. Expected “http:” Error

We can fix this error, which occurs when we try to make HTTPS requests with http .

We’ve to use https.get instead of http.get to make a request to a URL that starts with https .

Mongoose Connect Error Callback

We can pick up the error in the callback we pass into connect .

For example, we can write:

mongoose.connect('mongodb://localhost/db', (err) => {
  if (err) throw err;
});

err has the error object that’s set when there’s an error.

Run Some Code and Then Go Into Node REPL

We can use the repl module to start the REPL.

For instance, we can write:

const repl = require("repl");
const r = repl.start("node> ");

We call start the REPL with the command prompt of our choice.

Get the Node.js Version at Runtime

We can get the Node version at runtime with the process.version property.

To get more version information, we can use the process.versions property.

Then we get something like:

{
  node: '12.16.3',
  v8: '7.8.279.23-node.35',
  uv: '1.34.2',
  zlib: '1.2.11',
  brotli: '1.0.7',
  ares: '1.16.0',
  modules: '72',
  nghttp2: '1.40.0',
  napi: '5',
  llhttp: '2.0.4',
  http_parser: '2.9.3',
  openssl: '1.1.1g',
  cldr: '36.0',
  icu: '65.1',
  tz: '2019c',
  unicode: '12.1'
}

Unzip (Decompress) a NodeJS Request’s Module gzip Response Body

We can decompress a gzipped file by using the zlib library.

For instance, we can write:

const http = require("http"),
const zlib = require("zlib");

const getGzipped = (url, callback) => {
  const buffer = [];
  http.get(url, (res) => {
    const gunzip = zlib.createGunzip();
    res.pipe(gunzip);

    gunzip.on('data', (data) => {
      buffer.push(data.toString())
    })
    .on("end", () => {
      callback(null, buffer.join(""));
    })
    .on("error", (e) => {
      callback(e);
    })
  })
  .on('error', (e) => {
    callback(e);
  });
}

getGzipped(url, (err, data) => {
   console.log(data);
});

We created the getGzipped function by making the GET request to get a zipped file with the http.get method.

Then call createGunzip to create the gunzip object that we can pipe the res stream object to.

Then we can listen to the data event to get the data.

We call push on the buffer object to gather the data.

Then when the end event is emitted, we call the callback to send the data to the callback.

When an error event is emitted from the gunzip or http.get methods, we’ll call the callback to send the error.

Then we can use the function to get the error or the unzipped file’s data.

Conclusion

We can get the version with the process object.

To unzip files obtained from the http.get method, we can unzip it with zlib .

We’ve to hash and salt the password manually if we want to store it securely.

We can use the repl module to access the REPL.

Categories
Node.js Tips

Node.js Tips — Run Commands, Response Headers, Read Files, and More

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Node.js Execute System Command Synchronously

We can run system commands synchronously with the child_process module.

For instance, we can write:

const execSync = require('child_process').execSync;
const code = execSync('ls');

We run execSync to run the command we want.

Get Rid of header X-Powered-By:Express

We can remove the X-Powered-By header by using the app.disable method.

For instance, we can write:

app.disable('x-powered-by');

to stop the x-powered-by header from being added to the response.

We can also write:

app.set('x-powered-by', false);

to do the same thing.

Add a Custom Script to my package.json File that runs a JavaScript File

We can add the script to the package.json file’s scripts section.

For instance, we can write:

"scripts": {
  "start": "node app.js",
},

Then we can run npm start to run it.

Generate Unique ID with Node.js

To make creating unique IDs easy, we can add the uuid package to help us create UUIDs.

To install it, we run:

npm install uuid

Then we can use it by writing:

const { v1: uuidv1 } = require('uuid');`
const id = uuidv1();`

to create a v1 UUID.

And we can write:

const { v4: uuidv4 } = require('uuid');`
const id = uuidv4();`

to create a v4 UUID.

We can also use the crypto module to create a unique ID. However, it won’t be a UUID.

For instance, we can write:

const crypto = require("crypto");
const id = crypto.randomBytes(16).toString("hex");

id would be a 32-bit character string.

Replace a String in a File with Node.js

We can replace the content with new content with readFile and writeFile .

For instance, we can write:

const fs = require('fs');
const someFile = './file.txt';

fs.readFile(someFile, 'utf8', (err, data) => {
  if (err) {
    return console.log(err);
  }

  const result = data.replace('old string', 'new string');
  fs.writeFile(someFile, result, 'utf8', (err) => {
    if (err) {
      return console.log(err);
    }
  });
});

someFile is the path of the file.

We call fs.readFile to read the file with the UTF-8 encoding.

In the callback, we get the data , which has the content.

We call replace to replace 'old string' in data with 'new string' .

Then we call writeFile to write the file with the new content.

We also need the utf8 encoding to indicate that it’s a text file.

err will have the error object if it’s encountered in both callbacks.

Fix the ‘process.env.NODE_ENV is undefined’ Error

We can set the value of NODE_ENV by running:

SET NODE_ENV=development

in Windows or:

export NODE_ENV=development

in Linux to set the NODE_ENV environment variable.

Then process.env.NODE_ENV is set to 'development' when we run our app after those commands are run.

Also, we can put a script in the scripts section to set the NODE_ENV before running the script.

For instance, we can write:

"scripts": {
  "start": "set NODE_ENV=development && node app.js"
}

in package.json .

Set Response Header on Express.js Assets

We can call res.set to set the header on Express assets.

For instance, we can write:

res.set('Content-Type', 'text/plain');

where the first argument is the header key and the 2nd is the header value.

We can also pas sin an object to set multiple headers:

res.set({
  'Content-Type': 'text/plain',
  'Content-Length': '123'
})

We set the Content-Type and Content-Length all at once.

The res.header method is an alias of res.set .

To add more headers to the existing response header, we can use the res.append method.

For instance, we can write:

res.append('Set-Cookie', 'foo=bar; Path=/; HttpOnly')

We appended a response header for the cookie.

Read a Text File into an Array in a Node App

To read a text file into an array in a Node app, we can use the readFile method.

For example, we can write:

const fs = require('fs');

fs.readFile('file.txt', (err, data) => {
  if (err) {
    return console.log(err);
  }
  const array = data.toString().split("n");
  for(const a of array) {
    console.log(a);
  }
});

To read it in a synchronous way, we can use readFileSync :

constfs = require('fs');

const array = fs.readFileSync('file.txt').toString().split("n");
for(const a of array) {
  console.log(a);
}

Conclusion

We can set environment variables before running a Node app.

Also, we can read files in various ways.

We can use the child_process module to run commands in a Node app.

Response headers can be set in various ways.

Categories
Node.js Tips

Node.js Tips — MongoDB, Express and Socket.io, and Postgres

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Get the _id of Inserted Document in Mongo Database in NodeJS

We can get the _id of the inserted document in the Mongo database from the callback that’s called by the insert method.

For instance, we can write:

collection.insert(obj, (err, docsInserted) =>{
  console.log(docsInserted);
});

We pass in the obj to the collection .

Then we get the docsInserted parameter’s value to get the whole object including the ID with the _id.

Notify Node App when Data Values Change in Redis

We can use the Redis client for Node to watch for data value changes.

For instance, we can write:

const redis = require("redis");
const client = redis.createClient();

client.subscribe("pubsub");
client.on("message", (channel, message) => {
  console.log(channel, message);
});

We just call createClient to create a Redis client instance.

Then we subscribe to to the pubsub action to listen to published changes.

Finally, we listen to the 'message' event to get the messages.

Delete Query with Sequelize.js

We can make a delete query with Sequelize by writing:

Model.destroy({
  where: {
    id: 123
  }
})

where Model is the model class of the data table that has the entry we want to remove.

Then we call destroy to delete the entry.

where has the where. id is the ID field.

It can be replaced with other criteria.

Make Express.js App Work on a Different Port

We don’t have to hard code the port that the Express app listens to.

Instead, we can read the value from an environment variable.

For instance, we can write:

app.js

const express = require("express");
const app = express();

app.set('port', process.env.PORT || 3000);

app.get('/', (req, res) => {
  res.send('hello world');
});

app.listen(app.get('port'));

Then we get the port from the environment variables instead of hard-coding the port.

process.env.PORT is the value of the PORT environment variable.

Now we can set the port and run our app at the same time by running:

PORT=8080 node app.js

Connect to a Postgres Database with the Node Postgres Module

To connect to a Postgres database with the Node Postgres module, we can use the pg.connect method.

For instance, we can write:

module.exports = {
  query(text, values, cb) {
    pg.connect((err, client, done) => {
      client.query(text, values, (err, result) => {
        done();
        cb(err, result);
      })
    });
  }
}

We make use of the pg.connect method to connect to the database.

Then we use the client object, which is the database client object, to make a query.

text has the parameterized query.

values has the values for the query.

The callback is called when the query is finished.

result has the results.

We call our cb callback so that the results can be obtained when we call the query method.

Using socket.io in Express 4 and express-generator’s /bin/www

We can use socket.io by adding the io object to the app.js file:

const express = require("express");
const socketIo = require("socket.io");
const app = express();
const io = socketIo();
app.io = io;

io.on("connection", ( socket ) => {
  console.log('connected');
});

module.exports = app;

We attach the io object as the property of app .

We listen to the 'connection' event with the on method.

Then we can use it by writing the following in bin/www :

//...
const server = http.createServer(app);

const io = app.io;
io.attach(server);

We import the app object before calling createServer and then call io.attach to attach the socket.io listeners to the Express app.

Then in our route file, we can write:

const app = require('express');

module.exports = (io) => {
  const router = app.Router();

  io.on('connection', (socket) => {
    //...
  });
  return router;
}

Then we change app.js to:

const express = require("express");
const socketIo = require("socket.io");
const app = express();
const io = socketIo();
app.io = io;
io.on("connection", ( socket ) => {
  console.log('connected');
});

const routes = require('./routes/index')(io);

//...

module.exports = app;

Conclusion

We can get the ID of the inserted MongoDB document from the insert callback.

We can incorporate socket.io into the Express app generated by the express-generator by attaching the io object to the app .

Also, we can use the Postgres database with Node apps by connecting to the database and using the client object returned after creating the connection.

Categories
Node.js Tips

Node.js Tips — Global Objects, Synchronous Requests, Shared Code for Tests

Like any kind of apps, there are difficult issues to solve when we write Node apps.

In this article, we’ll look at some solutions to common problems when writing Node apps.

Synchronous Requests in Node.js

We can make synchronous requests in Node apps with the sync-request package.

For instance, we can write:

const request = require('sync-request');

try {
  const res1 = request('GET', '/yrl1');
  const res2 = request('GET', '/url2');
  //...
}
catch (e) {
  //...
}

We import the sync-request package an use its request function to make requests synchronously.

However, as usual, synchronous code is blocking, so it shouldn’t be used in production apps.

However, it may be good for scripts.

Get Schema of a MongoDB Database Which Defined in Another Model with Mongoose

We can get the Mongoose schema with the model method.

For instance, we can write:

const PersonSchema = require('mongoose').model('person').schema

to get the Mongoose schema for the person collection.

Detect Node.js or Browser Environments

To check in a piece of code whether it’s running in Node or the browser, we can check if the window object exists.

If it does, then we know it’s running in the browser.

Otherwise, it’s running in Node.

For instance, we can write:

const app = window ? window : global;

to get the global variable of the corresponding platforms.

window is the global object for the browser.

global is the global object for Node.

Difference Between readFile and readFileSync

fs.readFile takes a callback and reads the file asynchronously.

The callback is called with the result whenever it’s done.

readFileSync reads the file synchronously and returns the content of it when it’s done.

We should use readFile in most apps since it doesn’t block other parts of an app from running.

However, readFileSync are good for scripts where we aren’t running anything else other than the code in the script sequentially.

How to Detect the Environment in an Express.js App

We can use the app.get method to get the environment variable in an Express app.

For instance, we can write:

app.get('env')

to get the environment variable.

Then we can write:

if (app.get('env') === 'development') {
  //...
}

to check if the app is running in the 'development' environment.

Parsing Form Data Request Payloads in Express

The body-parser package lets us parse request payloads in Express.

For instance, we can write:

const bodyParser = require('body-parser');
app.use(bodyParser.json());
app.use(bodyParser.urlencoded());

app.use(bodyParser.urlencoded({ extended: true }));

app.post('/post',(req, res) => {
  console.log(req.body)
})

We require the body-parser so that we can call the json method to make it parse JSON.

And we call urlencoded so that it can parse form data payloads.

extended means that we can parse JSON that’s nested in form-data requests.

Then the parsed request payload will be available in req.body .

Alternatively, we can use the express-formidable middleware to do the same thing.

We can install it by running:

npm install express-formidable

Then we can use it by writing:

const express = require('express');
const formidable = require('express-formidable');

const app = express();

app.use(formidable());

app.post('/upload', (req, res) => {
  res.send(JSON.stringify(req.fields));
});

We just pass in the formidable middleware.

Then we get the parsed request payload with the req.fields object.

Run Async Mocha Shared Test Code in Order

We can run async Mocha tests in order by using the before hook to run code that’s shared between multiple tests.

For instance, we can write:

let someCondition = false;
//...

describe('is dependent on someCondition', () => {
  const beforeTest = (done) => {
    if (someCondition) {
      done();
    }
    else {
      setTimeout(() => beforeTest(done), 1000);
    }
  }

  before((done) => {
    beforeTest(done);
  });

  it('does something', () => {
    // ...
  });

})

We check that someCondition is met before we run our tests with the beforeTest function.

Otherwise, we call beforeTest again after 1 second.

We do that until someCondition is true .

We pass in done to the beforeTest function so that we can call it when someConditional finally becomes true .

Once done is called, the test will be run.

Conclusion

We can run some checks before a test is run and wait for a condition to become true before running it.

We can use the sync-request package to run synchronous requests.

However, it’s blocking the rest of the app so it should probably only be used in scripts.

We can detect the platform that a piece of code is run in by checking which global object is available.