Categories
Express JavaScript Nodejs

Easy Logging with the Morgan Express Middleware

Logging is an important part of any app. We want to know what activities are going on and look for information to fix problems when they arise.

In this article, we’ll look at how to use the morgan middleware for adding logging in Express apps.

Parameters

The morgan middleware lets us pass in 2 arguments. The first is the format and the second is the options object.

Format

The format is a string. For example, we can pass in 'tiny' to show minimal information.

Also, we can pass in a string with the fields we want to show lik1:

morgan(':method :url :status');

The format can also be a custom format function like the following:

const express = require('express');  
const bodyParser = require('body-parser');  
const morgan = require('morgan')  
const app = express();
app.use(bodyParser.json());  
app.use(bodyParser.urlencoded({ extended: true }));
morgan((tokens, req, res) => {  
  return [  
    tokens.method(req, res),  
    tokens.url(req, res),  
    tokens.status(req, res),  
  ].join(' ')  
})
app.get('/', (req, res) => {  
  res.send('foo');  
});
app.listen(3000, () => console.log('server started'));

We get the same thing as:

morgan(':method :url :status');

with the function that’s passed into the morgan function’s returned value.

Options

morgan accepts several properties in the options object.

immediate

immediate logs on request instead of response. The data from the response won’t be logged.

skip

A function to determine when logging is skipped.

stream

Output stream for writing log lines, defaults to process.stdout .

Predefined Formats

combined

Standard Apache combined log output:

:remote-addr - :remote-user [:date[clf]] ":method :url HTTP/:http-version" :status :res[content-length] ":referrer" ":user-agent"

common

Standard Apache common log output:

:remote-addr - :remote-user [:date[clf]] ":method :url HTTP/:http-version" :status :res[content-length]

dev

Logging for development use. The :status token will be colored red for server error code, yellow for client error codes, cyan for redirection codes and uncolored for other codes.

:method :url :status :response-time ms - :res[content-length]

short

Shorter than default and includes response time.

:remote-addr :remote-user :method :url HTTP/:http-version :status :res[content-length] - :response-time ms

tiny

Minimal output:

:method :url :status :res[content-length] - :response-time ms

Tokens

We can create new tokens to log the fields we want.

For example, we can write:

const express = require('express');  
const bodyParser = require('body-parser');  
const morgan = require('morgan')  
const app = express();
app.use(bodyParser.json());  
app.use(bodyParser.urlencoded({ extended: true }));
morgan.token('type', (req, res) => req.headers['content-type'])
app.get('/', (req, res) => {  
  res.send('foo');  
});
app.listen(3000);

Then :type will log req.headers[‘content-type’] .

Below are tokens built into morgan :

  • :date[format] — date with format in UTC. Formats available include clf for common long format (e.g. 10/Oct/2000:13:55:36 +0000 ), iso for ISO 8601 format (e.g. 2000–10–10T13:55:36.000 ), web for RFC 1123 format (e.g. Tue, 10 Oct 2000 13:55:36 GMT )
  • :http-version — HTTP version of the request
  • :method — HTTP method of the request
  • :referreer — Referrer header of the request. This will use the standard misspelled Referer header if it exists, otherwise will log Referrer
  • :remote-addr — the remote address of the request. This will use req.ip . Otherwise the standard req.connection.remoteAddress value
  • :remote-user — the user authenticated part for basic auth
  • :req[header] — the given header of the request. If it’s not present, it’ll be logged as -
  • :res[header] — the given header of the response. If it’s not present, it’ll be logged as -
  • :response-time[digits] — the time between a request coming into morgan and when the response headers are written in milliseconds.
  • :status — the response status. If the request/response cycle completes before a response was sent to the client, then the status will be empty
  • :url — the URL of the request. This will use req.originalUrl if it exists otherwise req.url will be used.
  • :user-agent — the contents of the User-Agent header of the request

morgan.compile(format)

This method compiles a format string into a format function for use by morgan .

A format string is a string that presents a single log line and can utilize token syntax.

Tokens are referred to by :token-name . If a token accepts arguments, we can pass it into [] .

Examples

Simple Example

Simple use of morgan is something like the following:

const express = require('express');  
const bodyParser = require('body-parser');  
const morgan = require('morgan')  
const app = express();
app.use(bodyParser.json());  
app.use(bodyParser.urlencoded({ extended: true }));
app.use(morgan('combined'));
app.get('/', (req, res) => {  
  res.send('foo');  
});
app.listen(3000);

Then we get something like:

::ffff:172.18.0.1 - - [28/Dec/2019:01:04:22 +0000] "GET / HTTP/1.1" 304 - "https://repl.it/languages/express" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"

Writing Logs to a File

We can write logs to a file by setting thestream option of the option object and pass it into the second argument as follows:

const express = require('express');  
const bodyParser = require('body-parser');  
const morgan = require('morgan')  
const app = express();  
const fs = require('fs')  
const path = require('path')  
const appLogStream = fs.createWriteStream(path.join(__dirname, 'app.log'), { flags: 'a' })
app.use(bodyParser.json());  
app.use(bodyParser.urlencoded({ extended: true }));
app.use(morgan('combined', { stream: appLogStream}));
app.get('/', (req, res) => {  
  res.send('foo');  
});
app.listen(3000);

Then we get:

::ffff:172.18.0.1 - - [28/Dec/2019:01:06:44 +0000] "GET / HTTP/1.1" 304 - "https://repl.it/languages/express" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"

In app.log .

Rotating Logs

We can rotate between logs bu setting the interval property of the object.

For example, we can write:

const express = require('express');  
const bodyParser = require('body-parser');  
const morgan = require('morgan')  
const app = express();  
const fs = require('fs')  
const path = require('path')  
const accessLogStream = fs.createWriteStream(path.join(__dirname, 'app.log'), { flags: 'a' })
app.use(bodyParser.json());  
app.use(bodyParser.urlencoded({ extended: true }));
app.use(morgan('combined', {  
  interval: '7d',  
  stream: accessLogStream  
}));
app.get('/', (req, res) => {  
  res.send('foo');  
});
app.listen(3000);

to rotate logs every week.

Custom Log Entry Format

We can define our own token and record it as follows:

const express = require('express');  
const bodyParser = require('body-parser');  
const morgan = require('morgan')  
const app = express();  
app.use(bodyParser.json());  
app.use(bodyParser.urlencoded({ extended: true }));  
app.use(morgan(':id :method :url :date[iso]'))
morgan.token('id', (req) => req.id);app.get('/', (req, res) => {  
  req.id = 1;  
  res.send('foo');  
});
app.listen(3000);

Then since we set req.id to 1, we get:

1 GET / 2019-12-28T01:11:07.646Z

1 is in the first column since we specified :id first in the format string before :method .

Conclusion

morgan is an easy to use logger for Express apps. It’s available as a middleware and we can specify our log format and data to log in each entry with tokens.

To log what we want that’s included in the preset tokens, we can define our own tokens using the token method.

Categories
JavaScript Vue

Introduction to Vue.js Watchers

Vue.js is an easy to use web app framework that we can use to develop interactive front end apps.

In this article, we’ll look at how to define and use Vue.js watchers.

When Should We Use Watchers?

We can use watchers when we want to watch for data changes in reactive properties and do some asynchronous or expensive options as the value changes.

For example, we can use it to watch for changes for an input and then get a joke from the Chuck Norris API when we type in something.

First, we put the following in src/index.js :

new Vue({  
  el: "#app",  
  data: {  
    message: "",  
    joke: ""  
  },  
  created() {  
    this.debounceGetAllCapsMessage = _.debounce(this.getAllCapsMessage, 500);  
  },  
  methods: {  
    async getAllCapsMessage() {  
      const res = await fetch("https://api.icndb.com/jokes/random");  
      const result = await res.json();  
      this.joke = result.value.joke;  
    }  
  },  
  watch: {  
    message(newMessage, oldMessage) {  
      this.debounceGetAllCapsMessage();  
    }  
  }  
});

The code above watches for the change in the message property and then call the debounceGetAllCapsMessage method, which gets a jokes after a half a second delay.

The watch method both takes the new value as the first parameter and the old value as the second parameter.

Then we put the following in index.html :

<!DOCTYPE html>  
<html>  
  <head>  
    <title>Hello</title>  
    <meta charset="UTF-8" />  
    <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>  
    <script src="https://cdn.jsdelivr.net/npm/lodash@4.13.1/lodash.min.js"></script>  
  </head> <body>  
    <div id="app">  
      <input type="text" v-model="message" />  
      <p>{{joke}}</p>  
    </div>  
    <script src="./src/index.js"></script>  
  </body>  
</html>

As we can see, we can run an asynchronous operation, add a delay before we perform the operation, and then get the final result.

We can also use vm.$watch to watch for a value. It takes up to 3 arguments.

The first is a path to a property, which is a string. It can also be a function with one or multiple things combined to watch.

The second argument is a function that runs when the value changes.

The third argument for vm.$watch is an object with some properties. They include:

  • immediate — watches for the setting of the initial value

The method returns a function to stop watching for values. This is an optional argument.

We can use it as follows. In src/index.js , we have:

const vm = new Vue({  
  el: "#app",  
  data: {  
    message: "",  
    joke: ""  
  }  
});

vm.$watch(  
  "message",  
  _.debounce(async function() {  
    const res = await fetch("https://api.icndb.com/jokes/random");  
    const result = await res.json();  
    this.joke = result.value.joke;  
  }, 500)  
);

Then index.html , we have:

<!DOCTYPE html>  
<html>  
  <head>  
    <title>Hello</title>  
    <meta charset="UTF-8" />  
    <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>  
    <script src="https://cdn.jsdelivr.net/npm/lodash@4.13.1/lodash.min.js"></script>  
  </head> <body>  
    <div id="app">  
      <input type="text" v-model="message" />  
      <p>{{joke}}</p>  
    </div>  
    <script src="./src/index.js"></script>  
  </body>  
</html>

which is the same as what we had before.

In both examples, we should get a new joke after half a second delay displayed.

Conclusion

We can use the watch method in the options and object and vm.$watch to watch for value changes.

The vm.$watch method watches a property given the path to it or a function that returns it or a combination of multiple properties.

vm.$watch takes a value changes handler and we can run code in it to handle when the value of a reactive property change.

The watch method runs code as the value changes.

Also, the watch method both takes the new value as the first parameter and the old value as the second parameter.

Categories
JavaScript JavaScript Basics

The JavaScript Global Variable — When Should We Use It?

In JavaScript, there’s a global variable depending on the environment. In the browser, there’s the window object. The worker context has the self object as the global object. Node.js has the global object.

In this article, we’ll look at the global object, what it does, and how and when to use it.

Scope of the Global Object

The global object is always in the topmost scope. There’s nothing else above it. Therefore, it can be accessed by anything. Consider the name, this makes sense.

In JavaScript, scopes don’t change at runtime, so we can get the scope of something from their location in the code.

Since the global object is in the topmost scope, it can be accessed by anything. For example, we can console.log the window object from anywhere and we can get the value for it.

The Global Object

A global object is an object where the properties of it are global variables. It’s called the window object in browsers, self in web workers, and global in Node.js

There’s also a proposal to have a globalThis global object that’s available in all contexts.

The global object contains all built-in global variables.

Creating Global Variables

We can create global variables by creating a property of the global object. In the browser context, this means that we create it as a property of the window object.

There’re a few ways to do this. One way is to define as a property of the window object as follows:

window.foo = 1;

Then we can reference as foo or window.foo anywhere in our code.

Another way is to use the var keyword in the top level of our code. That is, it’s not nested in any block. We can define it as follows:

var foo = 1;

Getting or Setting Variables

We can get and set global variables just like how we declared it. For example, if we declared:

window.foo = 1;

Then we can reference window.foo and assign values to them. If it’s declared with var at the top level as in:

var foo = 1;

Then we can either write:

foo = 1;

or:

window.foo = 1;

Module

Each module has its own environment that isn’t exposed to the outside unless something is explicitly exported. However, we can still access global variables from there like in any other environment.

Caveats

The global object is considered a mistake. It’s easy to create global variables that clash with the names of other variables.

Also, we can easily change existing built-in global variables or ones that are defined by us. This is a problem which is avoided by using newer constructs like let and const to declare variables and constants and also using modules to protect private data and namespace them so that we can’t modify anything accidentally.

It was only created to make simple scripting on the browser easy. Now that we’re building full-fledged apps with JavaScript, using the global object too much will create lots of problems like we described before.

However, the global object in the browser environment has lots of properties that we can use to add features to our apps like window.SubtleCrypto for cryptography, customElements variable to declare Web Components and many more properties. location is also a property of the window object.

Likewise Node.js’ global object has properties like __dirname and __filename, which is handy for accessing data about our current code. The console object is useful for debugging, and Process is useful for getting data about the currently running process.

setTimeout, setInterval are also part of the Node.js global object.

The full list of properties is at Mozilla Developer Network.

It was also used to access variables from external scripts before JavaScript has modules. But now that we have modules, we definitely don’t need to use the global object for that purpose.

Also, testing is a lot harder since global variables are changed by everything. So, it’s hard to reason what’s going on with them.

Conclusion

Nowadays, we just use global variables like the window object mostly for their built-in properties.

In our own code, we use what’s provided by the global object, but there isn’t any good use case to add any properties to the global object. It’s messy and hard to trace issues since everything modifies the global variable since everything can access it.

The better way is to modularize everything and then make expose the things that we need by exporting them.

However, the window object does have lots of useful properties that we can use. This is the same for Node.js’ global object.

Categories
JavaScript Nodejs

Testing Nodejs Apps that Interact with External APIs with Nock

Today’s web apps rarely live alone. They often make requests to external APIs via HTTP requests.

Fortunately, testing these kinds of code isn’t too hard since we can mock the HTTP responses. This means that our tests don’t have to make the requests in our tests, which is critical for writing repeatable and independent tests.

In this piece, we’ll look at how to test an Express app that interacts with an external API.

Creating a Simple App

First, we create the app that we’ll test. The app will get a random joke from the Chuck Norris Jokes API.

Next, we create an empty project folder and run:

npm init -y

To create an empty package.json file with the default answers.

Next, we install Express and Node-Fetch to create the app and get data from APIs, respectively.

We run:

npm i express node-fetch

To install the packages.

Then we create app.js and add:

To make the app that gets the joke and returns it as the response.

Adding the Test

To add the test, we use the Jest test runner to run the test and check the results, Nock to mock the response for the request that we made in app.js, and Supertest to run the Express app and get the response that we can check.

To do this, we run:

npm i nock supertest jest

Then we create app.test.js for the test code and then add:

In the file.

The code above starts by including our app from app.js and the libraries we use for building the test.

const app = require('./app');  
const nock = require('nock');  
const request = require('supertest');

The first line above imports the Express app instance from app.js. Then the other two lines import the Nock and Supertest libraries, respectively.

Then we add the skeleton for the test by calling test from Jest and pass in the description and async function since we’ll use the promise version of Supertest.

Then using Nock, we mock the response as follows:

As we can see, the mock request is a GET request, which is the same as what we had in app.js. Also, the URL is the same as what we had there.

The difference is that we return a 200 response with the mockResponse object in the test.

Now it doesn’t matter what the actual API returns. Nock will intercept the request and always return the content of mockResponse as the response body for the request made in our route in app.js.

Then we just have to call the route with Supertest as follows:

const res = await request(app).get('/');

To call the route in app.js and then check the response as follows:

expect(res.status).toBe(200);  
expect(res.body).toEqual(mockResponse);

The toEqual method checks for deep equality so we can pass in the whole object to check.

Running the Test

Now to run our test, we have to add:

"test": "jest --forceExit"

To the scripts section of package.json.

We need the --forceExit option so that the test will exit once the test is run. This is a bug that’s yet to be resolved.

Now we can run the test by running:

npm test

Then we should get:

We should get the same thing no matter how many times we run the test since we mocked the response of the API.

The real API returns something different every time we make a request to it, so we can’t use it for our tests.

Even if we could, it’d be much slower and the API might not always be available.

Conclusion

With Nock, we can mock responses easily for external API requests in our app. Then we can focus on just running our app’s code and checking the results.

Now we have tests that run fast and produce repeatable results that don’t depend on anything external to the tests.

Categories
JavaScript Nodejs

Node.js FS Module — Write Streams

Manipulating files and directories are basic operations for any program. Since Node.js is a server-side platform and can interact with the computer that it’s running on directly, being able to manipulate files is a basic feature.

Fortunately, Node.js has a fs module built into its library. It has many functions that can help with manipulating files and folders.

File and directory operation that are supported include basic ones like manipulating and opening files in directories.

Likewise, it can do the same for files. It can do this both synchronously and asynchronously. It has an asynchronous API that has functions that support promises.

Also, it can show statistics for a file. Almost all the file operations that we can think of can be done with the built-in fs module. In this article, we will create write streams to write a file’s data sequentially and listen to events from a write stream. Since Node.js WriteStreams are descendants of the Writable object, we will also listen to events to it.

Streams are collections of data that may not be available all at once and don’t have to fit in memory. This makes stream handy for processing large amounts of data.

It’s handy for files because files can be big and streams can let us get a small amount of data at one time. In the fs module, there are 2 kinds of streams. There’s the ReadStream and the WriteStream. We get the data into a WriteStream via a ReadStream.

WriteStream

A WriteStream is for used for writing data to a file. We can get the input into our WriteStream via the ReadStream’s pipe function. The data from the ReadStream is sent to the write stream in small chunks so that the host computer won’t run out of memory.

To define a WriteStream, we can use the fs.createWriteStream function. The function takes 2 arguments.

The first argument is the path of the file. The path can be in the form of a string, a Buffer object, or an URL object. The path can be in the form of a string, a Buffer object, or an URL object.

The second argument is an object that can have a variety of options as properties. The flag option is the file system flag for setting the mode for opening the file. The default flag is w. The list of flags are below:

  • 'a' – Opens a file for appending, which means adding data to the existing file. The file is created if it does not exist.
  • 'ax' – Like 'a' but exception is thrown if the path exists.
  • 'a+' – Open file for reading and appending. The file is created if it doesn’t exist.
  • 'ax+' – Like 'a+' but exception is thrown if the path exists.
  • 'as' – Opens a file for appending in synchronous mode. The file is created if it does not exist.
  • 'as+' – Opens a file for reading and appending in synchronous mode. The file is created if it does not exist.
  • 'r' – Opens a file for reading. An exception is thrown if the file doesn’t exist.
  • 'r+' – Opens a file for reading and writing. An exception is thrown if the file doesn’t exist.
  • 'rs+' – Opens a file for reading and writing in synchronous mode.
  • 'w' – Opens a file for writing. The file is created (if it does not exist) or overwritten (if it exists).
  • 'wx' – Like 'w' but fails if the path exists.
  • 'w+' – Opens a file for reading and writing. The file is created (if it does not exist) or overwritten (if it exists).
  • 'wx+' – Like 'w+' but exception is thrown if the path exists.

The encoding option is a string that sets the character encoding in the form of the string. The default value is null .

The fd option is the integer file descriptor which can be obtained with the open function and its variants. If the fd option is set, then the path argument will be ignored. The default value is null .

The mode option is the file permission and sticky bits of the file, which is an octal number that is the same as Unix or Linux file permissions. It’s only set if the file is created.

The default value is 0o666. The autoClose option specifies that the file descriptor will be closed automatically. The default value is true . If it’s false , then the file descriptor won’t be closed even if there’s an error. It’s completely up to us to close it autoClose is set to false to make sure there’s no file descriptor leak.

Otherwise, the file descriptor will be closed automatically if there’s an error or end event emitted. The emitClose option will emit the close event when the write stream ends.

The default value is false . The start and end options specifies the beginning and end parts of the file to write.

Everything in between will be written in addition to the start and end . start and end are numbers that are the starting and ending bytes of the file to write.

To create a WriteStream, we can use the createWriteStream like in the following code:

const fs = require("fs");  
const sourceFile = "./files/file.txt";  
const destFile = "./files/newFile.txt";
const readStream = fs.createReadStream(sourceFile);  
const writeStream = fs.createWriteStream(destFile, {  
  flags: "w",  
  encoding: "utf8",  
  mode: 0o666,  
  autoClose: true,  
  emitClose: true,  
  start: 0  
});

readStream.pipe(writeStream);

writeStream.on("open", () => {  
  console.log("Stream opened");  
});

writeStream.on("ready", () => {  
  console.log("Stream ready");  
});

writeStream.on("pipe", src => {  
  console.log(src);  
});

writeStream.on("unpipe", src => {  
  console.log(src);  
});

writeStream.on("finish", () => {  
  console.log("Stream finished");  
});

When we run the code above, we should get something like the following outputted to the screen:

Stream opened  
Stream ready  
ReadStream {  
  _readableState: ReadableState {  
    objectMode: false,  
    highWaterMark: 65536,  
    buffer: BufferList { head: null, tail: null, length: 0 },  
    length: 0,  
    pipes: null,  
    pipesCount: 0,  
    flowing: false,  
    ended: true,  
    endEmitted: true,  
    reading: false,  
    sync: false,  
    needReadable: false,  
    emittedReadable: false,  
    readableListening: false,  
    resumeScheduled: false,  
    paused: false,  
    emitClose: false,  
    autoDestroy: false,  
    destroyed: true,  
    defaultEncoding: 'utf8',  
    awaitDrain: 0,  
    readingMore: false,  
    decoder: null,  
    encoding: null  
  },  
  readable: false,  
  _events: [Object: null prototype] { end: [Function] },  
  _eventsCount: 1,  
  _maxListeners: undefined,  
  path: './files/file.txt',  
  fd: null,  
  flags: 'r',  
  mode: 438,  
  start: undefined,  
  end: Infinity,  
  autoClose: true,  
  pos: undefined,  
  bytesRead: 16,  
  closed: false  
}  
Stream finished

We have the Readable object sent when the pipe event is emitted. Also, we should get content that’s in file.txt in newFile.txt .

WriteStream Events

With a WriteStream, we can listen to the following events. There’s a close event which is emitted when the close event is emitted after the file is written. The open event is emitted when the stream is opened. The file descriptor number fd will be passed with the event when it’s emitted. The ready event is emitted when the WriteStream is ready to be used. It’s fired immediately after the open event is fired.

WriteStreams extend the Writable object, which emits events of its own. The close event is emitted when the stream any of its underlying resources like file descriptors have been closed.

This event indicates that there’re no more events to be emitted from the stream and nothing else will be run. If the emitClose option is set to true when creating the WriteStream then the close event will be emitted.

If a call to the stream.write function returns false , then the drain event will be emitted when writing data to the stream is resumed. The error event is emitted when an error occurred when piping data.

The listener callback function has an error object parameter to get the error information. The stream isn’t closed on error events unless the autoDestroy option is set to true when creating the stream.

After the error event is emitted, then no events other than close should be emitted. The finish event is emitted after the stream.end function is called and all data is flush to the underlying system like a file.

The pipe event is emitted when stream.pipe function is called with a readable stream passed in as an argument . The file specified when creating the ReadStream must exist before piping from it.

The unpipe event is emitted when the stream.unipipe function is called on the readable stream. It’s also emitted when the WriteStream emits an error event when a ReadStreanm is piped to it.

A WriteStream has a few properties. The bytesWritten property indicates the number of bytes being written by the WriteStream so far and doesn’t include data that’s still queued for writing.

The path property is the path of the file the WriteStream is writing to, which is the same as the first argument of the fs.createWriteStream function.

It will be of the same type as whatever we passed into the first argument of fs.createWriteStream . The pending property is a boolean that is true when the underlying file hasn’t been opened, that is, before the ready event is emitted.

By using the fs.createWriteStream function, we created read streams to read a file’s data sequentially and listen to events from a read stream. Since Node.js WriteStreams are descendants of the Writable object, we will also listen to events to it.

We have lots of control of how the write stream is created. We can set the path or file descriptor of the file. Also, we can set the mode of the file to be written and the permission and sticky bit of the file being read.

Also, we can choose to close the streams automatically or not or emit close event automatically.

We get data to a write stream by passing the WriteStream object as an argument of the ReadStream’s pipe function.