Categories
React

How to Add Authenticated Routes to Your React App

Authenticated Reacts routes are easy to make with higher order components.

Authenticated Reacts routes are easy to make with higher order components. Higher order components are components that takes components as props and return render a new component.

For example, to create an authenticated route, we can write:

import React from "react";
import { Redirect } from "react-router-dom";

function RequireAuth({ Component }) {
  if (!localStorage.getItem("token")) {
    return <Redirect to="/" />;
  }
  return <Component />;
}

export default RequireAuth;

RequireAuth is a component that takes a Component prop. Component is a prop that is a React component. We check if there’s an authentication token present in local storage, and if it is, then we render our route, the Component , that requires authentication. Otherwise we redirect to a route that doesn’t require authentication.

In this article, we will make a simple Bitbucket app with authenticated React routes. We will let users sign up for an account and set their Bitbucket username and password for their account. Once the user is logged in and set their Bitbucket credentials, they can view their repositories and the commits for each repository.

Bitbucket is a great repository hosting service that lets you host Git repositories for free. You can upgrade to their paid plans to get more features for a low price. It also has a comprehensive API for automation and getting data.

Developers have made Node.js clients for Bitbucket’s API. We can easily use it to do what we want like manipulating commits, adding, removing, or editing repositories, tracking issues, manipulating build pipelines and a lot more.

The Bitbucket.js package is one of the easiest packages to use for writing Node.js Bitbucket apps. All we have to do is log in with the Bitbucket instance with our Bitbucket username and password and call the built in functions listed at https://bitbucketjs.netlify.com/#api-_ to do almost anything we want with the Bitbucket packages.

Our app will consist of an back end and a front end. The back end handles the user authentication and communicates with Bitbucket via the Bitbucket API. The front end has the sign up form, log in form, a settings form for setting password and Bitbucket username and password.

To start building the app, we create a project folder with the backend folder inside. We then go into the backend folder and run the Express Generator by running npx express-generator . Next we install some packages ourselves. We need Babel to use import , BCrypt for hashing passwords, Bitbucket.js for using the Bitbucket API. Crypto-JS for encrypting and decrypting our Bitbucket password, Dotenv for storing hash and encryption secrets, Sequelize for ORM, JSON Web Token packages for authentication, CORS for cross domain communication and SQLite for database.

Run npm i @babel/cli @babel/core @babel/node @babel/preset-env bcrypt bitbucket cors crypto-js dotenv jsonwebtoken sequelize sqlite3 to install the packages.

In the script section of package.json , put:

"start": "nodemon --exec npm run babel-node --  ./bin/www",
"babel-node": "babel-node"

to start our app with Babel Node instead of the regular Node.js runtime so that we get the latest JavaScript features.

Then we create a file called .babelrc in the backend folder and add:

{
    "presets": [
        "@babel/preset-env"
    ]
}

to enable the latest features.

Then we have to add Sequelize code by running npx sequelize-cli init . After that we should get a config.json file in the config folder.

In config.json , we put:

{
  "development": {
    "dialect": "sqlite",
    "storage": "development.db"
  },
  "test": {
    "dialect": "sqlite",
    "storage": "test.db"
  },
  "production": {
    "dialect": "sqlite",
    "storage": "production.db"
  }
}

To use SQLite as our database.

Next we add a middleware for verifying the authentication token, add a middllewares folder in the backend folder, and in there add authCheck.js . In the file, add:

const jwt = require("jsonwebtoken");
const secret = process.env.JWT_SECRET;

export const authCheck = (req, res, next) => {
  if (req.headers.authorization) {
    const token = req.headers.authorization;
    jwt.verify(token, secret, (err, decoded) => {
      if (err) {
        res.send(401);
      } else {
        next();
      }
    });
  } else {
    res.send(401);
  }
};

We return 401 response if the token is invalid.

Next we create some migrations and models. Run:

npx sequelize-cli model:create --name User --attributes username:string,password:string,bitBucketUsername:string,bitBucketPassword:string

to create the model. Note that the attributes option has no spaces.

Then we add unique constraint to the username column of the Users table. To do this, run:

npx sequelize-cli migration:create addUniqueConstraintToUser

Then in newly created migration file, add:

"use strict";

module.exports = {
  up: (queryInterface, Sequelize) => {
    return queryInterface.addConstraint("Users", ["username"], {
      type: "unique",
      name: "usernameUnique"
    });
  },

down: (queryInterface, Sequelize) => {
    return queryInterface.removeConstraint("Users", "usernameUnique");
  }
};

Run npx sequelize-cli db:migrate to run all the migrations.

Next we create the routes. Create a file called bitbucket.js and add:

var express = require("express");
const models = require("../models");
const CryptoJS = require("crypto-js");
const Bitbucket = require("bitbucket");
const jwt = require("jsonwebtoken");
import { authCheck } from "../middlewares/authCheck";

const bitbucket = new Bitbucket();
var router = express.Router();

router.post("/setBitbucketCredentials", authCheck, async (req, res, next) => {
  const { bitBucketUsername, bitBucketPassword } = req.body;
  const token = req.headers.authorization;
  const decoded = jwt.verify(token, process.env.JWT_SECRET);
  const id = decoded.userId;
  const cipherText = CryptoJS.AES.encrypt(
    bitBucketPassword,
    process.env.CRYPTO_SECRET
  );
  await models.User.update(
    {
      bitBucketUsername,
      bitBucketPassword: cipherText.toString()
    },
    {
      where: { id }
    }
  );
  res.json({});
});

router.get("/repos/:page", authCheck, async (req, res, next) => {
  const page = req.params.page || 1;
  const token = req.headers.authorization;
  const decoded = jwt.verify(token, process.env.JWT_SECRET);
  const id = decoded.userId;
  const users = await models.User.findAll({ where: { id } });
  const user = users[0];
  const bytes = CryptoJS.AES.decrypt(
    user.bitBucketPassword.toString(),
    process.env.CRYPTO_SECRET
  );
  const password = bytes.toString(CryptoJS.enc.Utf8);
  bitbucket.authenticate({
    type: "basic",
    username: user.bitBucketUsername,
    password
  });
  let { data } = await bitbucket.repositories.list({
    username: user.bitBucketUsername,
    page,
    sort: "-updated_on"
  });
  res.json(data);
});

router.get("/commits/:repoName", authCheck, async (req, res, next) => {
  const token = req.headers.authorization;
  const decoded = jwt.verify(token, process.env.JWT_SECRET);
  const id = decoded.userId;
  const users = await models.User.findAll({ where: { id } });
  const user = users[0];
  const repoName = req.params.repoName;
  const bytes = CryptoJS.AES.decrypt(
    user.bitBucketPassword.toString(),
    process.env.CRYPTO_SECRET
  );
  const password = bytes.toString(CryptoJS.enc.Utf8);
  bitbucket.authenticate({
    type: "basic",
    username: user.bitBucketUsername,
    password
  });
  let { data } = await bitbucket.commits.list({
    username: user.bitBucketUsername,
    repo_slug: repoName,
    sort: "-date"
  });
  res.json(data);
});

module.exports = router;

In each route, we get the user from the token, since we will add the user ID into the token, and from there we get the Bitbucket username and password, which we use to log into the Bitbucket API. Note that we have to decrypt the password since we encrypted it before saving it to the database.

We set the Bitbucket credentials in the setBitbucketCredentials route. We encrypt the password before saving to keep it secure.

Then in the repos route, we get the repos of the user and sort by reversed update_on order since we specified -updated_on in the sort parameter. The commits are listed in reverse date order since we specified -date in the sort parameter.

Next we add the user.js in the routes folder and add:

const express = require("express");
const models = require("../models");
const bcrypt = require("bcrypt");
const jwt = require("jsonwebtoken");
import { authCheck } from "../middlewares/authCheck";
const router = express.Router();

router.post("/signup", async (req, res, next) => {
  const { username, password } = req.body;
  const hashedPassword = await bcrypt.hash(password, 10);
  const user = await models.User.create({ username, password: hashedPassword });
  res.json(user);
});

router.post("/login", async (req, res, next) => {
  const { username, password } = req.body;
  const users = await models.User.findAll({ where: { username } });
  const user = users[0];
  await bcrypt.compare(password, user.password);
  const token = jwt.sign({ userId: user.id }, process.env.JWT_SECRET);
  res.json({ token });
});

router.post("/changePassword", authCheck, async (req, res, next) => {
  const { password } = req.body;
  const token = req.headers.authorization;
  const decoded = jwt.verify(token, process.env.JWT_SECRET);
  const id = decoded.userId;
  const hashedPassword = await bcrypt.hash(password, 10);
  await models.User.update({ password: hashedPassword }, { where: { id } });
  res.json({});
});

router.get("/currentUser", authCheck, async (req, res, next) => {
  const token = req.headers.authorization;
  const decoded = jwt.verify(token, process.env.JWT_SECRET);
  const id = decoded.userId;
  const users = await models.User.findAll({ where: { id } });
  const { username, bitBucketUsername } = users[0];
  res.json({ username, bitBucketUsername });
});

module.exports = router;

We have routes for sign up, log in, and change password. We hash the password before saving when we sign up or changing password.

The currentUser route will be used for a settings form in the front end.

In app.js we replace the existing code with:

require("dotenv").config();
var createError = require("http-errors");
var express = require("express");
var path = require("path");
var cookieParser = require("cookie-parser");
var logger = require("morgan");
const cors = require("cors");

var indexRouter = require("./routes/index");
var usersRouter = require("./routes/users");
var bitbucketRouter = require("./routes/bitbucket");

var app = express();

// view engine setup
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "jade");

app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, "public")));
app.use(cors());

app.use("/", indexRouter);
app.use("/users", usersRouter);
app.use("/bitbucket", bitbucketRouter);

// catch 404 and forward to error handler
app.use(function(req, res, next) {
  next(createError(404));
});

// error handler
app.use(function(err, req, res, next) {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get("env") === "development" ? err : {};

// render the error page
  res.status(err.status || 500);
  res.render("error");
});

module.exports = app;

to add the CORS add-on to enable cross domain communication, and add the users and bitbucket routes in our app by adding:

app.use("/users", usersRouter);
app.use("/bitbucket", bitbucketRouter);

This finishes the back end portion of the app. Now we can build the front end. We will build it with React, so we start by running npx create-react-app frontend from the project’s root folder.

Next we have to install some packages. We will install Bootstrap for styling, React Router for routing, Formik and Yup for form value handling and form validation respectively and Axios for making HTTP requests.

To do this run npm i axios bootstrap formik react-bootstrap react-router-dom yup to install all the packages.

Next we modify the App.js folder by replacing the existing code with the following:

import React from "react";
import HomePage from "./HomePage";
import "./App.css";
import ReposPage from "./ReposPage";
import CommitsPage from "./CommitsPage";
import SettingsPage from "./SettingsPage";
import { createBrowserHistory as createHistory } from "history";
import { Router, Route } from "react-router-dom";
import SignUpPage from "./SignUpPage";
import RequireAuth from "./RequireAuth";
const history = createHistory();

function App() {
  return (
    <div className="App">
      <Router history={history}>
        <Route path="/" exact component={HomePage} />
        <Route path="/signup" exact component={SignUpPage} />
        <Route
          path="/settings"
          component={props => (
            <RequireAuth {...props} Component={SettingsPage} />
          )}
        />
        <Route
          path="/repos"
          exact
          component={props => <RequireAuth {...props} Component={ReposPage} />}
        />
        <Route
          path="/commits/:repoName"
          exact
          component={props => (
            <RequireAuth {...props} Component={CommitsPage} />
          )}
        />
      </Router>
    </div>
  );
}

export default App;

We add the routes for the pages in our app here. RequiredAuth is a higher order component which takes a component that requires authentication to access and return redirect if not authorized or the page if the user is authorized. We pass in the components that requires authentication in this route, like ReposPage , SettingsPage and CommitPage .

In App.css , we replace the existing code with:

.page {
    padding: 20px;
}

to add some padding.

Next we add CommitsPage.js in the src folder and add:

import React, { useState, useEffect } from "react";
import { withRouter } from "react-router-dom";
import { commits } from "./requests";
import Card from "react-bootstrap/Card";
import LoggedInTopBar from "./LoggedInTopBar";
const moment = require("moment");

function CommitsPage({ match: { params } }) {
  const [initialized, setInitialized] = useState(false);
  const [repoCommits, setRepoCommits] = useState([]);

const getCommits = async page => {
    const repoName = params.repoName;
    const response = await commits(repoName, page);
    setRepoCommits(response.data.values);
  };

useEffect(() => {
    if (!initialized) {
      getCommits(1);
      setInitialized(true);
    }
  });

return (
    <>
      <LoggedInTopBar />
      <div className="page">
        <h1 className="text-center">Commits</h1>
        {repoCommits.map(rc => {
          return (
            <Card style={{ width: "90vw", margin: "0 auto" }}>
              <Card.Body>
                <Card.Title>{rc.message}</Card.Title>
                <p>Message: {rc.author.raw}</p>
                <p>
                  Date:{" "}
                  {moment(rc.date).format("dddd, MMMM Do YYYY, h:mm:ss a")}
                </p>
                <p>Hash: {rc.hash}</p>
              </Card.Body>
            </Card>
          );
        })}
      </div>
    </>
  );
}

export default withRouter(CommitsPage);

We get the commits given the repository name in the URL parameter and display them in a list of Cards provided by React Bootstrap.

Next we create the Home Page, which lets us sign in. Create a HomePage.js file in the src folder and add:

import React, { useState, useEffect } from "react";
import { Formik } from "formik";
import Form from "react-bootstrap/Form";
import Col from "react-bootstrap/Col";
import Button from "react-bootstrap/Button";
import { Link } from "react-router-dom";
import * as yup from "yup";
import { logIn } from "./requests";
import { Redirect } from "react-router-dom";
import Navbar from "react-bootstrap/Navbar";

const schema = yup.object({
  username: yup.string().required("Username is required"),
  password: yup.string().required("Password is required")
});

function HomePage() {
  const [redirect, setRedirect] = useState(false);

const handleSubmit = async evt => {
    const isValid = await schema.validate(evt);
    if (!isValid) {
      return;
    }
    try {
      const response = await logIn(evt);
      localStorage.setItem("token", response.data.token);
      setRedirect(true);
    } catch (ex) {
      alert("Invalid username or password");
    }
  };

if (redirect) {
    return <Redirect to="/repos" />;
  }

return (
    <>
      <Navbar bg="primary" expand="lg" variant="dark">
        <Navbar.Brand href="#home">Bitbucket App</Navbar.Brand>
      </Navbar>
      <div className="page">
        <h1 className="text-center">Log In</h1>
        <Formik validationSchema={schema} onSubmit={handleSubmit}>
          {({
            handleSubmit,
            handleChange,
            handleBlur,
            values,
            touched,
            isInvalid,
            errors
          }) => (
            <Form noValidate onSubmit={handleSubmit}>
              <Form.Row>
                <Form.Group as={Col} md="12" controlId="username">
                  <Form.Label>Username</Form.Label>
                  <Form.Control
                    type="text"
                    name="username"
                    placeholder="Username"
                    value={values.username || ""}
                    onChange={handleChange}
                    isInvalid={touched.username && errors.username}
                  />
                  <Form.Control.Feedback type="invalid">
                    {errors.username}
                  </Form.Control.Feedback>
                </Form.Group>
                <Form.Group as={Col} md="12" controlId="password">
                  <Form.Label>Password</Form.Label>
                  <Form.Control
                    type="password"
                    name="password"
                    placeholder="Password"
                    value={values.password || ""}
                    onChange={handleChange}
                    isInvalid={touched.password && errors.password}
                  />

<Form.Control.Feedback type="invalid">
                    {errors.password}
                  </Form.Control.Feedback>
                </Form.Group>
              </Form.Row>
              <Button type="submit" style={{ marginRight: "10px" }}>
                Log In
              </Button>
              <Link
                className="btn btn-primary"
                to="/signup"
                style={{ marginRight: "10px", color: "white" }}
              >
                Sign Up
              </Link>
            </Form>
          )}
        </Formik>
      </div>
    </>
  );
}

export default HomePage;

We use the Formik component provided by Formik to get automatic form value handling. It will pipe the values directly to the parameter of the onSubmit handler. In the handleSubmit function, we run the schema.validate function to check for validity of the form values according to the schema object generated from the Yup library and if that’s successful, then we call login from the requests.js file which we will create.

Next we create the top bar. Create LoggedInTopBar.js in the src folder and add:

import React, { useState } from "react";
import Navbar from "react-bootstrap/Navbar";
import Nav from "react-bootstrap/Nav";
import { withRouter, Redirect } from "react-router-dom";

function LoggedInTopBar({ location }) {
  const [redirect, setRedirect] = useState(false);
  const { pathname } = location;

  const isLoggedIn = () => !!localStorage.getItem("token");

  if (redirect) {
    return <Redirect to="/" />;
  }

  return (
    <div>
      {isLoggedIn() ? (
        <Navbar bg="primary" expand="lg" variant="dark">
          <Navbar.Brand href="#home">Bitbucket App</Navbar.Brand>
          <Navbar.Toggle aria-controls="basic-navbar-nav" />
          <Navbar.Collapse id="basic-navbar-nav">
            <Nav className="mr-auto">
              <Nav.Link href="/settings" active={pathname == "/settings"}>
                Settings
              </Nav.Link>
              <Nav.Link href="/repos" active={pathname == "/repos"}>
                Repos
              </Nav.Link>
              <Nav.Link>
                <span
                  onClick={() => {
                    localStorage.clear();
                    setRedirect(true);
                  }}
                >
                  Log Out
                </span>
              </Nav.Link>
            </Nav>
          </Navbar.Collapse>
        </Navbar>
      ) : null}
    </div>
  );
}

export default withRouter(LoggedInTopBar);

This contains the React Bootstrap Navbar to show a top bar with a link to the home page and the name of the app. We only display it with the token present in local storage. We check the pathname to highlight the right links by setting the active prop.

Next create ReposPage.js in the src folder to display the list of repositories the user owns. We add:

import React, { useState, useEffect } from "react";
import { repos } from "./requests";
import Card from "react-bootstrap/Card";
import { Link } from "react-router-dom";
import Pagination from "react-bootstrap/Pagination";
import LoggedInTopBar from "./LoggedInTopBar";

function ReposPage() {
  const [initialized, setInitialized] = useState(false);
  const [repositories, setRepositories] = useState([]);
  const [page, setPage] = useState(1);
  const [totalPages, setTotalPages] = useState(1);

  const getRepos = async page => {
    const response = await repos(page);
    setRepositories(response.data.values);
    setTotalPages(Math.ceil(response.data.size / response.data.pagelen));
  };

  useEffect(() => {
    if (!initialized) {
      getRepos(1);
      setInitialized(true);
    }
  });

  return (
    <div>
      <LoggedInTopBar />
      <h1 className="text-center">Your Repositories</h1>
      {repositories.map((r, i) => {
        return (
          <Card style={{ width: "90vw", margin: "0 auto" }} key={i}>
            <Card.Body>
              <Card.Title>{r.slug}</Card.Title>
              <Link className="btn btn-primary" to={`/commits/${r.slug}`}>
                Go
              </Link>
            </Card.Body>
          </Card>
        );
      })}
      <br />
      <Pagination style={{ width: "90vw", margin: "0 auto" }}>
        <Pagination.First onClick={() => getRepos(1)} />
        <Pagination.Prev
          onClick={() => {
            let p = page - 1;
            getRepos(p);
            setPage(p);
          }}
        />
        <Pagination.Next
          onClick={() => {
            let p = page + 1;
            getRepos(p);
            setPage(p);
          }}
        />
        <Pagination.Last onClick={() => getRepos(totalPages)} />
      </Pagination>
      <br />
    </div>
  );
}

export default ReposPage;

We get the list repositories and we add pagination to get more repositories.

In requests.js , we add:

const axios = require("axios");
const APIURL = "http://localhost:3000";

axios.interceptors.request.use(
  config => {
    config.headers.authorization = localStorage.getItem("token");
    return config;
  },
  error => Promise.reject(error)
);

axios.interceptors.response.use(
  response => {
    return response;
  },
  error => {
    if (error.response.status == 401) {
      localStorage.clear();
    }
    return error;
  }
);

export const signUp = data => axios.post(`${APIURL}/users/signup`, data);

export const logIn = data => axios.post(`${APIURL}/users/login`, data);

export const changePassword = data =>
  axios.post(`${APIURL}/users/changePassword`, data);

export const currentUser = () => axios.get(`${APIURL}/users/currentUser`);

export const setBitbucketCredentials = data =>
  axios.post(`${APIURL}/bitbucket/setBitbucketCredentials`, data);

export const repos = page =>
  axios.get(`${APIURL}/bitbucket/repos/${page || 1}`);

export const commits = (repoName) =>
  axios.get(`${APIURL}/bitbucket/commits/${repoName}`);

This file has all the HTTP requests that we make for sign up, log in, set credentials, get repositories and commits, etc.

And for handling responses, if we get 401 responses then we clear local storage so we won’t be using an invalid token. The code is below:

axios.interceptors.response.use(
  response => {
    return response;
  },
  error => {
    if (error.response.status == 401) {
      localStorage.clear();
    }
    return error;
  }
);

We attach the token to the request headers with:

axios.interceptors.request.use(
  config => {
    config.headers.authorization = localStorage.getItem("token");
    return config;
  },
  error => Promise.reject(error)
);

so we don’t have to set it for each authenticated requests.

Next create a file called RequiredAuth.js in the src folder and add:

import React from "react";
import { Redirect } from "react-router-dom";

function RequireAuth({ Component }) {
  if (!localStorage.getItem("token")) {
    return <Redirect to="/" />;
  }
  return <Component />;
}

export default RequireAuth;

We check for presence of the token in the authenticated routes and render the Component prop passed in if the token is present.

Then we create a file called SettingsPage.js in the src folder and add:

import React, { useState, useEffect } from "react";
import { Formik } from "formik";
import Form from "react-bootstrap/Form";
import Col from "react-bootstrap/Col";
import Button from "react-bootstrap/Button";
import * as yup from "yup";
import {
  currentUser,
  setBitbucketCredentials,
  changePassword
} from "./requests";
import LoggedInTopBar from "./LoggedInTopBar";

const userFormSchema = yup.object({
  username: yup.string().required("Username is required"),
  password: yup.string().required("Password is required")
});

const bitBucketFormSchema = yup.object({
  bitBucketUsername: yup.string().required("Username is required"),
  bitBucketPassword: yup.string().required("Password is required")
});

function SettingsPage() {
  const [initialized, setInitialized] = useState(false);
  const [user, setUser] = useState({});
  const [bitbucketUser, setBitbucketUser] = useState({});

  const handleUserSubmit = async evt => {
    const isValid = await userFormSchema.validate(evt);
    if (!isValid) {
      return;
    }
    try {
      await changePassword(evt);
      alert("Password changed");
    } catch (error) {
      alert("Password change failed");
    }
  };

  const handleBitbucketSubmit = async evt => {
    const isValid = await bitBucketFormSchema.validate(evt);
    if (!isValid) {
      return;
    }
    try {
      await setBitbucketCredentials(evt);
      alert("Bitbucket credentials changed");
    } catch (error) {
      alert("Bitbucket credentials change failed");
    }
  };

  const getCurrentUser = async () => {
    const response = await currentUser();
    const { username, bitBucketUsername } = response.data;
    setUser({ username });
    setBitbucketUser({ bitBucketUsername });
  };

  useEffect(() => {
    if (!initialized) {
      getCurrentUser();
      setInitialized(true);
    }
  });

  return (
    <>
      <LoggedInTopBar />
      <div className="page">
        <h1 className="text-center">Settings</h1>
        <h2>User Settings</h2>
        <Formik
          validationSchema={userFormSchema}
          onSubmit={handleUserSubmit}
          initialValues={user}
          enableReinitialize={true}
        >
          {({
            handleSubmit,
            handleChange,
            handleBlur,
            values,
            touched,
            isInvalid,
            errors
          }) => (
            <Form noValidate onSubmit={handleSubmit}>
              <Form.Row>
                <Form.Group as={Col} md="12" controlId="username">
                  <Form.Label>Username</Form.Label>
                  <Form.Control
                    type="text"
                    name="username"
                    placeholder="Username"
                    value={values.username || ""}
                    onChange={handleChange}
                    isInvalid={touched.username && errors.username}
                    disabled
                  />
                  <Form.Control.Feedback type="invalid">
                    {errors.username}
                  </Form.Control.Feedback>
                </Form.Group>
                <Form.Group as={Col} md="12" controlId="password">
                  <Form.Label>Password</Form.Label>
                  <Form.Control
                    type="password"
                    name="password"
                    placeholder="Password"
                    value={values.password || ""}
                    onChange={handleChange}
                    isInvalid={touched.password && errors.password}
                  />

                  <Form.Control.Feedback type="invalid">
                    {errors.password}
                  </Form.Control.Feedback>
                </Form.Group>
              </Form.Row>
              <Button type="submit" style={{ marginRight: "10px" }}>
                Save
              </Button>
            </Form>
          )}
        </Formik>

        <br />
        <h2>BitBucket Settings</h2>
        <Formik
          validationSchema={bitBucketFormSchema}
          onSubmit={handleBitbucketSubmit}
          initialValues={bitbucketUser}
          enableReinitialize={true}
        >
          {({
            handleSubmit,
            handleChange,
            handleBlur,
            values,
            touched,
            isInvalid,
            errors
          }) => (
            <Form noValidate onSubmit={handleSubmit}>
              <Form.Row>
                <Form.Group as={Col} md="12" controlId="bitBucketUsername">
                  <Form.Label>BitBucket Username</Form.Label>
                  <Form.Control
                    type="text"
                    name="bitBucketUsername"
                    placeholder="BitBucket Username"
                    value={values.bitBucketUsername || ""}
                    onChange={handleChange}
                    isInvalid={
                      touched.bitBucketUsername && errors.bitBucketUsername
                    }
                  />
                  <Form.Control.Feedback type="invalid">
                    {errors.bitBucketUsername}
                  </Form.Control.Feedback>
                </Form.Group>
                <Form.Group as={Col} md="12" controlId="bitBucketPassword">
                  <Form.Label>Password</Form.Label>
                  <Form.Control
                    type="password"
                    name="bitBucketPassword"
                    placeholder="Bitbucket Password"
                    value={values.bitBucketPassword || ""}
                    onChange={handleChange}
                    isInvalid={
                      touched.bitBucketPassword && errors.bitBucketPassword
                    }
                  />

<Form.Control.Feedback type="invalid">
                    {errors.bitbucketPassword}
                  </Form.Control.Feedback>
                </Form.Group>
              </Form.Row>
              <Button type="submit" style={{ marginRight: "10px" }}>
                Save
              </Button>
            </Form>
          )}
        </Formik>
      </div>
    </>
  );
}

export default SettingsPage;

We have 2 forms. One for setting the password of the user and another for saving the Bitbucket credentials. We valid both in separate Yup schemas and make the request to back end if the data entered is valid.

Next we create the sign up page by creating SignUpPage.js in the src folder. We add:

import React, { useState, useEffect } from "react";
import { Formik } from "formik";
import Form from "react-bootstrap/Form";
import Col from "react-bootstrap/Col";
import Button from "react-bootstrap/Button";
import { Redirect } from "react-router-dom";
import * as yup from "yup";
import { signUp } from "./requests";
import Navbar from "react-bootstrap/Navbar";

const schema = yup.object({
  username: yup.string().required("Username is required"),
  password: yup.string().required("Password is required")
});

function SignUpPage() {
  const [redirect, setRedirect] = useState(false);

  const handleSubmit = async evt => {
    const isValid = await schema.validate(evt);
    if (!isValid) {
      return;
    }
    try {
      await signUp(evt);
      setRedirect(true);
    } catch (ex) {
      alert("Username already taken");
    }
  };

  if (redirect) {
    return <Redirect to="/" />;
  }

  return (
    <>
      <Navbar bg="primary" expand="lg" variant="dark">
        <Navbar.Brand href="#home">Bitbucket App</Navbar.Brand>
      </Navbar>
      <div className="page">
        <h1 className="text-center">Sign Up</h1>
        <Formik validationSchema={schema} onSubmit={handleSubmit}>
          {({
            handleSubmit,
            handleChange,
            handleBlur,
            values,
            touched,
            isInvalid,
            errors
          }) => (
            <Form noValidate onSubmit={handleSubmit}>
              <Form.Row>
                <Form.Group as={Col} md="12" controlId="username">
                  <Form.Label>Username</Form.Label>
                  <Form.Control
                    type="text"
                    name="username"
                    placeholder="Username"
                    value={values.username || ""}
                    onChange={handleChange}
                    isInvalid={touched.username && errors.username}
                  />
                  <Form.Control.Feedback type="invalid">
                    {errors.username}
                  </Form.Control.Feedback>
                </Form.Group>
                <Form.Group as={Col} md="12" controlId="password">
                  <Form.Label>Password</Form.Label>
                  <Form.Control
                    type="password"
                    name="password"
                    placeholder="Password"
                    value={values.password || ""}
                    onChange={handleChange}
                    isInvalid={touched.password && errors.password}
                  />

                  <Form.Control.Feedback type="invalid">
                    {errors.password}
                  </Form.Control.Feedback>
                </Form.Group>
              </Form.Row>
              <Button type="submit" style={{ marginRight: "10px" }}>
                Sign Up
              </Button>
            </Form>
          )}
        </Formik>
      </div>
    </>
  );
}

export default SignUpPage;

It’s similar to the other log in request, except we call the sign up request instead of log in.

Finally, in index.html , we replace the existing code with:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="shortcut icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="logo192.png" />
    <!--
      manifest.json provides metadata used when your web app is installed on a
      user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
    -->
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <!--
      Notice the use of %PUBLIC_URL% in the tags above.
      It will be replaced with the URL of the `public` folder during the build.
      Only files inside the `public` folder can be referenced from the HTML.

      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
      work correctly both with client-side routing and a non-root public URL.
      Learn how to configure a non-root public URL by running `npm run build`.
    -->
    <title>Bitbucket App</title>
    <link
      rel="stylesheet"
      href="https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
      integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T"
      crossorigin="anonymous"
    />
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <!--
      This HTML file is a template.
      If you open it directly in the browser, you will see an empty page.

      You can add webfonts, meta tags, or analytics to this file.
      The build step will place the bundled scripts into the <body> tag.

      To begin the development, run `npm start` or `yarn start`.
      To create a production bundle, use `npm run build` or `yarn build`.
    -->
  </body>
</html>

to add the Bootstrap CSS and change the title.

After writing all that code, we can run our app. Before running anything, install nodemon by running npm i -g nodemon so that we don’t have to restart back end ourselves when files change.

Then run back end by running npm start in the backend folder and npm start in the frontend folder, then choose ‘yes’ if you’re asked to run it from a different port.

Then you get:

Categories
JavaScript

Storing Key-Value Pairs With JavaScript Maps

With ES2015, we have a new data structure to store key-value pairs, called Maps, which are dictionaries that we can use to store any object as keys and any object as values.

Before we had Maps, we had to use objects as dictionaries that only use strings as keys, but values could be anything.

Using Maps is simple, we can define Maps like in the following code:

let messageMap = new Map();  
messageMap.set('hi', 'Hello');  
messageMap.set('greeting', 'How are you doing?');  
messageMap.set('bye', 'Bye');

Instead of using the set method to add our keys and values, we can also pass a nested array where each entry of the array has the key as the first element and the value as the second element.

For example, we can write:

const arrayOfKeysAndValues = [  
  ['a', 1],  
  ['b', 2],  
  ['c', 3],  
  ['d', 3],  
  ['e', 4],  
  ['f', 5],  
  ['g', 6],  
  ['h', 7],  
  ['i', 8],  
  ['j', 9],  
  ['k', 10],  
  ['l', 11],  
  ['m', 12],  
  ['n', 13],  
  ['o', 14],  
]  
let numMap = new Map(arrayOfKeysAndValues);  
console.log(numMap)

The code above will create a Map with the first element of each array as the key and the second element of each array as the corresponding value. So, after running the console.log line, we get all the entries of the Map listed in the same order as the array.

We can get the number of key-value pairs defined in a Map by using the size property:

messageMap.size // 3

To get the value by the key, we use the get function:

messageMap.get('hi'); // 'Hello'  
messageMap.get('hello'); // undefined

It returns the value if the key exists and undefined if an entry with the key doesn’t exist.

To remove an entry given the key, we can use the delete function with the key passed in:

messageMap.delete('hi');

If we call set with the same key passed in more than once, then the value in the later set call overwrites the earlier one. For example:

let messageMap = new Map();  
messageMap.set('hi', 'Hello');  
messageMap.set('greeting', 'How are you doing?');  
messageMap.set('bye', 'Bye');  
messageMap.set('hi', 'Hi');
console.log(messageMap.get('hi'))

If we run console.log(messageMap.get(‘hi’)), we get 'Hi' instead of 'Hello'.

To clear all the entries of a Map object, we can call the clear function, like in the following code:

let messageMap = new Map();  
messageMap.set('hi', 'Hello');  
messageMap.set('greeting', 'How are you doing?');  
messageMap.set('bye', 'Bye');  
messageMap.set('hi', 'Hi');  
messageMap.clear();  
console.log(messageMap)  
console.log(messageMap.size)

We see that messageMap should be empty when we log it and that its size should be 0.

The entries of a Map can be iterated over with a for...of loop. With the destructuring assignment of each entry, we can get the key and value of each entry with a for...of loop like in the following code:

let messageMap = new Map();  
messageMap.set('hi', 'Hello');  
messageMap.set('greeting', 'How are you doing?');  
messageMap.set('bye', 'Bye');

for (let [key, value] of messageMap) {  
  console.log(`${key} - ${value}`);  
}

Objects and Maps are different in multiple ways. The keys of objects can only be of the string type. In Maps, keys can be of any value.

We can get the size of a Map easily with the size property which isn’t available in objects. The iteration order of Maps is in the iteration order of the elements, while the iteration of the keys of an object isn’t guaranteed.

Object has a prototype, so there are default keys in an object, but not in a Map.

Maps are preferred is we want to store non-string keys, when keys are unknown until run time, and when all the keys are the same type and values are the same type which may or may not be the same type as the keys.

We can use the entries function to get the entries of a Map. For example, we can write:

let messageMap = new Map();  
messageMap.set('hi', 'Hello');  
messageMap.set('greeting', 'How are you doing?');  
messageMap.set('bye', 'Bye');

for (let [key, value] of messageMap.entries()) {  
  console.log(`${key} - ${value}`);  
}

To loop through the entries of the Map.

Keys are checked for equality by mostly following the rules of the triple equals operator, except that NaN is considered equal to itself and +0 and -0 are also considered equal. The algorithm is called the same-value-zero algorithm.

So, if we have:

const map = new Map();  
map.set(NaN, 'abc');

Then we run map.get(NaN), we get that 'abc' returned. Also, it’s important to note that we can use non-string values as keys like we did above. For example:

let messageMap = new Map();  
messageMap.set(1, 'Hello');  
messageMap.set(2, 'How are you doing?');  
messageMap.set(3, 'Bye');

If we call messageMap.get(1), we get 'Hello'.

However, since the keys are retrieved by checking with the triple equal operator with NaN considered equal to itself, we cannot get the value that you expect directly from object keys.

let messageMap = new Map();  
messageMap.set({ messageType: 'hi' }, 'Hello');  
messageMap.set({ messageType: 'greeting' }, 'How are you doing?');  
messageMap.set({ messageType: 'bye' }, 'Bye');  
const hi = messageMap.get({  
  messageType: 'hi'  
})
console.log(hi);

hi will be undefined as it doesn’t check the content of the object for equality.

We can use array methods on Map if we convert it to an array first and then convert it back to a Map. For example, if we want to multiply all the values in a Map by 2, then we can write the following:

let numMap = new Map();  
numMap.set('a', 1);  
numMap.set('b', 2);  
numMap.set('c', 3);

let multipliedBy2Array = [...numMap].map(([key, value]) => ([key, value * 2]));

let newMap = new Map(multipliedBy2Array);
console.log(newMap)

As we can see from the code above, we can use the spread operator to convert a Map to an Array directly by copying the values from the Map to the array.

Each entry consists of an array with the key as the first element and the value as the second element. Then we can call map on it since it’s an array. We can then multiply each value’s entry by 2, while keeping each entry’s array structure the same, with key first and value second.

Then we can pass it directly to the constructor of the Map object and we can see that the new Map object, newMap, has all the values multiplied by 2 while keeping the same keys as numMap.

Likewise, we can combine multiple Maps by first putting them in the same array with the spread operator and combining them into one, then passing it to the Map constructor to generate a new Map object.

For example, we can write:

let numMap1 = new Map();  
numMap1.set('a', 1);  
numMap1.set('b', 2);  
numMap1.set('c', 3);  
numMap1.set('d', 3);  
numMap1.set('e', 4);  
numMap1.set('f', 5);  
numMap1.set('g', 6);  
numMap1.set('h', 7);  
numMap1.set('i', 8);  
numMap1.set('j', 9);  
numMap1.set('k', 10);  
numMap1.set('l', 11);let numMap2 = new Map();  
numMap2.set('m', 1);  
numMap2.set('n', 2);  
numMap2.set('o', 3);  
numMap2.set('p', 3);  
numMap2.set('q', 4);  
numMap2.set('r', 5);  
numMap2.set('s', 6);  
numMap2.set('t', 7);  
numMap2.set('u', 8);  
numMap2.set('v', 9);  
numMap2.set('w', 10);  
numMap2.set('x', 11);let numMap3 = new Map();  
numMap3.set('y', 1);  
numMap3.set('z', 2);  
numMap3.set('foo', 3);  
numMap3.set('bar', 3);  
numMap3.set('baz', 4);

const combinedArray = [...numMap1, ...numMap2, ...numMap3];  
const combinedNumMap = new Map(combinedArray);

When we log combinedNumMap, we see that we have all the original keys and values intact, but they were combined into one Map.

If we have overlapping keys, then the one that’s inserted later overwrites the values that were inserted earlier. For example:

let numMap1 = new Map();  
numMap1.set('a', 1);  
numMap1.set('b', 2);  
numMap1.set('c', 3);  
numMap1.set('d', 3);  
numMap1.set('e', 4);  
numMap1.set('f', 5);  
numMap1.set('g', 6);  
numMap1.set('h', 7);  
numMap1.set('i', 8);  
numMap1.set('j', 9);  
numMap1.set('k', 10);  
numMap1.set('l', 11);

let numMap2 = new Map();  
numMap2.set('m', 1);  
numMap2.set('n', 2);  
numMap2.set('o', 3);  
numMap2.set('p', 3);  
numMap2.set('q', 4);  
numMap2.set('r', 5);  
numMap2.set('s', 6);  
numMap2.set('t', 7);  
numMap2.set('u', 8);  
numMap2.set('v', 9);  
numMap2.set('w', 10);  
numMap2.set('x', 11);let numMap3 = new Map();  
numMap3.set('y', 1);  
numMap3.set('z', 2);  
numMap3.set('a', 21);  
numMap3.set('b', 22);  
numMap3.set('c', 23);

const combinedArray = [...numMap1, ...numMap2, ...numMap3];  
const combinedNumMap = new Map(combinedArray);

When we log combinedNumMap, we see that we have most of the original keys and values intact, but they were combined into one Map. However, a now maps to 21, b now maps to 22, and c now maps to 23.

With Maps, we can have key-value pairs where the keys aren’t strings. Maps are collections that can be iterated in the order that the key-value pairs are inserted in.

They can be converted to arrays with each entry being an array with the key as the first element and the value as the second element, which means that we can convert them to arrays and then use arras operations like array methods.

We can use the spread operator to manipulate them in the way that we can do with any arrays, then convert them back to Maps again. We can use the set function to add more entries, and the get function to get the value of the given key.

The values are retrieved by the keys by the same-value-zero algorithm which means the keys are matched by the triple equals operator, with the exception that -0 and +0 are considered equal and NaN is considered equal to itself.

Categories
React

How to Build a Simple Video Converter with Node.js

Videos are an incredibly popular medium for sharing information, but in the past, converting videos has always been a problem for people. With FFMPEG, this can now be solved easily.

What is FFMPEG

FFMPEG is a command line video and audio processing program that has many capabilities. It also supports multiple formats for video conversion. The capabilities of FFMPEG include: getting video and audio metadata, changing the resolution of videos, changing audio quality, compressing videos, removing audio from videos, remove video from audio, extract frames from video, change video aspect ratio, and much more.

Developers have done the hard work for us by creating a Node.js wrapper for FFMPEG. The package is called fluent-ffmpeg. It is located at https://github.com/fluent-ffmpeg/node-fluent-ffmpeg. This package allows us to run FFMPEG commands by calling the built-in functions.

In this article, we will build a video converter to change the source video into the format of the user’s choice. We will use the fluent-ffmpeg package for running the conversions. Because the jobs are long-running, we will also create a job queue so that it will run in the background.

We will use Express for the back end and React for the front end.

Back End

To get started, create a project directory and a backend folder inside it. In the backend folder, run npx express-generator to generate the files for the Express framework.

Next run npm i in the backend folder to download the packages in package.json.

Then we have to install our own packages. We need Babel to use import in our app. Also, we will use the Bull package for background jobs, CORS package for cross domain requests with the front end, fluent-ffmpeg for converting videos, Multer for file upload, Dotenv for managing environment variables, Sequelize for ORM, and SQLite3 for or database.

Run npm i @babel/cli @babel/core @babel/node @babel/preset-env bull cors dotenv fluent-ffmpeg multer sequelize sqlite3 to install all the packages.

Next add the .babelrc file to the backend folder:

{
    "presets": [
        "@babel/preset-env"
    ]
}

This enables the latest JavaScript features.

In the scripts section of package.json, replace the existing code with:

"start": "nodemon --exec npm run babel-node --  ./bin/www",
"babel-node": "babel-node"

This runs our program with Babel instead of the regular Node runtime.

Next, we create the Sequelize code by running npx sequelize-cli init which creates a config.json file.

Then in config.json, we replace the existing code with:

{
  "development": {
    "dialect": "sqlite",
    "storage": "development.db"
  },
  "test": {
    "dialect": "sqlite",
    "storage": "test.db"
  },
  "production": {
    "dialect": "sqlite",
    "storage": "production.db"
  }
}

Next we need to create our model and migration. We run:

npx sequelize-cli --name VideoConversion --attributes filePath:string,convertedFilePath:string,outputFormat:string,status:enum

To create the model and migration for the VideoConversions table.

In the newly created migration file, replace the existing code with:

"use strict";
module.exports = {
  up: (queryInterface, Sequelize) => {
    return queryInterface.createTable("VideoConversions", {
      id: {
        allowNull: false,
        autoIncrement: true,
        primaryKey: true,
        type: Sequelize.INTEGER
      },
      filePath: {
        type: Sequelize.STRING
      },
      convertedFilePath: {
        type: Sequelize.STRING
      },
      outputFormat: {
        type: Sequelize.STRING
      },
      status: {
        type: Sequelize.ENUM,
        values: ["pending", "done", "cancelled"],
        defaultValue: "pending"
      },
      createdAt: {
        allowNull: false,
        type: Sequelize.DATE
      },
      updatedAt: {
        allowNull: false,
        type: Sequelize.DATE
      }
    });
  },
  down: (queryInterface, Sequelize) => {
    return queryInterface.dropTable("VideoConversions");
  }
};

This adds the constants for our enum.

In models/videoconversion.js, replace the existing code with:

"use strict";
module.exports = (sequelize, DataTypes) => {
  const VideoConversion = sequelize.define(
    "VideoConversion",
    {
      filePath: DataTypes.STRING,
      convertedFilePath: DataTypes.STRING,
      outputFormat: DataTypes.STRING,
      status: {
        type: DataTypes.ENUM("pending", "done", "cancelled"),
        defaultValue: "pending"
      }
    },
    {}
  );
  VideoConversion.associate = function(models) {
    // associations can be defined here
  };
  return VideoConversion;
};

This add the enum constants to the model.

Next run npx sequelize-init db:migrate to create our database.

Then create a files folder in the backend directory for storing the files.

Next we create our video processing job queue. Create a queues folder and inside it, create videoQueue.js file and add:

const Queue = require("bull");
const videoQueue = new Queue("video transcoding");
const models = require("../models");
var ffmpeg = require("fluent-ffmpeg");
const fs = require("fs");

const convertVideo = (path, format) => {
  const fileName = path.replace(/.[^/.]+$/, "");
  const convertedFilePath = `${fileName}_${+new Date()}.${format}`;
  return new Promise((resolve, reject) => {
    ffmpeg(`${__dirname}/../files/${path}`)
      .setFfmpegPath(process.env.FFMPEG_PATH)
      .setFfprobePath(process.env.FFPROBE_PATH)
      .toFormat(format)
      .on("start", commandLine => {
        console.log(`Spawned Ffmpeg with command: ${commandLine}`);
      })
      .on("error", (err, stdout, stderr) => {
        console.log(err, stdout, stderr);
        reject(err);
      })
      .on("end", (stdout, stderr) => {
        console.log(stdout, stderr);
        resolve({ convertedFilePath });
      })
      .saveToFile(`${__dirname}/../files/${convertedFilePath}`);
  });
};

videoQueue.process(async job => {
  const { id, path, outputFormat } = job.data;
  try {
    const conversions = await models.VideoConversion.findAll({ where: { id } });
    const conv = conversions[0];
    if (conv.status == "cancelled") {
      return Promise.resolve();
    }
    const pathObj = await convertVideo(path, outputFormat);
    const convertedFilePath = pathObj.convertedFilePath;
    const conversion = await models.VideoConversion.update(
      { convertedFilePath, status: "done" },
      {
        where: { id }
      }
    );
    Promise.resolve(conversion);
  } catch (error) {
    Promise.reject(error);
  }
});

export { videoQueue };

In the convertVideo function, we use fluent-ffmpeg to get the video file, then set the FFMPEG and FFProbe paths from the environment variables. Then we call toFormat to convert it to the format that the user wants. We add a log statement in the start, error, and end handlers to see the outputs and resolve our promise on the end event. When the conversion is complete, we save it to a new file.

videoQueue is a Bull queue that processes jobs in the background sequentially. Redis is required to run the queue, we will need a Ubuntu Linux installation. We run the following commands in Ubuntu to install and run Redis:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install redis-server
$ redis-server

In the callback of the videoQueue.process function, we call the convertVideo function and update the path of the converted file and the status of the given job when the job is done.

Next we create our routes. Create a conversions.js file in the routes folder and add:

var express = require("express");
var router = express.Router();
const models = require("../models");
var multer = require("multer");
const fs = require("fs").promises;
const path = require("path");
import { videoQueue } from "../queues/videoQueue";
const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, "./files");
  },
  filename: (req, file, cb) => {
    cb(null, `${+new Date()}_${file.originalname}`);
  }
});
const upload = multer({ storage });

router.get("/", async (req, res, next) => {
  const conversions = await models.VideoConversion.findAll();
  res.json(conversions);
});

router.post("/", upload.single("video"), async (req, res, next) => {
  const data = { ...req.body, filePath: req.file.path };
  const conversion = await models.VideoConversion.create(data);
  res.json(conversion);
});

router.delete("/:id", async (req, res, next) => {
  const id = req.params.id;
  const conversions = await models.VideoConversion.findAll({ where: { id } });
  const conversion = conversions[0];
  try {
    await fs.unlink(`${__dirname}/../${conversion.filePath}`);
    if (conversion.convertedFilePath) {
      await fs.unlink(`${__dirname}/../files/${conversion.convertedFilePath}`);
    }
  } catch (error) {
  } finally {
    await models.VideoConversion.destroy({ where: { id } });
    res.json({});
  }
});

router.put("/cancel/:id", async (req, res, next) => {
  const id = req.params.id;
  const conversion = await models.VideoConversion.update(
    { status: "cancelled" },
    {
      where: { id }
    }
  );
  res.json(conversion);
});

router.get("/start/:id", async (req, res, next) => {
  const id = req.params.id;
  const conversions = await models.VideoConversion.findAll({ where: { id } });
  const conversion = conversions[0];
  const outputFormat = conversion.outputFormat;
  const filePath = path.basename(conversion.filePath);
  await videoQueue.add({ id, path: filePath, outputFormat });
  res.json({});
});

module.exports = router;

In the POST / route, we accept the file upload with the Multer package. We add the job and save the file to the files folder that we created before. We save it with the file’s original name in the filename function in the object we passed into the diskStorage function and specified the file be saved in the files folder in the destination function.

The GET route / route gets the jobs added. DELETE / deletes the job with the given ID along with the original and converted file of the job. PUT /cancel/:id route sets status to cancelled .

The GET /start/:id route adds the job with the given ID to the queue we created earlier.

In app.js, we replace the existing code with:

require("dotenv").config();
var createError = require("http-errors");
var express = require("express");
var path = require("path");
var cookieParser = require("cookie-parser");
var logger = require("morgan");
var cors = require("cors");

var indexRouter = require("./routes/index");
var conversionsRouter = require("./routes/conversions");

var app = express();

// view engine setup
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "jade");

app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, "public")));
app.use(express.static(path.join(__dirname, "files")));
app.use(cors());

app.use("/", indexRouter);
app.use("/conversions", conversionsRouter);

// catch 404 and forward to error handler
app.use(function(req, res, next) {
  next(createError(404));
});

// error handler
app.use(function(err, req, res, next) {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get("env") === "development" ? err : {};

// render the error page
  res.status(err.status || 500);
  res.render("error");
});

module.exports = app;

This adds the CORS add-on to enable cross domain communication, exposes the files folder to the public, and exposes the conversions routes we created earlier to the public.

To add the environment variables, create an .env file in the backend folder and add:

FFMPEG_PATH='c:ffmpegbinffmpeg.exe'
FFPROBE_PATH='c:ffmpegbinffprobe.exe'

Front End

With back end done, we can move on to the front end. In the project’s root folder, run npx create-react-app frontend to create the front end files.

Next we install some packages. We need Axios for making HTTP requests, Formik for form value handling, MobX for state management, React Router for routing URLs to our pages, and Bootstrap for styling.

Run npm i axios bootstrap formik mobx mobx-react react-bootstrap react-router-dom to install the packages.

Next we replace the existing code in App.js with:

import React from "react";
import { Router, Route } from "react-router-dom";
import "./App.css";
import { createBrowserHistory as createHistory } from "history";
import HomePage from "./HomePage";
import { ConversionsStore } from "./store";
import TopBar from "./TopBar";
const conversionsStore = new ConversionsStore();
const history = createHistory();

function App() {
  return (
    <div className="App">
      <TopBar />
      <Router history={history}>
        <Route
          path="/"
          exact
          component={props => (
            <HomePage {...props} conversionsStore={conversionsStore} />
          )}
        />
      </Router>
    </div>
  );
}

export default App;

We add the top bar and the routes in this file.

In App.css, we replace the existing code with:

.page {
  padding: 20px;
}

.button {
  margin-right: 10px;
}

Next create HomePage.js in the src folder and add:

import React from "react";
import Table from "react-bootstrap/Table";
import Button from "react-bootstrap/Button";
import ButtonToolbar from "react-bootstrap/ButtonToolbar";
import { observer } from "mobx-react";
import {
  getJobs,
  addJob,
  deleteJob,
  cancel,
  startJob,
  APIURL
} from "./request";
import { Formik } from "formik";
import Form from "react-bootstrap/Form";
import Col from "react-bootstrap/Col";

function HomePage({ conversionsStore }) {
  const fileRef = React.createRef();
  const [file, setFile] = React.useState(null);
  const [fileName, setFileName] = React.useState("");
  const [initialized, setInitialized] = React.useState(false);

  const onChange = event => {
    setFile(event.target.files[0]);
    setFileName(event.target.files[0].name);
  };

  const openFileDialog = () => {
    fileRef.current.click();
  };

  const handleSubmit = async evt => {
    if (!file) {
      return;
    }
    let bodyFormData = new FormData();
    bodyFormData.set("outputFormat", evt.outputFormat);
    bodyFormData.append("video", file);
    await addJob(bodyFormData);
    getConversionJobs();
  };

  const getConversionJobs = async () => {
    const response = await getJobs();
    conversionsStore.setConversions(response.data);
  };

  const deleteConversionJob = async id => {
    await deleteJob(id);
    getConversionJobs();
  };

  const cancelConversionJob = async id => {
    await cancel(id);
    getConversionJobs();
  };

  const startConversionJob = async id => {
    await startJob(id);
    getConversionJobs();
  };

  React.useEffect(() => {
    if (!initialized) {
      getConversionJobs();
      setInitialized(true);
    }
  });

  return (
    <div className="page">
      <h1 className="text-center">Convert Video</h1>
      <Formik onSubmit={handleSubmit} initialValues={{ outputFormat: "mp4" }}>
        {({
          handleSubmit,
          handleChange,
          handleBlur,
          values,
          touched,
          isInvalid,
          errors
        }) => (
          <Form noValidate onSubmit={handleSubmit}>
            <Form.Row>
              <Form.Group
                as={Col}
                md="12"
                controlId="outputFormat"
                defaultValue="mp4"
              >
                <Form.Label>Output Format</Form.Label>
                <Form.Control
                  as="select"
                  value={values.outputFormat || "mp4"}
                  onChange={handleChange}
                  isInvalid={touched.outputFormat && errors.outputFormat}
                >
                  <option value="mov">mov</option>
                  <option value="webm">webm</option>
                  <option value="mp4">mp4</option>
                  <option value="mpeg">mpeg</option>
                  <option value="3gp">3gp</option>
                </Form.Control>
                <Form.Control.Feedback type="invalid">
                  {errors.outputFormat}
                </Form.Control.Feedback>
              </Form.Group>
            </Form.Row>

            <Form.Row>
              <Form.Group as={Col} md="12" controlId="video">
                <input
                  type="file"
                  style={{ display: "none" }}
                  ref={fileRef}
                  onChange={onChange}
                  name="video"
                />
                <ButtonToolbar>
                  <Button
                    className="button"
                    onClick={openFileDialog}
                    type="button"
                  >
                    Upload
                  </Button>
                  <span>{fileName}</span>
                </ButtonToolbar>
              </Form.Group>
            </Form.Row>

            <Button type="submit">Add Job</Button>
          </Form>
        )}
      </Formik>

      <br />
      <Table>
        <thead>
          <tr>
            <th>File Name</th>
            <th>Converted File</th>
            <th>Output Format</th>
            <th>Status</th>
            <th>Start</th>
            <th>Cancel</th>
            <th>Delete</th>
          </tr>
        </thead>
        <tbody>
          {conversionsStore.conversions.map(c => {
            return (
              <tr>
                <td>{c.filePath}</td>
                <td>{c.status}</td>
                <td>{c.outputFormat}</td>
                <td>
                  {c.convertedFilePath ? (
                    <a href={`${APIURL}/${c.convertedFilePath}`}>Open</a>
                  ) : (
                    "Not Available"
                  )}
                </td>
                <td>
                  <Button
                    className="button"
                    type="button"
                    onClick={startConversionJob.bind(this, c.id)}
                  >
                    Start
                  </Button>
                </td>
                <td>
                  <Button
                    className="button"
                    type="button"
                    onClick={cancelConversionJob.bind(this, c.id)}
                  >
                    Cancel
                  </Button>
                </td>
                <td>
                  <Button
                    className="button"
                    type="button"
                    onClick={deleteConversionJob.bind(this, c.id)}
                  >
                    Delete
                  </Button>
                </td>
              </tr>
            );
          })}
        </tbody>
      </Table>
    </div>
  );
}

export default observer(HomePage);

This is the home page of our app. We have a dropdown for selecting the format of the file, an upload button for select the file for conversion, and a table for displaying the video conversion jobs with the status and file names of the source and converted files.

We also have buttons to start, cancel, and delete each job.

To add file upload, we have a hidden file input and in the onChange handler of the file input. The Upload button’s onClick handler will click the file input to open the upload file dialog.

We get latest jobs by calling getConversionJobs when we first load the page, and also when we start, cancel, and delete jobs. The job data is stored in the MobX store that we will create later. We wrap observer in our HomePage in the last line to always get the latest values from the store.

Next create request.js and the src folder and add:

const axios = require("axios");
export const APIURL = "http://localhost:3000";

export const getJobs = () => axios.get(`${APIURL}/conversions`);

export const addJob = data =>
  axios({
    method: "post",
    url: `${APIURL}/conversions`,
    data,
    config: { headers: { "Content-Type": "multipart/form-data" } }
  });

export const cancel = id => axios.put(`${APIURL}/conversions/cancel/${id}`, {});

export const deleteJob = id =>
  axios.delete(`${APIURL}/conversions/${id}`);

export const startJob = id => axios.get(`${APIURL}/conversions/start/${id}`);

The HTTP requests that we make to the back end are all here. They were used on the HomePage .

Next create the MobX store by creating the store.js file in the src folder. In there add:

import { observable, action, decorate } from "mobx";

class ConversionsStore {
  conversions = [];

  setConversions(conversions) {
    this.conversions = conversions;
  }
}

ConversionsStore = decorate(ConversionsStore, {
  conversions: observable,
  setConversions: action
});

export { ConversionsStore };

This is a simple store that persists the contacts, and the conversions array is where we store the contacts for the whole app. The setConversions function lets us set contacts from any component where we pass in this store object to.

Next create TopBar.js in the src folder and add:

import React from "react";
import Navbar from "react-bootstrap/Navbar";
import Nav from "react-bootstrap/Nav";

function TopBar() {
  return (
    <Navbar bg="primary" expand="lg" variant="dark">
      <Navbar.Brand href="#home">Video Converter</Navbar.Brand>
      <Navbar.Toggle aria-controls="basic-navbar-nav" />
      <Navbar.Collapse id="basic-navbar-nav">
        <Nav className="mr-auto">
          <Nav.Link href="/">Home</Nav.Link>
        </Nav>
      </Navbar.Collapse>
    </Navbar>
  );
}

export default TopBar;

This contains the React Bootstrap Navbar to show a top bar with a link to the home page and the name of the app.

In index.html, we replace the existing code with:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="shortcut icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="logo192.png" />
    <!--
      manifest.json provides metadata used when your web app is installed on a
      user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
    -->
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <!--
      Notice the use of %PUBLIC_URL% in the tags above.
      It will be replaced with the URL of the `public` folder during the build.
      Only files inside the `public` folder can be referenced from the HTML.

      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
      work correctly both with client-side routing and a non-root public URL.
      Learn how to configure a non-root public URL by running `npm run build`.
    -->
    <title>Video Converter</title>
    <link
      rel="stylesheet"
      href="https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
      integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T"
      crossorigin="anonymous"
    />
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <!--
      This HTML file is a template.
      If you open it directly in the browser, you will see an empty page.

      You can add webfonts, meta tags, or analytics to this file.
      The build step will place the bundled scripts into the <body> tag.

      To begin the development, run `npm start` or `yarn start`.
      To create a production bundle, use `npm run build` or `yarn build`.
    -->
  </body>
</html>

This adds the Bootstrap CSS and changes the title.

After writing all that code, we can run our app. Before running anything, install nodemon by running npm i -g nodemon so that we don’t have to restart the back end ourselves when files change.

Then run back end by running npm start in the backend folder and npm start in the frontend folder, then choose ‘yes’ if you’re asked to run it from a different port.

Categories
JavaScript

Top Frameworks and Libraries to Make Developing Web Apps Easier

With plain JavaScript we can do a lot of things. However, there’s no set structure or if we write plain JavaScript code, so developers need a lot of discipline to structure their code properly. This takes a lot of effort and so it wastes a lot of developer’s time enforcing structure of every part of the code.

Manipulating the DOM is also very fragile because in today’s web apps, there are way too many things going on which can change the DOM. This means that elements appear and disappear in indeterminate amount of times. This create issues since we might be trying to manipulate DOM elements that aren’t there yet. Or multiple pieces of code might try to modify the same element at the same time.

It’s just too difficult to troubleshoot those issues and get your code working correctly. This means that a lot of time on these little issues, which takes away developer’s time that can be used to develop features. This is where frameworks come in to save the day.

Frameworks provide structure which allows teams of developers to develop complex web apps in a structured way. Most frameworks enforce their own conventions so structures can be enforced if multiple developers develop the same application. DOM manipulation is abstracted away in frameworks so that developers do not have to worry about that unless they develop features that need direct DOM manipulation like animations.

Libraries are small pieces of code that we can include in our app’s code to extend the functionality of our apps. Libraries make add functionality easier and some also provide shortcuts that does the equivalent of what’s available in plain JavaScript but better. There are also libraries that are made for specific frameworks to enhance their functionality.

Vue.js

Vue.js is a component based front end JavaScript framework that lets developers create high quality maintainable web apps in an easy way. The framework lets developers divide their app into Vue.js components, which are self contained parts that when put together, we get a full web app. Components are made to be nested within each other. Each component has a template for rendering the data, a script section for the logic, and a styles section for the CSS styles. Component support data binding between the template and script sections which means that changes from the template section is automatically reflected in the script section and vice versa. Each component has its own life cycle and has function to handle those life cycle events. Parent components can pass data into child components and child component can emit events to pass components into the parent.

With Vue.js, DOM manipulation is done in the background by getting the difference between the browser DOM and the virtual DOM tree it creates and make the changes from the virtual DOM to the actual DOM.

Also there’s a built in URL router called the Vue Router that lets developers map specific URLs into the pages of their apps. For centralize state management it uses the Vuex flux library which lets us use the flux architecture to store the app’s state centrally. Flux architecture basically means that data are set in a central data store and then components that need the data always get the latest data from the store by observing the values from the store.

New apps usually use the Vue CLI to create the project and select what to include.

Other feature it supports out of the box includes testing, CSS preprocessors like SCSS and SASS to make writing complex CSS easier, and building to web components or progressive web apps, and other advanced features. It can also be added incrementally to legacy web apps by using script tags. The Vue.js framework and libraries made for Vue.js all support this. So it’s a great choice for upgrading the functionality of legacy web apps.

There’s large ecosystem of libraries that we can use with Vue.js applications including component libraries, helper modules geolocation and webRTC modules and many others.

Angular

Like Vue.js, Angular is a component based framework. It’s made by Google. Most of the things that’s part of the Vue.js feature set is also in Angular, except the state management part. By default, Angular applications are written in TypeScript he framework lets developers divide their app into Angular components, which are self contained parts that when put together, we get a full web app. Components are made to be nested within each other. Each component has a template file for rendering the data, a TypeScript file for the logic, and a styles file for styling, which can be CSS, SASS or SCSS. Also, there’s a test file to add your unit tests for each component. Component support data binding between the template and script sections which means that changes from the template section is automatically reflected in the TypeScript file and vice versa. Each component has its own life cycle and has function to handle those life cycle events. Parent components can pass data into child components and child component can emit events to pass components into the parent.

Many parts of the library uses reactive programming, which means that parts of the app watches for data changes and does something when it changes. This is done mainly with observables.

One unique feature of Angular is that it includes dependency injection so that we don’t have to worry about resolving the dependencies of Angular libraries ourselves, which saves a lot of headaches since most web apps probably lots of dependencies.

Angular apps are divided into modules, which we can combine into one to make one whole app. Each module include the libraries that are needed to be used by that module.

With Angular, DOM manipulation is done in the background by getting the difference between the browser DOM and the virtual DOM tree it creates and make the changes from the virtual DOM to the actual DOM.

Also there’s a built in URL router called the Angular Router that lets developers map specific URLs into the pages of their apps.

Other feature it supports out of the box includes building to web components or progressive web apps, and other advanced features. There’s large ecosystem of libraries that we can use with React applications including component libraries, helper modules like NgRx store for state management many others.

React

React is a component based library that we can use to make dynamic web apps. It uses the JSX syntax for writing JavaScript which looks like HTML, except that you can mix it with JavaScript directly. Each component has its own life cycle and has function to handle those life cycle events. It also provides nesting for components and also allow for higher order components, which are components that returns component. There are 2 kinds of components in React. One is class based components, which are written as a class, with a render method at the end to render the HTML. The other is function components which returns the JSX directly and rendered into HTML. Originally function components cannot have dynamic logic, but now that React has hooks, function components can also have logic code in it.

With React, DOM manipulation is done in the background by getting the difference between the browser DOM and the virtual DOM tree it creates and make the changes from the virtual DOM to the actual DOM.

It does not include anything other than a view library to render JavaScript into HTML, so if we want to build a full single page application, we need to add other libraries like React Router and a state management like Redux. There’s large ecosystem of libraries that we can use with React applications including component libraries, helper modules like React Router and Redux and many others.

Lodash

Lodash is a library which contains many handy methods for manipulating data. Examples include array methods like flatten to squash a nested array into an ordinary array. There’s also a flattenDeep to flatten all levels of a nested array into a flat array. Other functions include:

  • findIndex — find the index of a given object
  • findLastIndex — find the last index of a given object
  • pull — removes the first given item from the array
  • pullAll — remove all the given item from the array
  • reverse — reverse an array
  • sortedIndex — get the index to insert the given item so that the array stays sorted
  • get — function used to traverse an object to get the item with the given property and return null if it can’t traverse the object to get the item or if it’s null or undefined

There are many more functions that aren’t yet available in plain JavaScript, even though some like find and findIndex did make it into JavaScript’s standard library, so Lodash is still useful.

Moment.js

Manipulating time and date in JavaScript is painful. Even with the functions available with Date objects, there are plenty of chances of bugs and errors. Also, there are no functions for formatting times and dates which is a big problem. To solve this, we use moment.js. This is the best library available for dealing with time.

There were issues with time zones with YYYY-MM-DD dates being parsed into UTC time as opposed to local time. Which creates bugs for developers who aren’t aware of this problem. See https://stackoverflow.com/questions/29174810/javascript-date-timezone-issue

Also there are difference in support for parts of Dates in different browsers. (See https://stackoverflow.com/questions/11253351/javascript-date-object-issue-in-safari-and-ie)

It is also hard to add and subtract timestamps with built in Date functions. There is no way to do this without writing a lot of code and doing a lot of checks. Also, there is no way to compare 2 times.

Formatting dates is also not available without writing your own code for using third party libraries.

Moment.js solves all of these issues by providing built in functions to do all these common operations. It provides functions for parsing and formatting dates.

The momentconstructor is where you can pass in a date string, and a moment object will be created. For example, you can pass in:

moment('2019-08-04')

and you will get back a moment which you can compare with other moment objects, and add or subtract by different time spans.

If you do not passing in anything to the moment constructor, you get the current date and time.

It also takes a second argument. If you want to make sure a date is parsed as a YYYY-MM-DD date, then write moment(‘2019–08–04’, 'YYYY-MM-DD') . If you don’t know the format of your date or time, then you can pass in an array of possible formats and Moment will pick the right one:

moment('2019–08–04', ['YYYY-MM-DD', 'DD-MM-YYYY']);

After you create a Moment object, then you can do many things like formatting dates:

const a = moment('2019–08–04', 'YYYY-MM-DD').format('MMMM Do YYYY, h:mm:ss a');
console.log(a);// August 4th 2019, 12:00:00 am

const b = moment('2019–08–04', 'YYYY-MM-DD').format('dddd');
console.log(b);// Sunday

const c = moment('2019–08–04', 'YYYY-MM-DD').format("MMM Do YY");
console.log(c);// Aug 4th 19

const d = moment('2019–08–04', 'YYYY-MM-DD').format('YYYY [escaped] YYYY');
console.log(d);// 2019 escaped 2019

const e = moment('2019–08–04', 'YYYY-MM-DD').format();
console.log(e);// 2019-08-04T00:00:00-07:00

From the above examples, we see that we can format dates in pretty much any way we want.

We can also tell what time span a date is relative to another date by writing:

const augDate = moment('2019–08–04', 'YYYY-MM-DD');
const sepDate = moment('2019–09–04', 'YYYY-MM-DD');console.log(augDate.from(sepDate)); // a month ago

We can also add or subtract Moment dates:

const augDate = moment('2019–08–04', 'YYYY-MM-DD');
const sepDate = moment('2019–09–04', 'YYYY-MM-DD');console.log(augDate.add(10, 'days').calendar()); // 08/14/2019
console.log(augDate.subtract(10, 'days').calendar()); // 07/25/2019

It is easy to compare 2 dates

moment('2010-01-01').isSame('2010-01-01', 'month'); // true
moment('2010-01-01').isSame('2010-05-01', 'day');   // false, different month
moment('2008-01-01').isSame('2011-01-01', 'month'); // false, different year

You can also check if a date is has Daylight Saving Time in effect or not:

const augDate = moment('2019–08–04', 'YYYY-MM-DD');
const decDate = moment('2019–12–04', 'YYYY-MM-DD');
console.log(augDate.isDST()) // true
console.log(decDate.isDST()) // false

And you can convert back to JavaScript date any time by calling the toDate() function on a Moment object.

As we can see. There’re lots of things that Moment.js can do that can’t be done in plain JavaScript, so we need this library badly.

All in all, developing front end JavaScript web apps feels almost like magic with all these libraries and frameworks. It makes development a lot easier if you’re working alone and in teams. There’s no way to live without all these modern day libraries and frameworks. A lot of these things that frameworks and libraries do aren’t possible with plain JavaScript.