diff --git a/1-express-basics-slides.html b/1-express-basics-slides.html new file mode 100644 index 0000000..b646124 --- /dev/null +++ b/1-express-basics-slides.html @@ -0,0 +1,457 @@ +Express Basics
+

Express Basics

+
+
+

JS HTTP server

+
+
+

A simple HTTP server in Node

+

A simple server in three steps:

+
    +
  • include "http" library
    • +
  • create server responses
    • +
  • define the port that you want to listen to.
    • +
+

Note: The _ before a variable name is used to indicate that the variable isn't used. This is to avoid the VSCode warning about unused variable.

+
+
+
import http from 'http'
+
+const server = http.createServer((\_req, res) => {
+  res.write('Some cool response!')
+  res.end()
+})
+
+const port = 3000
+server.listen(port)
+
+console.log('HTTP Server listening port', port)
+
+

For this code to work, remember "type": "module" in your package.json. Also, you may need to change your port number to something other than 3000.

+
+
+

PORT numbers

+

The port number (16 bit unsigned integer) defines a port through which the communication to your application happens. Ports allow multiple applications to be located on the same host (IP).

+

The client connects to this port in some server (defined by an IP address), so the client and the server are able to communicate.

+

Computers have several different ports at their disposal but some of them are being used and your application might not be able to assign those ports to itself.

+
+
+
+
+

Ports from 0 to 1023 are the system ports. There are restrictions in manually assigning applications to use them.

+

Generally speaking, ports from 1024 to 65535 are available for user to assign applications to use them.

+
+
+

+
+
+
+
+

Request and response

+

The request object is an object that is sent from the client to the server.

+

The response object is an object that is sent from the server to the client. When you modify the response object, you're essentially modifying how the server responds to the client.

+

+const server = http.createServer((req, res) => {
+  res.write("Some cool response!");
+  res.end();
+});
+
+
+
+

Exercise 1: Simple Request & Response

+ +

Create a basic HTTP-server application. The application should respond to GET requests by returning a response with the following content: "Hello world!".

+

Test your application either with client such as Insomnia or Postman, or by navigating your browser to http://localhost:<your_portnumber>

+
+
+

Express Server

+
+
+

Express

+

Express is the most popular server-library for Node

+

Setting up a server with node-http is already fairly simple, but Express tries to make this even simpler

+

In some cases you can manage without Express, but for larger applications it's recommended

+

Install Express by using

+
npm install express
+
+
+
+

A side note about dependencies

+
    +
  • By using npm install <package>, the package will become dependency for the application.
    • +
  • By using npm install --save-dev <package>, the package will become a development dependency for the application. +
      +
    • This means the package won't be included in the built application.
      • +
    +
    • +
  • A shortcut for ---save-dev is -D
    • +
  • Dependencies are needed to run the application, development dependencies are needed for development.
    • +
  • Additional info: https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/
    • +
+
+
+

Getting started with Express

+

Initialize a new TypeScript project with appropriate tsconfig.json file and scripts for building and/or running the project in development mode using ts-node.

+

Create a very simple express project in index.ts

+
import express from 'express'
+
+const server = express()
+
+server.listen(3000, () => {
+  console.log('Listening to port 3000')
+})
+
+

Then start the server as you would any other TS project and connect to http://localhost:3000 with your browser or using an API tool such as Insomnia or Postman.

+
+
+

Express - Requests

+

The server does not yet do anything except listens. That is because we have not defined how it should deal with incoming requests.

+

With Express you can easily process different kind of requests, e.g. GET, POST, DELETE etc.

+

An example of how to handle a GET request to the root of the project (/):

+
import { Request, Response } from "express";
+
+server.get("/", (req: Request, res: Response) => {
+  res.send("Just saying hello!")
+})
+
+
+
+

TypeScript Request and Response

+

Notice how we import Request and Response from express. These types are defined in express, but identically named types are also available from JavaScript basic library.

+

If you use the Request and Response types without importing them, JS mistakenly thinks they are different objects, and complains about incompatibility.

+
+
+

Express - Endpoints

+

To create a new endpoint we define what kind of request we're handling, then what the route to the endpoint is and finally what the endpoint does.

+

In the below example we are creating an endpoint that handles a GET request to route /foobar, that sends a response containing the string 'OK'.

+
server.get('/foobar', (req, res) => {
+  res.send('OK')
+})
+
+
+
+

Nodemon

+

Reminder : If you are not using Nodemon in development, you are making your life unnecessarily difficult. Use Nodemon.

+

Instructions are below as a refresher.

+
+
+

Nodemon with JavaScript

+

One of the handiest packages out there is nodemon, which monitors your node program. Whenever you save your files, it reloads the program, showing you immediately the results of your changes.

+

Nodemon is installed as a dev dependency npm install --save-dev nodemon

+

To use nodemon, you should have a development script "dev": "nodemon ./index.js"

+
+
+

Nodemon with TypeScript

+

To get Nodemon working with TypeScript we need additional package to run our TS code without waiting for it to compile. npm install --save-dev nodemon ts-node

+

After installing, nodemon works directly with .ts files. "dev": "nodemon ./index.ts"

+
+
+

Exercise 2: Simple Express Server

+ +

Create a basic Express application. The application should respond to GET requests by returns a response with the following content: "Hello world!".

+

Extra: Add a new endpoint to your application. So, in addition to accessing http://localhost:3000, try to send something back from http://localhost:3000/endpoint2.

+
+
+

Request params & query

+

If request params are defined, they're something user must send to the server. You can think parameters as the "path" of the endpoint.

+

https://localhost:5000/ThisIsParam

+

Request query is optional information that the user can send to the server. A question mark (?) is used to separate the request query.

+

https://localhost:5000?ThisIsQuery=123

+

Chaining query parameters is done with et symbol (&)

+

https://localhost:5000?first=123&second=456

+
+
+

Request params & request query

+
app.get("/:name/:surname", (req: Request, res: Response) => {
+  console.log("Params:");
+  console.log(request.params);
+  console.log("Query:");
+  console.log(request.query);
+  response.send("Hello " + request.params.name);
+});
+
+app.listen(5000);
+
+

Try connecting via browser!

+

http://localhost:5000/John/Doe
+http://localhost:5000/John/Doe?query1=123

+
+
+

Request query & param types

+

Request parameters and query parameters are always of type string. If a number is passed, it will be a string that contains a number.

+

The example below will always return 404, since the strict equality item.id === id will never be true. This is because the item.id is of type number and id being of type string.

+
+
+
import express, { Request, Response } from 'express'
+
+const server = express()
+
+const data = [
+  { id: 1, name: 'John' },
+  { id: 2, name: 'Jane' }
+]
+
+server.get('/:id', (req: Request, res: Response) => {
+  const id = req.params.id
+  const info = data.find(item => item.id === id)
+  if (info === undefined) {
+    return res.status(404).send()
+  }
+
+  res.send(info)
+})
+
+server.listen(3000)
+
+
+
+

Exercise 3: Counter Server

+ +

Create an API that consists of only one endpoint: /counter
+Whenever you enter this endpoint from the browser, it should respond with a JSON object with information on how many times the endpoint has been accessed.

+

Extra: Make it possible to set the counter to whichever integer with a query parameter /counter?number=5

+
+
+

Exercise 4: Advanced Counter Server

+

Expand the API from the previous assignment by accepting a name through the counter endpoint: /counter/:name

+

When entering this endpoint, the server should return the count of how many times this named endpoint has been visited.

+

For example

+
    +
  • Aaron enters /counter/Aaron 🠖 "Aaron was here 1 times"
    • +
  • Aaron enters /counter/Aaron 🠖 "Aaron was here 2 times"
    • +
  • Beatrice enters /counter/Beatrice 🠖 "Beatrice was here 1 times"
    • +
  • Aaron enters /counter/Aaron 🠖 "Aaron was here 3 times"
    • +
+
+
+

Extra: "url" library

+

With the "url" library, you can parse url fields (i.e., requests) the client sends to your server.

+
import  url from "url";
+// URL module usage example
+
+const addressString = 'http://localhost:5000/default.html?year=2017&month=february'; // Define the address
+const address = url.parse(addressString, true); // Parse through the address with url.parse() function.
+console.log(address.host); // returns 'localhost:5000'
+console.log(address.pathname); // returns '/default.html'
+console.log(address.search); // returns '?year=2017&month=february'
+const query = address.query; // returns an object: { year: 2017, month: 'february' }
+console_ .log(query.month); // returns 'february
+
+
+
+

Express Middleware

+
+
+

Middlewares

+

Generally speaking, middleware is an application that does something in between some other applications.

+

There are tons of middleware libraries available for Express.

+

What this means in Express: with a middleware, you can manipulate the client request and server response in the client-server communication.

+

Middleware functions are functions that have access to the request object (req), the response object (res), and the next function in the application's request-response cycle. When the next function is invoked, the next middleware function is executed.

+
+
+

Middleware can be defined as such

+
const authCheck = (req: Request, res: Response, next: NextFunction) => {
+  if (req.headers.Authentication === undefined) {    // Read the request object
+    res.status(401).send('Missing Authentication Header')    // Modify the response object
+    return
+  }
+
+  next()    // Execute the next middleware
+}
+
+

In Express, .use() method initializes a middleware (argument 2) in a given path (argument 1).

+

In the middleware, the next() function will jump to the next middleware or endpoint.

+
server.use('/customers', authCheck)
+server.get('/customers', (req, res) => {
+// we know that Authentication header is present
+...
+
+
+
+

Exercise 5: Logger Middleware

+ +

Create a new project called Student Registry. We will be developing this program in stages during this lecture. For now it should have a single GET endpoint /students that returns an empty list.

+

Create a logger middleware function that is used in all the project's endpoints. The middleware should log

+
    +
  1. the time the request was made
    2. +
  2. the method of the request
    3. +
  3. the url of the endpoint
    4. +
+
+
+

Middleware - Unknown endpoint (AKA 404)

+

As you might have noticed, all websites have some kind of a default functionality for unknown endpoints.

+

This can be handled with a middleware.

+

Unknown endpoint has to be taken into use after the actual endpoints!

+
const unknownEndpoint = (_req: Request, res: Response) => {
+  res.status(404).send({error: 'Page not found'})
+}
+
+app.use(unknownEndpoint)
+
+
+
+

Middleware - Error handler

+

An error handler is often needed to handle different kind of user or application errors.

+

We can use middleware to enable error handling for our application.

+

Has to be the last middleware to be taken into use!

+
const errorHandler = (error: Error, req: Request, res: Response, next: NextFunction) => {
+  console.error(error.message)
+  if (error.name === 'MyCustomError') {
+    res.status(400).send({error: 'My Custom Error Message'})
+  }
+
+  next(error)
+}
+
+
+
+
+

Middleware - Req & Res

+

A need to manually alter req & res is quite common. To avoid bloated code, the handling can be easily done with proper middlewares.

+

Setting a header in middleware:

+
app.use((req: Request, res: Response, next: Next) => {
+  res.setHeader('Content-Type', 'image/png');
+  next();
+});
+
+
+
+

Getting a header and storing its value. If the header is not present, return 403 with an 'Unauthorized' message:

+
app.use((req: Request, res: Response, next: NextFunction) => {
+  const someIdNeededInApplication = req.header('Some-Custom-Header');
+  if (!someIdNeededInApplication) {
+    res.status(403).send('Unauthorized');
+  }
+
+  someIdFunctionality(someIdNeededInApplication);
+  next();
+});
+
+
+
+

Exercise 6: 404 Not Found

+ +

Add a middleware function that sends a response with status code 404 and an appropriate error message, if user tries to use an endpoint that does not exist.

+

If you have not already done so, move all your middleware to a separate file, to keep your program clean and readable.

+
+
+

Express.js: Creating a REST API

+
+
+

REST HTTP requests

+

We have gone through what a GET request is. Let's briefly go through what other types of requests are necessary for you.

+
    +
  • GET /products/:id → Returns a single product identified by its id
    • +
  • GET /products →Returns all the products
    • +
  • POST /products → Creates a new product
    • +
  • DELETE /products/:id → Removes a single product identified by its id
    • +
  • PUT /products/:id → Updates an existing product
    • +
+

Basic RESTFUL functionality is often referenced as a CRUD (Create, Read, Update, Delete).

+
+
+

Handling request content

+

To properly handle some requests, e.g., POST requests, in Express, a body-parser middleware must be taken into use. It's as simple as:

+
const server = express();
+server.use(express.json());
+
+

You might encounter a separate body-parser being used. This is no longer necessary, as Express 4.16+ includes a body parser middleware built-in.

+
+
+

express.urlencoded() allows us to parse url-encoded forms by attaching the data to the request body.

+
const app = express();
+app.use(express.json());
+app.use(express.urlencoded({extended: false}));
+
+

The "extended" parameter's false value only says that we are not dealing with any complicated objects (objects with sub objects, etc).

+
+
+

Basic POST request

+
const app = express();
+app.use(express.json());
+
+app.get("/", (req: Request, res: Response) => {
+  console.log("GET request init!");
+  res.sendFile(__dirname + "/index.html");
+});
+
+app.post("/", (rreq: Request, res: Response) => {
+  console.log("POST request init!");
+  console.log(req.body);
+  res.redirect("/");
+});
+
+
+
+

Request Body

+

GET methods should not in general have body parameters. The GET method should only be used to get data, the request should not convey data.

+

Despite this, most modern implementations can send and receive request bodies in all request types.

+

The requests that should not include body are CONNECT, GET, HEAD, OPTIONS and TRACE.

+
+
+

Request Body Types

+

Request body is parsed by the express body parser, and therefore it can contain any basic JavaScript data types. Notice that this differs from the request parameters and queries.

+

This also means that you do not know what the data type of any given body parameter will be. It might be necessary to check this in some cases.

+
+
+

TypeScript Body Casting

+

If you want to enforce type safety, the solution is to validate the user input before using it.

+

This is usually a good idea even if not using TypeScript, since it is important to know what type all the variables are in your code.

+
+
+
interface Body {
+  name: string;
+  age: number;
+}
+
+const validate = (req: Request, res: Response, next: NextFunction ) => {
+  const { age, name } = req.body;
+  if (typeof(age) !== 'number' || typeof(name) !== 'string') {
+    return res.status(400).send('Missing or invalid parameters');
+  }
+  next();
+}
+
+server.post('/', validate, (req: Request, res: Response) => {
+  const body: Body = req.body; // this has been validated
+  console.log(body);
+  res.send(body.name.toUpperCase());
+});
+
+
+
+

Exercise 7: Body Logging

+ +

Enable body parsing in your application.

+

Modify your logger middleware so that in addition to existing functionality, it also logs the request body if it exists.

+
+
+

Exercise 8: POST Requests

+ +

Add two more endpoints to your app:

+

POST /student should expect the request body to include student id, name and email. If some of these parameters are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should store the student information in an array called students. The endpoint should return an empty response with status code 201.

+

GET /student/:id should return the information of a single student, identified by the id request parameter. The response should be in JSON format. If there is no such student, the endpoint should return 404.

+

Modify the GET /students endpoint to return the list of all student ids, without names or emails.

+
+
+

Exercise 9: PUT and DELETE

+ +

Add two more endpoints to your app:

+

PUT /student/:id should expect the request body to include student name or email. If both are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should update an existing student identified by the request parameter id.

+

DELETE /student/:id should remove a student identified by the request parameter id.

+

Both endpoints should return 404 if there is no student matching the request parameter id. On success both endpoint should return an empty response with status code 204.

+
+

^ Tuon extran ei pitäisi olla extra, vaan perustehtävä. Extraksi ehkä toisen serverin käynnistäminen eri porttiin samanaikaisesti

\ No newline at end of file diff --git a/1-express-basics.md b/1-express-basics.md index 4ab6024..6f99020 100644 --- a/1-express-basics.md +++ b/1-express-basics.md @@ -1,288 +1,276 @@ +--- +marp: true +paginate: true +math: mathjax +theme: buutti +title: Express Basics +--- + # Express Basics + + + # JS HTTP server # A simple HTTP server in Node -__A simple server in three steps:__ - -__include "http" library__ +A simple server in three steps: +- include "http" library +- create server responses +- define the port that you want to listen to. -__create server responses__ +Note: The `_` before a variable name is used to indicate that the variable isn't used. This is to avoid the VSCode warning about unused variable. -__define the port that you want to listen to.__ - -__Note: The \_ before a variable name is used to indicate that the variable isn't used. This is to avoid the VSCode warning about unused variable__ +--- +```ts import http from 'http' const server = http.createServer((\_req, res) => { - -res.write('Some cool response!') - -res.end() - + res.write('Some cool response!') + res.end() }) const port = 3000 - server.listen(port) console.log('HTTP Server listening port', port) +``` For this code to work, remember "type": "module" in your package.json. Also, you may need to change your port number to something other than 3000. # PORT numbers -__The port number (16 bit unsigned integer) defines a port through which the communication to your application happens. Ports allow multiple applications to be located on the same host (IP).__ +The port number (16 bit unsigned integer) defines a port through which the communication to your application happens. Ports allow multiple applications to be located on the same host (IP). -__The client connects to this port in some server (defined by an IP address), so the client and the server are able to communicate.__ +The client connects to this port in some server (defined by an IP address), so the client and the server are able to communicate. -__Computers have several different ports at their disposal __ _but_ __ some of them are being used and your application might not be able to assign those ports to itself.__ +Computers have several different ports at their disposal but some of them are being used and your application might not be able to assign those ports to itself. -__Ports from 0 to 1023 are the system ports. There are restrictions in manually assigning applications to use them.__ +--- -__Generally speaking, ports from 1024 to 65535 are available for user to assign applications to use them.__ +
+
-![](imgs/1-express-basics_0.png) +Ports from 0 to 1023 are the system ports. There are restrictions in manually assigning applications to use them. -# Request and response +Generally speaking, ports from 1024 to 65535 are available for user to assign applications to use them. + +
+
-__The request object is an object that is sent__ __ from the client to the server.__ +![](imgs/1-express-basics_0.png) + +
+
-__The response object is an object that is sent__ __ from the server to the client__ __. When you modify the response object, you're essentially modifying how the server responds to the client.__ +# Request and response -_const_ server = http.createServer(( _req_ , _res_ ) _=>_ { +The request object is an object that is sent ***from the client to the server.*** -res.write("Some cool response!"); +The response object is an object that is sent ***from the server to the client***. When you modify the response object, you're essentially modifying how the server responds to the client. -res.end(); +```ts +const server = http.createServer((req, res) => { + res.write("Some cool response!"); + res.end(); }); +``` # Exercise 1: Simple Request & Response + Create a basic HTTP-server application. The application should respond to GET requests by returning a response with the following content: "Hello world!". -Test your application either with _Postman,_ or by navigating your browser to http://localhost:< _your\_portnumber_ > +Test your application either with client such as Insomnia or Postman, or by navigating your browser to `http://localhost:` ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Express Server # Express -__Express is the most popular server-library for Node__ +Express is the most popular server-library for Node -__Setting up a server with node-http is already fairly simple, but Express tries to make this even simpler__ +Setting up a server with node-http is already fairly simple, but Express tries to make this even simpler -__In some cases you can manage without Express, but for larger applications it's recommended__ +In some cases you can manage without Express, but for larger applications it's recommended -__Install Express by using__ +Install Express by using -__npm install express__ +``` +npm install express +``` # A side note about dependencies -* __By using __ __npm install __ __, the package will become dependency for the application.__ -* __By using __ __npm install --save-dev __ __, the package will become __ __development dependency__ __ for the application.__ - * __This means the package won't be included in the built application.__ -* __A shortcut for __ __--__ __save-dev__ __ is __ __-D__ -* __Dependencies are needed to run the application, dev-dependencies are needed for development.__ -* __Additional info: __ [https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/](https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/) +* By using `npm install `, the package will become dependency for the application. +* By using `npm install --save-dev `, the package will become *a development dependency* for the application. + * This means the package won't be included in the built application. +* A shortcut for `---save-dev` is `-D` +* Dependencies are needed to run the application, development dependencies are needed for development. +* Additional info: [https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/](https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/) # Getting started with Express -__Create a very simple express project __ +Initialize a new TypeScript project with appropriate `tsconfig.json` file and scripts for building and/or running the project in development mode using `ts-node`. + +Create a very simple express project in `index.ts` +```ts import express from 'express' const server = express() server.listen(3000, () => { - -console.log('Listening to port 3000') - + console.log('Listening to port 3000') }) +``` -__Then start the server and connect to __ [http://localhost:3000](http://localhost:3000) __ with your browser.__ __ (See instructions on the next slide)_ - -# Starting the server - -__To make using the import statement possible, we need to edit the __ __package.json.__ - -__Add a new script "start" with the value "node index.js" under "scripts" object in the package. json__ +Then start the server as you would any other TS project and connect to http://localhost:3000 with your browser or using an API tool such as Insomnia or Postman. -__(you can name index.js to whatever file your server code is in)_ - -__Add "type": "module" property__ - -__Now you can run the server with __ _npm start_ # Express - Requests -__The server does not yet __ _do_ __ anything except listens. That is because we have not defined how it should deal with incoming __ _requests._ - -__With Express you can easily process different kind of requests, e.g.,__ +The server does not yet *do* anything except listens. That is because we have not defined how it should deal with incoming **requests**. -__server.get(), server.post(), server.delete() etc!__ +With Express you can easily process different kind of requests, e.g. GET, POST, DELETE etc. -__An example of a simple GET response below.__ +An example of how to handle a GET request to the root of the project (`/`): -server.get("/", (request, response) => { - -response.send("Just saying hello!") +```ts +import { Request, Response } from "express"; +server.get("/", (req: Request, res: Response) => { + res.send("Just saying hello!") }) +``` ---- +## TypeScript Request and Response -Käy läpi tämä esimerkki, ja anna sitten tehtäväksi ensinnäkin muuttaa portti ja palautusviesti muotoon "Hello world!". Tällä tavalla kaikkien pitää ymmärtää miten tämä toimii. -Anna heidän pähkäillä miten voi tehdä päätepisteen, johon pääsee näin localhost:5000/secondPage. Tämän jälkeen käy läpi yhdessä tämä heidän kanssa. +Notice how we import `Request` and `Response` from `express`. These types are defined in express, but identically named types are also available from JavaScript basic library. -# Express - Endpoints +If you use the Request and Response types without importing them, JS mistakenly thinks they are different objects, and complains about incompatibility. -__To create a new __ _endpoint _ __we define what __ _kind_ __ of request we're handling, then what the __ _route_ __ to the endpoint is and finally what the endpoint __ _does_ __.__ -__In the below example we are creating an endpoint that handles a __ __GET__ __ request to route __ __/foobar__ __, that sends a response containing the string 'OK'.__ -server.get('/foobar', (req, res) => { +# Express - Endpoints -res.send('OK') +To create a new **endpoint** we define what *kind* of request we're handling, then what the *route* to the endpoint is and finally what the endpoint *does*. +In the below example we are creating an endpoint that handles a GET request to route `/foobar`, that sends a response containing the string 'OK'. +```ts +server.get('/foobar', (req, res) => { + res.send('OK') }) - ---- +``` # Nodemon -__Reminder__ __: If you are not using Nodemon in development, you are making your life unnecessarily difficult. Use Nodemon. __ +Reminder : If you are not using Nodemon in development, you are making your life unnecessarily difficult. Use Nodemon. -__Instructions are __ __below as a refresher.__ +Instructions are below as a refresher. # Nodemon with JavaScript One of the handiest packages out there is nodemon, which monitors your node program. Whenever you save your files, it reloads the program, showing you immediately the results of your changes. -Nodemon is installed as a dev dependency __npm install --save-dev nodemon__ +Nodemon is installed as a dev dependency npm install --save-dev nodemon To use nodemon, you should have a development script "dev": "nodemon ./index.js" # Nodemon with TypeScript -To get Nodemon working with TypeScript we need additional package to run our TS code without waiting for it to compile. __npm install --save-dev nodemon ts-node__ +To get Nodemon working with TypeScript we need additional package to run our TS code without waiting for it to compile. npm install --save-dev nodemon ts-node After installing, nodemon works directly with .ts files. "dev": "nodemon ./index.ts" # Exercise 2: Simple Express Server + Create a basic Express application. The application should respond to GET requests by returns a response with the following content: "Hello world!". -Extra: Add a new endpoint to your application. So, in addition to accessing _http://localhost:3000_ , try to send something back from _http://localhost:3000/endpoint2_ . - ---- +**Extra**: Add a new endpoint to your application. So, in addition to accessing http://localhost:3000, try to send something back from http://localhost:3000/endpoint2. -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + # Request params & query -__If request params are defined, they're something user __ __must__ __ send to the server.__ +If request params are defined, they're something user **must** send to the server. You can think parameters as the "path" of the endpoint. -__https://localhost:5000/ThisIsParam__ +https://localhost:5000/ThisIsParam -__Request query is __ __optional__ __ information that the user can send to the server.__ +Request query is **optional** information that the user can send to the server. A question mark (`?`) is used to separate the request query. -__https://localhost:5000?ThisIsQuery=123__ +https://localhost:5000?ThisIsQuery=123 -__Chaining query parameters:__ +Chaining query parameters is done with et symbol (`&`) -__https://localhost:5000?first=123&second=456__ +https://localhost:5000?first=123&second=456 # Request params & request query -app.get("/:name/:surname", ( _request_ , _response_ ) _=>_ { - -_console_ .log("Params:"); - -_console_ .log(request.params); - -_console_ .log("Query:"); - -_console_ .log(request.query); - -response.send("Hello " + request.params.name); - +```ts +app.get("/:name/:surname", (req: Request, res: Response) => { + console.log("Params:"); + console.log(request.params); + console.log("Query:"); + console.log(request.query); + response.send("Hello " + request.params.name); }); app.listen(5000); +``` Try connecting via browser! -_localhost:5000_ _/John/Doe_ - -_localhost:5000_ _/John/Doe_ _?query1=123_ +http://localhost:5000/John/Doe +http://localhost:5000/John/Doe?query1=123 ---- - -Johdatteleva esimerkki vielä ennen seuraavaa tehtävää. # Request query & param types -__Request parameters and query parameters are always of type __ _string_ __. If a number is passed, it will be a string that contains a number. __ +Request parameters and query parameters are always of type **string**. If a number is passed, it will be a string that contains a number. -__The example on right will __ __always __ __return 404, since the strict equality __ item.id === id +The example below will always return 404, since the strict equality `item.id === id` will never be true. This is because the `item.id` is of type **number** and `id` being of type **string**. -__will never be true. This is because the__ __ item.id __ __is of type __ _number _ __and__ __ id __ __being of type __ _string_ __.__ +--- -import express from 'express' +```ts +import express, { Request, Response } from 'express' const server = express() const data = [ - -{ id: 1, name: 'John' }, - -{ id: 2, name: 'Jane' } - + { id: 1, name: 'John' }, + { id: 2, name: 'Jane' } ] -server.get('/:id', (req, res) => { - -const id = req.params.id - -const info = data.find(item => item.id === id) - -if (info === undefined) { - -return res.status(404).send() - -} - -res.send(info) +server.get('/:id', (req: Request, res: Response) => { + const id = req.params.id + const info = data.find(item => item.id === id) + if (info === undefined) { + return res.status(404).send() + } + res.send(info) }) server.listen(3000) +``` -# Exercise 3: Counter Server - -Create an API that consists of only one endpoint: - -/counter - -Whenever you enter this endpoint from the browser, it -should respond with a JSON object with information on how many times the endpoint has been accessed. +# Exercise 3: Counter Server + -Extra: Make it possible to set the counter to whichever +Create an API that consists of only one endpoint: `/counter` +Whenever you enter this endpoint from the browser, it should respond with a JSON object with information on how many times the endpoint has been accessed. -integer with a query parameter /counter?number=5 +**Extra**: Make it possible to set the counter to whichever integer with a query parameter /counter?number=5 ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Exercise 4: Advanced Counter Server @@ -292,97 +280,76 @@ When entering this endpoint, the server should return the count of how many time For example -Aaron enters /counter/Aaron 🠖 "Aaron was here 1 times" - -Aaron enters /counter/Aaron 🠖 "Aaron was here 2 times" - -Beatrice enters /counter/Beatrice 🠖 "Beatrice was here 1 times" - -Aaron enters /counter/Aaron 🠖 "Aaron was here 3 times" - ---- - -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js - -# "url" library +- Aaron enters /counter/Aaron 🠖 "Aaron was here 1 times" +- Aaron enters /counter/Aaron 🠖 "Aaron was here 2 times" +- Beatrice enters /counter/Beatrice 🠖 "Beatrice was here 1 times" +- Aaron enters /counter/Aaron 🠖 "Aaron was here 3 times" -__With the "url" library, you can parse url fields (i.e., requests) the client sends to your server.__ +# Extra: "url" library -_import_ url from 'url'; +With the "url" library, you can parse url fields (i.e., requests) the client sends to your server. +```ts +import url from "url"; // URL module usage example -_const_ adr = ' [http://localhost:5000/default.html?year=2017&month=february](http://localhost:5000/default.html?year=2017&month=february) '; // Define the address - -_const_ q = url.parse(adr, true); // Parse through the address with url.parse() function. - -_console_ .log(q.host); // returns 'localhost:5000' - -_console_ .log(q.pathname); // returns '/default.html' - -_console_ .log(q.search); // returns '?year=2017&month=february' - -_const_ qdata = q.query; // returns an object: { year: 2017, month: 'february' } - -_console_ .log(qdata.month); // returns 'february +const addressString = 'http://localhost:5000/default.html?year=2017&month=february'; // Define the address +const address = url.parse(addressString, true); // Parse through the address with url.parse() function. +console.log(address.host); // returns 'localhost:5000' +console.log(address.pathname); // returns '/default.html' +console.log(address.search); // returns '?year=2017&month=february' +const query = address.query; // returns an object: { year: 2017, month: 'february' } +console_ .log(query.month); // returns 'february +``` # Express Middleware # Middlewares -__Generally speaking, middleware is an application that does something in between some other applications. __ - -__There are tons of middleware libraries available for Express.__ - -__What this means in Express: with a middleware, you can manipulate the __ __the client request and server response __ __in the client-server communication.__ +Generally speaking, middleware is an application that does something in between some other applications. -__Middleware__ __ functions are functions that have access to the __ __request__ __ object (__ _req_ __), the __ __response__ __ object (__ _res_ __), and the __ _next_ __ function in the application's request-response cycle. When the __ _next_ __ function is invoked, the next middleware function is executed.__ +There are tons of middleware libraries available for Express. -__Middleware can be defined as such__ +What this means in Express: with a middleware, you can manipulate the client request and server response in the client-server communication. -const authCheck = (req, res, next) => { +Middleware functions are functions that have access to the request object (`req`), the response object (`res`), and the `next` function in the application's request-response cycle. When the `next` function is invoked, the next middleware function is executed. -if (req.headers.Authentication === undefined) { // Read the request object - -res.status(401).send('Missing Authentication Header') // Modify the response object - -} else { - -next() // Execute the next middleware +--- -} +Middleware can be defined as such +```ts +const authCheck = (req: Request, res: Response, next: NextFunction) => { + if (req.headers.Authentication === undefined) { // Read the request object + res.status(401).send('Missing Authentication Header') // Modify the response object + return + } + next() // Execute the next middleware } +``` -__In Express, .use() method initializes a middleware (argument 2) in a given path (argument 1).__ - -__In the middleware, the next() function will jump to the next middleware or endpoint.__ +In Express, .use() method initializes a middleware (argument 2) in a given path (argument 1). +In the middleware, the next() function will jump to the next middleware or endpoint. +```ts server.use('/customers', authCheck) - server.get('/customers', (req, res) => { - // we know that Authentication header is present - ... - -}) +``` # Exercise 5: Logger Middleware + -Create a new project called Student Registry. We will be developing this program in stages during this lecture. For now it should have a single GET endpoint /students that returns an empty list. +Create a new project called Student Registry. We will be developing this program in stages during this lecture. For now it should have a single GET endpoint `/students` that returns an empty list. Create a logger middleware function that is used in all the project's endpoints. The middleware should log -the time the request was made - -the method of the request +1) the time the request was made +2) the method of the request +3) the url of the endpoint -the url of the endpoint ---- - -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Middleware - Unknown endpoint (AKA 404) @@ -391,8 +358,15 @@ As you might have noticed, all websites have some kind of a default functionalit This can be handled with a middleware. Unknown endpoint has to be taken into use after the actual endpoints! +```ts +const unknownEndpoint = (_req: Request, res: Response) => { + res.status(404).send({error: 'Page not found'}) +} + +app.use(unknownEndpoint) +``` + -![](imgs/1-express-basics_1.png) # Middleware - Error handler @@ -401,212 +375,193 @@ An error handler is often needed to handle different kind of user or application We can use middleware to enable error handling for our application. Has to be the last middleware to be taken into use! +```ts +const errorHandler = (error: Error, req: Request, res: Response, next: NextFunction) => { + console.error(error.message) + if (error.name === 'MyCustomError') { + res.status(400).send({error: 'My Custom Error Message'}) + } + + next(error) +} -![](imgs/1-express-basics_2.png) - -# Middleware - Req & Res - -__A n__ __eed to manually alter req & res is quite common. To avoid bloated code, the handling can be easily done with proper middlewares.__ +``` -__Setting a header in middleware:__ -app.use((req, res, next) => { -res.setHeader('Content-Type', 'image/png') +# Middleware - Req & Res -next(); +A need to manually alter req & res is quite common. To avoid bloated code, the handling can be easily done with proper middlewares. +Setting a header in middleware: +```ts +app.use((req: Request, res: Response, next: Next) => { + res.setHeader('Content-Type', 'image/png'); + next(); }); +``` -}) - -__Getting a header and storing its value. If the header is not present, return 403 with an 'Unauthorized' message:__ - -app.use((req, res, next) => { - -const someIdNeededInApplication = req.header('Some-Custom-Header') - -if (!someIdNeededInApplication) { - -res.status(403).send('Unauthorized') - -} - -someIdFunctionality(someIdNeededInApplication) +--- -next() +Getting a header and storing its value. If the header is not present, return 403 with an 'Unauthorized' message: +```ts +app.use((req: Request, res: Response, next: NextFunction) => { + const someIdNeededInApplication = req.header('Some-Custom-Header'); + if (!someIdNeededInApplication) { + res.status(403).send('Unauthorized'); + } -}) + someIdFunctionality(someIdNeededInApplication); + next(); +}); +``` # Exercise 6: 404 Not Found + Add a middleware function that sends a response with status code 404 and an appropriate error message, if user tries to use an endpoint that does not exist. If you have not already done so, move all your middleware to a separate file, to keep your program clean and readable. ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Express.js: Creating a REST API # REST HTTP requests -__We have gone through what a GET request is. Let's briefly go through what other types of requests are necessary for you.__ +We have gone through what a GET request is. Let's briefly go through what other types of requests are necessary for you. -__GET /products/:id → Returns a single product identified by its id__ - -__GET /products →Returns all the products__ - -__POST /products → Creates a new product__ - -__DELETE /products/:id → Removes a single product identified by its id__ - -__PUT /products/:id → Updates an existing product__ +- GET `/products/:id` → Returns a single product identified by its id +- GET `/products` →Returns all the products +- POST `/products` → Creates a new product +- DELETE `/products/:id` → Removes a single product identified by its id +- PUT `/products/:id` → Updates an existing product Basic RESTFUL functionality is often referenced as a CRUD (Create, Read, Update, Delete). -# Handling request content -__To properly handle some requests, e.g., POST requests, in Express, a body-parser middleware must be taken into use. It's as simple as:__ -const server = express() +# Handling request content -server.use(express.json()) +To properly handle some requests, e.g., POST requests, in Express, a body-parser middleware must be taken into use. It's as simple as: +```ts +const server = express(); +server.use(express.json()); +``` You might encounter a separate body-parser being used. This is no longer necessary, as Express 4.16+ includes a body parser middleware built-in. -express.urlencoded() __allows us to parse url-encoded forms by attaching the data to the request body.__ +--- -const app = express() +`express.urlencoded()` allows us to parse url-encoded forms by attaching the data to the request body. +```ts +const app = express(); +app.use(express.json()); +app.use(express.urlencoded({extended: false})); +``` -app.use(express.json()) +The "extended" parameter's false value only says that we are not dealing with any complicated objects (objects with sub objects, etc). -app.use(express.urlencoded({extended: false})) -__The "extended" parameter's false value only says that we are not dealing with any complicated objects (objects with sub objects, etc)_ # Basic POST request -const app = express() - -app.use(express.json()) - -app.use(express.urlencoded({extended: false})) +```ts +const app = express(); +app.use(express.json()); -app.get("/", (req, res) => { - -console.log("GET request init!"); - -res.sendFile(\_\_dirname + "/index.html"); - -}) +app.get("/", (req: Request, res: Response) => { + console.log("GET request init!"); + res.sendFile(__dirname + "/index.html"); +}); -app.post("/", (req, res) => { +app.post("/", (rreq: Request, res: Response) => { + console.log("POST request init!"); + console.log(req.body); + res.redirect("/"); +}); +``` -console.log("POST request init!"); -console.log(req.body); -res.redirect("/"); +# Request Body -}) +GET methods should not in general have body parameters. The GET method should only be used to get data, the request should not convey data. -# Request Body +Despite this, most modern implementations can send and receive request bodies in all request types. -__GET__ __ methods should not in general have body parameters. The GET method should only be used to get data, the request should not convey data. __ +The requests that should not include body are CONNECT, GET, HEAD, OPTIONS and TRACE. -__Despite this, most modern implementations can send and receive request bodies in all request types.__ -__The requests that should not include body are __ __CONNECT__ __, __ __GET__ __, __ __HEAD__ __, __ __OPTIONS__ __, and __ __TRACE__ __. __ # Request Body Types -__Request body is parsed by the express body parser, and therefore it can contain any basic JavaScript data types. Notice that this differs from the request parameters and queries.__ +Request body is parsed by the express body parser, and therefore it can contain any basic JavaScript data types. Notice that this differs from the request parameters and queries. -__This also means that you do not know what the data type of any given body parameter will be. It might be necessary to check this in some cases.__ +This also means that you do not know what the data type of any given body parameter will be. It might be necessary to check this in some cases. -# TypeScript Body Casting -__If you want to enforce type safety, the solution is to validate the user input before using it.__ -__This is usually a good idea even if not using TypeScript, since it is important to know what type all the variables are in your code.__ +# TypeScript Body Casting -import express, { NextFunction, Request, Response} from 'express' +If you want to enforce type safety, the solution is to validate the user input before using it. -const server = express() +This is usually a good idea even if not using TypeScript, since it is important to know what type all the variables are in your code. -server.use(express.json()) +--- +```ts interface Body { - -name: string - -age: number - + name: string; + age: number; } const validate = (req: Request, res: Response, next: NextFunction ) => { - -const { age, name } = req.body - -if (typeof(age) !== 'number' || typeof(name) !== 'string') { - -return res.status(400).send('Missing or invalid parameters') - -} - -next() - + const { age, name } = req.body; + if (typeof(age) !== 'number' || typeof(name) !== 'string') { + return res.status(400).send('Missing or invalid parameters'); + } + next(); } server.post('/', validate, (req: Request, res: Response) => { + const body: Body = req.body; // this has been validated + console.log(body); + res.send(body.name.toUpperCase()); +}); +``` -const body: Body = req.body - -console.log(body) - -res.send(body.name.toUpperCase()) -}) - -server.listen(3000) # Exercise 7: Body Logging + Enable body parsing in your application. Modify your logger middleware so that in addition to existing functionality, it also logs the request body if it exists. ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Exercise 8: POST Requests + Add two more endpoints to your app: -POST /student should expect the request body to include student _id_ , _name _ and _email_ . If some of these parameters are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should store the student information in an array called _students_ . The endpoint should return an empty response with status code 201. +POST `/student` should expect the request body to include student *id*, *name* and *email*. If some of these parameters are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should store the student information in an array called **students**. The endpoint should return an empty response with status code 201. -GET /student/:id should return the information of a single student, identified by the _id_ request parameter. The response should be in JSON format. If there is no such student, the endpoint should return 404. +GET `/student/:id` should return the information of a single student, identified by the *id* request parameter. The response should be in JSON format. If there is no such student, the endpoint should return 404. -Modify the GET /students endpoint to return the list of all student _ids_ , without names or emails. +Modify the GET `/students` endpoint to return the list of all student *ids*, without names or emails. ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Exercise 9: PUT and DELETE + Add two more endpoints to your app: -PUT /student/:id should expect the request body to include student _name_ or _email_ . If both are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should update an existing student identified by the request parameter _id_ . - -DELETE /student/:id should remove a student identified by the request parameter _id_ . - -Both endpoints should return 404 if there is no student matching the request parameter _id_ . On success both endpoint should return an empty response with status code 204. - ---- +PUT `/student/:id` should expect the request body to include student *name* or *email*. If both are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should update an existing student identified by the request parameter *id*. -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js +DELETE `/student/:id` should remove a student identified by the request parameter *id*. +Both endpoints should return 404 if there is no student matching the request parameter *id*. On success both endpoint should return an empty response with status code 204. \ No newline at end of file diff --git a/2-express-router-authentication-and-testing-slides.html b/2-express-router-authentication-and-testing-slides.html new file mode 100644 index 0000000..534674f --- /dev/null +++ b/2-express-router-authentication-and-testing-slides.html @@ -0,0 +1,484 @@ +Express Basics
+

Express Router, Authentication & Testing

+
+
+

Static Content

+

We can make a web application out of our REST-API. First you need to create a folder at the root of your application, and name it for example "public". When you add an index.html file there, with the example below it will served when you send a GET request to our server.

+
import express from 'express'
+
+const server = express()
+server.use(express.static('public')) // defaults to root route '/'
+
+server.get('/route', (req: Request, res: Response) => {
+  res.send('OK')
+})
+
+server.listen(PORT, () => console.log('Listening to port', PORT))
+
+
+
+

Exercise 1: Static

+ +

Let's continue improving the Students API we created in the last lecture.

+

Add a static info page to your API.

+

The page should be reachable from the root path / and it should include some information about all the endpoints in the API.

+
+
+

Router

+
+
+

Express Router

+

Until now we have been building our routes to index.ts. This is fine for VERY small applications, but we'd like to keep our code readable even if our application grows in size.

+

Routers are Express classes that are dedicated to route handling. We can use them to group routes to meaningful units that share functionality. Routers do not require any installation since they are native express classes.

+

The following application, while still being extremely simple, already has a problem: index.ts file is doing several things.

+

It handles two kinds of routes and starts the server. Usually it also adds middleware and some configuration.

+

Most of the time it is best to separate the route handling to dedicated routers.

+
+
+
import express from 'express'
+import authors from './approvedAuthors.js'
+import articles from './articlesDao.js'
+
+const server = express()
+
+server.get('/authors', (req: Request, res: Response) => {
+  res.send(authors)
+})
+server.get('/authors/:id', (req: Request, res: Response) => {
+  const author = authors.find(author => author.id = req.params.id)
+  res.send(author)
+})
+server.get('/articles', (req: Request, res: Response) => {
+  res.send(articles.getAll())
+})
+server.post('/article', (req: Request, res: Response) => {
+  const { author, article } = req.body
+  const newArticle = articles.add({ author, article})
+  res.send(newArticle)
+})
+server.listen(3000, () => console.log('Listening to port 3000'))
+
+
+
+

We create two routers, one for each logical unit: authorsRouter.ts and articlesRouter.ts. The only difference in the route definition is in the route paths.

+
//authorsRouter.ts
+import express from 'express'
+import authors from './approvedAuthors.js'
+
+const router = express.Router()
+  router.get('/', (req: Request, res: Response) => {
+  res.send(authors)
+})
+
+router.get('/:id', (req: Request, res: Response) => {
+  const author = authors.find(author => author.id = req.params.id)
+  res.send(author)
+})
+
+export default router
+
+
+
+
//articlesRouter.ts
+import express from 'express'
+import articles from './articlesDao.js'
+
+const router = express.Router()
+
+router.get('/', (req: Request, res: Response) => {
+  res.send(articles.getAll())
+})
+
+router.post('/', (req: Request, res: Response) => {
+  const { author, article } = req.body
+  const newArticle = articles.add({ author, article})
+  res.send(newArticle)
+})
+
+export default router
+
+
+
+

Then we define index.ts to use those routers like any middleware.

+

Each route gets a path designation that will be used as prefix in the router.

+
import express from 'express'
+import authorsRouter from './authorsRouter.js'
+import articlesRouter from './articlesRouter.js'
+
+const server = express()
+server.use('/authors', authorsRouter)
+server.use('/articles', articlesRouter)
+
+server.listen(3000, () => {
+  console.log('Listening to port 3000')
+})
+
+
+
+

Exercise 2: Students Router

+ +

Let's continue improving the Students API we created on the last lecture.

+

Add a studentRouter.ts file that exports a router with all the five routes the app currently has.

+
+
+

Authentication & authorization

+
+
+

Token based login

+
    +
  • Currently our applications are accessible for anyone, which often is not a desired state of things.
    • +
  • If you don't restrict access to your endpoints, something like this might happen !!!
    • +
  • Authentication & Authorization help here +
      +
    • Authentication: Verifying that you are indeed you
      • +
    • Authorization: Determining access rights based on your privileges
      • +
    +
    • +
  • We shall first look at password authorization, and later token-based authentication.
    • +
+
+
+

Storing passwords

+
+
+

Password Hashing

+

Passwords should never be stored as plain text in to a database. Instead a password hash should be stored instead.

+

Hash function maps data of any size (in our case a password) to a hash of fixed length.

+

Hash functions are one-way functions, which means they can not be reversed. This means that you can not deduce the original password from the hash.

+

We can use (a variant of) the original hash function to compare a password and the hash and deduce if the hash matches the password or not.

+
+
+

Salting

+

Bare hash functions produce identical results from identical inputs.

+

To add complexity, a some random data is added to the input. This data is called salt and the process of adding the salt is called salting.

+

This means two identical passwords produce different hashes, since both have unique salt.

+

Salt can be stored in the hash or it can be stored separately. The hash function needs to know the salt in order to verify passwords against the hash.

+
+
+

Hashing a password

+

We will use Argon library to do our hashing and comparing. Install Argon with npm install argon2

+

Using Argon to hash passwords is very straightforward. Salting is done automatically (although you can configure it in more detail if you wish to do so) and the salt is added to the hash.

+
import argon2 from 'argon2'
+
+const password = process.argv[2]
+argon2.hash(password)
+  .then(result => console.log(result))
+
+

You can of course also use async/await syntax instead of .then()

+
+
+

Comparing hashes

+

Notice if you run the example from the previous slide multiple times, you always get a different result. In order to verify that a particular password matches the hash, Argon's verify function is used.

+

Verify returns a promise that resolves to either true, if the password matches the hash, or false if it doesn't.

+
import argon2 from 'argon2'
+
+const hash = '$argon2id$v=19$m=4096,t=3,p=1$+OM8yk1Kd3M/709t0Hy1vg$E3aii3UmOQOp6jFJVd9xDpakMOF1O6TDd1gS1i/98HE'
+const password = process.argv[2]
+
+argon2.verify(hash, password)
+  .then(passwordMatchesHash => console.log(passwordMatchesHash ? 'Correct' : 'Incorrect'))
+
+
+
+

Exercise 3: Registration

+ +

Create and attach a new router, userRouter.ts that has a single endpoint: POST /register.

+

The endpoint should

+
    +
  1. expect a request body with two parameters, username and password.
    2. +
  2. create a hash from the password using the Argon2 library.
    3. +
  3. store the username and the hash in an in-memory storage (e.g. let users = [ ... ]) and log the result to the console.
    4. +
  4. return a response with status code 201 (Created) on success
    5. +
+
+
+

Exercise 4: Login

+ +

Add another endpoint to the user router: POST /login that also expects two request body parameters, username and password.

+

The endpoint should

+
    +
  1. check that the user exists in the in-memory storage.
    2. +
  2. If the user exists, it should use the Argon2 library to verify that the given password matches the stored hash.
    3. +
  3. If they match, it should return a response with status code 204 (No Content).
    4. +
  4. If the user does not exist, or the password doesn't match the hash, it should return a response with status code 401 (Unauthorized).
    5. +
+
+
+

Environment variables

+
+
+

dotenv Library

+

With dotenv library, we can create environment variables that can be loaded into process.env. We can access the environment variables in our application with process.env.<variable_name>.

+

To use dotenv

+
    +
  1. Install dotenv package with npm install dotenv
    2. +
  2. Create a .env file in to the root folder of your application where the values are declared
    3. +
  3. Import the dotenv configuration in the project import 'dotenv/config'
    4. +
  4. Access the variables defined in .env file const PORT = process.env.PORT
    5. +
+
+
+

.env File

+

The custom environment variables are declared in the .env file. The file must be located in the folder from where the program is ran. This is usually the root folder.

+

The variables are declared as in the example below, separated by line changes. All values are strings (even if JS parses some automatically).

+

Notice there are no spaces around the equals (=) sign, nor any kind of quotation marks.

+
API_URL=https://cataas.com
+PORT=3000
+
+
+
+

dotenv in Development

+
    +
  • In many cases the environment variables are configured on the platform from where the program is ran. In those cases you might want to run dotenv only in development mode.
    • +
  • Install as dev dependency npm install --save-dev dotenv
    • +
  • Create .env file as usual. +
      +
    • Optionally include ENVIRONMENT=development variable
      • +
    +
    • +
  • Run your application from package.json script that preloads dotenv "dev": "nodemon -r dotenv/config index.js"
    • +
  • Access environment variables as usual const PORT = process.env.PORT
    • +
+
+
+

Exercise 5: Environmental Login

+ +

Add an admin login to the Students API. We want to store admin credentials in environment variables.

+

Add dotenv library as dependency. Then add a .env file that defines an admin username and an admin password hash.

+

Add an endpoint POST /admin that also expects two request body parameters, username and password. The endpoint should

+
    +
  1. check that the username matches the one defined in the .env file
    2. +
  2. check that the password matches the hash defined in the .env file
    3. +
  3. if they match, it should return a response with status code 204 (No Content)
    4. +
  4. if they do not match, it should return a response with status code 401 (Unauthorized)
    5. +
+
+
+

Token-based authentication

+
+
+

JSON Web Token

+

After verifying the identity of our user, we create a token that we send to the client. The client stores that token and attaches it to all future requests. This token is enough to identify the request as authorized.

+

Since the tokens are signed by us, we can also include data (such as user id) in the token. On all future requests we can rely that the id is correct since the JWT will be verified on every request.

+

Tokens can (and should be) equipped with expiration dates.

+
+
+

Creating a Token

+

Install the JSON Web Token package as dependency npm install jsonwebtoken

+

Tokens are created with the sign function that takes two parameters:

+
    +
  • Payload: the data we want to include in the token. It can be empty.
    • +
  • Secret: either a string or a private key.
    • +
+

It can also accept an optional options parameter. In that we can define the expiration date, compression algorithm, or many other things.

+
import 'dotenv/config'
+import jwt from 'jsonwebtoken'
+
+const payload = { username: 'sugarplumfairy' }
+const secret = process.env.SECRET
+const options = { expiresIn: '1h'}
+
+const token = jwt.sign(payload, secret, options)
+console.log(token)
+
+

The created JWT has three parts separated by period: header, payload and signature.

+
+
+

Exercise 6: Create a Token

+ +

Write a simple command line program that prints JSON Web tokens. Use the default algorithm (SHA256). Set the tokens to expire in fifteen minutes and include a payload of some JSON object.

+

Copy your JWT and paste it to https://jwt.io debugger. Verify that the debugger shows correct algorithm, data, and expiration date (iat).

+
+
+

Verifying a Token

+

Tokens are verified using the verify function that takes two parameters:

+
    +
  • Token: the token to be verified
    • +
  • Secret: the secret that was used to create the token
    • +
+

The verify function can also accept an optional options parameter.

+

The verify function returns a decoded token. Since it has been verified against our secret, we know that the information has not been altered.

+

If the token is invalid, the verify function throws an error. If this error is not caught, the program crashes.

+
+
+

Exercise 7: Verify a Token

+ +

Write a simple command line program that verifies JSON Web tokens.

+

The program should print the contents of the verified token. If the token is not valid, the program should print an error message and exit gracefully.

+

Create a JWT in https://jwt.io debugger. Remember to set your secret value. Use your program to verify the token and see that the data entered is as it should be.

+
+
+

Bearer Tokens

+

When a user has received a token, it can be used to authenticate all future requests.

+

Do this by adding the field Authorization to the request header.

+

The value is "Bearer " plus the token.

+

This header can be added to requests generated by Fetch, Axios, or any other JS request library, or by API clients like Postman or Insomnia.

+

Then protected routes can check that the authorization header is valid, and if not, send an error message. This is usually done using authorization middleware.

+
+
+

Authentication Middleware

+

A common way is to have user routes, such as login, logout or user creation in a separate router, which requires no token.

+

Then all protected routes are set to use the authentication middleware.

+

The request parameter user is set to the verified token data and can then be used in the actual route.

+
+
+
const authenticate = (req: AuthenticatedRequest, res: Response, next: NextFunction) => {
+  const auth = req.get('Authorization')
+  if (!auth?.startsWith('Bearer ')) {
+    return res.status(401).send('Invalid token')
+  }
+
+  const token = auth.substring(7)
+  const secret = process.env.SECRET
+
+  try {
+    const decodedToken = jwt.verify(token, secret)
+    req.user = decodedToken
+    next()
+  } catch (error) {
+    return res.status(401).send('Invalid token')
+  }
+}
+
+router.get('/protected', authenticate, (req: Request, res: Response) => {
+  res.send(`${req.user.username} accessed protected route`)
+})
+
+
+
+

TypeScript: Extending Request

+

The above example uses a custom AuthenticatedRequest instead of regular Request as the type of the req object. This is because in TypeScript, if we try to add a new parameter, in this case "user", to a Request object, we get an error

+
Property 'user' does not exist on type 
+'Request<ParamsDictionary, any, any, ParsedQs, Record<string, any>>'
+
+

We are trying to set a value to request property "user", but request objects are well defined objects that have no such property.

+

We can not add or use properties that an object does not have. This is part of the type safety that TypeScript is here to enforce. If an object is of type Request, we know exactly what the properties are.

+
+
+

The solution is to create an interface that extends the existing Request type. The new interface will have all the properties the Request has, plus the ones we define ourself.

+
interface AuthenticatedRequest extends Request {
+  user?: { username: string }
+}
+
+

Now our AuthenticatedRequest object can be used as a regular Request object (and hence is acceptable as a parameter to our Express route functions), and it can be used to store an object holding the authenticated username.

+
+
+

Exercise 8: Securing a Route

+ +

Modify the Students API /register and /login routes so that on success they return a response with status code 200 and a JWT with username as payload.

+

Secure all the routes in the students router so that they require the user to be logged in to use the routes.

+

Extra: Also modify the /admin route to return a JWT. Secure the POST, PUT and DELETE routes to require that in addition to being logged in, the user also needs to be an admin.

+

Hint: Here you have some resources on sending a token with Postman.

+

https://learning.postman.com/docs/sending-requests/authorization/

+
+
+

Supertest

+
+
+

Testing APIs with Supertest

+

Supertest is a library for testing API's. Let's see how to use it with Jest to test an Express app. Install supertest, ts-jest and the required TypeScript types with

+
npm install --save-dev supertest @types/supertest ts-jest
+
+

When using ESLint, also add "jest": true to .eslintrc's "env" object

+
import express from 'express'
+
+const server = express()
+
+server.get('/', (req: Request, res: Response) => {
+  res.send('Hello World!')
+})
+
+server.listen(3000, () => {
+  console.log('Listening to port 3000')
+})
+
+
+
+

Also, add to package.json:

+
"jest": {
+  "preset": "ts-jest",
+  "testEnvironment": "node"
+},
+
+
+
+

Preparing the App for Testing

+

We need to import our express application to the tests.

+

We will separate our application in two files. The server.ts defines the API server, and index.ts only starts the application.

+

Now we can test the server with Supertest. The startup file has no functionality we want to test.

+
//index.ts
+import server from './server.js'
+
+server.listen(3000, () => {
+  console.log('Listening to port 3000')
+})
+
+
+
+
//server.ts
+import express from 'express'
+
+const server = express()
+
+server.get('/', (req: Request, res: Response) => {
+  res.send('Hello World!')
+})
+
+export default server
+
+
+
+
+
+

We now create a new test file and import our server there.

+

We use Supertest request to get responses from our server. These responses can be tested just like any other Jest test.

+
+
+
//test/server.test.ts
+
+import request from 'supertest'
+import server from '../src/server.js'
+
+describe('Server', () => {
+  it('Returns 404 on invalid address', async () => {
+    const response = await request(server)
+      .get('/invalidaddress')
+    expect(response.statusCode).toBe(404)
+  })
+
+  it('Returns 200 on valid address', async () => {
+    const response = await request(server)
+      .get('/')
+    expect(response.statusCode).toBe(200)
+    expect(response.text).toEqual('Hello World!')
+  })
+})
+
+
+
+
+
+

Common Resources

+

https://www.npmjs.com/package/jest → Official repository

+

https://www.npmjs.com/package/supertest → Official repository

+

https://jestjs.io/docs/api → Jest API Docs. Very handy!

+

https://github.com/visionmedia/supertest → Supertest's github repository. Includes guides for basic use cases.

+
+
+

Exercise 9: Test with Supertest

+ +

Use Supertest with Jest to create tests for the Students API /user/register and /user/login routes.

+
+
\ No newline at end of file diff --git a/2-express-router-authentication-and-testing.md b/2-express-router-authentication-and-testing.md index 854a726..aa4d2a7 100644 --- a/2-express-router-authentication-and-testing.md +++ b/2-express-router-authentication-and-testing.md @@ -1,338 +1,323 @@ +--- +marp: true +paginate: true +math: mathjax +theme: buutti +title: Express Basics +--- + + # Express Router, Authentication & Testing + + + # Static Content We can make a web application out of our REST-API. First you need to create a folder at the root of your application, and name it for example "public". When you add an index.html file there, with the example below it will served when you send a GET request to our server. - +```ts import express from 'express' const server = express() +server.use(express.static('public')) // defaults to root route '/' -server.use(express.static('public')) - -server.get('/route', (req, res) => { - -res.send('OK') - +server.get('/route', (req: Request, res: Response) => { + res.send('OK') }) server.listen(PORT, () => console.log('Listening to port', PORT)) +``` + + # Exercise 1: Static + Let's continue improving the Students API we created in the last lecture. Add a static info page to your API. -The page should be reachable from the root path / and it should include some information about all the endpoints in the API. +The page should be reachable from the root path `/` and it should include some information about all the endpoints in the API. ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Router # Express Router -Until now we have been building our routes to index.js. This is fine for VERY small applications, but we'd like to keep our code readable even if our application grows in size. - -__Routers__ are Express classes that are dedicated to route handling. We can use them to group routes to meaningful units that share functionality. +Until now we have been building our routes to `index.ts`. This is fine for VERY small applications, but we'd like to keep our code readable even if our application grows in size. -Routers do not require any installation since they are native express classes. +**Routers** are Express classes that are dedicated to route handling. We can use them to group routes to meaningful units that share functionality. Routers do not require any installation since they are native express classes. -The following application, while still being extremely simple, already has a problem: index.js file is doing several things. +The following application, while still being extremely simple, already has a problem: `index.ts` file is doing several things. It handles two kinds of routes and starts the server. Usually it also adds middleware and some configuration. Most of the time it is best to separate the route handling to dedicated routers. +--- +```ts import express from 'express' - import authors from './approvedAuthors.js' - import articles from './articlesDao.js' const server = express() -server.get('/authors', (req, res) => { - -res.send(authors) - +server.get('/authors', (req: Request, res: Response) => { + res.send(authors) }) - -server.get('/authors/:id', (req, res) => { - -const author = authors.find(author => author.id = req.params.id) - -res.send(author) - +server.get('/authors/:id', (req: Request, res: Response) => { + const author = authors.find(author => author.id = req.params.id) + res.send(author) }) - -server.get('/articles', (req, res) => { - -res.send(articles.getAll()) - +server.get('/articles', (req: Request, res: Response) => { + res.send(articles.getAll()) }) - -server.post('/article', (req, res) => { - -const { author, article } = req.body - -const newArticle = articles.add({ author, article}) - -res.send(newArticle) - +server.post('/article', (req: Request, res: Response) => { + const { author, article } = req.body + const newArticle = articles.add({ author, article}) + res.send(newArticle) }) - server.listen(3000, () => console.log('Listening to port 3000')) +``` -We create two routers, one for each logical unit: _authorsRouter.js_ and _articlesRouter.js_ +--- -The only difference in the route definition is in the route paths. +We create two routers, one for each logical unit: `authorsRouter.ts` and `articlesRouter.ts`. The only difference in the route definition is in the route paths. +```ts +//authorsRouter.ts import express from 'express' - import authors from './approvedAuthors.js' const router = express.Router() - -router.get('/', (req, res) => { - -res.send(authors) - + router.get('/', (req: Request, res: Response) => { + res.send(authors) }) -router.get('/:id', (req, res) => { - -const author = authors.find(author => author.id = req.params.id) - -res.send(author) - +router.get('/:id', (req: Request, res: Response) => { + const author = authors.find(author => author.id = req.params.id) + res.send(author) }) -export default server +export default router +``` -import express from 'express' +--- +```ts +//articlesRouter.ts +import express from 'express' import articles from './articlesDao.js' const router = express.Router() -router.get('/', (req, res) => { - -res.send(articles.getAll()) - +router.get('/', (req: Request, res: Response) => { + res.send(articles.getAll()) }) -router.post('/', (req, res) => { - -const { author, article } = req.body - -const newArticle = articles.add({ author, article}) - -res.send(newArticle) - +router.post('/', (req: Request, res: Response) => { + const { author, article } = req.body + const newArticle = articles.add({ author, article}) + res.send(newArticle) }) export default router +``` -Then we define _index.js_ to use those routers like any middleware. +--- -Each route gets a path designation that will be used as _prefix_ in the router. +Then we define `index.ts` to use those routers like any middleware. -import express from 'express' +Each route gets a path designation that will be used as *prefix* in the router. +```ts +import express from 'express' import authorsRouter from './authorsRouter.js' - import articlesRouter from './articlesRouter.js' const server = express() - server.use('/authors', authorsRouter) - server.use('/articles', articlesRouter) server.listen(3000, () => { + console.log('Listening to port 3000') +}) +``` -console.log('Listening to port 3000') -}) # Exercise 2: Students Router + Let's continue improving the Students API we created on the last lecture. -Add a _studentRouter.js_ file that exports a router with all the five routes the app currently has. +Add a `studentRouter.ts` file that exports a router with all the five routes the app currently has. ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Authentication & authorization # Token based login * Currently our applications are accessible for anyone, which often is not a desired state of things. -* If you don't restrict access to your endpoints, something like [this](https://mastodon.gamedev.place/@badlogic/111246798083590676) [might happen](https://firesky.tv/) !!! +* If you don't restrict access to your endpoints, something like [this](https://mastodon.gamedev.place/@badlogic/111246798083590676) [might happen](https://firesky.tv/) !!! * Authentication & Authorization help here * Authentication: Verifying that you are indeed you * Authorization: Determining access rights based on your privileges * We shall first look at password authorization, and later token-based authentication. + + # Storing passwords # Password Hashing -Passwords should never be stored as plain text in to a database. Instead a password _hash_ should be stored instead. +Passwords should never be stored as plain text in to a database. Instead a password hash should be stored instead. + +**Hash function** maps data of any size (in our case a password) to a hash of fixed length. -_Hash function_ maps data of any size (in our case a password) to a _hash _ of fixed length. +Hash functions are *one-way functions*, which means they can not be reversed. This means that you can not deduce the original password from the hash. + +We can use (a variant of) the original hash function to compare a password and the hash and deduce if the hash matches the password or not. -Hash functions are _one-way functions_ , which means they can not be reversed. This means that you can not deduce the original password from the hash. -We can use (a variant of) the original hash function to compare a password and the hash and deduce if the hash _matches_ the password or not. # Salting Bare hash functions produce identical results from identical inputs. -To add complexity, a some random data is added to the input. This data is called _salt_ and the process of adding the salt is called _salting_ . +To add complexity, a some random data is added to the input. This data is called salt and the process of adding the salt is called salting. This means two identical passwords produce different hashes, since both have unique salt. Salt can be stored in the hash or it can be stored separately. The hash function needs to know the salt in order to verify passwords against the hash. + + # Hashing a password -We will use Argon library to do our hashing and comparing. Install Argon with __npm install argon2__ +We will use **Argon** library to do our hashing and comparing. Install Argon with `npm install argon2` Using Argon to hash passwords is very straightforward. Salting is done automatically (although you can configure it in more detail if you wish to do so) and the salt is added to the hash. +```ts import argon2 from 'argon2' const password = process.argv[2] - argon2.hash(password) + .then(result => console.log(result)) +``` + +You can of course also use async/await syntax instead of .then() -.then(result => console.log(result)) -You can of course also use _async/await_ syntax instead of _.then() # Comparing hashes -Notice if you run the example from the previous slide multiple times, you always get a different result. In order to verify that a particular password matches the hash, Argon's _verify_ function is used. +Notice if you run the example from the previous slide multiple times, you always get a different result. In order to verify that a particular password matches the hash, Argon's *verify* function is used. -Verify returns a promise that resolves to either _true,_ if the password matches the hash, or _false_ if it doesn't. +Verify returns a promise that resolves to either true, if the password matches the hash, or false if it doesn't. +```ts import argon2 from 'argon2' const hash = '$argon2id$v=19$m=4096,t=3,p=1$+OM8yk1Kd3M/709t0Hy1vg$E3aii3UmOQOp6jFJVd9xDpakMOF1O6TDd1gS1i/98HE' - const password = process.argv[2] argon2.verify(hash, password) + .then(passwordMatchesHash => console.log(passwordMatchesHash ? 'Correct' : 'Incorrect')) +``` + -.then(passwordMatchesHash => console.log(passwordMatchesHash ? 'Correct' : 'Incorrect')) # Exercise 3: Registration + -Create and attach a new router, _userRouter.js_ that has a single endpoint: POST /register . +Create and attach a new router, `userRouter.ts` that has a single endpoint: POST `/register`. The endpoint should -expect a request body with two parameters, _username _ and _password_ . +1) expect a request body with two parameters, username and password. +2) create a hash from the password using the Argon2 library. +3) store the username and the hash in an in-memory storage (e.g. `let users = [ ... ]`) and log the result to the console. +4) return a response with status code 201 (Created) on success -create a hash from the password using the Argon2 library. -store the username and the hash in an in-memory storage (e.g. let users = [ ... ]) and log the result to the console. - -return a response with status code 201 (Created) on success - ---- - -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Exercise 4: Login + -Add another endpoint to the user router: POST /login that also expects two request body parameters, _username _ and _password_ . +Add another endpoint to the user router: POST `/login` that also expects two request body parameters, username and password. The endpoint should +1) check that the user exists in the in-memory storage. +2) If the user exists, it should use the Argon2 library to verify that the given password matches the stored hash. +3) If they match, it should return a response with status code 204 (No Content). +4) If the user does not exist, or the password doesn't match the hash, it should return a response with status code 401 (Unauthorized). -check that the user exists in the in-memory storage. -If the user exists, it should use the Argon2 library to verify that the given password matches the stored hash. - -If they match, it should return a response with status code 204 (No Content). - -If the user does not exist, or the password doesn't match the hash, it should return a response with status code 401 (Unauthorized). - ---- - -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Environment variables # dotenv Library -With dotenv library, we can create environment variables that can be loaded into process.env. We can access the environment variables in our application with process.env.{variable\_name}. +With dotenv library, we can create environment variables that can be loaded into process.env. We can access the environment variables in our application with `process.env.`. To use dotenv -Install dotenv package with __npm install dotenv__ - -Create a __.env __ file in to the root folder of your application where the values are declared +1) Install dotenv package with `npm install dotenv` +2) Create a `.env` file in to the root folder of your application where the values are declared +3) Import the dotenv configuration in the project `import 'dotenv/config'` +4) Access the variables defined in `.env` file `const PORT = process.env.PORT` -Import the dotenv configuration in the projectimport 'dotenv/config' -Access the variables defined in __.env__ fileconst PORT = process.env.PORT # .env File The custom environment variables are declared in the .env file. The file must be located in the folder from where the program is ran. This is usually the root folder. -The variables are declared as in the example below, separated by line changes. All values are _strings _ (even if JS parses some automatically). +The variables are declared as in the example below, separated by line changes. All values are strings (even if JS parses some automatically). Notice there are no spaces around the equals (=) sign, nor any kind of quotation marks. +``` +API_URL=https://cataas.com +PORT=3000 +``` -API\_URL=https://cataas.com -PORT=3000 # dotenv in Development * In many cases the environment variables are configured on the platform from where the program is ran. In those cases you might want to run dotenv only in development mode. -* Install as dev dependency __npm install --save-dev dotenv__ -* Create __.env__ file as usual. - * Optionally include ENVIRONMENT=development variable -* Run your application from config.json script that preloads dotenv"dev": "nodemon -r dotenv/config index.js" -* Access environment variables as usualconst PORT = process.env.PORT - -# Exercise 5: Environmental Login - -Add an admin login to the Students API. We want to store admin credentials in environment variables. - -Add dotenv library as dependency. +* Install as dev dependency `npm install --save-dev dotenv` +* Create `.env` file as usual. + * Optionally include `ENVIRONMENT=development` variable +* Run your application from `package.json` script that preloads dotenv `"dev": "nodemon -r dotenv/config index.js"` +* Access environment variables as usual `const PORT = process.env.PORT` -Add a .env file that defines an admin username and an admin password hash. -Add an endpoint POST /admin that also expects two request body parameters, _username _ and _password_ . The endpoint should -check that the username matches the one defined in the .env file +# Exercise 5: Environmental Login + -check that the password matches the hash defined in the .env file +Add an admin login to the Students API. We want to store admin credentials in environment variables. -if they match, it should return a response with status code204 (No Content) +Add dotenv library as dependency. Then add a `.env` file that defines an admin username and an admin password hash. -if they do not match, it should return a response with status code 401 (Unauthorized) +Add an endpoint POST `/admin` that also expects two request body parameters, *username* and *password*. The endpoint should +1) check that the username matches the one defined in the `.env` file +2) check that the password matches the hash defined in the `.env` file +3) if they match, it should return a response with status code 204 (No Content) +4) if they do not match, it should return a response with status code 401 (Unauthorized) ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Token-based authentication # JSON Web Token -After verifying the identity of our user, we create a _token_ that we send to the client. The client stores that token and attaches it to all future requests. This token is enough to identify the request as authorized. +After verifying the identity of our user, we create a token that we send to the client. The client stores that token and attaches it to all future requests. This token is enough to identify the request as authorized. Since the tokens are signed by us, we can also include data (such as user id) in the token. On all future requests we can rely that the id is correct since the JWT will be verified on every request. @@ -340,69 +325,68 @@ Tokens can (and should be) equipped with expiration dates. # Creating a Token -Install the JSON Web Token package as dependency npm install jsonwebtoken +Install the JSON Web Token package as dependency `npm install jsonwebtoken` -Tokens are created with the _sign_ function that takes two parameters: _Payload:_ the data we want to include in the token. It can be empty. _Secret_ : either a string or a private key. +Tokens are created with the sign function that takes two parameters: +- Payload: the data we want to include in the token. It can be empty. +- Secret: either a string or a private key. -It can also accept an optional _options_ parameter. In that we can define the expiration date, compression algorithm, or many other things. +It can also accept an optional options parameter. In that we can define the expiration date, compression algorithm, or many other things. +```ts import 'dotenv/config' - import jwt from 'jsonwebtoken' const payload = { username: 'sugarplumfairy' } - const secret = process.env.SECRET - const options = { expiresIn: '1h'} const token = jwt.sign(payload, secret, options) - console.log(token) - +``` The created JWT has three parts separated by period: header, payload and signature. + + # Exercise 6: Create a Token + Write a simple command line program that prints JSON Web tokens. Use the default algorithm (SHA256). Set the tokens to expire in fifteen minutes and include a payload of some JSON object. -Copy your JWT and paste it to [https://jwt.io](https://jwt.io) debugger. Verify that the debugger shows correct algorithm, data, and expiration date (iat). +Copy your JWT and paste it to https://jwt.io debugger. Verify that the debugger shows correct algorithm, data, and expiration date (iat). ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Verifying a Token -Tokens are verified using the _verify_ function that takes two parameters: +Tokens are verified using the verify function that takes two parameters: +- Token: the token to be verified +- Secret: the secret that was used to create the token -_Token_ : the token to be verified +The verify function can also accept an optional options parameter. -_Secret_ : the secret that was used to create the token +The verify function returns a decoded token. Since it has been verified against our secret, we know that the information has not been altered. -The verify function can also accept an optional _options_ parameter. +If the token is invalid, the verify function throws an error. If this error is not caught, the program crashes. -The verify function returns a _decoded_ token. Since it has been verified against our secret, we know that the information has not been altered. -If the token is invalid, the verify function _throws an error_ . If this error is not caught, the program crashes. # Exercise 7: Verify a Token + Write a simple command line program that verifies JSON Web tokens. The program should print the contents of the verified token. If the token is not valid, the program should print an error message and exit gracefully. -Create a JWT in [https://jwt.io](https://jwt.io) debugger. Remember to set your secret value. Use your program to verify the token and see that the data entered is as it should be. +Create a JWT in https://jwt.io debugger. Remember to set your secret value. Use your program to verify the token and see that the data entered is as it should be. ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Bearer Tokens When a user has received a token, it can be used to authenticate all future requests. -Do this by adding the field _Authorization _ to the request header. +Do this by adding the field Authorization to the request header. The value is "Bearer " plus the token. @@ -410,295 +394,202 @@ This header can be added to requests generated by Fetch, Axios, or any other JS Then protected routes can check that the authorization header is valid, and if not, send an error message. This is usually done using authorization middleware. + + # Authentication Middleware A common way is to have user routes, such as login, logout or user creation in a separate router, which requires no token. Then all protected routes are set to use the authentication middleware. -The request parameter _user_ is set to the verified token data and can then be used in the actual route. - -const authenticate = (req, res, next) => { - -const auth = req.get('Authorization') - -if (!auth?.startsWith('Bearer ')) { - -return res.status(401).send('Invalid token') - -} - -const token = auth.substring(7) - -const secret = process.env.SECRET +The request parameter user is set to the verified token data and can then be used in the actual route. -try { - -const decodedToken = jwt.verify(token, secret) - -req.user = decodedToken - -next() - -} catch (error) { - -return res.status(401).send('Invalid token') - -} +--- +```ts +const authenticate = (req: AuthenticatedRequest, res: Response, next: NextFunction) => { + const auth = req.get('Authorization') + if (!auth?.startsWith('Bearer ')) { + return res.status(401).send('Invalid token') + } + + const token = auth.substring(7) + const secret = process.env.SECRET + + try { + const decodedToken = jwt.verify(token, secret) + req.user = decodedToken + next() + } catch (error) { + return res.status(401).send('Invalid token') + } } -router.get('/protected', authenticate, (req, res) => { - -res.send(\`${req.user.username} accessed protected route\`) - +router.get('/protected', authenticate, (req: Request, res: Response) => { + res.send(`${req.user.username} accessed protected route`) }) +``` # TypeScript: Extending Request -Let's look at the following example, written in JavaScript. There is a middleware that adds a _username _ property to the request object. Then the endpoint reads the username property and uses it to construct a response. - -What happens when we write the same code in TypeScript? - -import express from 'express' - -const server = express() - -const middleware = (req, \_res, next) => { - -req.username = 'sugarplumfairy' - -next() - -} - -server.use(middleware) - -server.get('/', (req, res) => { - -res.send(\`Welcome, ${req.username}!\`) - -}) - -server.listen(3000) - -Property 'username' does not exist on type 'Request>' +The above example uses a custom `AuthenticatedRequest` instead of regular `Request` as the type of the `req` object. This is because in TypeScript, if we try to add a new parameter, in this case "user", to a `Request` object, we get an error +``` +Property 'user' does not exist on type +'Request>' +``` -import express, { Request, Response, NextFunction } from 'express' - -const server = express() - -const middleware = (req: Request, res: Response, next: NextFunction) => { - -req.username = 'sugarplumfairy' - -next() - -} - -server.use(middleware) - -server.get('/', (req, res) => { - -res.send(\`Welcome, ${req.username}!\`) - -}) - -server.listen(3000) - -Property 'username' does not exist on type 'Request>' - -We are trying to set a value to request property "username", but request objects are well defined objects that have no such property. +We are trying to set a value to request property "user", but request objects are well defined objects that have no such property. We can not add or use properties that an object does not have. This is part of the type safety that TypeScript is here to enforce. If an object is of type Request, we know exactly what the properties are. -The solution is to create an interface that extends the existing Request type. The new interface will have all the properties the Request has, plus the ones we define ourself. - -interface CustomRequest extends Request { - -username?: string - -} - -Since the interface extends Request, CustomRequest objects can be used anywhere where Request object is required. - -Notice that the additional properties must be optional, signified by the question mark in the name. Otherwise existing Request objects can not be cast as CustomRequest objects. - -This example has only a single string, but the extended object can be as complex as needed. - -interface CustomRequest extends Request { - -username?: string - -} - -Now we can use the "username" property when we declare that we are using CustomRequest instead of basic Request object. - -import express, { Request, Response, NextFunction} from 'express' - -const server = express() - -interface CustomRequest extends Request { - -username?: string - -} - -const middleware = (req: CustomRequest, res: Response, next: NextFunction) => { - -req.username = 'sugarplumfairy' +--- -next() +The solution is to create an interface that extends the existing Request type. The new interface will have all the properties the Request has, plus the ones we define ourself. +```ts +interface AuthenticatedRequest extends Request { + user?: { username: string } } +``` -server.use(middleware) - -server.get('/', (req: CustomRequest, res: Response) => { +Now our `AuthenticatedRequest` object can be used as a regular `Request` object (and hence is acceptable as a parameter to our Express route functions), and it can be used to store an object holding the authenticated username. -res.send(\`Welcome, ${req.username}!\`) - -}) - -server.listen(3000) # Exercise 8: Securing a Route + -Modify the Students API /register and /login routes so that on success they return a response with status code 200 and a JWT with username as payload. +Modify the Students API `/register` and `/login` routes so that on success they return a response with status code 200 and a JWT with username as payload. -Secure all the routes in the _students _ router so that they require the user to be logged in to use the routes. +Secure all the routes in the students router so that they require the user to be logged in to use the routes. -Extra: Also modify the /admin route to return a JWT. Secure the POST, PUT and DELETE routes to require that in addition to being logged in, the user also needs to be an admin. +**Extra**: Also modify the `/admin` route to return a JWT. Secure the POST, PUT and DELETE routes to require that in addition to being logged in, the user also needs to be an admin. -Hint: Here you have some resources on sending a token with Postman. +*Hint*: Here you have some resources on sending a token with Postman. [https://learning.postman.com/docs/sending-requests/authorization/](https://learning.postman.com/docs/sending-requests/authorization/) ---- -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js # Supertest -# Testing APIs with Supertest - -Supertest is a library for testing API's. Let's see how to use it with Jest to test an Express app. Install supertest with +## Testing APIs with Supertest -npm install --save-dev supertest +Supertest is a library for testing API's. Let's see how to use it with Jest to test an Express app. Install supertest, ts-jest and the required TypeScript types with +``` +npm install --save-dev supertest @types/supertest ts-jest +``` When using ESLint, also add "jest": true to .eslintrc's "env" object -_index.js_ - +```ts import express from 'express' const server = express() -server.get('/', (req, res) => { - -res.send('Hello World!') - +server.get('/', (req: Request, res: Response) => { + res.send('Hello World!') }) server.listen(3000, () => { - -console.log('Listening to port 3000') - + console.log('Listening to port 3000') }) +``` -When using TypeScript, also install the required types from @types/supertest and @types/jest +--- -Also, install the ts-jest package, and add to package.json: +Also, add to package.json: +``` "jest": { - -"preset": "ts-jest", - -"testEnvironment": "node" - + "preset": "ts-jest", + "testEnvironment": "node" }, +``` + +## Preparing the App for Testing We need to import our express application to the tests. -We will separate our application in two files. The _server.js_ defines the API server, and _index.js_ only starts the application. +We will separate our application in two files. The `server.ts` defines the API server, and `index.ts` only starts the application. Now we can test the server with Supertest. The startup file has no functionality we want to test. -_index.js_ - +```ts +//index.ts import server from './server.js' server.listen(3000, () => { - -console.log('Listening to port 3000') - + console.log('Listening to port 3000') }) +``` -_server.js_ +--- +```ts +//server.ts import express from 'express' const server = express() -server.get('/', (req, res) => { - -res.send('Hello World!') - +server.get('/', (req: Request, res: Response) => { + res.send('Hello World!') }) export default server +``` -We now create a new test file and import our server there. - -We use Supertest _request_ to get responses from our server. These responses can be tested just like any other Jest test. +--- -_test/server.test.js_ +
+
-import request from 'supertest' +We now create a new test file and import our server there. -import server from '../src/server.js' +We use Supertest request to get responses from our server. These responses can be tested just like any other Jest test. -describe('Server', () => { +
+
-it('Returns 404 on invalid address', async () => { -const response = await request(server) -.get('/invalidaddress') +```ts +//test/server.test.ts -expect(response.statusCode).toBe(404) +import request from 'supertest' +import server from '../src/server.js' +describe('Server', () => { + it('Returns 404 on invalid address', async () => { + const response = await request(server) + .get('/invalidaddress') + expect(response.statusCode).toBe(404) + }) + + it('Returns 200 on valid address', async () => { + const response = await request(server) + .get('/') + expect(response.statusCode).toBe(200) + expect(response.text).toEqual('Hello World!') + }) }) +``` -it('Returns 200 on valid address', async () => { - -const response = await request(server) +
+
-.get('/') -expect(response.statusCode).toBe(200) - -expect(response.text).toEqual('Hello World!') - -}) - -}) # Common Resources -[https://www.npmjs.com/package/jest](https://www.npmjs.com/package/jest) → Official repository +https://www.npmjs.com/package/jest → Official repository -[https://www.npmjs.com/package/supertest](https://www.npmjs.com/package/supertest) → Official repository +https://www.npmjs.com/package/supertest → Official repository -[https://jestjs.io/docs/api](https://jestjs.io/docs/api) → Jest API Docs. Very handy! +https://jestjs.io/docs/api → Jest API Docs. Very handy! -[https://github.com/visionmedia/supertest](https://github.com/visionmedia/supertest) → Supertest's github repository. Includes guides for basic use cases. +https://github.com/visionmedia/supertest → Supertest's github repository. Includes guides for basic use cases. # Exercise 9: Test with Supertest + -Use Supertest with Jest to create tests for the Students API /user/register and /user/login routes. - ---- - -Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js - +Use Supertest with Jest to create tests for the Students API `/user/register` and `/user/login` routes. \ No newline at end of file diff --git a/3-databases-with-docker-slides.html b/3-databases-with-docker-slides.html new file mode 100644 index 0000000..db166ae --- /dev/null +++ b/3-databases-with-docker-slides.html @@ -0,0 +1,697 @@ +Databases
+

Databases

+
+
+

SQL

+
    +
  • Structured Query Language
    • +
  • A language to organize and manipulate data in a relational database
    • +
  • Originally developed by IBM in the 70s
    • +
  • Quickly became the most popular database language
    • +
+
SELECT id, email
+FROM users
+WHERE first_name = 'Teppo';
+
+
+
+

Relational Database Management Systems

+
    +
  • In relational databases, values are stored in tables +
      +
    • Each table has rows and columns
      • +
    • Data is displayed in a two-dimensional matrix
      • +
    +
    • +
  • Values in a table are related to each other +
      +
    • Values can also be related to values in other tables
      • +
    +
    • +
  • A relational database management system (RDBMS) is a program that executes queries to relational databases
    • +
+
+
+

Relational Database Management Systems

+

+

https://db-engines.com/en/ranking

+
+
+

PostgreSQL

+

A free, open-source, cross-platform relational database management system

+

Emphasizes extensibility and SQL compliance

+

Fully ACID-compliant (atomicity, consistency, isolation and durability)

+
+
+

Running Postgres in Docker

+

Using the official Postgres Docker image , let's create a locally running Postgres instance. You can choose the values for POSTGRES_PASSWORD and POSTGRES_USER freely.

+
docker run --name my-postgres --env POSTGRES_PASSWORD=pgpass 
+--env POSTGRES_USER=pguser -p 5432:5432 -d postgres:15.2
+
+

+
+
+

Managing the PostgreSQL database with psql

+

psql is a terminal-based front-end to PostgreSQL. It enables you to type in queries interactively, issue them to PostgreSQL, and see the query results. Alternatively, input can be from a file or from command line arguments. In addition, psql provides a number of meta-commands and various shell-like features to facilitate writing scripts and automating a wide variety of tasks.

+

https://www.postgresql.org/docs/current/app-psql.html

+

You can use psql directly from the container running the database

+
docker exec -it my-postgres psql -U pguser
+
+
+
+

Managing the PostgreSQL database with psql

+
    +
  • If you have connected to the server as user "pguser", you are by default connected to "pguser" database.
    • +
  • This is a database for user information. Do not use it for storing program data!!
    • +
  • All databases can be listed using the list command: \l
    • +
  • PostgreSQL uses a default database named "postgres"
    • +
  • Users can connect to a different database with connect command \c: +
      +
    • \c <database-name>
      • +
    +
    • +
  • A new database is created with CREATE command: +
      +
    • CREATE DATABASE <database-name>; <-- notice the semicolon!
      • +
    • After creating a new database, you still need to connect to it!
      • +
    +
    • +
  • Exit psql with the command exit
    • +
+
+
+

Exercise 1: Postgres Server

+ +

Start a local instance of Postgres in Docker

+

Connect to the server using psql

+

Use command \l to see what databases there are already created on the server.

+

Create a new database called "sqlpractice".

+

Connect to the newly created database.

+
+
+

pgAdmin

+

Administration and development platform for PostgreSQL

+

Cross-platform, features a web interface

+

Basically a control panel application for your PostgreSQL database

+

A graphical alternative to psql

+

Completely separate from PostgreSQL, so pgAdmin is not required for using PostgreSQL. It is only one example of how you can administer the database. Other options include the forementioned psql and a wide variety of other database management tools, like DBeaver.

+
+
+

Running pgAdmin in Docker

+

Using the official pgAdmin image, we'll run pgAdmin alongside Postgres

+
docker run --name my-pgadmin -p 5050:80 
+--env PGADMIN_DEFAULT_EMAIL=<your-email-address> 
+--env PGADMIN_DEFAULT_PASSWORD=<your-password> 
+-d dpage/pgadmin4
+
+

+
+
+

Logging into pgAdmin

+

With pgAdmin running, navigate your web browser to http://localhost:5050 and use the username and password you provided to log in.

+

+
+
+

PostgreSQL internal IP Address

+

Both PostgreSQL and pgAdmin are now running in our local Docker. To connect pgAdmin to PostgreSQL, we need the IP address used inside Docker.

+

Using Docker's inspect, we'll get Docker's internal IP for my-postgres -container. Since the command produces quite a lot of information, we pipe the result to grep (Linux) or findstr (Windows), to see only the rows that contain the word "IPAddress"

+
docker inspect <container-name> | grep IPAddress
+docker inspect <container-name> | findstr IPAddress
+
+

In the example output, the IP Address is 172.17.0.2

+

+
+
+

Connecting PgAdmin to our DB

+

Now we have all that we need for a connection. In pgAdmin, select "Object" > "Register" > "Server".

+

In the "General" tab, give the server a name that identifies the connection.

+

+
+
+

If Object menu is greyed out, click on Servers.

+

+
+
+
    +
  • In the "Connection" tab, enter
    • +
  • Host name/address: +
      +
    • the PostgreSQL internal Docker address
      • +
    +
    • +
  • Port, Username, Password +
      +
    • the values defined when running the PostgreSQL container
      • +
    +
    • +
  • Then click Save. You should now see all the databases available on this server.
    • +
+
+
+

+
+
+

+
+
+

Exercise 2: pgAdmin

+ +

Start a local instance of pgAdmin in Docker

+

Following lecture instructions, connect the pgAdmin to your already running PostgreSQL server.

+

Verify that you can see the database created in the previous assignment.

+
+
+

PostgreSQL: Querying

+

With psql: After connecting to a database, just type a query and hit enter.

+

With pgAdmin:

+

Right-click a database > Query tool

+

Insert a query into the Query Editor and hit Execute (F5)

+

+
+
+

+
+
+

Editing Data with pgAdmin

+
    +
  • Tables of Data in a DataBase are found under +
      +
    • Database > Schemas > Tables
      • +
    +
    • +
  • Inspect and edit data in pgAdmin by right-clicking a table and selecting View/Edit Data
    • +
+
+
+

+
+
+

+

Individual values in the table can be directly modified by double clicking the value and then editing the value in the visual user interface

+

Save the changes with the Save Data Changes button

+

+
+
+

Exercise 3: Preparing the Database

+ +

Using either PgAdmin or PSQL

+

Insert the provided query to the database you created previously.

+

Verify that the query has created new tables to your database.

+
+
+

Types of queries

+
    +
  • Select
    • +
  • Insert
    • +
  • Delete
    • +
  • Update
    • +
  • Create & Drop
    • +
+
+
+

Querying data with SELECT

+

Syntax:

+
SELECT column1, column2, column3 FROM table_name;
+
+

Examples:

+
    +
  • SELECT full_name, email FROM users;
    • +
  • SELECT full_name AS name, email FROM users;
    • +
  • SELECT * FROM users;
    • +
+
+
+

Filtering data with WHERE

+

Syntax:

+
SELECT column1, column2 FROM table_name WHERE condition;
+
+

Text is captured in single quotes. In a LIKE condition, % sign acts as a wildcard. IS and IS NOT are also valid comparison operators.

+

Example:

+
    +
  • SELECT full_name FROM users;
    • +
  • WHERE full_name = 'Teppo Testaaja';
    • +
  • SELECT * FROM books WHERE name LIKE '%rr%';
    • +
  • SELECT * FROM books WHERE author IS NOT null;
    • +
+
+
+

Ordering data with ORDER BY

+

Syntax:

+
SELECT column1 FROM table_name ORDER BY column1 ASC;
+
+

Examples:

+
    +
  • SELECT full_name FROM users ORDER BY full_name ASC;
    • +
  • SELECT full_name FROM users ORDER BY full_name DESC
    • +
+
+
+

Combining with JOIN

+

Also known as INNER JOIN

+

Corresponds to intersection from set theory

+

+
+
+

JOIN examples

+
SELECT users.id, users.full_name, borrows.id, borrows.user_id,
+borrows.due_date, borrows.returned_at
+FROM users JOIN borrows ON users.id = borrows.user_id;
+
+
SELECT U.full_name AS name, B.due_date AS due_date, B.returned_at AS returned_at
+FROM users AS U JOIN borrows AS B ON U.id = B.user_id;
+
+
+
+

Exercise 4: Querying the Library

+ +

Using SQL queries, get

+
    +
  1. All columns from loans that are loaned before 1.3.2000
    2. +
  2. All columns of loans that are returned
    3. +
  3. Columns user.full_name and borrows.borrowed_at of the user with an id of 1
    4. +
  4. Columns book.name, book.release_year and language.name of all books that are released after 1960
    5. +
+
+
+

INSERT

+

Syntax

+
INSERT INTO table_name (column1, column2, column3) VALUES (value1, value2, value3);
+
+

Example

+
    +
  • INSERT INTO users (full_name, email, created_at) VALUES ('Pekka Poistuja', 'pekka.poistuja@buutti.com', NOW());
    • +
+

Since id is not provided, it will be automatically generated.

+
+
+

UPDATE

+

Syntax

+
UPDATE table_nameSET column1 = value1, column2 = value2 WHERE condition;
+
+

Notice: if a condition is not provided, all rows will be updated! If updating only one row, it is usually best to use id.

+

Example

+
    +
  • UPDATE usersSET email = 'taija.testaaja@gmail.com' WHERE id = 2;
    • +
+
+
+

REMOVE

+

Syntax

+
DELETE FROM table_name WHERE condition;
+
+

Again, if the condition is not provided, DELETE affects all rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected.

+

Example:

+
    +
  • SELECT * FROM users WHERE id = 5; +
      +
    • First use SELECT to confirm the data you're about to delete
      • +
    +
    • +
  • DELETE FROM users WHERE id = 5; +
      +
    • Then delete the data
      • +
    +
    • +
+
+
+

Exercise 5: Editing Data

+ +

Postpone the due date of the loan with an id of 2 by two days in the borrows table

+

Add a couple of new books to the books table

+

Delete one of the loans.

+
+
+

CREATE TABLE

+

Before data can be manipulated, a database and its tables need to be initialized.

+

Syntax

+
CREATE TABLE table_name (
+  column1 datatype,	
+  column2 datatype,	
+  …
+);
+
+

Example:

+
CREATE TABLE "users" (
+  "id" SERIAL PRIMARY KEY,
+  "full_name" varchar NOT NULL,
+  "email" varchar UNIQUE NOT NULL,
+  "created_at" timestamp NOT NULL
+);
+
+
+
+

DROP

+

In order to remove tables or databases, we use a DROP statement. Syntax:

+
DROP TABLE table_name;
+DROP DATABASE database_name;
+
+

These statements do not ask for confirmation and there is no undo feature. Take care when using a drop statement.

+
+
+

NoSQL

+
    +
  • In addition to SQL databases, there are also NoSQL databases
    • +
  • Many differing definitions, but... +
      +
    • most agree that NoSQL databases store data in a format other than tables
      • +
    • They can still store relational data - just differently
      • +
    +
    • +
  • Four different database types: +
      +
    • Document databases
      • +
    • Key-value databases
      • +
    • Wide-column stores
      • +
    • Graph databases
      • +
    +
    • +
  • Example database engines include MongoDB, Redis and Cassandra
    • +
+
+
+

Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. MongoDB is consistently ranked as the world's most popular NoSQL database according to DB-engines and is an example of a document database. For more on document databases, visit What is a Document Database?.

+
+
+

Key-value databases are a simpler type of database where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don't need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Redis and DynanoDB are popular key-value databases.

+
+
+

Wide-column stores store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores.

+
+
+

Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Neo4j and JanusGraph are examples of graph databases.

+
+
+

Object-Relational Mappers

+
    +
  • ORMs allow developers to manipulate databases with code instead of SQL queries +
      +
    • For example, for performing CRUD operations on their database
      • +
    +
    • +
  • Some popular ORMs:
    • +
  • Hibernate (Java)
    • +
  • EFCore (.NET)
    • +
  • Sequelize (Node.js)
    • +
  • TypeORM (TypeScript) - documentation
    • +
+
+
+

PostgreSQL with Node

+
+
+

Local environment

+

Using our previously created, Dockerized Postgres instance, we'll create a Node application to connect to our database.

+

If you have deleted your postgres container, just create a new one with the same command.

+

If your container has shut down (after a reboot for example), you can start it with

+
docker start my-postgres
+
+
+
+

Preparing our Node application

+

Initialize a new TypeScript application.

+

Install express and PostgreSQL client for Node.JS, and their respective TypeScript types

+

Install dotenv, nodemon and ts-node development dependencies

+
+
+

Dotenv

+

Example of .env file:

+
PORT=3000
+PG_HOST=localhost
+PG_PORT=5432
+PG_USERNAME=pguser
+PG_PASSWORD=mypassword
+PG_DATABASE=postgres
+
+

These values must match the values declared when running the PostgreSQL container.

+

The database must exist as well.

+

Note that the example uses the default "postgres" database, but you can use any database you want.

+
+
+

Dotenv (continued)

+
{
+  "name": "products_api",
+  "version": "1.0.0",
+  "scripts": {
+    "dev": "nodemon -r dotenv/config ./src/index.ts"
+  },
+  "dependencies": {
+    "express": "^4.18.2",
+    "pg": "^8.9.0"
+  },
+  "devDependencies": {
+    "@types/express": "^4.17.17",
+    "@types/pg": "^8.6.6",
+    "dotenv": "^16.0.3",
+    "nodemon": "^2.0.20",
+    "ts-node": "^10.9.1",
+    "typescript": "^4.9.5"
+  }
+}
+
+
+
+

Dotenv is usually only used in development, not in production. In professional setting the dotenv config is often preloaded in the development startup script

+

You can require dotenv when running npm run dev_

+

-r is short for --require

+
+
+

Dotenv and Git

+
    +
  • .env files usually contain sensitive data that we do not want to store in Git repositories.
    • +
  • Thus, usually .env file is excluded from the Git repository +
      +
    • Add .env to _.gitignore
      • +
    • If you have auto-generated .gitignore with npx gitignore Node, environment files are excluded automatically
      • +
    +
    • +
+

+
+
+

Connecting to PostgreSQL

+

Our database file contains functions and configuration for initializing the Postgres pool, creating tables and running queries.

+

At the moment, we have only one query. It is a single-use query that creates a products table to the database if such table does not yet exist.

+
// db.ts
+import pg from "pg";
+const { PG_HOST, PG_PORT, PG_USERNAME, PG_PASSWORD, PG_DATABASE } = process.env;
+
+const pool = new pg.Pool({
+  host: PG_HOST,
+  port: Number(PG_PORT),
+  user: PG_USERNAME,
+  password: PG_PASSWORD,
+  database: PG_DATABASE,
+});
+
+const executeQuery = async (query: string, parameters?: Array<any>) => {
+  const client = await pool.connect();
+  try {
+    const result = await client.query(query, parameters);
+    return result;
+  } catch (error: any) {
+    console.error(error.stack);
+    error.name = "dbError";
+    throw error;
+  } finally {
+    client.release();
+  }
+};
+
+export const createProductsTable = async () => {
+  const query = `CREATE TABLE IF NOT EXISTS "products" (
+    "id" SERIAL PRIMARY KEY,
+    "name" VARCHAR(100) NOT NULL,
+    "price" REAL NOT NULL
+    )`;
+  await executeQuery(query);
+  console.log("Products table initialized");
+};
+
+
+
+

At the moment our index.ts does nothing but creates a single table for our database and launches express server.

+

It doesn't even have any endpoints, so it's not much of a server yet.

+

+import express from "express";
+import { createProductsTable } from "./db";
+
+const server = express();
+createProductsTable();
+
+const { PORT } = process.env;
+
+server.listen(PORT, () => {
+  console.log("Products API listening to port", PORT);
+});
+
+
+
+

Launching the application

+

Let's use our predefined npm run dev

+

+
+
+

…And check with psql that our application succeeds in connecting to the database and creating the table.

+

Epic success!

+

+
+
+

Exercise 6: Node & PostgreSQL

+ +

Following the lecture example, create an Express server that connects to your local PostgreSQL instance. The database information should be stored in environment variables. When the server starts, it should create a product table with three columns: id (serial, primary key), name (varchar) and price (real).

+
+
+

Creating Queries

+
    +
  • Next, we will create an actual CRUD API for communicating with the database.
    • +
  • For that we need endpoints for creating, reading, updating and deleting products. +
      +
    • All of these need their own queries.
      • +
    +
    • +
+
+
+

Using queries

+
    +
  • We'll use the pre-made executeQuery() function for querying the database from a few slides back
    • +
  • It takes in two arguments: +
      +
    • the actual query string
      • +
    • an optional parameters array
      • +
    +
    • +
  • When supplying parameters, the query string should have placeholders $1, $2, etc +
      +
    • These will be replaced with the contents of the parameters array.
      • +
    +
    • +
+
+
+

Parameterized queries example

+

+

When running executeQuery(query, parameters) with above values defined, the query would be parsed as

+
SELECT * FROM cats WHERE color = 'yellow' and age > 10;
+
+
+
+

Why not just use String templating?

+

…Because of SQL injections . Always use database library's built-in parameterization!

+

NEVER DO THIS!!!

+

+

+
+
+

Creating queries

+

We will create a Data Access Object, dao.ts that will handle interacting with our database.

+

The idea is that we want to just tell our DAO what we want done (e.g. "add this customer to the database") and the DAO will handle the details of that action.

+

The DAO will also return possible additional information that was created during the action.

+
+
+

Our insertProduct function

+

generates a new, unique ID for the product using uuid

+

constructs a parameters array containing said id, the name of the product and the price of the product

+

executes the query using db.executeQuery method

+

returns the database result object

+

+
+
+
+
+

+
+
+

+
+
+
+
+

The rest of the DAO operations work in similar fashion.

+

The router that declares the endpoints, uses the DAO to interact with the database.

+
+
+

Testing the API

+
+
+

Now we can use Insomnia to verify that all the endpoints work as expected.

+

We can also use psql to observe the changes in the database

+

+
+
+

+
+
+
+
+

Exercise 7: Creating Queries

+ +

Continue following the lecture example. Create a router and a database access object to handle

+
    +
  • Creating a product
    • +
  • Reading a product
    • +
  • Updating a product
    • +
  • Deleting a product
    • +
  • Listing all products
    • +
+
+
+

Dockerized PostgreSQL App

+
+
+

Setting Environment Variables

+

Docker has two kinds of environment variables: run-time and build-time.

+

In this scenario we want to set our environment variables at build time. This means that the Docker image will contain all the environment variable information, including sensitive things like passwords. This might be an issue in some scenarios. In those cases the environment variables need to be set at run time.

+

In the Dockerfile we set the build-time values by setting ARG parameters. Then we use these values to set the run-time environment variables by setting ENV parameters.

+

More information: https://vsupalov.com/docker-arg-env-variable-guide/

+
+
+

When the ARGs and ENVs have been set in the Dockerfile, we provide the ARG values when building the Docker image by using_--build-arg =_ flags. To build an image with these parameters, we'd use something like

+
docker build 
+--build-arg PORT=3000 
+--build-arg PG_HOST=https://my.postgres.server 
+--build-arg PG_PORT=5432 
+--build-arg PG_USERNAME=pguser 
+--build-arg PG_PASSWORD=pgpass 
+--build-arg PG_DATABASE=my-database 
+-t my-app .
+
+
+
+

And include the build-arg parameters in our Dockerfile with them mapped to environment variables.

+
ARG PORT
+ARG PG_HOST
+ARG PG_PORT
+ARG PG_USERNAME
+ARG PG_PASSWORD
+ARG PG_DATABASE
+
+ENV PORT=${PORT}
+ENV PG_HOST=${PG_HOST}
+ENV PG_PORT=${PG_PORT}
+ENV PG_USERNAME=${PG_USERNAME}
+ENV PG_PASSWORD=${PG_PASSWORD}
+ENV PG_DATABASE=${PG_DATABASE}
+
+

Docker documentation here!

+
+
+

Exercise 8: Dockerized PG App

+ +

Dockerize the application you have built. Build the docker image, run the app and test that it works using insomnia/postman.

+

Remember that when you run the application on your local Docker, both the app and the database are in the same Docker network, so you have to check the database IP address just like when running pgAdmin.

+
+
\ No newline at end of file diff --git a/3-databases-with-docker.md b/3-databases-with-docker.md index 3fbcaed..a5d182c 100644 --- a/3-databases-with-docker.md +++ b/3-databases-with-docker.md @@ -1,114 +1,169 @@ +--- +marp: true +paginate: true +math: mathjax +theme: default +theme: buutti +title: Databases +--- + # Databases + + + + + # SQL -* __Structured Query Language__ - * A language __ __ to organize and manipulate data in a relational database -* Originally developed by IBM in the 70s - * Quickly became the most popular database language -* SELECT id, email -* FROM users -* WHERE first\_name = 'Teppo'; +- Structured Query Language +- A language to organize and manipulate data in a relational database +- Originally developed by IBM in the 70s +- Quickly became the most popular database language + +``` +SELECT id, email +FROM users +WHERE first_name = 'Teppo'; +``` + -# Relational Database -Management Systems -* In relational databases, values are stored in __tables__ - * Each table has __rows __ and __columns__ +# Relational Database Management Systems + +* In relational databases, values are stored in **tables** + * Each table has **rows** and **columns** * Data is displayed in a two-dimensional matrix * Values in a table are related to each other * Values can also be related to values in other tables * A relational database management system (RDBMS) is a program that executes queries to relational databases + + +# Relational Database Management Systems + ![](imgs/3-databases-with-docker_0.png) [https://db-engines.com/en/ranking](https://db-engines.com/en/ranking) + + # PostgreSQL A free, open-source, cross-platform relational database management system Emphasizes extensibility and SQL compliance -Fully [ACID](https://en.wikipedia.org/wiki/ACID) -compliant (atomicity, consistency, isolation and durability) - -# pgAdmin - -Administration and development platform for PostgreSQL +Fully [ACID](https://en.wikipedia.org/wiki/ACID)-compliant (atomicity, consistency, isolation and durability) -Cross-platform, features a web interface -Basically a control panel application for your PostgreSQL database # Running Postgres in Docker -Using the official [Postgres Docker image](https://hub.docker.com/_/postgres) , let's create a locally running Postgres instance. You can choose the values for POSTGRES\_PASSWORD and POSTGRES\_USER freely. - -__docker run --name my-postgres --env POSTGRES\_PASSWORD=pgpass --env POSTGRES\_USER=pguser -p 5432:5432 -d postgres:15.2__ +Using the official [Postgres Docker image](https://hub.docker.com/_/postgres) , let's create a locally running Postgres instance. You can choose the values for POSTGRES_PASSWORD and POSTGRES_USER freely. +``` +docker run --name my-postgres --env POSTGRES_PASSWORD=pgpass +--env POSTGRES_USER=pguser -p 5432:5432 -d postgres:15.2 +``` ![](imgs/3-databases-with-docker_1.png) -# PostgreSQL: Using psql -If you have PostgreSQL installed -Win search _psql _ and open _SQL Shell (psql) +# Managing the PostgreSQL database with psql -Press enter four times to insert default values, or values matching to your configuration, and insert your password for the fifth prompt +*psql is a terminal-based front-end to PostgreSQL. It enables you to type in queries interactively, issue them to PostgreSQL, and see the query results. Alternatively, input can be from a file or from command line arguments. In addition, psql provides a number of meta-commands and various shell-like features to facilitate writing scripts and automating a wide variety of tasks.* -If you do not have PostgreSQL installed, you can use psql directly from the container running the database docker exec -it my-postgres psql -U pguser +https://www.postgresql.org/docs/current/app-psql.html + +You can use psql directly from the container running the database +``` +docker exec -it my-postgres psql -U pguser +``` + +# Managing the PostgreSQL database with psql * If you have connected to the server as user "pguser", you are by default connected to "pguser" database. -* This is a database for user information. -* You should not use it for storing program data!! -* All databases can be listed using the _list _ command: __ \\l __ +* This is a database for user information. Do not use it for storing program data!! +* All databases can be listed using the *list* command: `\l` * PostgreSQL uses a default database named "postgres" -* Users can connect to a different database with _connect _ command __\\c__ : - * __\\c __ +* Users can connect to a different database with *connect* command `\c`: + * `\c ` * A new database is created with CREATE command: - * __CREATE DATABASE ;__ - * Do not forget the __;__ at the end -* After creating a new database, you still need to connect to it! -* Exit psql with the command __exit__ + * `CREATE DATABASE ;` <-- notice the semicolon! + * After creating a new database, you still need to connect to it! +* Exit psql with the command `exit` + + # Exercise 1: Postgres Server + Start a local instance of Postgres in Docker Connect to the server using psql -Use command \\l to see what databases there are already created on the server. +Use command `\l` to see what databases there are already created on the server. Create a new database called "sqlpractice". Connect to the newly created database. -# Running pgAdmin in Docker -Using the official [pgAdmin](https://hub.docker.com/r/dpage/pgadmin4) image, we'll run pgAdmin alongside Postgres -__docker run --name my-pgadmin -p 5050:80 __ __-e PGADMIN\_DEFAULT\_EMAIL= __ __-e PGADMIN\_DEFAULT\_PASSWORD= __ __-d dpage/pgadmin4__ +# pgAdmin + +Administration and development platform for PostgreSQL + +Cross-platform, features a web interface + +Basically a control panel application for your PostgreSQL database + +A graphical alternative to psql + +Completely separate from PostgreSQL, so pgAdmin is not required for using PostgreSQL. It is only one example of how you can administer the database. Other options include the forementioned psql and a wide variety of other database management tools, like [DBeaver](https://dbeaver.io/). + + + +# Running pgAdmin in Docker + +Using the official [pgAdmin](https://hub.docker.com/r/dpage/pgadmin4) image, we'll run pgAdmin alongside Postgres +``` +docker run --name my-pgadmin -p 5050:80 +--env PGADMIN_DEFAULT_EMAIL= +--env PGADMIN_DEFAULT_PASSWORD= +-d dpage/pgadmin4 +``` ![](imgs/3-databases-with-docker_2.png) + + # Logging into pgAdmin -With pgAdmin running, navigate your web browser to [http://localhost:5050](http://localhost:5050) and use the username and password you provided to log in. +With pgAdmin running, navigate your web browser to http://localhost:5050 and use the username and password you provided to log in. ![](imgs/3-databases-with-docker_3.png) + + # PostgreSQL internal IP Address -Both __PostgreSQL__ and __pgAdmin __ are now running in our _local_ Docker. To connect pgAdmin to PostgreSQL, we need the IP address used _inside_ Docker. +Both **PostgreSQL** and **pgAdmin** are now running in our *local* Docker. To connect pgAdmin to PostgreSQL, we need the IP address used inside Docker. -Using Docker's inspect, we'll get Docker's internal IP for my-postgres -container. Since the command produces quite a lot of information, we pipe the result to grep, to see only the rows that contain the word "IPAddress" +Using Docker's inspect, we'll get Docker's internal IP for my-postgres -container. Since the command produces quite a lot of information, we pipe the result to grep (Linux) or findstr (Windows), to see only the rows that contain the word "IPAddress" -__docker inspect | grep IPAddress __ __<- unix__ __docker inspect | findstr IPAddress __ __<- windows__ +``` +docker inspect | grep IPAddress +docker inspect | findstr IPAddress +``` In the example output, the IP Address is 172.17.0.2 ![](imgs/3-databases-with-docker_4.png) + + # Connecting PgAdmin to our DB Now we have all that we need for a connection. In pgAdmin, select "Object" > "Register" > "Server". @@ -117,10 +172,14 @@ In the "General" tab, give the server a name that identifies the connection. ![](imgs/3-databases-with-docker_5.png) +--- + If Object menu is greyed out, click on Servers. ![](imgs/3-databases-with-docker_6.png) +--- + * In the "Connection" tab, enter * Host name/address: * the PostgreSQL internal Docker address @@ -128,11 +187,18 @@ If Object menu is greyed out, click on Servers. * the values defined when running the PostgreSQL container * Then click Save. You should now see all the databases available on this server. +--- + ![](imgs/3-databases-with-docker_7.png) +--- + ![](imgs/3-databases-with-docker_8.png) + + # Exercise 2: pgAdmin + Start a local instance of pgAdmin in Docker @@ -140,39 +206,52 @@ Following lecture instructions, connect the pgAdmin to your already running Post Verify that you can see the database created in the previous assignment. + + # PostgreSQL: Querying With psql: After connecting to a database, just type a query and hit enter. With pgAdmin: -Right-click a database _ _ > _Query tool_ +Right-click a database > Query tool -Insert a query into the _Query Editor_ and hit _Execute _ (F5) +Insert a query into the Query Editor and hit Execute (F5) ![](imgs/3-databases-with-docker_9.png) +--- + ![](imgs/3-databases-with-docker_10.png) ---- + + + # Editing Data with pgAdmin * Tables of Data in a DataBase are found under * Database > Schemas > Tables -* Inspect and edit data in pgAdmin by right-clicking a table and selecting _View/Edit Data_ +* Inspect and edit data in pgAdmin by right-clicking a table and selecting View/Edit Data + +--- ![](imgs/3-databases-with-docker_11.png) +--- + ![](imgs/3-databases-with-docker_12.png) Individual values in the table can be directly modified by double clicking the value and then editing the value in the visual user interface -Save the changes with the _Save Data Changes_ button +Save the changes with the Save Data Changes button ![](imgs/3-databases-with-docker_13.png) + + # Exercise 3: Preparing the Database + Using either PgAdmin or PSQL @@ -182,67 +261,59 @@ Verify that the query has created new tables to your database. # Types of queries -Select - -Insert +- Select +- Insert +- Delete +- Update +- Create & Drop -Delete -Update - -Create & Drop # Querying data with SELECT Syntax: -__SELECT column1, column2, column3 FROM table\_name;__ +``` +SELECT column1, column2, column3 FROM table_name; +``` Examples: +- `SELECT full_name, email FROM users;` +- `SELECT full_name AS name, email FROM users;` +- `SELECT * FROM users;` -SELECT full\_name, email FROM users; - -SELECT full\_name AS name, email FROM users; -SELECT * FROM users; # Filtering data with WHERE Syntax: -__SELECT column1, column2 FROM table\_name WHERE condition;__ - -Text is captured in _single quotes_ . - -In a LIKE condition, _%_ sign acts as a wildcard. +``` +SELECT column1, column2 FROM table_name WHERE condition; +``` -IS and IS NOT are also valid comparison operators. +Text is captured in **single** quotes. In a LIKE condition, % sign acts as a wildcard. IS and IS NOT are also valid comparison operators. Example: +- `SELECT full_name FROM users;` +- `WHERE full_name = 'Teppo Testaaja';` +- `SELECT * FROM books WHERE name LIKE '%rr%';` +- `SELECT * FROM books WHERE author IS NOT null;` -SELECT full\_name FROM users -WHERE full\_name = 'Teppo Testaaja'; - -SELECT * FROM books WHERE name LIKE '%rr%'; - -SELECT * FROM books WHERE author IS NOT null; # Ordering data with ORDER BY Syntax: - -__SELECT column1 FROM table\_name ORDER BY column1 ASC;__ +``` +SELECT column1 FROM table_name ORDER BY column1 ASC; +``` Examples: +- `SELECT full_name FROM users ORDER BY full_name ASC;` +- `SELECT full_name FROM users ORDER BY full_name DESC` -SELECT full\_name FROM users - -ORDER BY full\_name ASC; -SELECT full\_name FROM users - -ORDER BY full\_name DESC # Combining with JOIN @@ -252,146 +323,124 @@ Corresponds to intersection from set theory ![](imgs/3-databases-with-docker_14.png) -# JOIN examples - -SELECT - -users.id, users.full\_name, borrows.id, - -borrows.user\_id, borrows.due\_date, borrows.returned\_at - -FROM users - -JOIN borrows ON - -users.id = borrows.user\_id; - -SELECT - -U.full\_name AS name, - -B.due\_date AS due\_date, - -B.returned\_at AS returned\_at - -FROM users AS U - -JOIN borrows AS B ON - -U.id = B.user\_id; - -# Combining with LEFT JOIN - -Also known as LEFT OUTER JOIN - -Example: - -SELECT - -U.full\_name AS name, - -B.due\_date AS due\_date, -B.returned\_at AS returned\_at -FROM users AS U +# JOIN examples -LEFT JOIN borrows AS B ON +``` +SELECT users.id, users.full_name, borrows.id, borrows.user_id, +borrows.due_date, borrows.returned_at +FROM users JOIN borrows ON users.id = borrows.user_id; +``` +``` +SELECT U.full_name AS name, B.due_date AS due_date, B.returned_at AS returned_at +FROM users AS U JOIN borrows AS B ON U.id = B.user_id; +``` -U.id = B.user\_id; -![](imgs/3-databases-with-docker_15.png) # Exercise 4: Querying the Library + Using SQL queries, get -All columns from loans that are loaned before 1.3.2000 - -All columns of loans that are returned +1) All columns from loans that are loaned before 1.3.2000 +2) All columns of loans that are returned +3) Columns user.full_name and borrows.borrowed_at of the user with an id of 1 +4) Columns book.name, book.release_year and language.name of all books that are released after 1960 -Columns user.full\_name and borrows.borrowed\_at of the user with an id of 1 -Columns book.name, book.release\_year and language.name of all books that are released after 1960 # INSERT Syntax - -__INSERT INTO table\_name (column1, column2, column3)_ __VALUES (value1, value2, value3);__ - +``` +INSERT INTO table_name (column1, column2, column3) VALUES (value1, value2, value3); +``` Example +- `INSERT INTO users (full_name, email, created_at) VALUES ('Pekka Poistuja', 'pekka.poistuja@buutti.com', NOW());` -INSERT INTO users (full\_name, email, created\_at) +Since id is not provided, it will be automatically generated. -VALUES ('Pekka Poistuja', 'pekka.poistuja@buutti.com', NOW()); -Since id is not provided, it will be automatically generated. # UPDATE Syntax - -__UPDATE table\_name__ __SET column1 = value1, column2 = value2__ __WHERE condition;__ - -__Notice:__ if a condition is not provided, all rows will be updated!If updating only one row, it is usually best to use id. +``` +UPDATE table_nameSET column1 = value1, column2 = value2 WHERE condition; +``` +Notice: if a condition is not provided, all rows will be updated! If updating only one row, it is usually best to use id. Example +- `UPDATE usersSET email = 'taija.testaaja@gmail.com' WHERE id = 2;` + -UPDATE usersSET email = 'taija.testaaja@gmail.com'WHERE id = 2; # REMOVE Syntax - -__DELETE FROM table\_name WHERE condition;__ - -Again, if the _condition_ is not provided, DELETE affects _all_ rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected. +``` +DELETE FROM table_name WHERE condition; +``` +Again, if the condition is not provided, DELETE affects all rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected. Example: +- `SELECT * FROM users WHERE id = 5;` + - First use SELECT to confirm the data you're about to delete +- `DELETE FROM users WHERE id = 5;` + - Then delete the data -SELECT __*__ FROM __users __ WHERE __id = 5;__ -DELETE FROM users WHERE id = 5; # Exercise 5: Editing Data + -Postpone the due date of the loan with an id of 2 by two days in the _borrows_ _ _ table +Postpone the due date of the loan with an id of 2 by two days in the borrows table -Add a couple of new books to the _books _ table +Add a couple of new books to the books table Delete one of the loans. + + # CREATE TABLE Before data can be manipulated, a database and its tables need to be initialized. Syntax +``` +CREATE TABLE table_name ( + column1 datatype, + column2 datatype, + … +); +``` -__CREATE TABLE table\_name (__ - -__ column1 datatype,__ __ column2 datatype,__ __ …__ __);__ - -Example:CREATE TABLE "users" ( - -"id" SERIAL PRIMARY KEY, - -"full\_name" varchar NOT NULL, - -"email" varchar UNIQUE NOT NULL, +Example: +``` +CREATE TABLE "users" ( + "id" SERIAL PRIMARY KEY, + "full_name" varchar NOT NULL, + "email" varchar UNIQUE NOT NULL, + "created_at" timestamp NOT NULL +); +``` -"created\_at" timestamp NOT NULL -); # DROP -In order to remove tables or databases, we use a DROP statement - -__DROP TABLE table\_name;__ __DROP DATABASE database\_name;__ +In order to remove tables or databases, we use a DROP statement. Syntax: +``` +DROP TABLE table_name; +DROP DATABASE database_name; +``` These statements do not ask for confirmation and there is no undo feature. Take care when using a drop statement. + + # NoSQL * In addition to SQL databases, there are also NoSQL databases @@ -407,10 +456,21 @@ These statements do not ask for confirmation and there is no undo feature. Take --- Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. MongoDB is consistently ranked as the world's most popular NoSQL database according to DB-engines and is an example of a document database. For more on document databases, visit What is a Document Database?. + +--- + Key-value databases are a simpler type of database where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don't need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Redis and DynanoDB are popular key-value databases. + +--- + Wide-column stores store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores. + +--- + Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Neo4j and JanusGraph are examples of graph databases. + + # Object-Relational Mappers * ORMs allow developers to manipulate databases with code instead of SQL queries @@ -419,54 +479,48 @@ Graph databases store data in nodes and edges. Nodes typically store information * Hibernate (Java) * EFCore (.NET) * Sequelize (Node.js) -* TypeORM (TypeScript) -* We recommend using TypeORM instead of writing SQL yourself. You can find a good step-by-step guide within [their own documentation](https://typeorm.io/#installation) . - -# PostgreSQL with Node - -# Local environment +* TypeORM (TypeScript) - [documentation](https://typeorm.io/#installation) -Using our previously created, Dockerized Postgres instance, we'll create a Node.Js application to connect to our database. -If you have deleted your postgres container, a new one can be created with a following command: -__docker run --name my-postgres -e POSTGRES\_PASSWORD=mypassword -e POSTGRES\_USER=pguser -e POSTGRES\_DB=mydb -p 5432:5432 -d postgres:15.2__ - -With a previously created container, this command will start the container: - -__docker start my-postgres__ +# PostgreSQL with Node -# Preparing our Node application -Initialize the application with: __npm init -y__ -Install [express](https://www.npmjs.com/package/express) and [PostgreSQL client for Node.JS](https://www.npmjs.com/package/pg) __ npm install express pg__ +# Local environment -Install [dotenv](https://www.npmjs.com/package/dotenv) and nodemon as a development dependencies __npm install –-save-dev dotenv nodemon__ +Using our previously created, Dockerized Postgres instance, we'll create a Node application to connect to our database. -If using TypeScript, install types for express and pg, plus ts-node for running nodemon. __npm install --save-dev @types/express @types/pg ts-node__ +If you have deleted your postgres container, just create a new one with the same command. -# Dotenv +If your container has shut down (after a reboot for example), you can start it with +``` +docker start my-postgres +``` -Example of .env file: -In development, it's customary to store ports, database login info, etc. as _environment variables_ in a separate .env file -Previously installed package _dotenv_ is used to load variables from .env to the main program +# Preparing our Node application -dotenv can be imported with +Initialize a new TypeScript application. -PORT=3000 +Install [express](https://www.npmjs.com/package/express) and [PostgreSQL client for Node.JS](https://www.npmjs.com/package/pg), and their respective TypeScript types -PG\_HOST=localhost +Install [dotenv](https://www.npmjs.com/package/dotenv), nodemon and ts-node development dependencies -PG\_PORT=5432 -PG\_USERNAME=pguser -PG\_PASSWORD=mypassword +# Dotenv -PG\_DATABASE=postgres +Example of `.env` file: +``` +PORT=3000 +PG_HOST=localhost +PG_PORT=5432 +PG_USERNAME=pguser +PG_PASSWORD=mypassword +PG_DATABASE=postgres +``` These values must match the values declared when running the PostgreSQL container. @@ -474,179 +528,151 @@ The database must exist as well. Note that the example uses the default "postgres" database, but you can use any database you want. -import dotenv from 'dotenv' -dotenv.config() # Dotenv (continued) +```json { - -"name": "products\_api", - -"version": "1.0.0", - -"scripts": { - -"dev": "nodemon -r dotenv/config ./src/index.ts" - -}, - -"dependencies": { - -"express": "^4.18.2", - -"pg": "^8.9.0" - -}, - -"devDependencies": { - -"@types/express": "^4.17.17", - -"@types/pg": "^8.6.6", - -"dotenv": "^16.0.3", - -"nodemon": "^2.0.20", - -"ts-node": "^10.9.1", - -"typescript": "^4.9.5" - + "name": "products_api", + "version": "1.0.0", + "scripts": { + "dev": "nodemon -r dotenv/config ./src/index.ts" + }, + "dependencies": { + "express": "^4.18.2", + "pg": "^8.9.0" + }, + "devDependencies": { + "@types/express": "^4.17.17", + "@types/pg": "^8.6.6", + "dotenv": "^16.0.3", + "nodemon": "^2.0.20", + "ts-node": "^10.9.1", + "typescript": "^4.9.5" + } } +``` -} +--- Dotenv is usually only used in development, not in production. In professional setting the dotenv config is often preloaded in the development startup script -You can require dotenv when running _npm run dev_ +You can require dotenv when running npm run dev_ -r is short for --require + + # Dotenv and Git -* .env files usually contain sensitive data that we do __not__ want to store in Git repositories. -* Thus, usually __.env__ file is excluded from the Git repository - * Add __.env__ to __.gitignore__ - * If you have auto-generated .gitignore with npx gitignore Node, environment files are excluded automatically +* .env files usually contain sensitive data that we do _not_ want to store in Git repositories. +* Thus, usually _.env_ file is excluded from the Git repository + * Add _.env_ to _.gitignore + * If you have auto-generated .gitignore with `npx gitignore Node`, environment files are excluded automatically ![](imgs/3-databases-with-docker_16.png) + + # Connecting to PostgreSQL Our database file contains functions and configuration for initializing the Postgres pool, creating tables and running queries. At the moment, we have only one query. It is a single-use query that creates a products table to the database if such table does not yet exist. +```ts // db.ts - import pg from "pg"; - -const { PG\_HOST, PG\_PORT, PG\_USERNAME, PG\_PASSWORD, PG\_DATABASE } = process.env; +const { PG_HOST, PG_PORT, PG_USERNAME, PG_PASSWORD, PG_DATABASE } = process.env; const pool = new pg.Pool({ - -host: PG\_HOST, - -port: Number(PG\_PORT), - -user: PG\_USERNAME, - -password: String(PG\_PASSWORD), - -database: PG\_DATABASE, - + host: PG_HOST, + port: Number(PG_PORT), + user: PG_USERNAME, + password: PG_PASSWORD, + database: PG_DATABASE, }); const executeQuery = async (query: string, parameters?: Array) => { - -const client = await pool.connect(); - -try { - -const result = await client.query(query, parameters); - -return result; - -} catch (error: any) { - -console.error(error.stack); - -error.name = "dbError"; - -throw error; - -} finally { - -client.release(); - -} - + const client = await pool.connect(); + try { + const result = await client.query(query, parameters); + return result; + } catch (error: any) { + console.error(error.stack); + error.name = "dbError"; + throw error; + } finally { + client.release(); + } }; export const createProductsTable = async () => { - -const query = \` - -CREATE TABLE IF NOT EXISTS "products" ( - -"id" SERIAL PRIMARY KEY, - -"name" VARCHAR(100) NOT NULL, - -"price" REAL NOT NULL - -)\`; - -await executeQuery(query); - -console.log("Products table initialized"); - + const query = `CREATE TABLE IF NOT EXISTS "products" ( + "id" SERIAL PRIMARY KEY, + "name" VARCHAR(100) NOT NULL, + "price" REAL NOT NULL + )`; + await executeQuery(query); + console.log("Products table initialized"); }; +``` -At the moment our __index.js__ does nothing but creates a single table for our database and launches express server. +--- + +At the moment our `index.ts` does nothing but creates a single table for our database and launches express server. It doesn't even have any endpoints, so it's not much of a server yet. -import express from "express"; +```ts +import express from "express"; import { createProductsTable } from "./db"; const server = express(); - createProductsTable(); const { PORT } = process.env; server.listen(PORT, () => { + console.log("Products API listening to port", PORT); +}); +``` -console.log("Products API listening to port", PORT); -}); # Launching the application -Let's use our predefined _npm run dev_ +Let's use our predefined `npm run dev` ![](imgs/3-databases-with-docker_17.png) +--- + …And check with psql that our application succeeds in connecting to the database and creating the table. Epic success! ![](imgs/3-databases-with-docker_18.png) + + # Exercise 6: Node & PostgreSQL + Following the lecture example, create an Express server that connects to your local PostgreSQL instance. The database information should be stored in environment variables. When the server starts, it should create a product table with three columns: id (serial, primary key), name (varchar) and price (real). + + # Creating Queries * Next, we will create an actual CRUD API for communicating with the database. * For that we need endpoints for creating, reading, updating and deleting products. * All of these need their own queries. + + # Using queries * We'll use the pre-made executeQuery() function for querying the database from a few slides back @@ -656,13 +682,17 @@ Following the lecture example, create an Express server that connects to your lo * When supplying parameters, the query string should have placeholders $1, $2, etc * These will be replaced with the contents of the parameters array. -# Parameterized queries example -When running executeQuery(query, parameters) with above values defined, the query would be parsed as + +# Parameterized queries example ![](imgs/3-databases-with-docker_19.png) +When running executeQuery(query, parameters) with above values defined, the query would be parsed as + +``` SELECT * FROM cats WHERE color = 'yellow' and age > 10; +``` # Why not just use String templating? @@ -674,37 +704,57 @@ _NEVER DO THIS!!!_ ![](imgs/3-databases-with-docker_21.png) + + # Creating queries -We will create a Data Access Object, __dao.js__ that will handle interacting with our database. +We will create a Data Access Object, `dao.ts` that will handle interacting with our database. The idea is that we want to just tell our DAO what we want done (e.g. "add this customer to the database") and the DAO will handle the details of that action. The DAO will also return possible additional information that was created during the action. -Our _insertProduct _ function +--- + +Our insertProduct function generates a new, unique ID for the product using [uuid](https://www.npmjs.com/package/uuid) constructs a parameters array containing said id, the name of the product and the price of the product -executes the query using _db.executeQuery_ method +executes the query using db.executeQuery method returns the database result object ![](imgs/3-databases-with-docker_22.png) +--- + +
+
+ ![](imgs/3-databases-with-docker_23.png) +
+
+ ![](imgs/3-databases-with-docker_24.png) +
+
+ +--- + The rest of the DAO operations work in similar fashion. The router that declares the endpoints, uses the DAO to interact with the database. + + # Testing the API -![](imgs/3-databases-with-docker_25.png) +
+
Now we can use Insomnia to verify that all the endpoints work as expected. @@ -712,19 +762,29 @@ We can also use psql to observe the changes in the database ![](imgs/3-databases-with-docker_26.png) -# Exercise 7: Creating Queries +
+
+ +![](imgs/3-databases-with-docker_25.png) + +
+
+ -Continue following the lecture example. Create a router and a database access object to handle -Creating a product -Reading a product -Updating a product +# Exercise 7: Creating Queries + + +Continue following the lecture example. Create a router and a database access object to handle +- Creating a product +- Reading a product +- Updating a product +- Deleting a product +- Listing all products -Deleting a product -Listing all products # Dockerized PostgreSQL App @@ -732,57 +792,50 @@ Listing all products Docker has two kinds of environment variables: run-time and build-time. -In this scenario we want to set our environment variables at __build time__ . This means that _the Docker image will contain all the environment variable information_ , including sensitive things like passwords. This might be an issue in some scenarios. In those cases the environment variables need to be set at __run time__ . +In this scenario we want to set our environment variables at _build time_. This means that the Docker image will contain all the environment variable information, including sensitive things like passwords. This might be an issue in some scenarios. In those cases the environment variables need to be set at _run time_. In the Dockerfile we set the build-time values by setting ARG parameters. Then we use these values to set the run-time environment variables by setting ENV parameters. More information: [https://vsupalov.com/docker-arg-env-variable-guide/](https://vsupalov.com/docker-arg-env-variable-guide/) -When the ARGs and ENVs have been set in the Dockerfile, we provide the ARG values when building the Docker image by using __--build-arg =__ flags. To build an image with these parameters, we'd use something like - -__docker build __ - -__--build-arg PORT=3000 __ - -__--build-arg PG\_HOST=https://my.postgres.server__ - -__--build-arg PG\_PORT=5432 __ - -__--build-arg PG\_USERNAME=pguser __ +--- -__--build-arg PG\_PASSWORD=pgpass __ +When the ARGs and ENVs have been set in the Dockerfile, we provide the ARG values when building the Docker image by using_--build-arg =_ flags. To build an image with these parameters, we'd use something like +``` +docker build +--build-arg PORT=3000 +--build-arg PG_HOST=https://my.postgres.server +--build-arg PG_PORT=5432 +--build-arg PG_USERNAME=pguser +--build-arg PG_PASSWORD=pgpass +--build-arg PG_DATABASE=my-database +-t my-app . +``` -__--build-arg PG\_DATABASE=my-database __ +--- -__-t my-app .__ +And include the build-arg parameters in our Dockerfile with them mapped to environment variables. +``` ARG PORT - -ARG PG\_HOST - -ARG PG\_PORT - -ARG PG\_USERNAME - -ARG PG\_PASSWORD - -ARG PG\_DATABASE +ARG PG_HOST +ARG PG_PORT +ARG PG_USERNAME +ARG PG_PASSWORD +ARG PG_DATABASE ENV PORT=${PORT} - -ENV PG\_HOST=${PG\_HOST} - -ENV PG\_PORT=${PG\_PORT} - -ENV PG\_USERNAME=${PG\_USERNAME} - -ENV PG\_PASSWORD=${PG\_PASSWORD} - -ENV PG\_DATABASE=${PG\_DATABASE} +ENV PG_HOST=${PG_HOST} +ENV PG_PORT=${PG_PORT} +ENV PG_USERNAME=${PG_USERNAME} +ENV PG_PASSWORD=${PG_PASSWORD} +ENV PG_DATABASE=${PG_DATABASE} +``` [Docker documentation here!](https://www.docker.com/blog/how-to-use-the-postgres-docker-official-image/) # Exercise 8: Dockerized PG App + Dockerize the application you have built. Build the docker image, run the app and test that it works using insomnia/postman. diff --git a/imgs/1-express-basics_1.png b/imgs/1-express-basics_1.png deleted file mode 100644 index e657163..0000000 Binary files a/imgs/1-express-basics_1.png and /dev/null differ diff --git a/imgs/1-express-basics_2.png b/imgs/1-express-basics_2.png deleted file mode 100644 index 62d291d..0000000 Binary files a/imgs/1-express-basics_2.png and /dev/null differ diff --git a/imgs/3-databases-with-docker_14.png b/imgs/3-databases-with-docker_14.png index 35a1812..4df2e42 100644 Binary files a/imgs/3-databases-with-docker_14.png and b/imgs/3-databases-with-docker_14.png differ diff --git a/imgs/3-databases-with-docker_23.png b/imgs/3-databases-with-docker_23.png index 53666b9..5ae7089 100644 Binary files a/imgs/3-databases-with-docker_23.png and b/imgs/3-databases-with-docker_23.png differ diff --git a/imgs/3-databases-with-docker_24.png b/imgs/3-databases-with-docker_24.png index f35cdb2..ef6f060 100644 Binary files a/imgs/3-databases-with-docker_24.png and b/imgs/3-databases-with-docker_24.png differ