diff --git a/1-express-basics.md b/1-express-basics.md new file mode 100644 index 0000000..4ab6024 --- /dev/null +++ b/1-express-basics.md @@ -0,0 +1,612 @@ +# Express Basics + +# JS HTTP server + +# A simple HTTP server in Node + +__A simple server in three steps:__ + +__include "http" library__ + +__create server responses__ + +__define the port that you want to listen to.__ + +__Note: The \_ before a variable name is used to indicate that the variable isn't used. This is to avoid the VSCode warning about unused variable__ + +import http from 'http' + +const server = http.createServer((\_req, res) => { + +res.write('Some cool response!') + +res.end() + +}) + +const port = 3000 + +server.listen(port) + +console.log('HTTP Server listening port', port) + +For this code to work, remember "type": "module" in your package.json. Also, you may need to change your port number to something other than 3000. + +# PORT numbers + +__The port number (16 bit unsigned integer) defines a port through which the communication to your application happens. Ports allow multiple applications to be located on the same host (IP).__ + +__The client connects to this port in some server (defined by an IP address), so the client and the server are able to communicate.__ + +__Computers have several different ports at their disposal __ _but_ __ some of them are being used and your application might not be able to assign those ports to itself.__ + +__Ports from 0 to 1023 are the system ports. There are restrictions in manually assigning applications to use them.__ + +__Generally speaking, ports from 1024 to 65535 are available for user to assign applications to use them.__ + +![](imgs/1-express-basics_0.png) + +# Request and response + +__The request object is an object that is sent__ __ from the client to the server.__ + +__The response object is an object that is sent__ __ from the server to the client__ __. When you modify the response object, you're essentially modifying how the server responds to the client.__ + +_const_ server = http.createServer(( _req_ , _res_ ) _=>_ { + +res.write("Some cool response!"); + +res.end(); + +}); + +# Exercise 1: Simple Request & Response + +Create a basic HTTP-server application. The application should respond to GET requests by returning a response with the following content: "Hello world!". + +Test your application either with _Postman,_ or by navigating your browser to http://localhost:< _your\_portnumber_ > + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Express Server + +# Express + +__Express is the most popular server-library for Node__ + +__Setting up a server with node-http is already fairly simple, but Express tries to make this even simpler__ + +__In some cases you can manage without Express, but for larger applications it's recommended__ + +__Install Express by using__ + +__npm install express__ + +# A side note about dependencies + +* __By using __ __npm install __ __, the package will become dependency for the application.__ +* __By using __ __npm install --save-dev __ __, the package will become __ __development dependency__ __ for the application.__ + * __This means the package won't be included in the built application.__ +* __A shortcut for __ __--__ __save-dev__ __ is __ __-D__ +* __Dependencies are needed to run the application, dev-dependencies are needed for development.__ +* __Additional info: __ [https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/](https://www.geeksforgeeks.org/difference-between-dependencies-devdependencies-and-peerdependencies/) + +# Getting started with Express + +__Create a very simple express project __ + +import express from 'express' + +const server = express() + +server.listen(3000, () => { + +console.log('Listening to port 3000') + +}) + +__Then start the server and connect to __ [http://localhost:3000](http://localhost:3000) __ with your browser.__ __ (See instructions on the next slide)_ + +# Starting the server + +__To make using the import statement possible, we need to edit the __ __package.json.__ + +__Add a new script "start" with the value "node index.js" under "scripts" object in the package. json__ + +__(you can name index.js to whatever file your server code is in)_ + +__Add "type": "module" property__ + +__Now you can run the server with __ _npm start_ + +# Express - Requests + +__The server does not yet __ _do_ __ anything except listens. That is because we have not defined how it should deal with incoming __ _requests._ + +__With Express you can easily process different kind of requests, e.g.,__ + +__server.get(), server.post(), server.delete() etc!__ + +__An example of a simple GET response below.__ + +server.get("/", (request, response) => { + +response.send("Just saying hello!") + +}) + +--- + +Käy läpi tämä esimerkki, ja anna sitten tehtäväksi ensinnäkin muuttaa portti ja palautusviesti muotoon "Hello world!". Tällä tavalla kaikkien pitää ymmärtää miten tämä toimii. +Anna heidän pähkäillä miten voi tehdä päätepisteen, johon pääsee näin localhost:5000/secondPage. Tämän jälkeen käy läpi yhdessä tämä heidän kanssa. + +# Express - Endpoints + +__To create a new __ _endpoint _ __we define what __ _kind_ __ of request we're handling, then what the __ _route_ __ to the endpoint is and finally what the endpoint __ _does_ __.__ + +__In the below example we are creating an endpoint that handles a __ __GET__ __ request to route __ __/foobar__ __, that sends a response containing the string 'OK'.__ + +server.get('/foobar', (req, res) => { + +res.send('OK') + +}) + +--- + +# Nodemon + +__Reminder__ __: If you are not using Nodemon in development, you are making your life unnecessarily difficult. Use Nodemon. __ + +__Instructions are __ __below as a refresher.__ + +# Nodemon with JavaScript + +One of the handiest packages out there is nodemon, which monitors your node program. Whenever you save your files, it reloads the program, showing you immediately the results of your changes. + +Nodemon is installed as a dev dependency __npm install --save-dev nodemon__ + +To use nodemon, you should have a development script "dev": "nodemon ./index.js" + +# Nodemon with TypeScript + +To get Nodemon working with TypeScript we need additional package to run our TS code without waiting for it to compile. __npm install --save-dev nodemon ts-node__ + +After installing, nodemon works directly with .ts files. "dev": "nodemon ./index.ts" + +# Exercise 2: Simple Express Server + +Create a basic Express application. The application should respond to GET requests by returns a response with the following content: "Hello world!". + +Extra: Add a new endpoint to your application. So, in addition to accessing _http://localhost:3000_ , try to send something back from _http://localhost:3000/endpoint2_ . + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Request params & query + +__If request params are defined, they're something user __ __must__ __ send to the server.__ + +__https://localhost:5000/ThisIsParam__ + +__Request query is __ __optional__ __ information that the user can send to the server.__ + +__https://localhost:5000?ThisIsQuery=123__ + +__Chaining query parameters:__ + +__https://localhost:5000?first=123&second=456__ + +# Request params & request query + +app.get("/:name/:surname", ( _request_ , _response_ ) _=>_ { + +_console_ .log("Params:"); + +_console_ .log(request.params); + +_console_ .log("Query:"); + +_console_ .log(request.query); + +response.send("Hello " + request.params.name); + +}); + +app.listen(5000); + +Try connecting via browser! + +_localhost:5000_ _/John/Doe_ + +_localhost:5000_ _/John/Doe_ _?query1=123_ + +--- + +Johdatteleva esimerkki vielä ennen seuraavaa tehtävää. + +# Request query & param types + +__Request parameters and query parameters are always of type __ _string_ __. If a number is passed, it will be a string that contains a number. __ + +__The example on right will __ __always __ __return 404, since the strict equality __ item.id === id + +__will never be true. This is because the__ __ item.id __ __is of type __ _number _ __and__ __ id __ __being of type __ _string_ __.__ + +import express from 'express' + +const server = express() + +const data = [ + +{ id: 1, name: 'John' }, + +{ id: 2, name: 'Jane' } + +] + +server.get('/:id', (req, res) => { + +const id = req.params.id + +const info = data.find(item => item.id === id) + +if (info === undefined) { + +return res.status(404).send() + +} + +res.send(info) + +}) + +server.listen(3000) + +# Exercise 3: Counter Server + +Create an API that consists of only one endpoint: + +/counter + +Whenever you enter this endpoint from the browser, it + +should respond with a JSON object with information on how many times the endpoint has been accessed. + +Extra: Make it possible to set the counter to whichever + +integer with a query parameter /counter?number=5 + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Exercise 4: Advanced Counter Server + +Expand the API from the previous assignment by accepting a name through the counter endpoint: /counter/:name + +When entering this endpoint, the server should return the count of how many times this named endpoint has been visited. + +For example + +Aaron enters /counter/Aaron 🠖 "Aaron was here 1 times" + +Aaron enters /counter/Aaron 🠖 "Aaron was here 2 times" + +Beatrice enters /counter/Beatrice 🠖 "Beatrice was here 1 times" + +Aaron enters /counter/Aaron 🠖 "Aaron was here 3 times" + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# "url" library + +__With the "url" library, you can parse url fields (i.e., requests) the client sends to your server.__ + +_import_ url from 'url'; + +// URL module usage example + +_const_ adr = ' [http://localhost:5000/default.html?year=2017&month=february](http://localhost:5000/default.html?year=2017&month=february) '; // Define the address + +_const_ q = url.parse(adr, true); // Parse through the address with url.parse() function. + +_console_ .log(q.host); // returns 'localhost:5000' + +_console_ .log(q.pathname); // returns '/default.html' + +_console_ .log(q.search); // returns '?year=2017&month=february' + +_const_ qdata = q.query; // returns an object: { year: 2017, month: 'february' } + +_console_ .log(qdata.month); // returns 'february + +# Express Middleware + +# Middlewares + +__Generally speaking, middleware is an application that does something in between some other applications. __ + +__There are tons of middleware libraries available for Express.__ + +__What this means in Express: with a middleware, you can manipulate the __ __the client request and server response __ __in the client-server communication.__ + +__Middleware__ __ functions are functions that have access to the __ __request__ __ object (__ _req_ __), the __ __response__ __ object (__ _res_ __), and the __ _next_ __ function in the application's request-response cycle. When the __ _next_ __ function is invoked, the next middleware function is executed.__ + +__Middleware can be defined as such__ + +const authCheck = (req, res, next) => { + +if (req.headers.Authentication === undefined) { // Read the request object + +res.status(401).send('Missing Authentication Header') // Modify the response object + +} else { + +next() // Execute the next middleware + +} + +} + +__In Express, .use() method initializes a middleware (argument 2) in a given path (argument 1).__ + +__In the middleware, the next() function will jump to the next middleware or endpoint.__ + +server.use('/customers', authCheck) + +server.get('/customers', (req, res) => { + +// we know that Authentication header is present + +... + +}) + +# Exercise 5: Logger Middleware + +Create a new project called Student Registry. We will be developing this program in stages during this lecture. For now it should have a single GET endpoint /students that returns an empty list. + +Create a logger middleware function that is used in all the project's endpoints. The middleware should log + +the time the request was made + +the method of the request + +the url of the endpoint + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Middleware - Unknown endpoint (AKA 404) + +As you might have noticed, all websites have some kind of a default functionality for unknown endpoints. + +This can be handled with a middleware. + +Unknown endpoint has to be taken into use after the actual endpoints! + +![](imgs/1-express-basics_1.png) + +# Middleware - Error handler + +An error handler is often needed to handle different kind of user or application errors. + +We can use middleware to enable error handling for our application. + +Has to be the last middleware to be taken into use! + +![](imgs/1-express-basics_2.png) + +# Middleware - Req & Res + +__A n__ __eed to manually alter req & res is quite common. To avoid bloated code, the handling can be easily done with proper middlewares.__ + +__Setting a header in middleware:__ + +app.use((req, res, next) => { + +res.setHeader('Content-Type', 'image/png') + +next(); + +}); + +}) + +__Getting a header and storing its value. If the header is not present, return 403 with an 'Unauthorized' message:__ + +app.use((req, res, next) => { + +const someIdNeededInApplication = req.header('Some-Custom-Header') + +if (!someIdNeededInApplication) { + +res.status(403).send('Unauthorized') + +} + +someIdFunctionality(someIdNeededInApplication) + +next() + +}) + +# Exercise 6: 404 Not Found + +Add a middleware function that sends a response with status code 404 and an appropriate error message, if user tries to use an endpoint that does not exist. + +If you have not already done so, move all your middleware to a separate file, to keep your program clean and readable. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Express.js: Creating a REST API + +# REST HTTP requests + +__We have gone through what a GET request is. Let's briefly go through what other types of requests are necessary for you.__ + +__GET /products/:id → Returns a single product identified by its id__ + +__GET /products →Returns all the products__ + +__POST /products → Creates a new product__ + +__DELETE /products/:id → Removes a single product identified by its id__ + +__PUT /products/:id → Updates an existing product__ + +Basic RESTFUL functionality is often referenced as a CRUD (Create, Read, Update, Delete). + +# Handling request content + +__To properly handle some requests, e.g., POST requests, in Express, a body-parser middleware must be taken into use. It's as simple as:__ + +const server = express() + +server.use(express.json()) + +You might encounter a separate body-parser being used. This is no longer necessary, as Express 4.16+ includes a body parser middleware built-in. + +express.urlencoded() __allows us to parse url-encoded forms by attaching the data to the request body.__ + +const app = express() + +app.use(express.json()) + +app.use(express.urlencoded({extended: false})) + +__The "extended" parameter's false value only says that we are not dealing with any complicated objects (objects with sub objects, etc)_ + +# Basic POST request + +const app = express() + +app.use(express.json()) + +app.use(express.urlencoded({extended: false})) + +app.get("/", (req, res) => { + +console.log("GET request init!"); + +res.sendFile(\_\_dirname + "/index.html"); + +}) + +app.post("/", (req, res) => { + +console.log("POST request init!"); + +console.log(req.body); + +res.redirect("/"); + +}) + +# Request Body + +__GET__ __ methods should not in general have body parameters. The GET method should only be used to get data, the request should not convey data. __ + +__Despite this, most modern implementations can send and receive request bodies in all request types.__ + +__The requests that should not include body are __ __CONNECT__ __, __ __GET__ __, __ __HEAD__ __, __ __OPTIONS__ __, and __ __TRACE__ __. __ + +# Request Body Types + +__Request body is parsed by the express body parser, and therefore it can contain any basic JavaScript data types. Notice that this differs from the request parameters and queries.__ + +__This also means that you do not know what the data type of any given body parameter will be. It might be necessary to check this in some cases.__ + +# TypeScript Body Casting + +__If you want to enforce type safety, the solution is to validate the user input before using it.__ + +__This is usually a good idea even if not using TypeScript, since it is important to know what type all the variables are in your code.__ + +import express, { NextFunction, Request, Response} from 'express' + +const server = express() + +server.use(express.json()) + +interface Body { + +name: string + +age: number + +} + +const validate = (req: Request, res: Response, next: NextFunction ) => { + +const { age, name } = req.body + +if (typeof(age) !== 'number' || typeof(name) !== 'string') { + +return res.status(400).send('Missing or invalid parameters') + +} + +next() + +} + +server.post('/', validate, (req: Request, res: Response) => { + +const body: Body = req.body + +console.log(body) + +res.send(body.name.toUpperCase()) + +}) + +server.listen(3000) + +# Exercise 7: Body Logging + +Enable body parsing in your application. + +Modify your logger middleware so that in addition to existing functionality, it also logs the request body if it exists. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Exercise 8: POST Requests + +Add two more endpoints to your app: + +POST /student should expect the request body to include student _id_ , _name _ and _email_ . If some of these parameters are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should store the student information in an array called _students_ . The endpoint should return an empty response with status code 201. + +GET /student/:id should return the information of a single student, identified by the _id_ request parameter. The response should be in JSON format. If there is no such student, the endpoint should return 404. + +Modify the GET /students endpoint to return the list of all student _ids_ , without names or emails. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Exercise 9: PUT and DELETE + +Add two more endpoints to your app: + +PUT /student/:id should expect the request body to include student _name_ or _email_ . If both are missing, the endpoint should return a response with status code 400 and an error message. The endpoint should update an existing student identified by the request parameter _id_ . + +DELETE /student/:id should remove a student identified by the request parameter _id_ . + +Both endpoints should return 404 if there is no student matching the request parameter _id_ . On success both endpoint should return an empty response with status code 204. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + diff --git a/2-express-router-authentication-and-testing.md b/2-express-router-authentication-and-testing.md new file mode 100644 index 0000000..854a726 --- /dev/null +++ b/2-express-router-authentication-and-testing.md @@ -0,0 +1,704 @@ +# Express Router, Authentication & Testing + +# Static Content + +We can make a web application out of our REST-API. First you need to create a folder at the root of your application, and name it for example "public". When you add an index.html file there, with the example below it will served when you send a GET request to our server. + +import express from 'express' + +const server = express() + +server.use(express.static('public')) + +server.get('/route', (req, res) => { + +res.send('OK') + +}) + +server.listen(PORT, () => console.log('Listening to port', PORT)) + +# Exercise 1: Static + +Let's continue improving the Students API we created in the last lecture. + +Add a static info page to your API. + +The page should be reachable from the root path / and it should include some information about all the endpoints in the API. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Router + +# Express Router + +Until now we have been building our routes to index.js. This is fine for VERY small applications, but we'd like to keep our code readable even if our application grows in size. + +__Routers__ are Express classes that are dedicated to route handling. We can use them to group routes to meaningful units that share functionality. + +Routers do not require any installation since they are native express classes. + +The following application, while still being extremely simple, already has a problem: index.js file is doing several things. + +It handles two kinds of routes and starts the server. Usually it also adds middleware and some configuration. + +Most of the time it is best to separate the route handling to dedicated routers. + +import express from 'express' + +import authors from './approvedAuthors.js' + +import articles from './articlesDao.js' + +const server = express() + +server.get('/authors', (req, res) => { + +res.send(authors) + +}) + +server.get('/authors/:id', (req, res) => { + +const author = authors.find(author => author.id = req.params.id) + +res.send(author) + +}) + +server.get('/articles', (req, res) => { + +res.send(articles.getAll()) + +}) + +server.post('/article', (req, res) => { + +const { author, article } = req.body + +const newArticle = articles.add({ author, article}) + +res.send(newArticle) + +}) + +server.listen(3000, () => console.log('Listening to port 3000')) + +We create two routers, one for each logical unit: _authorsRouter.js_ and _articlesRouter.js_ + +The only difference in the route definition is in the route paths. + +import express from 'express' + +import authors from './approvedAuthors.js' + +const router = express.Router() + +router.get('/', (req, res) => { + +res.send(authors) + +}) + +router.get('/:id', (req, res) => { + +const author = authors.find(author => author.id = req.params.id) + +res.send(author) + +}) + +export default server + +import express from 'express' + +import articles from './articlesDao.js' + +const router = express.Router() + +router.get('/', (req, res) => { + +res.send(articles.getAll()) + +}) + +router.post('/', (req, res) => { + +const { author, article } = req.body + +const newArticle = articles.add({ author, article}) + +res.send(newArticle) + +}) + +export default router + +Then we define _index.js_ to use those routers like any middleware. + +Each route gets a path designation that will be used as _prefix_ in the router. + +import express from 'express' + +import authorsRouter from './authorsRouter.js' + +import articlesRouter from './articlesRouter.js' + +const server = express() + +server.use('/authors', authorsRouter) + +server.use('/articles', articlesRouter) + +server.listen(3000, () => { + +console.log('Listening to port 3000') + +}) + +# Exercise 2: Students Router + +Let's continue improving the Students API we created on the last lecture. + +Add a _studentRouter.js_ file that exports a router with all the five routes the app currently has. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Authentication & authorization + +# Token based login + +* Currently our applications are accessible for anyone, which often is not a desired state of things. +* If you don't restrict access to your endpoints, something like [this](https://mastodon.gamedev.place/@badlogic/111246798083590676) [might happen](https://firesky.tv/) !!! +* Authentication & Authorization help here + * Authentication: Verifying that you are indeed you + * Authorization: Determining access rights based on your privileges +* We shall first look at password authorization, and later token-based authentication. + +# Storing passwords + +# Password Hashing + +Passwords should never be stored as plain text in to a database. Instead a password _hash_ should be stored instead. + +_Hash function_ maps data of any size (in our case a password) to a _hash _ of fixed length. + +Hash functions are _one-way functions_ , which means they can not be reversed. This means that you can not deduce the original password from the hash. + +We can use (a variant of) the original hash function to compare a password and the hash and deduce if the hash _matches_ the password or not. + +# Salting + +Bare hash functions produce identical results from identical inputs. + +To add complexity, a some random data is added to the input. This data is called _salt_ and the process of adding the salt is called _salting_ . + +This means two identical passwords produce different hashes, since both have unique salt. + +Salt can be stored in the hash or it can be stored separately. The hash function needs to know the salt in order to verify passwords against the hash. + +# Hashing a password + +We will use Argon library to do our hashing and comparing. Install Argon with __npm install argon2__ + +Using Argon to hash passwords is very straightforward. Salting is done automatically (although you can configure it in more detail if you wish to do so) and the salt is added to the hash. + +import argon2 from 'argon2' + +const password = process.argv[2] + +argon2.hash(password) + +.then(result => console.log(result)) + +You can of course also use _async/await_ syntax instead of _.then() + +# Comparing hashes + +Notice if you run the example from the previous slide multiple times, you always get a different result. In order to verify that a particular password matches the hash, Argon's _verify_ function is used. + +Verify returns a promise that resolves to either _true,_ if the password matches the hash, or _false_ if it doesn't. + +import argon2 from 'argon2' + +const hash = '$argon2id$v=19$m=4096,t=3,p=1$+OM8yk1Kd3M/709t0Hy1vg$E3aii3UmOQOp6jFJVd9xDpakMOF1O6TDd1gS1i/98HE' + +const password = process.argv[2] + +argon2.verify(hash, password) + +.then(passwordMatchesHash => console.log(passwordMatchesHash ? 'Correct' : 'Incorrect')) + +# Exercise 3: Registration + +Create and attach a new router, _userRouter.js_ that has a single endpoint: POST /register . + +The endpoint should + +expect a request body with two parameters, _username _ and _password_ . + +create a hash from the password using the Argon2 library. + +store the username and the hash in an in-memory storage (e.g. let users = [ ... ]) and log the result to the console. + +return a response with status code 201 (Created) on success + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Exercise 4: Login + +Add another endpoint to the user router: POST /login that also expects two request body parameters, _username _ and _password_ . + +The endpoint should + +check that the user exists in the in-memory storage. + +If the user exists, it should use the Argon2 library to verify that the given password matches the stored hash. + +If they match, it should return a response with status code 204 (No Content). + +If the user does not exist, or the password doesn't match the hash, it should return a response with status code 401 (Unauthorized). + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Environment variables + +# dotenv Library + +With dotenv library, we can create environment variables that can be loaded into process.env. We can access the environment variables in our application with process.env.{variable\_name}. + +To use dotenv + +Install dotenv package with __npm install dotenv__ + +Create a __.env __ file in to the root folder of your application where the values are declared + +Import the dotenv configuration in the projectimport 'dotenv/config' + +Access the variables defined in __.env__ fileconst PORT = process.env.PORT + +# .env File + +The custom environment variables are declared in the .env file. The file must be located in the folder from where the program is ran. This is usually the root folder. + +The variables are declared as in the example below, separated by line changes. All values are _strings _ (even if JS parses some automatically). + +Notice there are no spaces around the equals (=) sign, nor any kind of quotation marks. + +API\_URL=https://cataas.com + +PORT=3000 + +# dotenv in Development + +* In many cases the environment variables are configured on the platform from where the program is ran. In those cases you might want to run dotenv only in development mode. +* Install as dev dependency __npm install --save-dev dotenv__ +* Create __.env__ file as usual. + * Optionally include ENVIRONMENT=development variable +* Run your application from config.json script that preloads dotenv"dev": "nodemon -r dotenv/config index.js" +* Access environment variables as usualconst PORT = process.env.PORT + +# Exercise 5: Environmental Login + +Add an admin login to the Students API. We want to store admin credentials in environment variables. + +Add dotenv library as dependency. + +Add a .env file that defines an admin username and an admin password hash. + +Add an endpoint POST /admin that also expects two request body parameters, _username _ and _password_ . The endpoint should + +check that the username matches the one defined in the .env file + +check that the password matches the hash defined in the .env file + +if they match, it should return a response with status code204 (No Content) + +if they do not match, it should return a response with status code 401 (Unauthorized) + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Token-based authentication + +# JSON Web Token + +After verifying the identity of our user, we create a _token_ that we send to the client. The client stores that token and attaches it to all future requests. This token is enough to identify the request as authorized. + +Since the tokens are signed by us, we can also include data (such as user id) in the token. On all future requests we can rely that the id is correct since the JWT will be verified on every request. + +Tokens can (and should be) equipped with expiration dates. + +# Creating a Token + +Install the JSON Web Token package as dependency npm install jsonwebtoken + +Tokens are created with the _sign_ function that takes two parameters: _Payload:_ the data we want to include in the token. It can be empty. _Secret_ : either a string or a private key. + +It can also accept an optional _options_ parameter. In that we can define the expiration date, compression algorithm, or many other things. + +import 'dotenv/config' + +import jwt from 'jsonwebtoken' + +const payload = { username: 'sugarplumfairy' } + +const secret = process.env.SECRET + +const options = { expiresIn: '1h'} + +const token = jwt.sign(payload, secret, options) + +console.log(token) + +The created JWT has three parts separated by period: header, payload and signature. + +# Exercise 6: Create a Token + +Write a simple command line program that prints JSON Web tokens. Use the default algorithm (SHA256). Set the tokens to expire in fifteen minutes and include a payload of some JSON object. + +Copy your JWT and paste it to [https://jwt.io](https://jwt.io) debugger. Verify that the debugger shows correct algorithm, data, and expiration date (iat). + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Verifying a Token + +Tokens are verified using the _verify_ function that takes two parameters: + +_Token_ : the token to be verified + +_Secret_ : the secret that was used to create the token + +The verify function can also accept an optional _options_ parameter. + +The verify function returns a _decoded_ token. Since it has been verified against our secret, we know that the information has not been altered. + +If the token is invalid, the verify function _throws an error_ . If this error is not caught, the program crashes. + +# Exercise 7: Verify a Token + +Write a simple command line program that verifies JSON Web tokens. + +The program should print the contents of the verified token. If the token is not valid, the program should print an error message and exit gracefully. + +Create a JWT in [https://jwt.io](https://jwt.io) debugger. Remember to set your secret value. Use your program to verify the token and see that the data entered is as it should be. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Bearer Tokens + +When a user has received a token, it can be used to authenticate all future requests. + +Do this by adding the field _Authorization _ to the request header. + +The value is "Bearer " plus the token. + +This header can be added to requests generated by Fetch, Axios, or any other JS request library, or by API clients like Postman or Insomnia. + +Then protected routes can check that the authorization header is valid, and if not, send an error message. This is usually done using authorization middleware. + +# Authentication Middleware + +A common way is to have user routes, such as login, logout or user creation in a separate router, which requires no token. + +Then all protected routes are set to use the authentication middleware. + +The request parameter _user_ is set to the verified token data and can then be used in the actual route. + +const authenticate = (req, res, next) => { + +const auth = req.get('Authorization') + +if (!auth?.startsWith('Bearer ')) { + +return res.status(401).send('Invalid token') + +} + +const token = auth.substring(7) + +const secret = process.env.SECRET + +try { + +const decodedToken = jwt.verify(token, secret) + +req.user = decodedToken + +next() + +} catch (error) { + +return res.status(401).send('Invalid token') + +} + +} + +router.get('/protected', authenticate, (req, res) => { + +res.send(\`${req.user.username} accessed protected route\`) + +}) + +# TypeScript: Extending Request + +Let's look at the following example, written in JavaScript. There is a middleware that adds a _username _ property to the request object. Then the endpoint reads the username property and uses it to construct a response. + +What happens when we write the same code in TypeScript? + +import express from 'express' + +const server = express() + +const middleware = (req, \_res, next) => { + +req.username = 'sugarplumfairy' + +next() + +} + +server.use(middleware) + +server.get('/', (req, res) => { + +res.send(\`Welcome, ${req.username}!\`) + +}) + +server.listen(3000) + +Property 'username' does not exist on type 'Request>' + +import express, { Request, Response, NextFunction } from 'express' + +const server = express() + +const middleware = (req: Request, res: Response, next: NextFunction) => { + +req.username = 'sugarplumfairy' + +next() + +} + +server.use(middleware) + +server.get('/', (req, res) => { + +res.send(\`Welcome, ${req.username}!\`) + +}) + +server.listen(3000) + +Property 'username' does not exist on type 'Request>' + +We are trying to set a value to request property "username", but request objects are well defined objects that have no such property. + +We can not add or use properties that an object does not have. This is part of the type safety that TypeScript is here to enforce. If an object is of type Request, we know exactly what the properties are. + +The solution is to create an interface that extends the existing Request type. The new interface will have all the properties the Request has, plus the ones we define ourself. + +interface CustomRequest extends Request { + +username?: string + +} + +Since the interface extends Request, CustomRequest objects can be used anywhere where Request object is required. + +Notice that the additional properties must be optional, signified by the question mark in the name. Otherwise existing Request objects can not be cast as CustomRequest objects. + +This example has only a single string, but the extended object can be as complex as needed. + +interface CustomRequest extends Request { + +username?: string + +} + +Now we can use the "username" property when we declare that we are using CustomRequest instead of basic Request object. + +import express, { Request, Response, NextFunction} from 'express' + +const server = express() + +interface CustomRequest extends Request { + +username?: string + +} + +const middleware = (req: CustomRequest, res: Response, next: NextFunction) => { + +req.username = 'sugarplumfairy' + +next() + +} + +server.use(middleware) + +server.get('/', (req: CustomRequest, res: Response) => { + +res.send(\`Welcome, ${req.username}!\`) + +}) + +server.listen(3000) + +# Exercise 8: Securing a Route + +Modify the Students API /register and /login routes so that on success they return a response with status code 200 and a JWT with username as payload. + +Secure all the routes in the _students _ router so that they require the user to be logged in to use the routes. + +Extra: Also modify the /admin route to return a JWT. Secure the POST, PUT and DELETE routes to require that in addition to being logged in, the user also needs to be an admin. + +Hint: Here you have some resources on sending a token with Postman. + +[https://learning.postman.com/docs/sending-requests/authorization/](https://learning.postman.com/docs/sending-requests/authorization/) + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + +# Supertest + +# Testing APIs with Supertest + +Supertest is a library for testing API's. Let's see how to use it with Jest to test an Express app. Install supertest with + +npm install --save-dev supertest + +When using ESLint, also add "jest": true to .eslintrc's "env" object + +_index.js_ + +import express from 'express' + +const server = express() + +server.get('/', (req, res) => { + +res.send('Hello World!') + +}) + +server.listen(3000, () => { + +console.log('Listening to port 3000') + +}) + +When using TypeScript, also install the required types from @types/supertest and @types/jest + +Also, install the ts-jest package, and add to package.json: + +"jest": { + +"preset": "ts-jest", + +"testEnvironment": "node" + +}, + +We need to import our express application to the tests. + +We will separate our application in two files. The _server.js_ defines the API server, and _index.js_ only starts the application. + +Now we can test the server with Supertest. The startup file has no functionality we want to test. + +_index.js_ + +import server from './server.js' + +server.listen(3000, () => { + +console.log('Listening to port 3000') + +}) + +_server.js_ + +import express from 'express' + +const server = express() + +server.get('/', (req, res) => { + +res.send('Hello World!') + +}) + +export default server + +We now create a new test file and import our server there. + +We use Supertest _request_ to get responses from our server. These responses can be tested just like any other Jest test. + +_test/server.test.js_ + +import request from 'supertest' + +import server from '../src/server.js' + +describe('Server', () => { + +it('Returns 404 on invalid address', async () => { + +const response = await request(server) + +.get('/invalidaddress') + +expect(response.statusCode).toBe(404) + +}) + +it('Returns 200 on valid address', async () => { + +const response = await request(server) + +.get('/') + +expect(response.statusCode).toBe(200) + +expect(response.text).toEqual('Hello World!') + +}) + +}) + +# Common Resources + +[https://www.npmjs.com/package/jest](https://www.npmjs.com/package/jest) → Official repository + +[https://www.npmjs.com/package/supertest](https://www.npmjs.com/package/supertest) → Official repository + +[https://jestjs.io/docs/api](https://jestjs.io/docs/api) → Jest API Docs. Very handy! + +[https://github.com/visionmedia/supertest](https://github.com/visionmedia/supertest) → Supertest's github repository. Includes guides for basic use cases. + +# Exercise 9: Test with Supertest + +Use Supertest with Jest to create tests for the Students API /user/register and /user/login routes. + +--- + +Solution: https://gitlab.com/bctjs/week3-day2/-/blob/master/examples/day2/countdown.js + diff --git a/3-databases-with-docker.md b/3-databases-with-docker.md new file mode 100644 index 0000000..3fbcaed --- /dev/null +++ b/3-databases-with-docker.md @@ -0,0 +1,790 @@ +# Databases + +# SQL + +* __Structured Query Language__ + * A language __ __ to organize and manipulate data in a relational database +* Originally developed by IBM in the 70s + * Quickly became the most popular database language +* SELECT id, email +* FROM users +* WHERE first\_name = 'Teppo'; + +# Relational Database +Management Systems + +* In relational databases, values are stored in __tables__ + * Each table has __rows __ and __columns__ + * Data is displayed in a two-dimensional matrix +* Values in a table are related to each other + * Values can also be related to values in other tables +* A relational database management system (RDBMS) is a program that executes queries to relational databases + +![](imgs/3-databases-with-docker_0.png) + +[https://db-engines.com/en/ranking](https://db-engines.com/en/ranking) + +# PostgreSQL + +A free, open-source, cross-platform relational database management system + +Emphasizes extensibility and SQL compliance + +Fully [ACID](https://en.wikipedia.org/wiki/ACID) -compliant (atomicity, consistency, isolation and durability) + +# pgAdmin + +Administration and development platform for PostgreSQL + +Cross-platform, features a web interface + +Basically a control panel application for your PostgreSQL database + +# Running Postgres in Docker + +Using the official [Postgres Docker image](https://hub.docker.com/_/postgres) , let's create a locally running Postgres instance. You can choose the values for POSTGRES\_PASSWORD and POSTGRES\_USER freely. + +__docker run --name my-postgres --env POSTGRES\_PASSWORD=pgpass --env POSTGRES\_USER=pguser -p 5432:5432 -d postgres:15.2__ + +![](imgs/3-databases-with-docker_1.png) + +# PostgreSQL: Using psql + +If you have PostgreSQL installed + +Win search _psql _ and open _SQL Shell (psql) + +Press enter four times to insert default values, or values matching to your configuration, and insert your password for the fifth prompt + +If you do not have PostgreSQL installed, you can use psql directly from the container running the database docker exec -it my-postgres psql -U pguser + +* If you have connected to the server as user "pguser", you are by default connected to "pguser" database. +* This is a database for user information. +* You should not use it for storing program data!! +* All databases can be listed using the _list _ command: __ \\l __ +* PostgreSQL uses a default database named "postgres" +* Users can connect to a different database with _connect _ command __\\c__ : + * __\\c __ +* A new database is created with CREATE command: + * __CREATE DATABASE ;__ + * Do not forget the __;__ at the end +* After creating a new database, you still need to connect to it! +* Exit psql with the command __exit__ + +# Exercise 1: Postgres Server + +Start a local instance of Postgres in Docker + +Connect to the server using psql + +Use command \\l to see what databases there are already created on the server. + +Create a new database called "sqlpractice". + +Connect to the newly created database. + +# Running pgAdmin in Docker + +Using the official [pgAdmin](https://hub.docker.com/r/dpage/pgadmin4) image, we'll run pgAdmin alongside Postgres + +__docker run --name my-pgadmin -p 5050:80 __ __-e PGADMIN\_DEFAULT\_EMAIL= __ __-e PGADMIN\_DEFAULT\_PASSWORD= __ __-d dpage/pgadmin4__ + +![](imgs/3-databases-with-docker_2.png) + +# Logging into pgAdmin + +With pgAdmin running, navigate your web browser to [http://localhost:5050](http://localhost:5050) and use the username and password you provided to log in. + +![](imgs/3-databases-with-docker_3.png) + +# PostgreSQL internal IP Address + +Both __PostgreSQL__ and __pgAdmin __ are now running in our _local_ Docker. To connect pgAdmin to PostgreSQL, we need the IP address used _inside_ Docker. + +Using Docker's inspect, we'll get Docker's internal IP for my-postgres -container. Since the command produces quite a lot of information, we pipe the result to grep, to see only the rows that contain the word "IPAddress" + +__docker inspect | grep IPAddress __ __<- unix__ __docker inspect | findstr IPAddress __ __<- windows__ + +In the example output, the IP Address is 172.17.0.2 + +![](imgs/3-databases-with-docker_4.png) + +# Connecting PgAdmin to our DB + +Now we have all that we need for a connection. In pgAdmin, select "Object" > "Register" > "Server". + +In the "General" tab, give the server a name that identifies the connection. + +![](imgs/3-databases-with-docker_5.png) + +If Object menu is greyed out, click on Servers. + +![](imgs/3-databases-with-docker_6.png) + +* In the "Connection" tab, enter +* Host name/address: + * the PostgreSQL internal Docker address +* Port, Username, Password + * the values defined when running the PostgreSQL container +* Then click Save. You should now see all the databases available on this server. + +![](imgs/3-databases-with-docker_7.png) + +![](imgs/3-databases-with-docker_8.png) + +# Exercise 2: pgAdmin + +Start a local instance of pgAdmin in Docker + +Following lecture instructions, connect the pgAdmin to your already running PostgreSQL server. + +Verify that you can see the database created in the previous assignment. + +# PostgreSQL: Querying + +With psql: After connecting to a database, just type a query and hit enter. + +With pgAdmin: + +Right-click a database _ _ > _Query tool_ + +Insert a query into the _Query Editor_ and hit _Execute _ (F5) + +![](imgs/3-databases-with-docker_9.png) + +![](imgs/3-databases-with-docker_10.png) + +--- + +# Editing Data with pgAdmin + +* Tables of Data in a DataBase are found under + * Database > Schemas > Tables +* Inspect and edit data in pgAdmin by right-clicking a table and selecting _View/Edit Data_ + +![](imgs/3-databases-with-docker_11.png) + +![](imgs/3-databases-with-docker_12.png) + +Individual values in the table can be directly modified by double clicking the value and then editing the value in the visual user interface + +Save the changes with the _Save Data Changes_ button + +![](imgs/3-databases-with-docker_13.png) + +# Exercise 3: Preparing the Database + +Using either PgAdmin or PSQL + +Insert the [provided query](https://gitea.buutti.com/education/academy-assignments/src/branch/master/Databases/initdb.txt) to the database you created previously. + +Verify that the query has created new tables to your database. + +# Types of queries + +Select + +Insert + +Delete + +Update + +Create & Drop + +# Querying data with SELECT + +Syntax: + +__SELECT column1, column2, column3 FROM table\_name;__ + +Examples: + +SELECT full\_name, email FROM users; + +SELECT full\_name AS name, email FROM users; + +SELECT * FROM users; + +# Filtering data with WHERE + +Syntax: + +__SELECT column1, column2 FROM table\_name WHERE condition;__ + +Text is captured in _single quotes_ . + +In a LIKE condition, _%_ sign acts as a wildcard. + +IS and IS NOT are also valid comparison operators. + +Example: + +SELECT full\_name FROM users + +WHERE full\_name = 'Teppo Testaaja'; + +SELECT * FROM books WHERE name LIKE '%rr%'; + +SELECT * FROM books WHERE author IS NOT null; + +# Ordering data with ORDER BY + +Syntax: + +__SELECT column1 FROM table\_name ORDER BY column1 ASC;__ + +Examples: + +SELECT full\_name FROM users + +ORDER BY full\_name ASC; + +SELECT full\_name FROM users + +ORDER BY full\_name DESC + +# Combining with JOIN + +Also known as INNER JOIN + +Corresponds to intersection from set theory + +![](imgs/3-databases-with-docker_14.png) + +# JOIN examples + +SELECT + +users.id, users.full\_name, borrows.id, + +borrows.user\_id, borrows.due\_date, borrows.returned\_at + +FROM users + +JOIN borrows ON + +users.id = borrows.user\_id; + +SELECT + +U.full\_name AS name, + +B.due\_date AS due\_date, + +B.returned\_at AS returned\_at + +FROM users AS U + +JOIN borrows AS B ON + +U.id = B.user\_id; + +# Combining with LEFT JOIN + +Also known as LEFT OUTER JOIN + +Example: + +SELECT + +U.full\_name AS name, + +B.due\_date AS due\_date, + +B.returned\_at AS returned\_at + +FROM users AS U + +LEFT JOIN borrows AS B ON + +U.id = B.user\_id; + +![](imgs/3-databases-with-docker_15.png) + +# Exercise 4: Querying the Library + +Using SQL queries, get + +All columns from loans that are loaned before 1.3.2000 + +All columns of loans that are returned + +Columns user.full\_name and borrows.borrowed\_at of the user with an id of 1 + +Columns book.name, book.release\_year and language.name of all books that are released after 1960 + +# INSERT + +Syntax + +__INSERT INTO table\_name (column1, column2, column3)_ __VALUES (value1, value2, value3);__ + +Example + +INSERT INTO users (full\_name, email, created\_at) + +VALUES ('Pekka Poistuja', 'pekka.poistuja@buutti.com', NOW()); + +Since id is not provided, it will be automatically generated. + +# UPDATE + +Syntax + +__UPDATE table\_name__ __SET column1 = value1, column2 = value2__ __WHERE condition;__ + +__Notice:__ if a condition is not provided, all rows will be updated!If updating only one row, it is usually best to use id. + +Example + +UPDATE usersSET email = 'taija.testaaja@gmail.com'WHERE id = 2; + +# REMOVE + +Syntax + +__DELETE FROM table\_name WHERE condition;__ + +Again, if the _condition_ is not provided, DELETE affects _all_ rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected. + +Example: + +SELECT __*__ FROM __users __ WHERE __id = 5;__ + +DELETE FROM users WHERE id = 5; + +# Exercise 5: Editing Data + +Postpone the due date of the loan with an id of 2 by two days in the _borrows_ _ _ table + +Add a couple of new books to the _books _ table + +Delete one of the loans. + +# CREATE TABLE + +Before data can be manipulated, a database and its tables need to be initialized. + +Syntax + +__CREATE TABLE table\_name (__ + +__ column1 datatype,__ __ column2 datatype,__ __ …__ __);__ + +Example:CREATE TABLE "users" ( + +"id" SERIAL PRIMARY KEY, + +"full\_name" varchar NOT NULL, + +"email" varchar UNIQUE NOT NULL, + +"created\_at" timestamp NOT NULL + +); + +# DROP + +In order to remove tables or databases, we use a DROP statement + +__DROP TABLE table\_name;__ __DROP DATABASE database\_name;__ + +These statements do not ask for confirmation and there is no undo feature. Take care when using a drop statement. + +# NoSQL + +* In addition to SQL databases, there are also NoSQL databases +* Many differing definitions, but... + * most agree that NoSQL databases store data in a format other than tables + * They can still store relational data - just differently +* Four different database types: + * Document databases + * Key-value databases + * Wide-column stores + * Graph databases +* Example database engines include MongoDB, Redis and Cassandra +--- + +Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. MongoDB is consistently ranked as the world's most popular NoSQL database according to DB-engines and is an example of a document database. For more on document databases, visit What is a Document Database?. +Key-value databases are a simpler type of database where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don't need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Redis and DynanoDB are popular key-value databases. +Wide-column stores store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores. +Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Neo4j and JanusGraph are examples of graph databases. + +# Object-Relational Mappers + +* ORMs allow developers to manipulate databases with code instead of SQL queries + * For example, for performing CRUD operations on their database +* Some popular ORMs: +* Hibernate (Java) +* EFCore (.NET) +* Sequelize (Node.js) +* TypeORM (TypeScript) +* We recommend using TypeORM instead of writing SQL yourself. You can find a good step-by-step guide within [their own documentation](https://typeorm.io/#installation) . + +# PostgreSQL with Node + +# Local environment + +Using our previously created, Dockerized Postgres instance, we'll create a Node.Js application to connect to our database. + +If you have deleted your postgres container, a new one can be created with a following command: + +__docker run --name my-postgres -e POSTGRES\_PASSWORD=mypassword -e POSTGRES\_USER=pguser -e POSTGRES\_DB=mydb -p 5432:5432 -d postgres:15.2__ + +With a previously created container, this command will start the container: + +__docker start my-postgres__ + +# Preparing our Node application + +Initialize the application with: __npm init -y__ + +Install [express](https://www.npmjs.com/package/express) and [PostgreSQL client for Node.JS](https://www.npmjs.com/package/pg) __ npm install express pg__ + +Install [dotenv](https://www.npmjs.com/package/dotenv) and nodemon as a development dependencies __npm install –-save-dev dotenv nodemon__ + +If using TypeScript, install types for express and pg, plus ts-node for running nodemon. __npm install --save-dev @types/express @types/pg ts-node__ + +# Dotenv + +Example of .env file: + +In development, it's customary to store ports, database login info, etc. as _environment variables_ in a separate .env file + +Previously installed package _dotenv_ is used to load variables from .env to the main program + +dotenv can be imported with + +PORT=3000 + +PG\_HOST=localhost + +PG\_PORT=5432 + +PG\_USERNAME=pguser + +PG\_PASSWORD=mypassword + +PG\_DATABASE=postgres + +These values must match the values declared when running the PostgreSQL container. + +The database must exist as well. + +Note that the example uses the default "postgres" database, but you can use any database you want. + +import dotenv from 'dotenv' + +dotenv.config() + +# Dotenv (continued) + +{ + +"name": "products\_api", + +"version": "1.0.0", + +"scripts": { + +"dev": "nodemon -r dotenv/config ./src/index.ts" + +}, + +"dependencies": { + +"express": "^4.18.2", + +"pg": "^8.9.0" + +}, + +"devDependencies": { + +"@types/express": "^4.17.17", + +"@types/pg": "^8.6.6", + +"dotenv": "^16.0.3", + +"nodemon": "^2.0.20", + +"ts-node": "^10.9.1", + +"typescript": "^4.9.5" + +} + +} + +Dotenv is usually only used in development, not in production. In professional setting the dotenv config is often preloaded in the development startup script + +You can require dotenv when running _npm run dev_ + +-r is short for --require + +# Dotenv and Git + +* .env files usually contain sensitive data that we do __not__ want to store in Git repositories. +* Thus, usually __.env__ file is excluded from the Git repository + * Add __.env__ to __.gitignore__ + * If you have auto-generated .gitignore with npx gitignore Node, environment files are excluded automatically + +![](imgs/3-databases-with-docker_16.png) + +# Connecting to PostgreSQL + +Our database file contains functions and configuration for initializing the Postgres pool, creating tables and running queries. + +At the moment, we have only one query. It is a single-use query that creates a products table to the database if such table does not yet exist. + +// db.ts + +import pg from "pg"; + +const { PG\_HOST, PG\_PORT, PG\_USERNAME, PG\_PASSWORD, PG\_DATABASE } = process.env; + +const pool = new pg.Pool({ + +host: PG\_HOST, + +port: Number(PG\_PORT), + +user: PG\_USERNAME, + +password: String(PG\_PASSWORD), + +database: PG\_DATABASE, + +}); + +const executeQuery = async (query: string, parameters?: Array) => { + +const client = await pool.connect(); + +try { + +const result = await client.query(query, parameters); + +return result; + +} catch (error: any) { + +console.error(error.stack); + +error.name = "dbError"; + +throw error; + +} finally { + +client.release(); + +} + +}; + +export const createProductsTable = async () => { + +const query = \` + +CREATE TABLE IF NOT EXISTS "products" ( + +"id" SERIAL PRIMARY KEY, + +"name" VARCHAR(100) NOT NULL, + +"price" REAL NOT NULL + +)\`; + +await executeQuery(query); + +console.log("Products table initialized"); + +}; + +At the moment our __index.js__ does nothing but creates a single table for our database and launches express server. + +It doesn't even have any endpoints, so it's not much of a server yet. + +import express from "express"; + +import { createProductsTable } from "./db"; + +const server = express(); + +createProductsTable(); + +const { PORT } = process.env; + +server.listen(PORT, () => { + +console.log("Products API listening to port", PORT); + +}); + +# Launching the application + +Let's use our predefined _npm run dev_ + +![](imgs/3-databases-with-docker_17.png) + +…And check with psql that our application succeeds in connecting to the database and creating the table. + +Epic success! + +![](imgs/3-databases-with-docker_18.png) + +# Exercise 6: Node & PostgreSQL + +Following the lecture example, create an Express server that connects to your local PostgreSQL instance. The database information should be stored in environment variables. When the server starts, it should create a product table with three columns: id (serial, primary key), name (varchar) and price (real). + +# Creating Queries + +* Next, we will create an actual CRUD API for communicating with the database. +* For that we need endpoints for creating, reading, updating and deleting products. + * All of these need their own queries. + +# Using queries + +* We'll use the pre-made executeQuery() function for querying the database from a few slides back +* It takes in two arguments: + * the actual query string + * an optional parameters array +* When supplying parameters, the query string should have placeholders $1, $2, etc + * These will be replaced with the contents of the parameters array. + +# Parameterized queries example + +When running executeQuery(query, parameters) with above values defined, the query would be parsed as + +![](imgs/3-databases-with-docker_19.png) + +SELECT * FROM cats WHERE color = 'yellow' and age > 10; + +# Why not just use String templating? + +…Because of [SQL injections](https://fi.wikipedia.org/wiki/SQL-injektio) . Always use database library's built-in parameterization! + +_NEVER DO THIS!!!_ + +![](imgs/3-databases-with-docker_20.png) + +![](imgs/3-databases-with-docker_21.png) + +# Creating queries + +We will create a Data Access Object, __dao.js__ that will handle interacting with our database. + +The idea is that we want to just tell our DAO what we want done (e.g. "add this customer to the database") and the DAO will handle the details of that action. + +The DAO will also return possible additional information that was created during the action. + +Our _insertProduct _ function + +generates a new, unique ID for the product using [uuid](https://www.npmjs.com/package/uuid) + +constructs a parameters array containing said id, the name of the product and the price of the product + +executes the query using _db.executeQuery_ method + +returns the database result object + +![](imgs/3-databases-with-docker_22.png) + +![](imgs/3-databases-with-docker_23.png) + +![](imgs/3-databases-with-docker_24.png) + +The rest of the DAO operations work in similar fashion. + +The router that declares the endpoints, uses the DAO to interact with the database. + +# Testing the API + +![](imgs/3-databases-with-docker_25.png) + +Now we can use Insomnia to verify that all the endpoints work as expected. + +We can also use psql to observe the changes in the database + +![](imgs/3-databases-with-docker_26.png) + +# Exercise 7: Creating Queries + +Continue following the lecture example. Create a router and a database access object to handle + +Creating a product + +Reading a product + +Updating a product + +Deleting a product + +Listing all products + +# Dockerized PostgreSQL App + +# Setting Environment Variables + +Docker has two kinds of environment variables: run-time and build-time. + +In this scenario we want to set our environment variables at __build time__ . This means that _the Docker image will contain all the environment variable information_ , including sensitive things like passwords. This might be an issue in some scenarios. In those cases the environment variables need to be set at __run time__ . + +In the Dockerfile we set the build-time values by setting ARG parameters. Then we use these values to set the run-time environment variables by setting ENV parameters. + +More information: [https://vsupalov.com/docker-arg-env-variable-guide/](https://vsupalov.com/docker-arg-env-variable-guide/) + +When the ARGs and ENVs have been set in the Dockerfile, we provide the ARG values when building the Docker image by using __--build-arg =__ flags. To build an image with these parameters, we'd use something like + +__docker build __ + +__--build-arg PORT=3000 __ + +__--build-arg PG\_HOST=https://my.postgres.server__ + +__--build-arg PG\_PORT=5432 __ + +__--build-arg PG\_USERNAME=pguser __ + +__--build-arg PG\_PASSWORD=pgpass __ + +__--build-arg PG\_DATABASE=my-database __ + +__-t my-app .__ + +ARG PORT + +ARG PG\_HOST + +ARG PG\_PORT + +ARG PG\_USERNAME + +ARG PG\_PASSWORD + +ARG PG\_DATABASE + +ENV PORT=${PORT} + +ENV PG\_HOST=${PG\_HOST} + +ENV PG\_PORT=${PG\_PORT} + +ENV PG\_USERNAME=${PG\_USERNAME} + +ENV PG\_PASSWORD=${PG\_PASSWORD} + +ENV PG\_DATABASE=${PG\_DATABASE} + +[Docker documentation here!](https://www.docker.com/blog/how-to-use-the-postgres-docker-official-image/) + +# Exercise 8: Dockerized PG App + +Dockerize the application you have built. Build the docker image, run the app and test that it works using insomnia/postman. + +Remember that when you run the application on your local Docker, both the app and the database are in the same Docker network, so you have to check the database IP address just like when running pgAdmin. + diff --git a/example-lecture-slides.html b/example-lecture-slides.html deleted file mode 100644 index dcc8159..0000000 --- a/example-lecture-slides.html +++ /dev/null @@ -1,112 +0,0 @@ -N. Example Lecture
-

Example Lecture

-
-
-

Section

-
    -
  • This line appears instantly
    • -
-
    -
  • This line appears by pressing spacebar (preferred)
    • -
  • This line has an inline code variable
    • -
-
    -
  1. This line appears instantly
    2. -
-
    -
  1. This line appears by pressing spacebar
    2. -
-
-
-

Code and maths

-
    -
  • code code code:
    console.log("Here's a syntax-highlighted JavaScript code block");
-console.log("Remember indentation so it's revealed after the bullet point.");
-
    -
    • -
  • This line has an inline LaTeX maths equation
    • -
  • Here's a maths block:
    • -
-

Footers are exclusive to presentation; they are not shown in the webpage markdown document
-
-
-

Columns

-
-
-

-
    -
  • Basic image example
    • -
-
-
-

-
    -
  • Wider image
    • -
-
-
-
    -
  • This line is outside the columns and goes from left all the way to the right
    • -
-
-
-

Columns 2

-
-
-
    -
  • Another column example with a wider left panel
    • -
-
-
-
    -
  • Change class name to change proportions
    • -
  • If suitable proportions not available, add to buutti.css
    • -
-
-
-
-
-

Setup

- -
-
\ No newline at end of file diff --git a/example-lecture.md b/example-lecture.md deleted file mode 100644 index 6c11c6d..0000000 --- a/example-lecture.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -marp: true -paginate: true -math: mathjax -theme: buutti -title: N. Example Lecture ---- - -# Example Lecture - - - - -## Section - -- This line appears instantly -* This line appears by pressing spacebar (preferred) -* This line has an inline code `variable` -1. This line appears instantly -2) This line appears by pressing spacebar - -## Code and maths - -* code code code: - ```js - console.log("Here's a syntax-highlighted JavaScript code block"); - console.log("Remember indentation so it's revealed after the bullet point."); - ``` -* This line has an inline LaTeX maths equation $c = \frac{a^2}{\sqrt{b}}$ -* Here's a maths block: - -$$ -F(x) = \int_a^b f(x) dx -$$ - - - -## Columns - -
-
- -![](imgs/buuttilogo.png) - -* Basic image example - -
-
- -![width:800px](imgs/buuttilogo.png) -* Wider image - -
-
- -* This line is outside the columns and goes from left all the way to the right - -## Columns 2 - -
-
- -* Another column example with a wider left panel - -
-
- -* Change `class` name to change proportions -* If suitable proportions not available, add to `buutti.css` - -
-
- -## Setup - -* In VS Code, install the extensions - * [Marp for VS code](https://marketplace.visualstudio.com/items?itemName=marp-team.marp-vscode) - * So you can see the slideshow preview when editing. - * [Markdown all in one](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one) - * [Markdown table formatter](https://marketplace.visualstudio.com/items?itemName=fcrespo82.markdown-table-formatter) - * *Right click > Format document* makes tables pretty - * [Save and run](https://marketplace.visualstudio.com/items?itemName=wk-j.save-and-run) - * An HTML version of the lecture is created on save - * See [settings.json](./.vscode/settings.json) - * Add filenames to `notMatch` if a HTML on save is not needed \ No newline at end of file diff --git a/imgs/.gitkeep b/imgs/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/imgs/1-express-basics_0.png b/imgs/1-express-basics_0.png new file mode 100644 index 0000000..a710d49 Binary files /dev/null and b/imgs/1-express-basics_0.png differ diff --git a/imgs/1-express-basics_1.png b/imgs/1-express-basics_1.png new file mode 100644 index 0000000..e657163 Binary files /dev/null and b/imgs/1-express-basics_1.png differ diff --git a/imgs/1-express-basics_2.png b/imgs/1-express-basics_2.png new file mode 100644 index 0000000..62d291d Binary files /dev/null and b/imgs/1-express-basics_2.png differ diff --git a/imgs/3-databases-with-docker_0.png b/imgs/3-databases-with-docker_0.png new file mode 100644 index 0000000..3fb3c4a Binary files /dev/null and b/imgs/3-databases-with-docker_0.png differ diff --git a/imgs/3-databases-with-docker_1.png b/imgs/3-databases-with-docker_1.png new file mode 100644 index 0000000..fcb3736 Binary files /dev/null and b/imgs/3-databases-with-docker_1.png differ diff --git a/imgs/3-databases-with-docker_10.png b/imgs/3-databases-with-docker_10.png new file mode 100644 index 0000000..e60521c Binary files /dev/null and b/imgs/3-databases-with-docker_10.png differ diff --git a/imgs/3-databases-with-docker_11.png b/imgs/3-databases-with-docker_11.png new file mode 100644 index 0000000..03a1e2f Binary files /dev/null and b/imgs/3-databases-with-docker_11.png differ diff --git a/imgs/3-databases-with-docker_12.png b/imgs/3-databases-with-docker_12.png new file mode 100644 index 0000000..2d57d7c Binary files /dev/null and b/imgs/3-databases-with-docker_12.png differ diff --git a/imgs/3-databases-with-docker_13.png b/imgs/3-databases-with-docker_13.png new file mode 100644 index 0000000..23fd825 Binary files /dev/null and b/imgs/3-databases-with-docker_13.png differ diff --git a/imgs/3-databases-with-docker_14.png b/imgs/3-databases-with-docker_14.png new file mode 100644 index 0000000..35a1812 Binary files /dev/null and b/imgs/3-databases-with-docker_14.png differ diff --git a/imgs/3-databases-with-docker_15.png b/imgs/3-databases-with-docker_15.png new file mode 100644 index 0000000..4190de0 Binary files /dev/null and b/imgs/3-databases-with-docker_15.png differ diff --git a/imgs/3-databases-with-docker_16.png b/imgs/3-databases-with-docker_16.png new file mode 100644 index 0000000..1cd9252 Binary files /dev/null and b/imgs/3-databases-with-docker_16.png differ diff --git a/imgs/3-databases-with-docker_17.png b/imgs/3-databases-with-docker_17.png new file mode 100644 index 0000000..7620fae Binary files /dev/null and b/imgs/3-databases-with-docker_17.png differ diff --git a/imgs/3-databases-with-docker_18.png b/imgs/3-databases-with-docker_18.png new file mode 100644 index 0000000..816232b Binary files /dev/null and b/imgs/3-databases-with-docker_18.png differ diff --git a/imgs/3-databases-with-docker_19.png b/imgs/3-databases-with-docker_19.png new file mode 100644 index 0000000..b8a4ab2 Binary files /dev/null and b/imgs/3-databases-with-docker_19.png differ diff --git a/imgs/3-databases-with-docker_2.png b/imgs/3-databases-with-docker_2.png new file mode 100644 index 0000000..1872b7b Binary files /dev/null and b/imgs/3-databases-with-docker_2.png differ diff --git a/imgs/3-databases-with-docker_20.png b/imgs/3-databases-with-docker_20.png new file mode 100644 index 0000000..868aebb Binary files /dev/null and b/imgs/3-databases-with-docker_20.png differ diff --git a/imgs/3-databases-with-docker_21.png b/imgs/3-databases-with-docker_21.png new file mode 100644 index 0000000..2c46206 Binary files /dev/null and b/imgs/3-databases-with-docker_21.png differ diff --git a/imgs/3-databases-with-docker_22.png b/imgs/3-databases-with-docker_22.png new file mode 100644 index 0000000..95e64a4 Binary files /dev/null and b/imgs/3-databases-with-docker_22.png differ diff --git a/imgs/3-databases-with-docker_23.png b/imgs/3-databases-with-docker_23.png new file mode 100644 index 0000000..53666b9 Binary files /dev/null and b/imgs/3-databases-with-docker_23.png differ diff --git a/imgs/3-databases-with-docker_24.png b/imgs/3-databases-with-docker_24.png new file mode 100644 index 0000000..f35cdb2 Binary files /dev/null and b/imgs/3-databases-with-docker_24.png differ diff --git a/imgs/3-databases-with-docker_25.png b/imgs/3-databases-with-docker_25.png new file mode 100644 index 0000000..e4aa590 Binary files /dev/null and b/imgs/3-databases-with-docker_25.png differ diff --git a/imgs/3-databases-with-docker_26.png b/imgs/3-databases-with-docker_26.png new file mode 100644 index 0000000..74d7a35 Binary files /dev/null and b/imgs/3-databases-with-docker_26.png differ diff --git a/imgs/3-databases-with-docker_3.png b/imgs/3-databases-with-docker_3.png new file mode 100644 index 0000000..6ffb1fb Binary files /dev/null and b/imgs/3-databases-with-docker_3.png differ diff --git a/imgs/3-databases-with-docker_4.png b/imgs/3-databases-with-docker_4.png new file mode 100644 index 0000000..b71d5fa Binary files /dev/null and b/imgs/3-databases-with-docker_4.png differ diff --git a/imgs/3-databases-with-docker_5.png b/imgs/3-databases-with-docker_5.png new file mode 100644 index 0000000..db7b5f4 Binary files /dev/null and b/imgs/3-databases-with-docker_5.png differ diff --git a/imgs/3-databases-with-docker_6.png b/imgs/3-databases-with-docker_6.png new file mode 100644 index 0000000..ff11198 Binary files /dev/null and b/imgs/3-databases-with-docker_6.png differ diff --git a/imgs/3-databases-with-docker_7.png b/imgs/3-databases-with-docker_7.png new file mode 100644 index 0000000..40550a2 Binary files /dev/null and b/imgs/3-databases-with-docker_7.png differ diff --git a/imgs/3-databases-with-docker_8.png b/imgs/3-databases-with-docker_8.png new file mode 100644 index 0000000..4a45325 Binary files /dev/null and b/imgs/3-databases-with-docker_8.png differ diff --git a/imgs/3-databases-with-docker_9.png b/imgs/3-databases-with-docker_9.png new file mode 100644 index 0000000..2c8f562 Binary files /dev/null and b/imgs/3-databases-with-docker_9.png differ diff --git a/imgs/buuttilogo.png b/imgs/buuttilogo.png deleted file mode 100644 index 8c35578..0000000 Binary files a/imgs/buuttilogo.png and /dev/null differ