A free, open-source, cross-platform relational database management system
Emphasizes extensibility and SQL compliance
Fully [ACID](https://en.wikipedia.org/wiki/ACID)-compliant (atomicity, consistency, isolation and durability)
# Running Postgres in Docker
Using the official [Postgres Docker image](https://hub.docker.com/_/postgres) , let's create a locally running Postgres instance. You can choose the values for POSTGRES_PASSWORD and POSTGRES_USER freely.
```
docker run --name my-postgres --env POSTGRES_PASSWORD=pgpass
*psql is a terminal-based front-end to PostgreSQL. It enables you to type in queries interactively, issue them to PostgreSQL, and see the query results. Alternatively, input can be from a file or from command line arguments. In addition, psql provides a number of meta-commands and various shell-like features to facilitate writing scripts and automating a wide variety of tasks.*
* After creating a new database, you still need to connect to it!
* Exit psql with the command `exit`
# Exercise 1: Postgres Server
<!--_class: "exercise invert" -->
Start a local instance of Postgres in Docker
Connect to the server using psql
Use command `\l` to see what databases there are already created on the server.
Create a new database called "sqlpractice".
Connect to the newly created database.
# pgAdmin
Administration and development platform for PostgreSQL
Cross-platform, features a web interface
Basically a control panel application for your PostgreSQL database
A graphical alternative to psql
Completely separate from PostgreSQL, so pgAdmin is not required for using PostgreSQL. It is only one example of how you can administer the database. Other options include the forementioned psql and a wide variety of other database management tools, like [DBeaver](https://dbeaver.io/).
# Running pgAdmin in Docker
Using the official [pgAdmin](https://hub.docker.com/r/dpage/pgadmin4) image, we'll run pgAdmin alongside Postgres
```
docker run --name my-pgadmin -p 5050:80
--env PGADMIN_DEFAULT_EMAIL=<your-email-address>
--env PGADMIN_DEFAULT_PASSWORD=<your-password>
-d dpage/pgadmin4
```
# Logging into pgAdmin
With pgAdmin running, navigate your web browser to http://localhost:5050 and use the username and password you provided to log in.
# PostgreSQL internal IP Address
Both **PostgreSQL** and **pgAdmin** are now running in our *local* Docker. To connect pgAdmin to PostgreSQL, we need the IP address used inside Docker.
Using Docker's inspect, we'll get Docker's internal IP for my-postgres -container. Since the command produces quite a lot of information, we pipe the result to grep (Linux) or findstr (Windows), to see only the rows that contain the word "IPAddress"
In the example output, the IP Address is 172.17.0.2
# Connecting PgAdmin to our DB
Now we have all that we need for a connection. In pgAdmin, select "Object" > "Register" > "Server".
In the "General" tab, give the server a name that identifies the connection.
---
If Object menu is greyed out, click on Servers.
---
* In the "Connection" tab, enter
* Host name/address:
* the PostgreSQL internal Docker address
* Port, Username, Password
* the values defined when running the PostgreSQL container
* Then click Save. You should now see all the databases available on this server.
---
---
# Exercise 2: pgAdmin
<!--_class: "exercise invert" -->
Start a local instance of pgAdmin in Docker
Following lecture instructions, connect the pgAdmin to your already running PostgreSQL server.
Verify that you can see the database created in the previous assignment.
# PostgreSQL: Querying
With psql: After connecting to a database, just type a query and hit enter.
With pgAdmin:
Right-click a database > Query tool
Insert a query into the Query Editor and hit Execute (F5)
---
# Editing Data with pgAdmin
* Tables of Data in a DataBase are found under
* Database > Schemas > Tables
* Inspect and edit data in pgAdmin by right-clicking a table and selecting View/Edit Data
---
---
Individual values in the table can be directly modified by double clicking the value and then editing the value in the visual user interface
Save the changes with the Save Data Changes button
# Exercise 3: Preparing the Database
<!--_class: "exercise invert" -->
Using either PgAdmin or PSQL
Insert the [provided query](https://gitea.buutti.com/education/academy-assignments/src/branch/master/Databases/initdb.txt) to the database you created previously.
Verify that the query has created new tables to your database.
# Types of queries
- Select
- Insert
- Delete
- Update
- Create & Drop
# Querying data with SELECT
Syntax:
```
SELECT column1, column2, column3 FROM table_name;
```
Examples:
- `SELECT full_name, email FROM users;`
- `SELECT full_name AS name, email FROM users;`
- `SELECT * FROM users;`
# Filtering data with WHERE
Syntax:
```
SELECT column1, column2 FROM table_name WHERE condition;
```
Text is captured in **single** quotes. In a LIKE condition, % sign acts as a wildcard. IS and IS NOT are also valid comparison operators.
Example:
- `SELECT full_name FROM users;`
- `WHERE full_name = 'Teppo Testaaja';`
- `SELECT * FROM books WHERE name LIKE '%rr%';`
- `SELECT * FROM books WHERE author IS NOT null;`
# Ordering data with ORDER BY
Syntax:
```
SELECT column1 FROM table_name ORDER BY column1 ASC;
```
Examples:
- `SELECT full_name FROM users ORDER BY full_name ASC;`
- `SELECT full_name FROM users ORDER BY full_name DESC`
Since id is not provided, it will be automatically generated.
# UPDATE
Syntax
```
UPDATE table_nameSET column1 = value1, column2 = value2 WHERE condition;
```
Notice: if a condition is not provided, all rows will be updated! If updating only one row, it is usually best to use id.
Example
- `UPDATE usersSET email = 'taija.testaaja@gmail.com' WHERE id = 2;`
# REMOVE
Syntax
```
DELETE FROM table_name WHERE condition;
```
Again, if the condition is not provided, DELETE affects all rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected.
Example:
- `SELECT * FROM users WHERE id = 5;`
- First use SELECT to confirm the data you're about to delete
- `DELETE FROM users WHERE id = 5;`
- Then delete the data
# Exercise 5: Editing Data
<!--_class: "exercise invert" -->
Postpone the due date of the loan with an id of 2 by two days in the borrows table
Add a couple of new books to the books table
Delete one of the loans.
# CREATE TABLE
Before data can be manipulated, a database and its tables need to be initialized.
Syntax
```
CREATE TABLE table_name (
column1 datatype,
column2 datatype,
…
);
```
Example:
```
CREATE TABLE "users" (
"id" SERIAL PRIMARY KEY,
"full_name" varchar NOT NULL,
"email" varchar UNIQUE NOT NULL,
"created_at" timestamp NOT NULL
);
```
# DROP
In order to remove tables or databases, we use a DROP statement. Syntax:
```
DROP TABLE table_name;
DROP DATABASE database_name;
```
These statements do not ask for confirmation and there is no undo feature. Take care when using a drop statement.
# NoSQL
* In addition to SQL databases, there are also NoSQL databases
* Many differing definitions, but...
* most agree that NoSQL databases store data in a format other than tables
* They can still store relational data - just differently
* Four different database types:
* Document databases
* Key-value databases
* Wide-column stores
* Graph databases
* Example database engines include MongoDB, Redis and Cassandra
---
Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. MongoDB is consistently ranked as the world's most popular NoSQL database according to DB-engines and is an example of a document database. For more on document databases, visit What is a Document Database?.
---
Key-value databases are a simpler type of database where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don't need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Redis and DynanoDB are popular key-value databases.
---
Wide-column stores store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores.
---
Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Neo4j and JanusGraph are examples of graph databases.
# Object-Relational Mappers
* ORMs allow developers to manipulate databases with code instead of SQL queries
* For example, for performing CRUD operations on their database
Using our previously created, Dockerized Postgres instance, we'll create a Node application to connect to our database.
If you have deleted your postgres container, just create a new one with the same command.
If your container has shut down (after a reboot for example), you can start it with
```
docker start my-postgres
```
# Preparing our Node application
Initialize a new TypeScript application.
Initialize a new TypeScript application.
@ -508,9 +31,7 @@ Install [express](https://www.npmjs.com/package/express) and [PostgreSQL client
Install [dotenv](https://www.npmjs.com/package/dotenv), nodemon and ts-node development dependencies
Install [dotenv](https://www.npmjs.com/package/dotenv), nodemon and ts-node development dependencies
## Dotenv
# Dotenv
Example of `.env` file:
Example of `.env` file:
```
```
@ -530,7 +51,7 @@ Note that the example uses the default "postgres" database, but you can use any
# Dotenv (continued)
## Dotenv (continued)
```json
```json
{
{
@ -558,24 +79,21 @@ Note that the example uses the default "postgres" database, but you can use any
Dotenv is usually only used in development, not in production. In professional setting the dotenv config is often preloaded in the development startup script
Dotenv is usually only used in development, not in production. In professional setting the dotenv config is often preloaded in the development startup script
You can require dotenv when running npm run dev_
You can require dotenv when running `npm run dev`
-r is short for --require
`-r` is short for `--require`
# Dotenv and Git
## Dotenv and Git
* .env files usually contain sensitive data that we do _not_ want to store in Git repositories.
* `.env` files usually contain sensitive data that we do _not_ want to store in Git repositories.
* Thus, usually _.env_ file is excluded from the Git repository
* Thus, usually `.env` file is excluded from the Git repository
* Add _.env_ to _.gitignore
* Add `.env` to `.gitignore`
* If you have auto-generated .gitignore with `npx gitignore Node`, environment files are excluded automatically
* If you have auto-generated .gitignore with `npx gitignore Node`, environment files are excluded automatically
## Connecting to PostgreSQL
# Connecting to PostgreSQL
Our database file contains functions and configuration for initializing the Postgres pool, creating tables and running queries.
Our database file contains functions and configuration for initializing the Postgres pool, creating tables and running queries.
@ -626,7 +144,6 @@ At the moment our `index.ts` does nothing but creates a single table for our dat
It doesn't even have any endpoints, so it's not much of a server yet.
It doesn't even have any endpoints, so it's not much of a server yet.
```ts
```ts
import express from "express";
import express from "express";
import { createProductsTable } from "./db";
import { createProductsTable } from "./db";
@ -642,7 +159,7 @@ server.listen(PORT, () => {
# Launching the application
## Launching the application
Let's use our predefined `npm run dev`
Let's use our predefined `npm run dev`
@ -655,17 +172,17 @@ Let's use our predefined `npm run dev`
Epic success!
Epic success!
b
## Exercise 1: Node & PostgreSQL
# Exercise 6: Node & PostgreSQL
<!--_class: "exercise invert" -->
<!--_class: "exercise invert" -->
Following the lecture example, create an Express server that connects to your local PostgreSQL instance. The database information should be stored in environment variables. When the server starts, it should create a product table with three columns: id (serial, primary key), name (varchar) and price (real).
Following the lecture example, create an Express server that connects to your local PostgreSQL instance. The database information should be stored in environment variables. When the server starts, it should create a product table with three columns: id (serial, primary key), name (varchar) and price (real).
# Creating Queries
## Creating Queries
* Next, we will create an actual CRUD API for communicating with the database.
* Next, we will create an actual CRUD API for communicating with the database.
* For that we need endpoints for creating, reading, updating and deleting products.
* For that we need endpoints for creating, reading, updating and deleting products.
@ -673,7 +190,7 @@ Following the lecture example, create an Express server that connects to your lo
# Using queries
## Using queries
* We'll use the pre-made executeQuery() function for querying the database from a few slides back
* We'll use the pre-made executeQuery() function for querying the database from a few slides back
* It takes in two arguments:
* It takes in two arguments:
@ -684,17 +201,17 @@ Following the lecture example, create an Express server that connects to your lo
# Parameterized queries example
## Parameterized queries example
When running executeQuery(query, parameters) with above values defined, the query would be parsed as
When running `executeQuery(query, parameters)` with the above values defined, the query would be parsed as
```
```sql
SELECT * FROM cats WHERE color = 'yellow' and age > 10;
SELECT * FROM cats WHERE color = 'yellow' and age > 10;
```
```
# Why not just use String templating?
## Why not just use String templating?
…Because of [SQL injections](https://fi.wikipedia.org/wiki/SQL-injektio) . Always use database library's built-in parameterization!
…Because of [SQL injections](https://fi.wikipedia.org/wiki/SQL-injektio) . Always use database library's built-in parameterization!
@ -706,7 +223,7 @@ _NEVER DO THIS!!!_
# Creating queries
## Creating queries
We will create a Data Access Object, `dao.ts` that will handle interacting with our database.
We will create a Data Access Object, `dao.ts` that will handle interacting with our database.
@ -751,7 +268,7 @@ The router that declares the endpoints, uses the DAO to interact with the databa
# Testing the API
## Testing the API
<divclass='columns'markdown='1'>
<divclass='columns'markdown='1'>
<divmarkdown='1'>
<divmarkdown='1'>
@ -774,7 +291,7 @@ We can also use psql to observe the changes in the database
# Exercise 7: Creating Queries
## Exercise 2: Creating Queries
<!--_class: "exercise invert" -->
<!--_class: "exercise invert" -->
Continue following the lecture example. Create a router and a database access object to handle
Continue following the lecture example. Create a router and a database access object to handle
@ -786,9 +303,9 @@ Continue following the lecture example. Create a router and a database access ob
# Dockerized PostgreSQL App
## Dockerized PostgreSQL App
# Setting Environment Variables
## Setting Environment Variables
Docker has two kinds of environment variables: run-time and build-time.
Docker has two kinds of environment variables: run-time and build-time.
borb
4 weeks ago
