diff --git a/3-databases-with-docker-slides.html b/3-databases-with-docker-slides.html index db166ae..6d64d3a 100644 --- a/3-databases-with-docker-slides.html +++ b/3-databases-with-docker-slides.html @@ -1,4 +1,5 @@ -
Administration and development platform for PostgreSQL
-Cross-platform, features a web interface
-Basically a control panel application for your PostgreSQL database
-A graphical alternative to psql
-Completely separate from PostgreSQL, so pgAdmin is not required for using PostgreSQL. It is only one example of how you can administer the database. Other options include the forementioned psql and a wide variety of other database management tools, like DBeaver.
-Using the official pgAdmin image, we'll run pgAdmin alongside Postgres
-docker run --name my-pgadmin -p 5050:80
---env PGADMIN_DEFAULT_EMAIL=<your-email-address>
---env PGADMIN_DEFAULT_PASSWORD=<your-password>
--d dpage/pgadmin4
-
-With pgAdmin running, navigate your web browser to http://localhost:5050 and use the username and password you provided to log in.
-Both PostgreSQL and pgAdmin are now running in our local Docker. To connect pgAdmin to PostgreSQL, we need the IP address used inside Docker.
-Using Docker's inspect, we'll get Docker's internal IP for my-postgres -container. Since the command produces quite a lot of information, we pipe the result to grep (Linux) or findstr (Windows), to see only the rows that contain the word "IPAddress"
-docker inspect <container-name> | grep IPAddress
-docker inspect <container-name> | findstr IPAddress
-
-In the example output, the IP Address is 172.17.0.2
-Now we have all that we need for a connection. In pgAdmin, select "Object" > "Register" > "Server".
-In the "General" tab, give the server a name that identifies the connection.
-If Object menu is greyed out, click on Servers.
-Start a local instance of pgAdmin in Docker
-Following lecture instructions, connect the pgAdmin to your already running PostgreSQL server.
-Verify that you can see the database created in the previous assignment.
-With psql: After connecting to a database, just type a query and hit enter.
-With pgAdmin:
-Right-click a database > Query tool
-Insert a query into the Query Editor and hit Execute (F5)
-Individual values in the table can be directly modified by double clicking the value and then editing the value in the visual user interface
-Save the changes with the Save Data Changes button
-Using either PgAdmin or PSQL
-Insert the provided query to the database you created previously.
-Verify that the query has created new tables to your database.
-Syntax:
-SELECT column1, column2, column3 FROM table_name;
-
-Examples:
-SELECT full_name, email FROM users;
SELECT full_name AS name, email FROM users;
SELECT * FROM users;
Syntax:
-SELECT column1, column2 FROM table_name WHERE condition;
-
-Text is captured in single quotes. In a LIKE condition, % sign acts as a wildcard. IS and IS NOT are also valid comparison operators.
-Example:
-SELECT full_name FROM users;
WHERE full_name = 'Teppo Testaaja';
SELECT * FROM books WHERE name LIKE '%rr%';
SELECT * FROM books WHERE author IS NOT null;
Syntax:
-SELECT column1 FROM table_name ORDER BY column1 ASC;
-
-Examples:
-SELECT full_name FROM users ORDER BY full_name ASC;
SELECT full_name FROM users ORDER BY full_name DESC
Also known as INNER JOIN
-Corresponds to intersection from set theory
-SELECT users.id, users.full_name, borrows.id, borrows.user_id,
-borrows.due_date, borrows.returned_at
-FROM users JOIN borrows ON users.id = borrows.user_id;
-
-SELECT U.full_name AS name, B.due_date AS due_date, B.returned_at AS returned_at
-FROM users AS U JOIN borrows AS B ON U.id = B.user_id;
-
-Using SQL queries, get
-Syntax
-INSERT INTO table_name (column1, column2, column3) VALUES (value1, value2, value3);
-
-Example
-INSERT INTO users (full_name, email, created_at) VALUES ('Pekka Poistuja', 'pekka.poistuja@buutti.com', NOW());
Since id is not provided, it will be automatically generated.
-Syntax
-UPDATE table_nameSET column1 = value1, column2 = value2 WHERE condition;
-
-Notice: if a condition is not provided, all rows will be updated! If updating only one row, it is usually best to use id.
-Example
-UPDATE usersSET email = 'taija.testaaja@gmail.com' WHERE id = 2;
Syntax
-DELETE FROM table_name WHERE condition;
-
-Again, if the condition is not provided, DELETE affects all rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected.
-Example:
+SELECT * FROM users WHERE id = 5;
+DELETE FROM users WHERE id = 5;
+Postpone the due date of the loan with an id of 2 by two days in the borrows table
-Add a couple of new books to the books table
-Delete one of the loans.
-Before data can be manipulated, a database and its tables need to be initialized.
-Syntax
-CREATE TABLE table_name (
- column1 datatype,
- column2 datatype,
- …
-);
+If your container has shut down (after a reboot for example), you can start it withdocker start my-postgres
-Example:
-CREATE TABLE "users" (
- "id" SERIAL PRIMARY KEY,
- "full_name" varchar NOT NULL,
- "email" varchar UNIQUE NOT NULL,
- "created_at" timestamp NOT NULL
-);
-
-
In order to remove tables or databases, we use a DROP statement. Syntax:
-DROP TABLE table_name;
-DROP DATABASE database_name;
-
-These statements do not ask for confirmation and there is no undo feature. Take care when using a drop statement.
-Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. MongoDB is consistently ranked as the world's most popular NoSQL database according to DB-engines and is an example of a document database. For more on document databases, visit What is a Document Database?.
-Key-value databases are a simpler type of database where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don't need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Redis and DynanoDB are popular key-value databases.
-Wide-column stores store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores.
-Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Neo4j and JanusGraph are examples of graph databases.
-Using our previously created, Dockerized Postgres instance, we'll create a Node application to connect to our database.
-If you have deleted your postgres container, just create a new one with the same command.
-If your container has shut down (after a reboot for example), you can start it with
-docker start my-postgres
-
-Initialize a new TypeScript application.
Install express and PostgreSQL client for Node.JS, and their respective TypeScript types
Install dotenv, nodemon and ts-node development dependencies
Example of .env
file:
PORT=3000
+PORT=3000
PG_HOST=localhost
PG_PORT=5432
PG_USERNAME=pguser
@@ -437,9 +55,9 @@ PG_DATABASE=postgres
The database must exist as well.
Note that the example uses the default "postgres" database, but you can use any database you want.
{
+
{
"name": "products_api",
"version": "1.0.0",
"scripts": {
@@ -460,29 +78,29 @@ PG_DATABASE=postgres
}
Dotenv is usually only used in development, not in production. In professional setting the dotenv config is often preloaded in the development startup script
-You can require dotenv when running npm run dev_
--r is short for --require
+You can require dotenv when running npm run dev
-r
is short for --require
.env
files usually contain sensitive data that we do not want to store in Git repositories..env
file is excluded from the Git repository
npx gitignore Node
, environment files are excluded automatically.env
to .gitignore
npx gitignore Node
, environment files are excluded automaticallyOur database file contains functions and configuration for initializing the Postgres pool, creating tables and running queries.
At the moment, we have only one query. It is a single-use query that creates a products table to the database if such table does not yet exist.
-// db.ts
+// db.ts
import pg from "pg";
const { PG_HOST, PG_PORT, PG_USERNAME, PG_PASSWORD, PG_DATABASE } = process.env;
@@ -494,7 +112,7 @@ PG_DATABASE=postgres
database: PG_DATABASE,
});
-const executeQuery = async (query: string, parameters?: Array<any>) => {
+const executeQuery = async (query: string, parameters?: Array<any>) => {
const client = await pool.connect();
try {
const result = await client.query(query, parameters);
@@ -519,11 +137,10 @@ PG_DATABASE=postgres
};
At the moment our index.ts
does nothing but creates a single table for our database and launches express server.
It doesn't even have any endpoints, so it's not much of a server yet.
-
-import express from "express";
+import express from "express";
import { createProductsTable } from "./db";
const server = express();
@@ -536,23 +153,24 @@ server.listen(
-Launching the application
+
Let's use our predefined npm run dev
…And check with psql that our application succeeds in connecting to the database and creating the table.
Epic success!
-
+b
Following the lecture example, create an Express server that connects to your local PostgreSQL instance. The database information should be stored in environment variables. When the server starts, it should create a product table with three columns: id (serial, primary key), name (varchar) and price (real).
When running executeQuery(query, parameters) with above values defined, the query would be parsed as
-SELECT * FROM cats WHERE color = 'yellow' and age > 10;
+When running executeQuery(query, parameters)
with the above values defined, the query would be parsed as
+SELECT * FROM cats WHERE color = 'yellow' and age > 10;
…Because of SQL injections . Always use database library's built-in parameterization!
NEVER DO THIS!!!
We will create a Data Access Object, dao.ts
that will handle interacting with our database.
The idea is that we want to just tell our DAO what we want done (e.g. "add this customer to the database") and the DAO will handle the details of that action.
The DAO will also return possible additional information that was created during the action.
Our insertProduct function
generates a new, unique ID for the product using uuid
constructs a parameters array containing said id, the name of the product and the price of the product
@@ -607,7 +225,7 @@ server.listen(The rest of the DAO operations work in similar fashion.
The router that declares the endpoints, uses the DAO to interact with the database.
Now we can use Insomnia to verify that all the endpoints work as expected.
@@ -634,8 +252,8 @@ server.listen(Continue following the lecture example. Create a router and a database access object to handle
Docker has two kinds of environment variables: run-time and build-time.
In this scenario we want to set our environment variables at build time. This means that the Docker image will contain all the environment variable information, including sensitive things like passwords. This might be an issue in some scenarios. In those cases the environment variables need to be set at run time.
In the Dockerfile we set the build-time values by setting ARG parameters. Then we use these values to set the run-time environment variables by setting ENV parameters.
More information: https://vsupalov.com/docker-arg-env-variable-guide/
When the ARGs and ENVs have been set in the Dockerfile, we provide the ARG values when building the Docker image by using_--build-arg
docker build
+docker build
--build-arg PORT=3000
--build-arg PG_HOST=https://my.postgres.server
--build-arg PG_PORT=5432
@@ -668,9 +286,9 @@ server.listen(
+
And include the build-arg parameters in our Dockerfile with them mapped to environment variables.
-ARG PORT
+ARG PORT
ARG PG_HOST
ARG PG_PORT
ARG PG_USERNAME
@@ -686,12 +304,12 @@ ENV PG_DATABASE=${PG_DATABASE}
Dockerize the application you have built. Build the docker image, run the app and test that it works using insomnia/postman.
Remember that when you run the application on your local Docker, both the app and the database are in the same Docker network, so you have to check the database IP address just like when running pgAdmin.