22 KiB
SQL
- __Language __ to organize and manipulate data in a database
- Originally developed by IBM in the 70's
- Quickly became the most popular database language
- SELECT id, email
- FROM users
- WHERE first_name = 'Teppo';
Relational Database Management Systems
- In relational databases, values are stored in tables
- Each table has __rows __ and columns
- Data is displayed in a two-dimensional matrix
- Values in a table are related to each other
- Values can also be related to values in other tables
- A relational database management system (RDBMS) is a program, which executes queries to relational databases
https://db-engines.com/en/ranking
PostgreSQL
Free open-source, cross-platform relational database management system
Emphasizes extensibility and SQL compliance
Fully ACID-compliant (atomicity, consistency, isolation and durability)
pgAdmin
Administration and development platform for PostgreSQL
Cross-platform, features a web interface
Basically a control panel application for your PostgreSQL database
PostgreSQL: Creating a Database
With pgAdmin:
Right-click Servers > my-postgres > Databases
Select _Create _ > Database…
Insert a name for the database _ _ and hit Save
With psql:
Enter commandCREATE DATABASE ;
PostgreSQL: Querying
With pgAdmin:
Right-click _sqlpractice _ > Query tool...
Insert a query into the Query Editor and hit _Execute _ (F5)
With psql:
List all available databases with \l
Connect to the created database with \c
List all tables in the database with \dt
Type a query and press enter
https://drive.google.com/file/d/1E5TqY5yt8DNcWHVkb8gxvBpgLtWB15DC/view?usp=sharing "provided query"
Editing Data with pgAdmin
Inspect and edit data in pgAdmin by right-clicking a table and selecting View/Edit Data
Individual values in the table can be directly modified by double clicking the value and then editing the value in the visual user interface
Save the changes with the Save Data Changes button
Exercise 1: Preparing Database
Using either PgAdmin or PSQL
Create a new database called sqlpractice
Insert the provided query to the new database
Verify that the query has created new tables to your database
Types of queries
Select
Insert
Delete
Update
Create & Drop
Querying data with SELECT
Syntax:
SELECT column1, column2, column3 FROM table_name;
Examples:
SELECT full_name, email FROM users;
SELECT full_name AS name, email FROM users;
SELECT * FROM users;
Filtering data with WHERE
Syntax:
SELECT column1, column2 FROM table_name WHERE condition;
Text is captured in single quotes .
LIKE condition uses % sign as wildcard.
IS and IS NOT are also valid operators.
Example:
SELECT full_name FROM users
WHERE full_name = 'Teppo Testaaja';
SELECT * FROM books WHERE name LIKE '%rr%';
SELECT * FROM books WHERE author IS NOT null;
Ordering data with ORDER BY
Syntax:
SELECT column1 FROM table_name ORDER BY column1 ASC;
Examples:
SELECT full_name FROM users
ORDER BY full_name ASC;
SELECT full_name FROM users
ORDER BY full_name DESC
Combining with JOIN
Also known as INNER JOIN
Corresponds to intersection from set theory
JOIN examples
SELECT
users.id, users.full_name, borrows.id,
borrows.user_id, borrows.due_date, borrows.returned_at
FROM users
JOIN borrows ON
users.id = borrows.user_id;
SELECT
U.full_name AS name,
B.due_date AS due_date,
B.returned_at AS returned_at
FROM users AS U
JOIN borrows AS B ON
U.id = B.user_id;
Combining with LEFT JOIN
Also known as LEFT OUTER JOIN
Example:
SELECT
U.full_name AS name,
B.due_date AS due_date,
B.returned_at AS returned_at
FROM users AS U
LEFT JOIN borrows AS B ON
U.id = B.user_id;
Exercise 2: Querying the Library
Using SQL queries, get
all columns of borrows that are borrowed before 2020-10-27
all columns of borrows that are returned
columns user.full_name and borrows.borrowed_at of the user with an id of 1
columns book.name, book.release_year and language.name of all books that are released after 1960
INSERT
Syntax
_INSERT INTO table_name (column1, column2, column3) VALUES (value1, value2, value3);
Example
INSERT INTO users (full_name, email, created_at)
VALUES ('Pekka Poistuja', 'pekka.poistuja@buutti.com', NOW());
Since id is not provided, it will be automatically generated.
UPDATE
Syntax
UPDATE table_name SET column1 = value1, column2 = value2 WHERE condition;
Notice: if a condition is not provided, all rows will be updated!If updating only one row, it is usually best to use id.
Example
UPDATE usersSET email = 'taija.testaaja@gmail.com'WHERE id = 2;
REMOVE
Syntax
DELETE FROM table_name WHERE condition;
Again, if the condition is not provided, DELETE affects all rows. Before deleting, it is a good practice to execute an equivalent SELECT query to make sure that only the proper rows will be affected.
Example:
SELECT * FROM __users __ WHERE id = 5;
DELETE FROM users WHERE id = 5;
Exercise 3: Editing Data
Postpone the due date of the borrow with an id of 1 by two days in the _borrows _ table
Add a couple of new books to the _books _ table
Delete one of the borrows.
CREATE TABLE
Before data can be manipulated, a database and its tables need to be initialized.
Syntax
CREATE TABLE table_name (
__ column1 datatype,__ __ column2 datatype,__ __ …__ );
Example:CREATE TABLE "users" (
"id" SERIAL PRIMARY KEY,
"full_name" varchar NOT NULL,
"email" varchar UNIQUE NOT NULL,
"created_at" timestamp NOT NULL
);
DROP
In order to remove tables or databases, we use a DROP statement
DROP TABLE table_name; DROP DATABASE database_name;
These statements do not ask for confirmation and there is no undo feature. Take care when using a drop statement.
NoSQL
- Many differing definitions, but...
- most agree that NoSQL databases store data in a format other than tables
- They can still store relational data - just differently
- Four different database types:
- Document databases
- Key-value databases
- Wide-column stores
- Graph databases
- Example database engines include MongoDB, Redis and Cassandra
Document databases store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. MongoDB is consistently ranked as the world's most popular NoSQL database according to DB-engines and is an example of a document database. For more on document databases, visit What is a Document Database?. Key-value databases are a simpler type of database where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don't need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Redis and DynanoDB are popular key-value databases. Wide-column stores store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores. Graph databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Neo4j and JanusGraph are examples of graph databases.
Object-Relational Mappers
Allows a developer to write code instead of SQL to perform CRUD operations on their database
Some popular ORMs:
Hibernate (Java)
EFCore (.NET)
Sequelize (Node.js)
TypeORM (TypeScript)
Databases with Entity Framework
Entity Framework (EF)
- ORM made by Microsoft for the .NET framework
- Object-Relational Mapping: converting from database representation to objects in a programming language
- Allows creation of CRUD operations without writing SQL
Entity Framework Core (EFC)
Cross-platform, can be used outside of the .NET framework unlike normal Entity Framework
Open-source, lightweight, extensible
Supports many database engines, such as MySQL, PostgreSQL, and so on
This is what we'll be using
Notes: ORM - Object Relational Mapper CRUD - CREATE READ UPDATE DELETE
Code First, Database First & Model First
Three approaches through which Entity Framework is implemented
Database First and Code First are the most used ones and will be taught in this lecture
Code First approach
Code First
- Entity framework will create databases and tables based on defined entity classes
- Good for small applications
- Other advantages include:
- You can create database and tables from your business objects
- You can specify which related collections are…
- eager loaded
- not serialized at all
- Database version control
- Not preferred for data intensive applications
Required Packages
Install and add the following packages to your project:
Microsoft.EntityFrameworkCore
Microsoft.EntityFrameworkCore.Tools
Npgsql.EntityFrameworkCore.PostgreSQL
Code First: DbContext
- Let's begin with the Code First Approach
- The DbContext class of EFCore is the bridge between the code representation of your data (entities) and the database
- DbContext holds methods to form the database schema with Code First approach and classes to keep the database up-to-date with CRUD operations
- DATABASE -> CODE: DbSet class property in DbContext can be queried directly with LINQ and this results in an object in your code
- CODE -> DATABASE: DbSet also has methods like Add, Update and Remove to make changes to the database from your code
DbContext
- Create a context that inherits from DbContext
- Commonly located in the Models folder, but ideally should be in a separate abstraction/repository folder (for example Repositories)
- The class needs to have a constructor which calls the base constructor with : base(options)
- Create a DbSet property for each resource
- public class ContactsContext : DbContext
- {
- public DbSet Contacts { get; set; }
- public ContactsContext(DbContextOptions options) : base(options){ }
- }
DbContext (continued)
To further configure how the database will be structured, override the OnModelCreating method
In this example, one table named Contact with columns Id, Name and Email will be created:
public class ContactsContext : DbContext
{
public DbSet Contacts { get; set; }
public ContactsContext(DbContextOptions options) : base(options){ }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity().ToTable("Contact");
}
}
In this example, the Contact table will be created with some starting values for Id, Name and Email columns:
public class ContactsContext : DbContext
{
public DbSet Contacts { get; set; }
public ContactsContext(DbContextOptions options) : base(options){ }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity().HasData(
new Contact { Id = 1, Name = "Johannes Kantola", Email = "johkant@example.com" },
new Contact { Id = 2, Name = "Rene Orosz", Email = "rene_king@example.com" }
);
}
}
DbContext as a Service
- In Program.cs, add the context to services with AddDbContext method
- This is where you set the DB management system you want to use (MySQL, PostgreSQL, SQLite)
- The EFCore support for PostgreSQL is called Npgsql __ __ as in the package name
- Add the server, host, port, username, password and database name of the existing database inside options.UseNpgsql as a connection string:
- services.AddDbContext(options => options.UseNpgsql(
- @"Server=PostgreSQL 12;Host=localhost;Port=5432;Username=postgres;Password=1234;Database=contacts"));
- services.AddScoped<IContactRepository, ContactRepository>();
- services.AddControllers().AddNewtonsoftJson();
Migrations
- As the development progresses, models and database schemas change over time
- This means that both the database and the code needs to be updated to match each other
- Migrations allow for the database to keep in sync with the code schematically
- The data stored in the database is also preserved
- EFCore migrations have built-in version control; a snapshot of each version of the schema is stored
Migrations (continued)
- Open the Package Manager Console in Visual Studio
- If the tab is not in the bottom of the window, open it from _View _ -> Other windows -> Package Manager Console
- Add your initial migration by entering the command Add-Migration to the console, for example
- Add-Migration InitialMigration
- This now creates the first "blueprint" of how the database should be structured
- Update the database by entering the command Update-Database to the console
- This will update the existing database according to the ModelBuilder options
At this point, the values you have entered (Contacts table in this example) should show up in the database. You can check it up e.g. in pgAdmin.
- Notice that the table and column names are initialized with a capital letter
- The value naming in psql is case sensitive, so all names have to be in quotation marks
Exercise 4: Adding Context
Continue working on the CourseAPI.
Create a new empty database course_db in pgAdmin or psql
Create a DbContext for the courses. Name it CoursesContext, and add a DbSet of type Course to it, named Courses
Add the OnModelCreating method to the context and add a couple of courses with some starting values to the modelBuilder
Add the CoursesContext to the services in Program.cs with a connection string pointing to course_db
Add the first migration and update the database from the Package Manager Console
Check that the Course table with the starting values has appeared to the database
Using DbContext in the API
- Because DbContext is added to services, it can be accessed from any other service, such as the repository
- Using the DbSet for each model in your project, CRUD operations can be applied to the database from the repository with LINQ and DbSet methods
- Add()
- Update()
- Remove()
- After modifying the DbSet, update the changes to the database with DbContext.SaveChanges() method
Injecting DbContext
Inject the DbContext to your repositories as you would any other service:
public class ContactRepository : IContactRepository
{
private readonly ContactsContext _context;
public ContactRepository(ContactsContext context)
{
_context = context;
}
//...
}
DbContext: Read Operations
public class ContactRepository : IContactRepository
{
private readonly ContactsContext _context;
public ContactRepository(ContactsContext context) { ... }
public Contact GetContact(int id) =>
_context.Contacts.FirstOrDefault(c => c.Id == id);
public List GetContacts() =>
_context.Contacts.ToList();
}
DbContext: Create Operations
public class ContactRepository : IContactRepository
{
private readonly ContactsContext _context;
public ContactRepository(ContactsContext context) { ... }
// Read operations
// …
public void AddContact(Contact contact)
{
_context.Contacts.Add(contact);
_context.SaveChanges();
}
}
DbContext: Update Operations
public class ContactRepository : IContactRepository
{
private readonly ContactsContext _context;
public ContactRepository(ContactsContext context) { ... }
// Read & create operations
// ...
public void UpdateContact(int id, Contact newContact)
{
var contact = GetContact(id);
contact.Email = newContact.Email;
contact.Name = newContact.Name;
_context.Contacts.Update(contact);
_context.SaveChanges();
}
}
DbContext: Delete Operations
public class ContactRepository : IContactRepository
{
private readonly ContactsContext _context;
public ContactRepository(ContactsContext context) { ... }
// Read, create & update operations
// ...
public void DeleteContact(int id)
{
_context.Contacts.Remove(GetContact(id));
_context.SaveChanges();
}
}
Exercise 5: CRUD on the DB
Continue working on CourseAPI.
Modify the CourseRepository to create, read, update and delete from the database instead of the locally stored list of courses
Test with Postman. Keep refreshing the DB in pgAdmin or creating queries with psql to make sure the request work as intended
Summing Things Up
Now the API has been hooked up to a PostgreSQL database
Changes to the schema are kept up-to-date with migrations
Repository is processing CRUD operations to the database
Controllers accepting HTTP requests have access to the repository
EFCore Code First Checklist
Install required packages
Create DbContext for database
Add DbContext to services
Add-Migration & Update-Database
Add CRUD operations to repository for DB
Modifying the Relations
- Let's change the structure of our Contacts API by adding a new class, Account
- Instead of Contact directly having an Email, it will have an Account instead
- Account holds the information about the Email, as well as a Description about the nature of the account (personal, work, school etc.)
- Emails will be removed from the Contacts table
Modifying the Relations (continued)
public class Contact
{
public int Id { get; set; }
public string Name { get; set; }
public ICollection Accounts { get; set; }
}
public class Account
{
public int Id { get; set; }
public string Email { get; set; }
public string Description { get; set; }
public int ContactId { get; set; }
public Contact Contact { get; set; }
}
Adding a migration at this point will result in a warning:
- In the generated migration file, you can find Up and Down methods
- The Up method describes the changes that will be made with the migration
- In this case, removing the Email column from Contacts table, and creating the new Accounts table
- The Down method describes the changes that will be made if the migration is reverted
- The Up method describes the changes that will be made with the migration
- Updating the database will still work, and the database will have a new table, Accounts
Exercise 6: Adding Migrations
Continue working on CourseAPI.
Add a new model Lecture with properties int Id, DateTime StartTime, int Length, Course Course, and int CourseId
Add a new property ICollection Lectures to the Course model
Add a new migration, named AddLectures
Update the database. Check that the changes show up in the database with pgAdmin
Database First approach
Database First
- This is the other approach for creating a connection between the database and the application
- Databases and tables are created first, then you create entity data model using the created database
- Preferred for data intense, large applications
- Other advantages include:
- Data model is simple to create
- GUI
- You do not need to write any code to create your database
Database First (continued)
- Use the Package Manager Console to "reverse engineer" the code for an existing database
- This is called scaffolding
- Scaffold the database with the following command:
- Scaffold-DbContext "Server=PostgreSQL 12;Host=localhost;Port=5432;Username=postgres;Password=1234;Database=sqlpractice" Npgsql.EntityFrameworkCore.PostgreSQL -OutputDir Models
- Using the connection string corresponding to your database, this will create all the classes for the entities in the DB as well as the context class
Exercise 7: Database First
Create a new ASP.NET Core web app using the API template.
Install the required NuGet packages for using EFCore, EFCore Tools and PostgreSQL by using the package manager, or by copying the lines from the .csproj file of the previous assignment to this projects .csproj file
Scaffold the _sqlpractice _ database (which was created in the Exercise 1) to the project using the Database First approach. If you have not yet created the database in PostGreSQL, it can be found here
Reading: Authentication with roles
- Here's an example how to do Role-based Authentication by using JWT tokens
Extra: Exercise 8: Connection
[under construction]
Continuing the previous exercise,
Create and connect Postgres database to API and create a second entity with a relation to the first entity. Test your solution.