Mariadb-Developer-Examples/bookings

Attention - Please Read!

This application is no longer being maintained here. It has been moved to this repository.

Bookings

Bookings is a web application written in React.js and Node.js that, backed by the power of the MariaDB Node.js Connector and the MariaDB X4 Platform, unleashes the power of smart transactions on hundreds of millions of records with sub-second query performance without having to add any indexes!

The following will walk you through the steps for getting this application up and running (locally) within minutes! This application is completely open source. Please feel free to use it and the source code as you see fit.

Table of Contents

1. Environment and Compatibility
2. Getting started with MariaDB and Hybrid Transactional-Analytical Processing
   1. The Basics
   2. Deploying MariaDB HTAP
   3. Clone repo
   4. Retrieving flight data
   5. Create the schema
   6. Load airport, airline, and flight data
   7. Setting up replication
3. Requirements to run the app
4. Getting started with the app
   1. Grab the code
   2. Build the code
   3. Run the app
   4. Adding app data
5. Smart Transactions
6. Cross-Engine Queries
7. Support and Contribution
8. License

Environment and Compatibility

This sample was created using the following techologies:

• MariaDB X4 HTAP (v10.4)
• React.js (v.16.12.0)
• Node.js (v.12.x)
• NPM (v.6.11.3)

This sample was tested on macOS v.10.14.6.

Overview

Introduction to MariaDB

MariaDB platform unifies transactional and analytical information so applications can leverage powerful, real-time analytics in order to provide data-driven customers with more information, actionable insight and greater value.

Deploying MariaDB Hybrid Transactional-Analytical Processing (HTAP)

MariaDB Platform supports Hybrid Transactional-Analytical Processing (HTAP) through a combination of MariaDB Enterprise Server, MariaDB ColumnStore, and MariaDB MaxScale.

Here's a simple architecture diagram of MariaDB X4 Platform.

For this application we'll be targeting a single instance of MariaDB HTAP, and instructions for setting such an environment can be found here:

https://github.com/mariadb-corporation/mariadb-columnstore-htap

Once you have created your HTAP instance you will be able to create the schemas and load the data. If you have elected to use the method above you simply need to use

$ vagrant ssh node1

to access the database instance.

Note: You can also find more details on how to manually deploy MariaDB X4 here.

Clone repo

Next git clone this repository to the machine that contains your database instance, and then proceed to the following steps for retrieving flight data, creating schemas, and loading airports/airlines/flight data.

Retrieving flight data

This application uses flight record data provided by the Bureau of Transportation.

The following script will retrieve the data set by year and month creating CSV data files under the data directory. By default the script will retrieve data for all months from 1990 to 2020. The script can be edited to retrieve smaller or larger data ranges as needed:

$ ./get_flight_data.sh

2018-011
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   654  100   183  100   471     12     33  0:00:15  0:00:14  0:00:01     0
100 14.9M  100 14.9M    0     0   415k      0  0:00:36  0:00:36 --:--:--  736k
Archive:  data.zip
  inflating: 544122566_.csv
2018-02

Note: The script makes use of curl and unzip which may need to be installed if not already present on your Linux OS.

Create the schema

The following script will create (and drop if it already exists) the flights database:

$ ./create_schemas.sh

This includes the following 6 tables within the schema innodb_schema:

• airlines
• airports
• flights
• tickets
• trips

This also includes the following table within the schema columnstore_schema:

• flights

Loading Flights data into ColumnStore

The airlines and airports table can be populated from the airlines.csv and airports.csv files in the schema directory. Use the following script to do this:

$ ./load_airports_airlines.sh

The flights table is populated using a script which will load each CSV file from the data directory into the flights table:

$ ./load_flight_data.sh

Setting up HTAP Replication

Using MariaDB Replication, MariaDB Enterprise Server replicates writes from InnoDB tables to the MariaDB ColumnStore tables, ensuring that the application can perform analytical processing on current data. Combining MariaDB Replication with MariaDB MaxScale configured as a Binlog Server, MariaDB Enterprise Server can host InnoDB and ColumnStore on the same Server.

This application uses replication on a single table called flights, which exists innodb_schema and columnstore_schema. In order to set up replication add the following to /etc/maxscale.cnf for your HTAP instance.

[replication-filter]
type         = filter
module       = binlogfilter
match        = /[.]flights/
rewrite_src  = innodb
rewrite_dest = columnstore

For more information on configuring MariaDB HTAP please review the official Enterprise Documentation.

Requirements to run the app

This project assumes you have familiarity with building web applications using ReactJS and NodeJS technologies.

The following is required to run this application:

1. Download and install MariaDB HTAP.
2. Download and install NodeJS.
3. git (Optional) - this is required if you would prefer to pull the source code from GitHub repo.
   • Create a free github account if you don’t already have one
   • git can be downloaded from git-scm.org

Getting started with the app

Grab the code

Download this code directly or use git (through CLI or a client) to retrieve the code.

Configure the code

Configure the MariaDB connection by adding an .env file to the Node.js project.

Example implementation:

DB_HOST=<host_address>
DB_PORT=<port_number>
DB_USER=<username>
DB_PASS=<password>
DB_NAME=<database>

The environmental variables from .env are used within the db.js for the MariaDB Node.js Connector configuration pool settings:

var mariadb = require('mariadb');
require('dotenv').config();

const pool = mariadb.createPool({
    host: process.env.DB_HOST, 
    user: process.env.DB_USER, 
    password: process.env.DB_PASS,
    port: process.env.DB_PORT,
    multipleStatements: true,
    connectionLimit: 5
});

Build the code

Once you have retrieved a copy of the code you're ready to build and run the project! However, before running the code it's important to point out that the application uses several Node Packages.

Executing the CLI command

npm install

within

• src: the Node.js project
• client: the React.js project

folders will target the the relative package.json file and install all dependencies.

Run the app

Once you've pulled down the code and have verified that all of the required Node packages are installed you're ready to run the application! It's as easy as 1,2,3.

1. Using a command line interface (CLI) navigate to the src directory.

2. Run the command:

npm start

3. Open a browser window and navigate to http://localhost:3000.

Adding app data

Upcoming flight data

Upon running the application you will notice that searching for flights and viewin upcoming trips yields no results. This is because there currently no transactional flights, tickets, or trips data. Because this application is merely meant for demonstration purposes only you will need to provide relevant data within the following:

• innodb_schema.flights
• innodb_schema.tickets
• innodb_schema.trips

The following are sample datasets for:

An upcoming flight (option) from LAX to ORD on February 5th, 2020.

INSERT INTO `flights` (`year`, `month`, `day`, `day_of_week`, `fl_date`, `carrier`, `tail_num`, `fl_num`, `origin`, `dest`, `crs_dep_time`, `dep_time`, `dep_delay`, `taxi_out`, `wheels_off`, `wheels_on`, `taxi_in`, `crs_arr_time`, `arr_time`, `arr_delay`, `cancelled`, `cancellation_code`, `diverted`, `crs_elapsed_time`, `actual_elapsed_time`, `air_time`, `distance`, `carrier_delay`, `weather_delay`, `nas_delay`, `security_delay`, `late_aircraft_delay`) VALUES (2020, 2, 5, 5, '2020-02-05', 'DL', NULL, 1280, 'LAX', 'ORD', '0600', '0600', NULL, NULL, NULL, NULL, NULL, '0913', '0913', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO `tickets` (`id`, `fl_date`, `fl_num`, `carrier`, `origin`, `dest`, `price`) VALUES (1, '2020-02-05', 1280, 'DL', 'LAX', 'ORD', 240.00);

An upcoming trip from ORD to LAX on February 6th, 2020.

INSERT INTO `flights` (`year`, `month`, `day`, `day_of_week`, `fl_date`, `carrier`, `tail_num`, `fl_num`, `origin`, `dest`, `crs_dep_time`, `dep_time`, `dep_delay`, `taxi_out`, `wheels_off`, `wheels_on`, `taxi_in`, `crs_arr_time`, `arr_time`, `arr_delay`, `cancelled`, `cancellation_code`, `diverted`, `crs_elapsed_time`, `actual_elapsed_time`, `air_time`, `distance`, `carrier_delay`, `weather_delay`, `nas_delay`, `security_delay`, `late_aircraft_delay`) VALUES (2020, 2, 6, 2, '2020-02-06', 'DL', NULL, 1170, 'ORD', 'LAX', '1420', '1420', NULL, NULL, NULL, NULL, NULL, '1730', '1730', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO `tickets` (`id`, `fl_date`, `fl_num`, `carrier`, `origin`, `dest`, `price`) VALUES (2, '2020-02-06', 1170, 'DL', 'ORD', 'LAX', 276.00);

INSERT INTO `trips` (`id`, `ticket_id`) VALUES (1, 2);

Weather forecast data

You also have the option of hard-coding your weather forecast information or integrating a Weather Forecast API of your choice. Currently the data is hard-coded, but can facilitate either approach.

Check out tripRoutes.js to learn more.

var forecasts = {
    "ORD_2020-02-06": {
        description: "Snow",
        icon: "snow",
        temp_low: "28°F",
        temp_high: "29°F",
        precip_probability: 0.6,
        wind_speed: 15
    },
    "LAX_2020-02-08": {
        description: "Clear",
        icon: "clear-day",
        temp_low: "56°F",
        temp_high: "65°F",
        precip_probability: 0,
        wind_speed: 5
    }
};

Smart Transactions

At this point you're probably wondering, what are smart transactions?

At their core, smart transactions are the standard transactions that databases have been performing for decades – ultimately powering the online interactions we’ve become accustomed to. The difference with modern applications is the use of real-time analytics before, during and/or after these transactions.

Pre-transaction

This application uses real-time analytics before a flight is booked. Each flight ticket option contains information calculated from the historical records (average delay, average duration, flight score, etc.) within the flights table.

Post-transaction

This application also uses real-time analytics after a flight has been booked, and a trip has been created.

Cross-Engine Queries

This application uses cross-engine queries to maximize the potentials of the MariaDB X4 Platform. Cross-engine querying is the ability to access, via MaxScale, both the transactional and analytics data within a single query.

Support and Contribution

Thanks so much for taking a look at the Bookings app! As this is a very simple example, there's plenty of potential for customization. Please feel free to submit PR's to the project to include your modifications!

If you have any questions, comments, or would like to contribute to this or future projects like this please reach out to us directly at developers@mariadb.com or on Twitter.

License