Database RDB APIv1

Scaleway Database is a service that make your database's management easy as pie. It provides simple and reliable solutions to create and run your database, dealing with security and high availability to ensure the durability of your data.

The following sections define concepts such as Instance, Database Engine, Configuration, ACL, Failover, Configuration, Backup and Restore in Scaleway Database.

A Database Engine is the software component that stores and retrieves your data from a database. For example, PostgreSQL is a database engine and it implements the SQL standard. Database Engines have versions. Different major versions represents changes that can break backward compatibility.

See "Engine Specifics" for supported engines and versions.

A Database Instance is composed of one or more computing nodes, depending on the chosen HA setting (1 leader for Standalone, 1 leader + 1 replica for High Availability). Replica nodes are linked to a leader, using synchronous replication. Synchronous replication offers the ability to confirm that all changes intended by a transaction have been transferred and applied to one or more synchronous replica servers, thus providing durability to the data.

A Database Instance has settings that let you tune the behavior of its Database Engine to better fit your needs. Available settings depend on the Database Engine and its version. Each instance setting entry has a default value that the user can override. The deletion of a setting entry will restore the setting to the entry's default value. Some of the defaults values can be different from the engine defaults, as we optimize these defaults for the Scaleway platform.

A Database Instance Endpoint is composed of an IP and a TCP port. These are the coordinates that are used to connect to the service. For now, there is only one endpoint, always pointing to the leader node of the Instance.

Endpoints IPs are public (No access is available using private IPs for now, but this might change in the future). This might raise security concerns to some, but here are a few points that must be remembered:

• When connecting to a Database's public IP address from a Scaleway Compute Instance from the same region, the traffic does not leave Scaleway's network.
• All Scaleway Database Endpoints have SSL enabled. Be sure to use it to make your data and your passwords unreadable in transit to anyone but you.
• Access to your Endpoint can be restricted to a set of trusted hosts or networks that you choose, see "ACL Rules" below.

Please note that RDB architecture uses LoadBalancers to forward the traffic to the right node based on the node state (active/hot standby). LoadBalancers configuration is set to cut off inactive connections if no TCP traffic is sent within a 6-hour timeframe. We recommend using a connection pooling on application side in order to renew database connections regularly.

Access Control List Rules (ACL Rules) define which host and networks can connect to your Database Instance Endpoint. You can add, edit or delete rules from your ACLs. The initial setup of an instance allows full network access from anywhere (0.0.0.0/0).

Access control is handled directly at network-level by our load-balancers, so the Database does not need to do it itself, making the filtering more efficient, universal and relieving the Database from this task.

You can order a high availability database instance. With this configuration, we install the RDBMS (Relational Database Management System) on two dedicated compute nodes. We also make sure those instances are on two physically separate racks.

On a High Availability Database Instance, replication is synchronous. With this replication mode, write operations are routed to one database node. The other node (standby node) replicates these data. Write operations are acknowledged to the user only when they are applied on the master AND on the standby node. If a node goes down, data integrity is guaranteed. You can't be in the situation where the leader node has more data than the replica. This replication mode introduces a bit of overhead on the database system, but for us, data integrity is the most important.

High availability instances are monitored continuously and got an AutoHealing feature. If a database node goes down, the standby node will take write operations in less than 10 seconds letting you be able to continue your business as usual with read and write operations. During this phase, your instance switch to AUTOHEALING status (blue dot in the console). During this period, your High Availability Instance is working on the remaining database node for at most 700 seconds. After 700 seconds, if your database node is still considered as down, a new node is spawned and automatically configured as a standby database node. Once configured, your database instance get the READY status back. With this system, your database instance is unavailable only for a few seconds letting your app work as usual.

Logical Databases can be saved to and restored from our storage facilities, located in a different physical region than your Database. Each logical database inside of a database instance is backed up and therefore restorable separately.

A User is an account protected by a password, allowing access to your Database Instance.

There are two ways to manage user permssions: from the API/Console or using SQL. All supported engines have a rich permission model and as most users only need a part of it, so Scaleway Database provides a simplified and unified permission model through the API and the console to make things easier to manage and understand. Beside this, it is perfectly fine to use engine-specific ways of managing users and permissions, see each engine documentation to find mode about the corresponding SQL commands.

The following is only about the simplified workflow.

Each User has associated Permissions that give it access to zero or more Logical Databases. A User can have the "admin" flag, the behaviour is slightly different depending on the engine, see below in "Engine Specifics".

Logs can contain useful information for debugging or to know more about the behaviour and activity of your database. They are available for download, but must first be gathered from the Database Nodes.

Memory, CPU and Disk usage are important metrics that you should keep an eye on. Graphs are available on the "Metrics" section in the console. You can also track these value from the API to inject them in you own system and raise custom alerts.

Scaleway's infrastructure is divided into regions and zones.

Scaleway Database is available in the following regions:

• fr-par
• nl-ams
• pl-waw

You can use cURL to send request to APIs, we will provide curl example for each step.

Ubuntu

?> sudo apt-get install curl

Mac OS

?> brew install curl

Go to the following link: https://console.scaleway.com/register Provide a valid e-mail address and a password.

Login to your Scaleway account and create a token from the credentials page. A SECRET_KEY and an access_key will show on your screen, the SECRET_KEY will be used as X-Auth-Token.

reference: https://www.scaleway.com/docs/generate-an-api-token/

In order to use Scaleway API you need to have curl installed. It is also a good idea to have jq, which will help you to read and parse JSON output. Make sure you have these two tools before you begin. Otherwise use your package manager to install them.

To call Scaleway API, you need an API key. If you don't have one yet, you can create it with the Scaleway console.

Using this token, get your project ID:

export ACCESS_KEY="<Access key of your API key>"
export SECRET_KEY="<Secret key of your API key>"
export REGION="<choose your location (nl-ams/fr-par/pl-waw)>"

curl -s -H "X-Auth-Token: $SECRET_KEY" https://account.scaleway.com/tokens/$ACCESS_KEY | jq -r '.token.project_id'

export PROJECT_ID="<your project ID>"

You can also find your project ID in the Scaleway web console.

This documentation shows how to create and connect to a PostgreSQL Database using Scaleway Database product.

Requirements:

You have to complete the "Setup" explained on upper sections, you are logged in scaleway and have your API Token and Project ID. We are going to create, connect to and delete a database.

Create your database instance

• endpoint: https://api.scaleway.com/rdb/v1/regions/$REGION/instances

{
    "project_id": "$PROJECT_ID",
    "name": "myDB",
    "engine": "PostgreSQL-11",
    "tags": ["donnerstag"],
    "is_ha_cluster": true,
    "node_type": "db-dev-s",
    "disable_backup": true,
    "user_name": "my_initial_user",
    "password": "thiZ_is_v&ry_s3cret"
}

Example:

?> curl -X POST -H "Content-Type: application/json" \
-H "X-Auth-Token: $SECRET_KEY" https://api.scaleway.com/rdb/v1/regions/$REGION/instances
-d "{
    \"project_id\": \"$PROJECT_ID\",
    \"name\": \"myDB\",
    \"engine\": \"PostgreSQL-11\",
    \"tags\": [\"donnerstag\"],
    \"is_ha_cluster\": true,
    \"node_type\": \"db-dev-s\",
    \"disable_backup\": true,
    \"user_name\": \"my_initial_user\",
    \"password\": \"thiZ_is_v&ry_s3cret\"
}"

Now you've created your database instance, you can open a connection between a database client and your database. You will need a PostgreSQL client such as psql

List instances

• endpoint: https://api.scaleway.com/rdb/v1/regions/$REGION/instances

example:

?> curl -X GET -H "Content-Type: application/json" \
-H "X-Auth-Token: $SECRET_KEY" https://api.scaleway.com/rdb/v1/regions/$REGION/instances

{
  "instances": [
    {
      "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "name": "myDB",
      "organization_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "project_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "status": "ready",
      "engine": "PostgreSQL-11",
      "endpoint": {
        "ip": "xxx.xxx.xxx.xxx",
        "port": 22245,
        "name": null
      },
      "tags": [
        "donnerstag"
      ],
      "settings": [],
      "backup_schedule": {
        "frequency": 24,
        "retention": 7,
        "disabled": true
      },
      "is_ha_cluster": true,
      "read_replicas": [],
      "node_type": "db-dev-s",
      "volume": {
        "type": "lssd",
        "size": 30000000000
      }
      "created_at": "2019-04-19T16:24:52.591417Z",
      "region": "fr-par"
    }
  ],
  "total_count": 1
}

Get the endpoint, port

• endpoint: https://api.scaleway.com/rdb/v1/regions/$REGION/instances/{instance_id}

?> curl -X GET -H "Content-Type: application/json" \
-H "X-Auth-Token: $SECRET_KEY" https://api.scaleway.com/rdb/v1/regions/$REGION/instances/{instance_id}

{
  "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "name": "myDB",
  "organization_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "project_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "status": "ready",
  "engine": "PostgreSQL-11",
  "endpoint": {
    "ip": "xxx.xxx.xxx.xxx",
    "port": 22245,
    "name": null
  },
  "tags": [
    "donnerstag"
  ],
  "settings": [],
  "backup_schedule": {
    "frequency": 24,
    "retention": 7,
    "disabled": false
  },
  "volume": {
    "type": "local",
    "size": {
      "bytes": 50000000000,
    }
  },
  "is_ha_cluster": false,
  "read_replicas": [],
  "node_type": {
    "name": "db-dev-s",
    "stock_status": "available",
    "description": "2 GB ram; 2 cpu; 50 GB LSSD storage",
    "region": "fr-par"
  },
  "created_at": "2019-04-18T15:27:24.177854Z",
  "region": "fr-par"
}

Connect to your database with psql client

?> psql -h 192.0.2.250 -p 28991 -U my_initial_user -d rdb
password:

postsql>

Congratulations! You have done your first steps on Database product !

• api token: get your api token
• organization id: get your organization id

In order to migrate your PostgreSQL instance to a Scaleway Database for PostgreSQL instance, you will need the following:

• pg_dump command-line installed
• psql command-line installed
• scaleway database instance ip
• scaleway database instance port
• scaleway database instance user and password
• source instance user
• source instance port if required
• list of the database that you want to migrate

You can export your data from your source instance to a local file by performing the following command:

?> pg_dump –-host=<host> --port=<port> --username=<name> --dbname=<database_name> --file=<database>.sql

If your source postgreSQL is in a docker container, the command will be:

?> docker exec -it $CONTAINER_NAME pg_dump --username=<name> --dbname=<database_name> > <database>.sql

In order to restore a database, you have to create the destination database and users of that database in your instance. You can create your database and roles by using the web console, or by using the API. Ensure that your user permission is correctly set to all.

• create database: https://developers.scaleway.com/en/products/rdb/api/#post-560a84
• create user: https://developers.scaleway.com/en/products/rdb/api/#post-944e53

Once you have created each destination database and its users, you can import your data from the <database>.sql file to your scaleway database instance by performing the following command:

?> psql --file=<database>.sql --host=<Scaleway Database Instance Ip> --port=<Scaleway Database Instance Port> --username=<user> --dbname=<destination database name>

Scaleway Database for PostgreSQL supports PostgreSQL versions 9.6, 10, 11 and 12.

Some modules are available for installation, including the popular TimescaleDB and PostGIS. See FAQ for a list of allowed extensions.

Logical databases are created either using Console/API or SQL.

• Databases created using the Console/API will be owned by an internal system user. These are called "managed databases".
• Databases created using SQL will be owned by the creator. These are called "unmanaged databases".

• Users with the admin level have the CREATEROLE and CREATEDB privileges
• Users do NOT have the SUPERUSER privileges
• Permissions management through the Console/API is only possible on "managed databases".

Database will be restored with the permissions they were backuped with. However, privileges will be ignored for deleted users, and if objects cannot be restored to the original owner, they will be owned by an internal system user.

Scaleway Database for MySQL supports MySQL 8.

• Only the InnoDB engine is supported
• Global Transaction Identifier (GTID) is enabled.
• mysql_native_password and caching_sha2_password authentication are supported. Default is mysql_native_password for compatibility.

• User with the admin level have access to all logical databases and can create new ones.
• Users created through the API are authenticated using the default authentication plugin, which can be changed in the settings.

Even if we manage your database storage space, you're responsible for its usage.

In the case you're reaching the space provisioned for your instance, the instance will shift to the disk_full status, informing you that there is only 1GB left on your database instance. This lets you either scale up the disk on the console or leaves you enough room to do some cleanup.

In disk_full mode, the default transaction mode will shift to read only, allowing your application to still access its data, while preventing new data from being inserted.

In order to remove data from your instance, you must connect to it as usual, but explicitly shift to a read write transaction, like this:

-- On PostgreSQL
BEGIN;
SET TRANSACTION READ WRITE;
-- do some cleanup
COMMIT;

-- On MySQL
SET transaction_read_only=0;
BEGIN;
-- do some cleanup
COMMIT;

When enough space is available on the instance, the default transaction mode will change back to read write shortly after, allowing normal operation again.

Failing to cleanup or forcing insertions while the database is in disk_full mode will lead to downtime.