Cockpit APIv1beta1

Scaleway Cockpit is a platform to monitor applications and their infrastructure. It ingests metrics, using a Prometheus-like API, and logs, using a Grafana Loki API. It also provides a Grafana Dashboard to visualize the metrics and logs.

• The retention is limited to 32 days. It will be configurable later on.
• The number of active metrics time series is limited to 100 thousand per cockpit.
• The number of active log streams is limited to 5000 per cockpit.
• No downsampling (yet) for the metrics.

Warning: The cockpit activation may take few seconds. To know if the cockpit is ready, you can GET your cockpit and look status, if the status is ready then you can use it.

You can reach out to us using our community slack, on #observability or #observability-beta channels. If you don't have access to this Slack, you can sign up to it by following this tutorial.

Before continuing, please make sure you are familiar with the following terminology:

• Cockpit: A cockpit is an instance of the Cockpit product. A Scaleway Project can have only one cockpit. A cockpit has a dedicated Grafana instance.
• Prometheus: Prometheus is a popular time series database. See https://prometheus.io/ for more information.
• Loki: Loki is a logs management platform made by Grafana Labs. See https://grafana.com/oss/loki/ for more information.
• Pusher: A pusher is any tool that can push metrics and/or logs, using the right format, to the platform.

You can use a software like Postman to try the Cockpit product. The OpenAPI specification file (available at the top of this page) can be loaded into Postman. You can find documentation on how to do this here.

Here's the plan for this quickstart:

• First, we create a token which will be used to push logs & metrics
• Next, we configure Grafana Agent to push metrics and logs to our cockpit
• Finally, we import a dashboard to visualize our metrics

For this quickstart, you will need the following software:

• Docker
• Docker Compose
• A linux machine to run the container on. Note that you can still go through this quickstart by running the container on macOS or Windows, but you won't see any logs in your cockpit.

If you don't have them, you can either install Docker Desktop on your local machine using this link, or install Docker Community Edition using the official documentation.

Note: If you've chosen Docker Community Edition, Docker Compose won't be installed automatically for you. You will need to follow the instructions given on this page to install it.

Before executing any of the curl examples in the quickstart, you need to prepare your environment. To do so you will need an API key. If don't have one yet, please follow this tutorial to get one.

Now, we need to set your API key in your environment. To do so, paste your API secret key (the one that looks like a UUID) into the following code and execute the command.

A cockpit has 3 endpoints which are given when creating or getting a cockpit. Here's a quick example:

The metrics_url is a domain that exposes a Prometheus-like API to manage metrics, and the logs_url exposes a Loki API to manage logs. But to be able to send metrics and logs, we need the exact URL to use with our pushers.

Here are the URL paths to append to the endpoints:

• For metrics: /api/v1/push, which gives us https://metrics.prd.obs.fr-par.scw.cloud/api/v1/push
• For logs: /loki/api/v1/push, which gives us https://logs.prd.obs.fr-par.scw.cloud/loki/api/v1/push

To be able to push logs and metrics into your cockpit, you need a token. Here is an example request to create a token for a given cockpit:

Note: The name of the token must be unique within the cockpit.

Running this query should output the following JSON response:

Note: The secret_key will only be displayed once. Any subsequent request on this token will return this field empty.

Important: There are multiple permission scopes available for tokens. We strongly recommend that you only give the minimal amount of permissions to a given token. Hence metric pushers should only have the write_metrics scope, and log pushers the write_logs one. Here we are going to configure a single agent that does both, so we need a token with both scopes.

Now that the cockpit is ready, we can install Grafana Agent on a machine. Before going further, make sure that you have:

• Your push URLs (for logs and metrics)
• A cockpit token that is able to write logs and metrics

Using the following yaml template, create a new file named config.yaml that will contain the Grafana Agent configuration. This log configuration will scrape the logs from the /var/log directory. Make sure to replace the URLs with the valid ones and to put your own token in both locations.

Note: The promtail configuration for the logs doesn't support custom headers. The tenant_id field correspond to the header X-Scope-OrgID which is one of the supported headers. For more details on the different supported headers, see the section at the end of the document here

Create docker-compose.yaml file next to the config.yaml file that you've just created.

Next, use the docker-compose up command to bring the Grafana Agent container up. You should see some logs indicating that it started to watch files in the /host/root/var/log directory, and then stays silently up and running.

Now it's time to have a look at the metrics and logs the agent is pushing.

To be able to visualize your data, a Grafana instance is started when you create a cockpit. The Grafana instance already has its data sources set up so that it is ready to use. To continue this quickstart, go to the grafana_url using your browser, and log in using the grafana_username and grafana_password values (those values were returned when you created your cockpit).

Now that we're on your Grafana instance, we need a Grafana dashboard to be able to make use of the metrics that the agent is now pushing. But luckily for us, node-exporter is a well known exporter and there are dashboards already made for it. This will save us a lot of time. To take advantage of it, you can search the Grafana dashboard database here or use the one we already picked for you.

To import a Grafana dashboard:

1. Go to the dashboard's page: https://grafana.com/grafana/dashboards/1860
2. Note down the ID of the dashboard. In this case, it is 1860.
3. Go back to your Grafana instance
4. Hover your mouse over the "+" button on the left of your screen.
5. On the menu that appeared, click on the Import item.
6. Put the dashboard ID in the text box and click Load
7. Select the Scaleway Metrics Prometheus instance as the data source.
8. Click the Import button.

After doing so, you will be taken directly to the dashboard where you will be able to visualize your metrics.

Finally, let's explore the logs that the agent is also sending.

To do this:

1. Click on the "compass" icon on the left of your screen
2. At the top of the screen, select the Scaleway Logs data source.
3. Now click on the Log browser button.
4. Follow the on-screen instructions to select your labels and their values
5. Click the Show logs button

On the screen that just opened, you will be able to see the logs of your system! Cherry on the top: if you click on the Live button at the top right corner of the screen, you will be able to see your logs arrive in real time.

When a Grafana instance is created, it is already configured with logs and metrics data sources. But a third one is also configured: the Alertmanager data source. This means that right from Grafana, you will be able to:

• Configure your alerting rules, for metrics and logs.
• Configure your contact points & notification policies.
• Manage your alert silences.

To do so, on your Grafana instance, click on the "bell" icon on the left of your screen and you will see the Alerting page and a number of tabs. It makes use of the Prometheus alerting system, so if you are not familiar with the alertmanager or alert rules, here are some nice resources:

• Introduction to alerting using Prometheus
• How to create a Mimir or Loki managed alerting rule using Grafana
• How to manage alerting rules using Grafana
• How to manage contact points using Grafana
• How to manage notification policies using Grafana
• How to manage silences using Grafana

Important: For the alerts & rules to be created in your cockpit, you must use the Scaleway provided data sources. This means two things:

1. When creating alert rules (using the Alert rules tab), in the Rule type field you need to select Cortex/Loki managed alert. Then, on the Select data source field that appeared on the right, select the data source you wish to create the rule on. Then proceed as usual.
2. For all the other tabs, from Contact points to Alert groups, you will be working on the Alertmanager configuration. When doing so, there will be a field at the top of the screen (just under the tabs) named Choose Alertmananger. Make sure to always select Scaleway Alerts there, as the alerts are managed by the platform. If you use Grafana you will work with the local Alertmanager which has been deactivated.

If you've finished this quickstart and want to go further, we have prepared a list of links that we think could be interesting to read:

• Introduction to prometheus. This page explains what Prometheus is and gives a general introduction to it.
• Introduction to Promtail. Promtail is an agent that collects and forwards logs to a Grafana Loki instance.
• Introduction to Grafana Agent. Grafana Agent is an agent that embeds multiple other agents like Prometheus or Promtail.
• Pushing logs using Fluentbit. Fluentbit is another commonly used agent that collects and forwards logs.
• Pushing metrics using Prometheus's agent mode. Prometheus can be used in an "agent-mode" that can scrape local targets and forward them using remote-write. This is useful for scenarios where targets are exposed in a private network (like a VPN or a single host) but need to be forwarded outside of it.
• Prometheus remote write configuration. This page gives the exact configuration parameters that can be used to configure Prometheus's remote-write.
• Prometheus data model. This page explains the Prometheus data model as it's important to understand the concepts of metrics, time series, labels and samples, and how they work together.
• Article on Prometheus label cardinality & performance. When working with metrics, the cardinality, which is the number time series for a given metric, is extremely important. This article explains why.
• Integrations supported by Grafana Agent. Grafana Agent has many other embedded agents that can be used to monitor various software, like PostgreSQL or MongoDB. All of them can be found on the left menu of this page.

Yes. To do that you will need a recent Grafana version (8+) which has alpha plugins enabled, and a token for the cockpit you want to use.

Make sure to add the right permissions on your token. To be able to make use of all the features that Grafana has to offer, we recommend that you enable the following permissions:

• query_metrics
• query_logs
• setup_alerts
• setup_metrics_rules
• setup_logs_rules

Before adding the data source, note that you will need to specify a token for it to work. To do this, while creating a data source, add a Custom HTTP Header, set the Header field to X-Token and put your token in the Value field. If everything works, clicking on the Save & test button should yield a green banner saying Data source is working.

Now you will need to add the 3 different data sources:

1. A Prometheus data source using the https://metrics.prd.obs.fr-par.scw.cloud/api/prom URL.
2. An alert manager data source using the https://alertmanager.prd.obs.fr-par.scw.cloud/api/prom URL.
3. A Loki data source using the https://logs.prd.obs.fr-par.scw.cloud URL.

Once done, your Grafana instance is set up and ready for use. Don't forget to select the right data source when working with the alerting section!

Using this request:

In response, you will receive a brand-new password for your user.

Note: The username is always the same, and you can get it by GET'ing your cockpit.

If your pusher doesn't support the X-Token header, there are two other ways to pass it:

• Through the X-Scope-OrgID header. This means that when using Promtail, you can set the tenant_id field of the clients configuration to pass the token.
• Through the Authorization: Bearer <token> header. This is for all the other clients that usually have support for these bearer tokens.
• Through the Authorization: Basic header. With basic authentication you will have to use the token as password with any username.

If none of these methods are available for you, do not hesitate to reach out!

Activate a cockpit associated with the given project ID.

Get the cockpit associated with the given project ID.

Deactivate a cockpit associated with the given project ID.

Reset the Grafana of your cockpit associated with the given project ID.

List tokens associated with the given project ID.

Create a token associated with the given project ID.

Get the token associated with the given ID.

Delete the token associated with the given ID.

List alert contact points associated with the given cockpit ID.

Create an alert contact point for the default receiver.

Delete an alert contact point for the default receiver.

Disable managed alerts.

Enable managed alerts.

Trigger a test alert to all receivers.

List grafana users who are able to connect to your grafana instance.

Create a grafana user for your grafana instance.

Delete a grafana user from your grafana instance.

Reset the Grafana user password from your grafana instance.