Search for a command to run...
Learn how to easily export data from Prometheus into other popular databases
A technical data engineer / passionate technologist who loves tackling interesting problems with code, exploration & persistence.
No comments yet. Be the first to comment.
Configure REST API extraction the same way you configure database replication
A Step-by-Step Guide to Using Sling for Ingesting data into DuckLake
Simplify Your Data Journey
Leveraging DuckDB for Seamless Delta Table Processing
Leveraging DuckDB for Efficient Iceberg Table Processing
Sling aims to augment the exporting/loading data process into a positive and potentially enjoyable experience. It focuses on 3 of data types interfaces:
The list of connections that Sling supports continues to grow. You can see the full list here, but it supports all the major platforms including Clickhouse, DuckDB, Google BigQuery, Google BigTable, MariaDB, MongoDB, MotherDuck, MySQL, Oracle, PostgreSQL, Prometheus, Redshift, Snowflake, SQL Server, SQLite, StarRocks and Trino.
Prometheus is an open-source time-series database originally built by SoundCloud. It is a powerful and flexible monitoring solution ideal for gathering, querying, and alerting on metrics data in dynamic and distributed environments. Its pull-based model, multi-dimensional data model, PromQL query language, and alerting capabilities make it a popular choice for monitoring modern cloud-native applications and infrastructure.
An issue faced dealing with Prometheus is the difficulty in exporting data from it. Fortunately, Sling handles this without a skip.
See here for details on how to install Sling. It is usually a simple command, such as:
# On Mac
brew install slingdata-io/sling/sling
# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling
# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
&& tar xf sling_linux_amd64.tar.gz \
&& rm -f sling_linux_amd64.tar.gz \
&& chmod +x sling
You should be able to run the sling command at this point.
Let's now set up the connections. In this blog post, we will use Postgres as the destination database, however the steps are the same if you'd like to load into a different database.
See here for more details on the keys accepted. But we can simply do:
$ sling conns set prometheus type=prometheus http_url="http://localhost:9090" api_key="xxxxxxxxxxxxxxxxxxxxxx"
$ sling conns test prometheus
6:01PM INF success!
# get list of metrics
$ sling conns discover prometheus --column
+------------+------------+------------+-----+----------------------------------+-------------+--------------+
| DATABASE | SCHEMA | TABLE | ID | COLUMN | NATIVE TYPE | GENERAL TYPE |
+------------+------------+------------+-----+----------------------------------+-------------+--------------+
| prometheus | prometheus | prometheus | 1 | go_gc_duration_seconds | summary | bigint |
| prometheus | prometheus | prometheus | 2 | go_goroutines | gauge | bigint |
| prometheus | prometheus | prometheus | 3 | go_info | gauge | bigint |
| prometheus | prometheus | prometheus | 4 | go_memstats_alloc_bytes | gauge | bigint |
| prometheus | prometheus | prometheus | 5 | go_memstats_alloc_bytes_total | counter | bigint |
| prometheus | prometheus | prometheus | 6 | go_memstats_buck_hash_sys_bytes | gauge | bigint |
| prometheus | prometheus | prometheus | 7 | go_memstats_frees_total | counter | bigint |
.....
Similarly, we will again use the sling conns set command, this time for our PG connection. See here for more details.
$ sling conns set postgres url="postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"
$ sling conns test postgres
6:03PM INF success!
Great, we are now ready to move data!
# export results to stdout, with start time from 2 months ago: "now-2M"
$ ./sling run --src-conn prometheus \
--src-stream 'sum(go_gc_duration_seconds) by (job, instance, quantile) # {"start": "now-2M"}' \
--stdout --limit 10 -d
# load into PG
$ ./sling run --src-conn prometheus \
--src-stream 'sum(go_gc_duration_seconds) by (job, instance, quantile) # {"start": "now-2M"}' \
--tgt-conn postgres --tgt-object public.gc_duration_by_job \
--mode full-refresh
Adding time filters, simply add a suffix to your query. Accepts a JSON value with keys start, end and step:
# {"start": "now-2M"}
# {"start": "now-2M", "end": "now-1d"}
# {"start": "now-2M", "end": "now-1d", "step": "1d"}
We can also use a replication. Replications are the best way to use sling in a reusable manner. The defaults key allows reusing your inputs with the ability to override any of them in a particular stream. Both YAML or JSON files are accepted.
# replication.yaml
source: prometheus
target: postgres
defaults:
object: prometheus.{stream_name}
mode: full-refresh
streams:
gc_duration_by_job:
sql: 'sum(go_gc_duration_seconds) by (job, instance, quantile) # {"start": "now-2M", "end": "now-1d", "step": "1d"}'
# incremental load, last 2 days of data, hourly
go_memstats_alloc_bytes_total:
sql: 'sum(go_memstats_alloc_bytes_total) by (job, instance, quantile) # {"start": "now-2d"}'
primary_key: [timestamp, job, instance, quantile]
update_key: timestamp
mode: incremental
We can run the replication like this:
sling run -r replication.yaml
We went over on how easy it was to export data with Sling from Prometheus. Feel free to check out other examples here: https://docs.slingdata.io.