In the ever-evolving landscape of data engineering, the tools we use are constantly getting better, faster, and more efficient. DuckLake is one such innovation, building on the phenomenal success of DuckDB to offer a robust, ACID-compliant data lake format. It's designed for scalability and flexibility, supporting various backends for both its catalog and data storage.

But how do you get your data into DuckLake? Whether your data lives in a production PostgreSQL database, a MySQL instance, or any other database, you need a simple and powerful way to extract and load it.

This is where Sling comes in. Sling is a modern data movement tool designed to make transferring data between different sources and destinations as easy as possible. In this article, we'll walk you through how to use Sling to extract data from most databases and load it directly into your DuckLake instance. The list of connections that Sling supports continues to grow. You can see the full list here, but it supports all the major platforms including Clickhouse, DuckDB, Google BigQuery, Google BigTable, MariaDB, MongoDB, MotherDuck, MySQL, Oracle, PostgreSQL, Prometheus, Redshift, Snowflake, SQL Server, SQLite, StarRocks, Databricks and more.

What is DuckLake?

Before we dive in, let's quickly recap what makes DuckLake special. DuckLake is a data lake format specification that combines the power of DuckDB with flexible catalog backends and scalable data storage. It provides versioned, ACID-compliant tables, which brings database-like reliability to your data lake. See Ducklake's website for more details: https://ducklake.select/.

Key features include:

• Flexible Catalog: Use DuckDB, SQLite, PostgreSQL, or MySQL as your catalog backend.
• Scalable Storage: Store your data files locally, or in cloud storage like AWS S3, Azure Blob Storage, or Google Cloud Storage.

Step 1: Configure Your Connections

First things first, we need to tell Sling how to connect to our source database and our target DuckLake instance. We'll use the sling conns command, which makes managing connections a breeze.

Source Database: PostgreSQL

Let's assume our source data is in a PostgreSQL database. See the complete list of databases sling can connect to here.

We can set up a connection named PG_CONN like this:

sling conns set PG_CONN type=postgres host=mypg.host user=myuser password=mypass port=5432 database=analytics

# or use environment variable
export PG_CONN='postgresql://myuser:mypass@mypg.host:5432/analytics?sslmode=require'

You can then run sling conns test pg_conn to ensure it can successfully connect.

Target: DuckLake

Configuring DuckLake involves specifying the catalog and the data storage path. See the documentation for details here.

For this example, we'll use a local SQLite file for the catalog and a local directory for the data. This setup is great for multi-client access on the same machine.

Here's how to set up a connection named DUCKLAKE_CONN:

sling conns set DUCKLAKE_CONN type=ducklake catalog_type=sqlite catalog_conn_string=ducklake_catalog.db data_path=./ducklake_data

# or use environment variable
export DUCKLAKE_CONN='{ 
  type: ducklake, 
  catalog_type: sqlite,
  catalog_conn_string: "ducklake_catalog.db",
  data_path: "./ducklake_data"
}'

After setting these, you can verify they're configured correctly by running sling conns list.

If you used the sling conns set command, your ~/.sling/env.yaml file should now contain these configurations:

connections:
  PG_CONN:
    type: postgres
    host: mypg.host
    user: myuser
    password: mypass
    port: 5432
    database: analytics

  DUCKLAKE_CONN:
    type: ducklake
    catalog_type: sqlite
    catalog_conn_string: ducklake_catalog.db
    data_path: ./ducklake_data

Step 2: Create the Replication YAML

Now for the fun part. We'll define our data movement task in a simple YAML file. Sling's replication configs are powerful because you can define defaults and replicate many streams (e.g., tables) at once. See here for full documentation.

Let's create a file named db_to_ducklake.yaml:

source: pg_conn
target: ducklake_conn

defaults:
  mode: full-refresh
  object: '{stream_schema}.{stream_table}' # Dynamically name tables in DuckLake

streams:
  # Replicate all tables from the 'public' schema
  public.*:

  # You can also add specific tables or disable some
  public.forbidden:
    disabled: true

  analytics.users:

A few things to note here:

• source and target refer to the connection names we just set up.
• defaults applies the full-refresh mode to all our streams. This means the target tables in DuckLake will be dropped and recreated on each run.
• The object name is the target table, and uses runtime variables {stream_schema} and {stream_table}. Sling will dynamically replace these with the actual schema and table names from the source.
• The real power move is public.*. This single line tells Sling to find all tables in the public schema of our PostgreSQL database and replicate every single one of them.

Step 3: Run the Replication

With our connections and replication file ready, all that's left is to run it:

sling run -r db_to_ducklake.yaml

That's it! Sling will connect to your PostgreSQL database, read the tables from the public and analytics schemas, and write the data into your DuckLake instance, creating the tables and structuring the data files in the ./ducklake_data directory.

Going Further: Incremental Loads and Cloud Storage

This example is just the beginning. You can easily adapt this for more advanced use cases:

• Incremental Loads: Change the mode to incremental and specify a primary_key and/or update_key to only process new or updated records, making your pipelines much more efficient.
• Cloud Storage: To use cloud storage for your data, simply update your DuckLake connection's data_path to an S3, GCS, or Azure URI and provide the necessary credentials.

For example, to use an S3 bucket, your DuckLake connection in env.yaml might look like this:

connections:
  DUCKLAKE_CONN_S3:
    type: ducklake
    catalog_type: postgres
    catalog_conn_string: "host=db.example.com port=5432 user=ducklake password=secret dbname=ducklake_catalog"
    data_path: "s3://my-data-lake-bucket/data/"
    s3_access_key_id: "AKIA..."
    s3_secret_access_key: "xxxx"

And here's a replication with incremental mode and custom SQL:

source: pg_conn
target: ducklake_conn

streams:

  analytics.users:
    object: '{stream_schema}.{stream_table}'
    mode: incremental
    primary_key: id
    update_key: updated_at

  custom_stream:
    object: another_schema.report
    sql: |
      select ...
      from ...

Conclusion

DuckLake brings exciting new capabilities to the world of data lakes, and with Sling, populating it from your existing databases is incredibly straightforward. With just a few lines of configuration, you can build scalable, repeatable, and robust data pipelines to feed your modern data stack.

Ready to give it a try? Install the Sling CLI and check out the official documentation to get started.

This is where Sling comes in. As a modern data movement and transformation platform, Sling dramatically simplifies the entire process of setting up and managing data pipelines. Whether you're moving data between databases, synchronizing data warehouses, or transforming data during transfer, Sling provides an elegant solution that works right out of the box.

The Data Pipeline Challenge

Building data pipelines traditionally involves numerous challenges:

• Complex setup procedures requiring extensive configuration

• Managing multiple tools and technologies

• Ensuring data consistency and reliability

• Handling different data formats and schemas

• Monitoring and maintaining pipelines in production

• Scaling operations as data volumes grow

These challenges often lead to increased development time, higher maintenance costs, and reliability issues. Teams spend more time troubleshooting infrastructure than focusing on valuable data insights.

Enter Sling: A Modern Solution

Sling addresses these challenges head-on by providing:

• A unified platform for all database/file system movement needs

• Simple, intuitive interfaces through both CLI and UI

• Built-in support for numerous databases and storage systems

• Automatic schema handling and data type mapping

• Production-ready features like monitoring and scheduling

• Scalable architecture that grows with your needs

Let's dive deeper into the Sling platform and discover how it can transform your data operations.

Understanding Sling Data Platform

Sling is a comprehensive data movement and transformation platform designed with modern data needs in mind. At its core, Sling combines powerful functionality with user-friendly interfaces, making it accessible to both developers and data teams.

Key Benefits

1. Simplified Setup

   • Simple configuration for many common scenarios

   • Intuitive YAML-based configuration for complex cases

   • Visual interface for pipeline creation and management

2. Reduced Development Time

   • Pre-built connectors for popular databases and storage systems

   • Automated schema handling and type mapping

   • Built-in transformation capabilities

3. Enhanced Reliability

   • Robust error handling and retry mechanisms

   • Comprehensive logging and monitoring

   • Production-grade performance

4. Scalable Operations

   • Distributed agent architecture

   • Efficient resource utilization

Let's explore the main components that make up the Sling platform and see how they work together to provide a seamless data movement experience.

Platform Architecture

Sling's architecture consists of three main components that work together seamlessly: the Platform UI, the Control Server, and the Agents.

Sling Platform UI

The Sling Platform provides a modern, intuitive web interface for managing your data operations at scale. It's designed for teams who need a centralized way to manage, monitor, and collaborate on data movement tasks.

Using the Platform

The Platform UI makes it easy to:

1. Manage Connections

   • Create and test database connections

   • Store credentials securely

   • Share connections with team members

   • Monitor connection health

2. Design Replications

   • Real-time validation and feedback via Editor (IDE)

   • Create new replications visually

   • Configure source and target settings

   • Set up transformations

   • Define scheduling and triggers

3. Monitor Operations

   • Track replication status

   • View detailed execution logs

   • Analyze performance metrics

   • Set up alerts and notifications

Sling Platform Agents

Sling Agents are the workers that execute your data operations. They can be deployed anywhere in your infrastructure, providing flexibility and security.

Key Features

• Flexible Deployment

  • Run in your own infrastructure

  • Secure access to data sources

• Smart Resource Management

  • Concurrent streams handling

  • Efficient memory utilization

• Security First

  • Encrypted communication

  • No inbound connections required

  • Credential isolation

Deployment Options

Agents can be deployed in various ways:

1. Local Development

   • Run alongside CLI for testing

   • Quick setup and configuration

   • Direct debugging capabilities

2. Production Environment

   • Container-based deployment

   • BYOC or Cloud Hosting

   • Resource optimization

Getting Started with Sling Platform

1. Sign Up

   • Visit platform.slingdata.io

   • Create your account

   • Set up your project

2. Deploy an Agent

   • Install the agent in your environment

   • Configure connection to platform

   • Test connectivity

3. Create Connections

   • Add your data sources

   • Configure credentials

   • Test connections

4. Create Your First Pipeline

   • Use the editor

   • Create a new Replication

   • Create a Job and test it

   • Deploy with a schedule and monitor

Next Steps

To learn more about Sling's capabilities:

Explore Documentation

• CLI Documentation

• Platform Guide

• Connection Types

Start small with simple replications and gradually explore more advanced features as you become comfortable with the platform. Sling's flexibility means you can grow your usage alongside your data needs.

In the ever-evolving landscape of big data, Delta Lake has emerged as a powerful open-source storage layer that brings reliability and performance to data lakes. Today, we're thrilled to announce that Sling, our versatile data integration tool, now supports reading Delta Lake format, opening up new avenues for data engineers and analysts to harness the power of Delta tables.

What is Sling?

Sling aims to augment the exporting/loading data process into a positive and potentially enjoyable experience. It offers both CLI and YAML-based configurations for easy setup and management of data flows, by focusing on 3 data types interfaces:

• From File Systems to Databases

• From Databases to Databases

• From Databases to File Systems

The list of connections that Sling supports continues to grow. You can see the full list here, but it supports all the major platforms including Clickhouse, DuckDB, Google BigQuery, Google BigTable, MariaDB, MongoDB, MotherDuck, MySQL, Oracle, PostgreSQL, Prometheus, Redshift, Snowflake, SQL Server, SQLite, StarRocks and Trino.

Delta Lake: A Game-Changer in Data Lakes

Delta Lake, developed by Databricks, addresses many of the challenges faced by traditional data lakes. It introduces ACID transactions, scalable metadata handling, and time travel capabilities to big data workloads. These features make Delta Lake an attractive choice for organizations dealing with large-scale data processing and analytics.

Key benefits of Delta Lake include:

1. ACID Transactions: Ensures data consistency even with concurrent reads and writes.
2. Schema Evolution and Enforcement: Allows for easy schema changes and maintains data quality.
3. Time Travel: Enables querying data as it existed at a specific point in time.
4. Unified Batch and Streaming: Seamlessly handles both batch and real-time data processing.
5. Optimized Performance: Leverages various optimizations for faster queries on large datasets.

Sling's Delta Lake Integration: Read-Only Power

Sling now offers robust support for reading Delta Lake tables, leveraging the power of DuckDB under the hood. This integration allows users to easily incorporate Delta Lake data into their existing data pipelines and analytics workflows.

It's important to note that Sling's current implementation is read-only. While you can't write or modify Delta tables using Sling, you can efficiently extract data from Delta Lake for further processing or analysis.

How Sling Reads Delta Tables

Sling utilizes DuckDB's Delta Lake reader to efficiently process Delta tables. Here's a brief overview of how it works:

1. DuckDB Integration: Sling uses DuckDB's built-in Delta reader, allowing for direct querying of Delta tables without additional dependencies.
2. Delta Scan Function: Sling leverages DuckDB's delta_scan function to read Delta metadata and data files.
3. Query Optimization: Sling constructs optimized SQL queries using the delta_scan function, ensuring efficient data retrieval.
4. Streaming Results: Results are streamed from DuckDB, enabling efficient processing of large Delta tables without loading the entire dataset into memory.

Using Sling with Delta Lake: Practical Examples

Let's explore how you can use Sling to read Delta Lake tables in various scenarios.

Reading Delta Tables with Sling CLI

To read Delta Lake files using Sling's command-line interface, you can use the following commands:

# Read a local Delta table
sling run --src-stream file://path/to/table \
  --src-options '{format: delta}' \
  --stdout --limit 100

# Read a Delta table from AWS S3
sling run --src-conn aws_s3 \
  --src-stream path/to/table \
  --src-options '{format: delta}' \
  --stdout --limit 100

These commands will read the specified Delta table and output the first 100 rows to the console.

Incorporating Delta Lake in Replication YAML

For more complex data integration tasks, you can specify Delta as a format in your replication.yaml file:

source: aws_s3
target: postgres

defaults:
  mode: full-refresh
  source_options:
    format: delta

streams:
  path/to/delta/table:
    object: my_schema.delta_table

  path/to/delta_tables/*:
    object: my_schema.{stream_file_name}

This configuration reads data from Delta tables stored in AWS S3 and loads it into PostgreSQL tables. The full-refresh mode indicates that the target table will be completely replaced with the data from the source Delta table during each replication run.

To execute the replication, use:

sling run -d -r replication.yaml

See docs here to get started with Sling!

Real-World Use Case: Analytics on E-commerce Data

Imagine you're working with a large e-commerce platform that stores its transaction data in Delta format on AWS S3. You need to perform daily analytics on this data using your PostgreSQL data warehouse. Here's how you could use Sling to streamline this process:

1. Set up a replication YAML file to read from your Delta tables and write to PostgreSQL.
2. Schedule daily Sling runs to keep your analytics database up-to-date.
3. Leverage Delta Lake's time travel feature by specifying a timestamp in your Sling configuration to analyze historical data.
4. Use Sling's column selection feature to optimize data transfer by only reading the columns you need for your analytics.

This setup allows you to take advantage of Delta Lake's reliability and performance while using Sling's simplicity and flexibility for your data integration needs.

Conclusion

Sling's new capability to read Delta Lake format opens up exciting possibilities for data engineers and analysts. By combining Delta Lake's robust features with Sling's efficient data integration capabilities, organizations can streamline their data workflows and gain valuable insights from their data lakes.

While the current implementation is read-only, it provides a powerful tool for extracting and analyzing data stored in Delta format. As we continue to develop Sling, we're excited about the potential for expanding our Delta Lake support in the future via DuckDB.

What is Sling?

Sling aims to augment the exporting/loading data process into a positive and potentially enjoyable experience. It offers both CLI and YAML-based configurations for easy setup and management of data flows, by focusing on 3 data types interfaces:

• From File Systems to Databases

• From Databases to Databases

• From Databases to File Systems

The list of connections that Sling supports continues to grow. You can see the full list here, but it supports all the major platforms including Clickhouse, DuckDB, Google BigQuery, Google BigTable, MariaDB, MongoDB, MotherDuck, MySQL, Oracle, PostgreSQL, Prometheus, Redshift, Snowflake, SQL Server, SQLite, StarRocks and Trino.

What is Apache Iceberg?

Apache Iceberg is an open table format for huge analytic datasets. It's designed to improve on the limitations of older table formats, offering better performance, reliability, and flexibility for data lakes.

Advantages of Using Iceberg

1. Schema Evolution: Iceberg allows for easy schema changes without the need for data migration.

2. Partition Evolution: You can change partition schemes without rewriting data.

3. Time Travel: Query data as it existed at a specific point in time.

4. ACID Transactions: Ensures data consistency and reliability.

5. Performance: Optimized for fast queries on large datasets.

Iceberg's Popularity

Iceberg has gained significant traction in the data engineering community. It's used by major companies like Netflix, Apple, and Adobe, and is supported by popular data processing tools like Spark, Flink, and Presto.

How Sling Uses DuckDB to Read Iceberg

Under the hood, Sling leverages DuckDB's powerful Iceberg integration to read and process Iceberg tables efficiently. DuckDB is working on adding support for writing to Iceberg tables, and we're excited to see what new features this will bring to the table. Here's a brief overview of how this works:

1. DuckDB Integration: Sling utilizes DuckDB's built-in Iceberg reader, which allows for direct querying of Iceberg tables without the need for additional dependencies.

2. Iceberg Scan Function: When reading an Iceberg table, Sling uses DuckDB's iceberg_scan function. This function is capable of reading Iceberg metadata and data files directly.

3. Query Generation: Sling constructs a SQL query using the iceberg_scan function. For example:

    SELECT * FROM iceberg_scan('path/to/iceberg/table', allow_moved_paths = true)
   

   The allow_moved_paths option is set to true to handle cases where data files might have been moved.

4. Column Projection: When specific columns are requested, Sling modifies the query to select only those columns, optimizing read performance.

5. Type Mapping: Sling maps Iceberg/DuckDB types to its internal column types for consistent data handling across different sources.

6. Metadata Retrieval: Before executing the main query, Sling uses DuckDB to fetch table metadata, including schema information, by running a DESCRIBE query on the Iceberg scan.

7. Streaming Results: Sling streams the results from DuckDB, allowing for efficient processing of large Iceberg tables without loading the entire dataset into memory.

This approach allows Sling to provide seamless support for Iceberg tables, leveraging DuckDB's optimized Iceberg reader while maintaining Sling's flexible and user-friendly interface.

Reading Iceberg with Sling CLI

To work with Iceberg files in Sling, you can use the following CLI flags:

# read a local iceberg table
sling run --src-stream file://path/to/table \
  --src-options '{format: iceberg}' \
  --stdout --limit 100

# read an iceberg table from aws s3
sling run --src-conn aws_s3 \
  --src-stream path/to/table \
  --src-options '{format: iceberg}' \
  --stdout --limit 100

This command reads an Iceberg table located at path/to/table and outputs the results to the console (limited to 100 rows).

# read a local iceberg table, write to bigquery
sling run --src-stream file://path/to/table \
  --src-options '{format: iceberg}' \
  --tgt-conn bigquery \
  --tgt-object bq_schema.bq_table

This command reads an Iceberg table located at path/to/table and writes it to our BigQuery connection.

Reading Iceberg in Replication YAML

You can also specify Iceberg as a format in your replication.yaml file:

source: aws_s3
target: postgres

defaults:
  mode: full-refresh
  source_options:
    format: iceberg

streams:
  path/to/iceberg/table:
    object: my_schema.iceberg_table

This configuration will read data from an Iceberg table stored in AWS S3 and load it into a PostgreSQL table named my_schema.iceberg_table. The full-refresh mode indicates that the target table will be completely replaced with the data from the source Iceberg table during each replication run. We can also use wildcards to read multiple Iceberg tables. See the variables docs for a complete list of variables.

source: aws_s3
target: postgres

defaults:
  mode: full-refresh
  source_options:
    format: iceberg

streams:
  path/to/iceberg_tables/*:
    object: my_schema.{stream_file_name}

  path/to/more_iceberg_tables/prefix*:
    object: my_schema.{stream_file_name}

Running a replication is easily done with the sling run command:

sling run -d -r replication.yaml

See docs here to get started with Sling!

Conclusion

Iceberg support in Sling, powered by DuckDB, offers a powerful and efficient way to work with large-scale data stored in the Iceberg format. By leveraging Sling's intuitive configuration and DuckDB's performance, users can easily integrate Iceberg tables into their data pipelines and analytics workflows.

As the Apache Iceberg ecosystem continues to grow and evolve, we anticipate expanding Sling's capabilities to include writing to Iceberg tables and supporting more advanced Iceberg features. This will further enhance Sling's position as a versatile tool for modern data engineering tasks.

We encourage users to explore the Iceberg integration in Sling and provide feedback. Your input is valuable in shaping the future development of this feature and ensuring it meets the diverse needs of the data community.

Sling aims to augment the exporting/loading data process into a positive and potentially enjoyable experience. It focuses on 3 of data types interfaces:

• From File Systems to Databases
• From Databases to Databases
• From Databases to File Systems

The list of connections that Sling supports continues to grow. You can see the full list here, but it supports all the major platforms including Clickhouse, DuckDB, Google BigQuery, Google BigTable, MariaDB, MongoDB, MotherDuck, MySQL, Oracle, PostgreSQL, Prometheus, Redshift, Snowflake, SQL Server, SQLite, StarRocks and Trino.

Prometheus

Prometheus is an open-source time-series database originally built by SoundCloud. It is a powerful and flexible monitoring solution ideal for gathering, querying, and alerting on metrics data in dynamic and distributed environments. Its pull-based model, multi-dimensional data model, PromQL query language, and alerting capabilities make it a popular choice for monitoring modern cloud-native applications and infrastructure.

Export data from Prom

An issue faced dealing with Prometheus is the difficulty in exporting data from it. Fortunately, Sling handles this without a skip.

Install Sling

See here for details on how to install Sling. It is usually a simple command, such as:

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

You should be able to run the sling command at this point.

Set up Connections

Let's now set up the connections. In this blog post, we will use Postgres as the destination database, however the steps are the same if you'd like to load into a different database.

Setting up Prometheus

See here for more details on the keys accepted. But we can simply do:

$ sling conns set prometheus type=prometheus http_url="http://localhost:9090" api_key="xxxxxxxxxxxxxxxxxxxxxx"

$ sling conns test prometheus
6:01PM INF success!

# get list of metrics
$ sling conns discover prometheus --column
+------------+------------+------------+-----+----------------------------------+-------------+--------------+
| DATABASE   | SCHEMA     | TABLE      |  ID | COLUMN                           | NATIVE TYPE | GENERAL TYPE |
+------------+------------+------------+-----+----------------------------------+-------------+--------------+
| prometheus | prometheus | prometheus |   1 | go_gc_duration_seconds           | summary     | bigint       |
| prometheus | prometheus | prometheus |   2 | go_goroutines                    | gauge       | bigint       |
| prometheus | prometheus | prometheus |   3 | go_info                          | gauge       | bigint       |
| prometheus | prometheus | prometheus |   4 | go_memstats_alloc_bytes          | gauge       | bigint       |
| prometheus | prometheus | prometheus |   5 | go_memstats_alloc_bytes_total    | counter     | bigint       |
| prometheus | prometheus | prometheus |   6 | go_memstats_buck_hash_sys_bytes  | gauge       | bigint       |
| prometheus | prometheus | prometheus |   7 | go_memstats_frees_total          | counter     | bigint       |
.....

Setting up Postgres

Similarly, we will again use the sling conns set command, this time for our PG connection. See here for more details.

$ sling conns set postgres url="postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"

$ sling conns test postgres
6:03PM INF success!

Great, we are now ready to move data!

Run with CLI Flags

# export results to stdout, with start time from 2 months ago: "now-2M"
$ ./sling run --src-conn prometheus \
    --src-stream 'sum(go_gc_duration_seconds) by (job, instance, quantile) # {"start": "now-2M"}' \
    --stdout --limit 10 -d

# load into PG
$ ./sling run --src-conn prometheus \
    --src-stream 'sum(go_gc_duration_seconds) by (job, instance, quantile) # {"start": "now-2M"}' \
    --tgt-conn postgres --tgt-object public.gc_duration_by_job \
    --mode full-refresh

Time Filters

Adding time filters, simply add a suffix to your query. Accepts a JSON value with keys start, end and step:

• # {"start": "now-2M"}

• # {"start": "now-2M", "end": "now-1d"}

• # {"start": "now-2M", "end": "now-1d", "step": "1d"}

Run with Replication

We can also use a replication. Replications are the best way to use sling in a reusable manner. The defaults key allows reusing your inputs with the ability to override any of them in a particular stream. Both YAML or JSON files are accepted.

# replication.yaml
source: prometheus
target: postgres

defaults:
  object: prometheus.{stream_name}
  mode: full-refresh

streams:
  gc_duration_by_job:
    sql: 'sum(go_gc_duration_seconds) by (job, instance, quantile) # {"start": "now-2M", "end": "now-1d", "step": "1d"}'

# incremental load, last 2 days of data, hourly
  go_memstats_alloc_bytes_total:
    sql: 'sum(go_memstats_alloc_bytes_total) by (job, instance, quantile) # {"start": "now-2d"}'
    primary_key: [timestamp, job, instance, quantile]
    update_key: timestamp
    mode: incremental

We can run the replication like this:

sling run -r replication.yaml

Conclusion

We went over on how easy it was to export data with Sling from Prometheus. Feel free to check out other examples here: https://docs.slingdata.io.

Let's look at how we can easily export data from StarRocks into a local DuckDB database with Sling, a flexible command-line interface (CLI) data integration tool that enables rapid extraction and loading of data directly from the terminal.

StarRocks

StarRocks is a powerful distributed, columnar storage database system designed for real-time analytics. It can scale to Petabytes and connects to systems like HDFS, Apache Spark, Apache Flink and Apache Kafka. It is backed by CelerData, a $60 million VC-funded startup, and aims to be an open-source replacement for Snowflake, BigQuery, and Redshift. This makes it suitable for many analytics use cases, such as business intelligence, ad hoc querying, real-time events and even machine learning / AI-driven data processing.

DuckDB

DuckDB on the other hand, is a lightweight database dubbed as the SQLite of data warehouses. It is a local database saved in a single file (like SQLite) and enables many kinds of analytical use cases right on your machine, since the data is stored in columnar fashion. This is especially convenient when one wants to join several tables and do local analytics without dealing with external latencies and workloads.

Export Data with Sling

As with have it, Sling can Read/Write from both of these databases. Let's go over the steps.

First, let's install Sling. See here to see how to do so. It is usually a simple command, such as:

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

Next, we'll set up StarRocks. If you don't have a StarRocks instance running, you can quickly launch a development instance on your machine with docker and load data into it.

docker run --rm -p 9030:9030 -p 8030:8030 -p 8040:8040 -it starrocks/allin1-ubuntu

Now we can create a connection for our StarRocks database with Sling and test connectivity. See here for more details on configuration.

$ sling conns set starrocks url="starrocks://root:@localhost:9030/sling"
10:01PM INF connection `starrocks` has been set in /Users/me/.sling/env.yaml. Please test with `sling conns test starrocks`

$ sling conns test starrocks
10:01PM INF success!

Now let's set up our DuckDB connection. Since it is an embedded database, we don't need to install anything. In fact, Sling will auto-download the binary to interface with DuckDB the first time. See here for more details on configuring DuckDB.

$ sling conns set duckdb url="duckdb:///tmp/sling/duck.db"
10:02PM INF connection `duckdb` has been set in /Users/me/.sling/env.yaml. Please test with `sling conns test duckdb`

$ sling conns test duckdb
10:02PM INF success!

Great, we're ready to export some data into DuckDB.

Replication

Let's assume you already have data in your StarRocks instance, in the database public(see here on how to load data into StarRocks). If you'd like to export all the tables in that database, you'd create a replication like this:

# replication.yaml
source: starrocks
target: duckdb

defaults:
    object: main.{stream_schema}_{stream_table}
    mode: full-refresh

streams:
  # all tables in schema `public`
  public.*:

 # only one table
  salesforce.account:
    object: main.salesforce_account
    mode: full-refresh

We can run the replication like this:

sling run -r replication.yaml

10:56PM INF Sling Replication [24 streams] | starrocks -> duckdb

10:56PM INF [1 / 24] running stream "public"."call_center"
10:56PM INF connecting to source database (starrocks)
10:56PM INF connecting to target database (duckdb)
10:56PM INF reading from source database
10:56PM INF writing to target database [mode: full-refresh]
10:56PM INF streaming data
10:56PM INF created table main.public_call_center
10:56PM INF inserted 62347 rows into main.public_call_center in 10 secs [6234 r/s] [20.0 MB]
10:56PM INF execution succeeded
10:56PM INF [2 / 24] running stream "public"."catalog_page"
10:56PM INF connecting to source database (starrocks)
10:56PM INF connecting to target database (duckdb)
10:56PM INF reading from source database
10:56PM INF writing to target database [mode: full-refresh]
10:56PM INF streaming data
10:56PM INF created table main.public_catalog_page
10:56PM INF inserted 11718 rows into main.public_catalog_page in 0 secs [14,126 r/s] [2.0 MB]
10:56PM INF execution succeeded
.......
11:15PM INF [24 / 24] running stream "salesforce"."account"
11:15PM INF connecting to source database (starrocks)
11:15PM INF connecting to target database (duckdb)
11:15PM INF reading from source database
11:15PM INF writing to target database [mode: full-refresh]
11:15PM INF streaming data
11:15PM INF created table main.salesforce_account
11:15PM INF inserted 71654 rows into main.salesforce_account in 2 secs [30,073 r/s] [12 MB]
11:15PM INF execution succeeded
11:20PM INF Sling Replication Completed in 19m 34s | starrocks -> duckdb | 24 Successes | 0 Failures

So easy! See many more examples here: https://docs.slingdata.io/sling-cli/run/examples

Conclusion

We went over on how easy it was to install Sling, and export data from StarRocks into DuckDB. Feel free to check out other examples here: https://docs.slingdata.io.

Let's look at how we can easily export load data into a StarRocks from most major databases with Sling, a versatile CLI data integration tool which allows you to quickly extract and load data right from the terminal.

Sling is a tool with a goal of making the experience of ingesting data a positive, even pleasant one. Sling focuses on 3 of data types interfaces:

• From File Systems to Databases

• From Databases to Databases

• From Databases to File Systems

The list of connections that Sling supports continues to grow. You can see the full list here, but it supports all the major platforms including Clickhouse, DuckDB, Google BigQuery, Google BigTable, MariaDB, MongoDB, MotherDuck, MySQL, Oracle, PostgreSQL, Prometheus, Redshift, Snowflake, SQL Server, SQLite, StarRocks and Trino.

StarRocks

StarRocks is a powerful distributed, columnar storage database system designed for real-time analytics. One of the key features is that it supports several table type designs which can meet varying business requirements:

• Duplicate Key table: each of the records as a separate row
• Aggregate table: the aggregated record as a row
• Primary Key table: only the most recently loaded record as a row (based on key)

This makes it suitable for many analytics use cases, such as business intelligence, ad hoc querying, real-time events and even machine learning / AI-driven data processing.

Load Data with Sling

First, let us install Sling. See here on details how to do so. It is usually a simple command, such as:

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

You should be able to run the sling command at this point.

Next, we'll set up StarRocks. If you don't have a StarRocks instance running, you can quickly launch a development instance on your machine with docker and load data into it.

docker run --rm -p 9030:9030 -p 8030:8030 -p 8040:8040 -it starrocks/allin1-ubuntu

Now we can create a connection for our StarRocks database with Sling and test connectivity. It is important to set the fe_url so we can Stream Load into StarRocks See here for more details on configuration.

$ sling conns set starrocks url="starrocks://root:@localhost:9030/sling" fe_url="http://root:@localhost:8030"
8:01PM INF connection `starrocks` has been set in /Users/me/.sling/env.yaml. Please test with `sling conns test starrocks`

# test our connection
$ sling conns test starrocks
8:01PM INF success!

Great. In this tutorial, we will be extracting data from a MySQL database. We can set up the connection as shown below (see here for more details).

$ sling conns set mysql url="mysql://admin:password@localhost:3306/mysql"
8:02PM INF connection `mysql` has been set in /Users/me/.sling/env.yaml. Please test with `sling conns test mysql`

# test our connection
$ sling conns test mysql
8:02PM INF success!

# discover our tables
$ sling conns discover mysql
+-----+--------------------+----------------------------------------------+-------+---------+
|   # | SCHEMA             | NAME                                         | TYPE  | COLUMNS |
+-----+--------------------+----------------------------------------------+-------+---------+
|   1 | information_schema | ADMINISTRABLE_ROLE_AUTHORIZATIONS            | table |       9 |
|   2 | information_schema | APPLICABLE_ROLES                             | table |       9 |
|   3 | information_schema | CHARACTER_SETS                               | table |       4 |
|   4 | information_schema | CHECK_CONSTRAINTS                            | table |       4 |
|   5 | information_schema | COLLATIONS                                   | table |       7 |
|   6 | information_schema | COLLATION_CHARACTER_SET_APPLICABILITY        | table |       2 |
.....

Great, we're ready to load some data into StarRocks!

Replication

We'll be using a replication to define what tables Sling should load. Replications are the best way to use sling in a reusable manner. The defaults key allows reusing your inputs with the ability to override any of them in a particular stream. Both YAML or JSON files are accepted.

# replication.yaml
source: mysql
target: starrocks

defaults:
    object: main.{stream_schema}_{stream_table}
    mode: full-refresh

streams:
  # all tables in schema `mysql`
  mysql.*:

 # only one table with specific duplicate keys
  finance.account_sales:
    mode: truncate
    target_options:
      table_keys:
        duplicate: [account_id, sale_id]

We can run the replication like this:

sling run -r replication.yaml

09:14PM INF Sling Replication [5 streams] | mysql -> starrocks

09:14PM INF [1 / 5] running stream "mysql"."accounts"
09:14PM INF connecting to source database (mysql)
09:14PM INF connecting to target database (starrocks)
09:14PM INF reading from source database
09:14PM INF writing to target database [mode: full-refresh]
09:14PM INF streaming data
09:14PM INF importing into StarRocks via stream load
09:14PM INF created table main.mysql_accounts
09:14PM INF inserted 62347 rows into main.mysql_accounts in 10 secs [6234 r/s] [20.0 MB]
09:14PM INF execution succeeded
09:14PM INF [2 / 5] running stream "mysql"."orders"
09:14PM INF connecting to source database (mysql)
09:14PM INF connecting to target database (starrocks)
09:14PM INF reading from source database
09:14PM INF writing to target database [mode: full-refresh]
09:14PM INF streaming data
09:14PM INF importing into StarRocks via stream load
09:14PM INF created table main.mysql_orders
09:14PM INF inserted 716540 rows into main.mysql_orders in 40 secs [17,973 r/s] [120 MB]
09:14PM INF execution succeeded
.......
09:15PM INF [5 / 5] running stream "finance"."account_sales"
09:15PM INF connecting to source database (mysql)
09:15PM INF connecting to target database (starrocks)
09:15PM INF reading from source database
09:15PM INF writing to target database [mode: full-refresh]
09:15PM INF streaming data
09:15PM INF importing into StarRocks via stream load
09:15PM INF created table main.finance_account_sales
09:15PM INF inserted 11718 rows into main.finance_account_sales in 0 secs [14,126 r/s] [2.0 MB]
09:15PM INF execution succeeded
09:15PM INF Sling Replication Completed in 1m 34s | mysql -> starrocks | 5 Successes | 0 Failures

So easy! See many more examples here: https://docs.slingdata.io/sling-cli/run/examples

Conclusion

We went over on how easy it was to install Sling, and export data from MySQL into StarRocks. Feel free to check out other examples here: https://docs.slingdata.io.

Sling is an easy-to-use, lightweight data loading tool, typically run from the CLI. It focuses on data movement between Database to Database, File System to Database and Database to File System. See here for the list of Connectors.

Today, we're going to be looking at parsing a complex JSON file (a dbt manifest file), extracting a sub-set of data and writing it into a CSV file for further analysis.

JMESPath

JMESPath is the most popular query / transformation language for JSON. It has many libraries ready to use, including Go, which is what Sling is built with. Some key features of JMESPath:

1. Filtering and Projection: You can use JMESPath expressions to filter and project specific elements or attributes from JSON data. This allows you to focus on the relevant parts of a JSON structure.

2. Functions: JMESPath includes a set of built-in functions that can be used in expressions for various tasks, such as string manipulation, mathematical operations, and more. These functions enhance the flexibility of JMESPath queries.

3. Multi-level Queries: JMESPath supports querying JSON documents with nested structures. You can navigate through arrays and objects to access the data at different levels within the JSON hierarchy.

4. Pipes and Operators: JMESPath expressions can include pipes (|) and various operators for combining and transforming data. This allows you to create more complex queries and transformations.

Running Sling

Let us assume we are working from our dbt project folder, and that we have run the dbt compile command. This would have generated a target folder, containing the beefy manifest.json file. We will be extracting the models dataset from that file.

After installing sling, we are good to go. From the root of our dbt project (folder containing the dbt_project.yaml), run the following command:

sling run \
  --src-stream file://./target/manifest.json \
  --tgt-object file://./target/models.csv \
  --src-options '{
      flatten: true,
      jmespath: "nodes.*.{resource_type: resource_type, database: database, schema: schema, name: name, relation_name: relation_name, original_file_path: original_file_path, materialized: config.materialized }",
    }'

Let's go over the inputs we provided:

• --src-stream: this is the source stream that we want to read from, which is the dbt manifest file in the target folder.

• --tgt-object: this is the destination file path that we want to write to. Here we are writing to a CSV file, but Sling can write to JSON and Parquet as well. We'd just need to just change the extension to .json or .parquet.

• --src-options: Here, we specify the source options for Sling to use.

  • flatten: this indicates sling to flatten on nested data. This basically creates columns for individual nested nodes.

  • jmespath: this is where we define the JMESPath transform logic.

Let's take a look at the output.

6:34AM INF reading from source file system (file)
6:34AM INF writing to target file system (file)
6:34AM INF wrote 41 rows to file://./target/models.csv [1,432 r/s]
6:34AM INF execution succeeded

Great! Your data is ready for further analysis. Let look at a sample of the output CSV file:

$ head ./target/models.csv
database,materialized,name,original_file_path,relation_name,resource_type,schema
MY_DATABASE,incremental,track_events_raw,models/track_events_raw.sql,MY_DATABASE.dbt_dev.track_events_raw,model,dbt_dev
MY_DATABASE,test,test_mapping_global_graph_uuid,tests/test_mapping_global_graph_uuid.sql,,test,dev_dbt_test__audit
MY_DATABASE,incremental,track_events,models/track_events.sql,MY_DATABASE.dbt_dev.track_events,model,dbt_dev
MY_DATABASE,table,mapping_invalid_shopify,models/mapping/mapping_invalid_shopify.sql,MY_DATABASE.dbt_dev.mapping_invalid_shopify,model,dbt_dev
MY_DATABASE,table,mapping_global,models/mapping/mapping_global.sql,MY_DATABASE.dbt_dev.mapping_global,model,dbt_dev
MY_DATABASE,test,not_null_edges_edge_b,models/schema.yml,,test,dev_dbt_test__audit
MY_DATABASE,test,unique_id_graph_edge,models/schema.yml,,test,dev_dbt_test__audit
MY_DATABASE,table,mapping_invalid_anonymous,models/mapping/mapping_invalid_anonymous.sql,MY_DATABASE.dbt_dev.mapping_invalid_anonymous,model,dbt_dev
MY_DATABASE,test,test_mapping_global_graph_unique,tests/test_mapping_global_graph_unique.sql,,test,dev_dbt_test__audit

Neat! In seconds, we were able to create a perfect CSV file, with the columns that we want from the JSON nested nodes. And that's not it, sling can readily load this into a target database as well. See the docs for more examples!

Sling is the go-to tool to quickly extract and load data, whether you're handling CSV, Parquet, Avro or JSON files, moving data across databases, or even extracting custom queries, right from your command line. For more details, feel free to browse the docs. When it comes to a drifting table schema, Sling is no exception in detecting those changes.

What is schema evolution?

To put it simply, schema evolution in databases refers to the process of modifying the structure of a database schema over time to accommodate changes in requirements, business rules, or application enhancements. This is something that happens frequently, and needs to be handled carefully.

Slinging like a champ

When using Sling to extract/load data in a incremental manner, it will attempt to match whatever columns are present in both the source stream and target table. If an extra column is present in the source stream, it will add it in the target table. If no columns match from source stream at all, it will error. At least the primary_key or update_key must be present in the target table.

See below for a simple example, mimicking the addition and removal of columns.

# Initial data

$ echo 'a,b,c
1,2,3
4,5,6' > test1.csv

$ sling run \
  --src-stream file://./test1.csv \
  --tgt-conn postgres \
  --tgt-object public.test1

<...log output omitted>

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at
1,2,3,1707869559
4,5,6,1707869559

# test2.csv is missing column b

echo 'a,c
7,8' > test2.csv

$ sling run \
  --src-stream file://./test2.csv \
  --tgt-conn postgres \
  --tgt-object public.test1 \
  --mode incremental \
  --primary-key a

<...log output omitted>

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at
1,2,3,1707869559
4,5,6,1707869559
7,,8,1707869689

# test3.csv is missing column b, c and has extra column d

$ echo 'a,d
9,10' > test3.csv

$ sling run \
  --src-stream file://./test3.csv \
  --tgt-conn postgres \
  --tgt-object public.test1 \
  --mode incremental \
  --primary-key a

<...log output omitted>

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at,d
1,2,3,1707869559,
4,5,6,1707869559,
7,,8,1707869689,
9,,,1707870320,10

We can see that sling handled the changes properly, in a non-destructive manner. If the source stream were from a database, the same rules would apply, whether a column disappeared or appeared.

While MySQL is no longer a young buck, it is surprisingly still a widely-used open-source relational database management system (RDBMS). Some of the key features of MySQL include its ability to support high-performance data handling and large-scale data processing, its support for multiple storage engines, and its ability to be easily integrated with other software and applications.

JSON on the other hand, is a lightweight, human-readable, and easy-to-use data interchange format which is based on a subset of the JavaScript programming language. One of the key advantages of using JSON is that it is easy to work with, both for developers and for machines. It is a hierarchical data format, which means that data is organized into a tree-like structure with nested elements. This makes it easy to represent complex data structures and relationships, and allows for efficient data manipulation and querying. It is used everywhere these days!

Your Loading Situation

So you are in a situation where you not only need to load your JSON files into a MySQL database, but you also need to flatten each file. Well, wonder no more! Sling can be a useful tool to help you accomplish this task, as it is a command-line tool that allows you to efficiently transfer data between files and databases.

By using Sling, it can flatten your JSON file, auto-create your table DDL and then load the data into it from your local drive. This can be a quick and easy way to get your data into a format that is ready for analysis and processing.

Installing Sling CLI

Sling is very easy to install, no matter which operating system you are using. It's written in the go programming language, so it compiles into a single binary file. It also supports many databases. Please see here for the full list of compatible connectors.

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

Please see here for additional installation options (such as downloading binaries). There is also a Python wrapper library, which is useful if you prefer interacting with Sling inside of Python. Once installed, we should be able to run the sling command.

Loading from our Local Drive

Let us assume that we desire to ingest the following JSON array file, which includes nested objects:

[
 {
   "_id": "638f4cab1c024be3cadd3ca5",
   "isActive": true,
   "balance": "3,148.57",
   "picture": "http://placehold.it/32x32",
   "age": 35,
   "name": "Joann Kim",
   "company": {
     "name": "PROXSOFT",
     "email": "joannkim@proxsoft.com",
     "phone": "+1 (836) 517-2388",
     "address": "951 Ellery Street, Norwood, Palau, 2947",
     "about": "Labore id et sunt cupidatat dolore aute. Sit laborum nulla pariatur nisi dolore consectetur ex exercitation cupidatat ex ex reprehenderit duis. Eiusmod ut aliquip laborum enim proident ex cupidatat ut velit qui amet dolor tempor enim.\r\n",
     "registered": "2020-01-15T02:07:00 +03:00",
     "latitude": -60.00954,
     "longitude": -55.92312
   },
   "tags": [
     "elit",
     "veniam"
   ]
 },
...
]

In our example, our file will be located at path /tmp/records.json. And if you'd like to ingest various similarly structured files inside a folder (say /path/to/my/folder), you could just input that instead of the file path, Sling will read all files in the folder! Only make sure to add the file:// prefix. See below.

# first let's set our MYSQL connection. Sling can pick up connection URLs from environment variables
$ export MYSQL=' mysql://admin:password@mysql.host:3306/mysql'

# let's check and test our MYSQL connection
$ sling conns list
+------------+-------------+---------------+
| CONN NAME  | CONN TYPE   | SOURCE        |
+------------+-------------+---------------+
| MYSQL      | DB - MySQL  | env variable  |
+------------+-------------+---------------+

$ sling conns test MYSQL
11:19PM INF success!

# awesome, now we can run our task
$ sling run --src-stream file:///tmp/records.json --tgt-conn MYSQL --tgt-object mysql.records --mode full-refresh
11:19PM INF connecting to target database (mysql)
11:19PM INF reading from source file system (file)
11:19PM INF writing to target database [mode: full-refresh]
11:19PM INF streaming data
11:19PM INF dropped table mysql.records
11:19PM INF created table mysql.records
11:19PM INF inserted 500 rows in 2 secs
11:19PM INF execution succeeded

How easy was that? Now let's repeat this again with debug mode enabled (flag -d) to see the created table DDL, and this time let's pipe in the data with the cat command:

$ cat /tmp/records.json | sling run -d --tgt-conn MYSQL --tgt-object mysql.records --mode full-refresh
11:20PM INF connecting to target database (mysql)
11:20PM INF reading from stream (stdin)
11:20PM INF writing to target database [mode: full-refresh]
11:20PM DBG drop table if exists mysql.records_tmp
11:20PM DBG table mysql.records_tmp dropped
11:20PM DBG create table if not exists mysql.records_tmp (`data` json)
11:20PM INF streaming data
11:20PM DBG select count(1) cnt from mysql.records_tmp
11:20PM DBG drop table if exists mysql.records
11:20PM DBG table mysql.records dropped
11:20PM INF dropped table mysql.records
11:20PM DBG create table if not exists mysql.records (`data` json)
11:20PM INF created table mysql.records
11:20PM DBG insert into `mysql`.`records` (`data`) select `data` from `mysql`.`records_tmp`
11:20PM DBG inserted rows into `mysql.records` from temp table `mysql.records_tmp`
11:20PM INF inserted 500 rows in 2 secs [170 r/s]
11:20PM DBG drop table if exists mysql.records_tmp
11:20PM DBG table mysql.records_tmp dropped
11:20PM INF execution succeeded

We can see that the DDL used was create table if not exists mysql.records (`data` json).

Flattening our JSON Records

This time, let's flatten the records when ingesting. When we flatten, Sling will create individual columns for each of the keys in the record. We can do so by adding --src-options 'flatten: true' as a flag. See here for all options:

$ sling run -d --src-stream file:///tmp/records.json --src-options 'flatten: true' --tgt-conn MYSQL --tgt-object mysql.records --mode full-refresh
11:23PM INF connecting to target database (mysql)
11:23PM INF reading from source file system (file)
11:23PM DBG reading datastream from /tmp/records.json
11:23PM INF writing to target database [mode: full-refresh]
11:23PM DBG drop table if exists mysql.records_tmp
11:23PM DBG table mysql.records_tmp dropped
11:23PM DBG create table if not exists mysql.records_tmp (`_id` mediumtext,
`age` bigint,
`balance` mediumtext,
`company__about` mediumtext,
`company__address` mediumtext,
`company__email` mediumtext,
`company__latitude` decimal(30,9),
`company__longitude` decimal(30,9),
`company__name` mediumtext,
`company__phone` mediumtext,
`company__registered` mediumtext,
`isactive` char(5),
`name` mediumtext,
`picture` mediumtext,
`tags` json,
`_sling_loaded_at` bigint)
11:23PM INF streaming data
11:23PM DBG select count(1) cnt from mysql.records_tmp
11:23PM DBG drop table if exists mysql.records
11:23PM DBG table mysql.records dropped
11:23PM INF dropped table mysql.records
11:23PM DBG create table if not exists mysql.records (`_id` mediumtext,
`age` bigint,
`balance` mediumtext,
`company__about` mediumtext,
`company__address` mediumtext,
`company__email` mediumtext,
`company__latitude` decimal(30,9),
`company__longitude` decimal(30,9),
`company__name` mediumtext,
`company__phone` mediumtext,
`company__registered` mediumtext,
`isactive` varchar(255),
`name` mediumtext,
`picture` mediumtext,
`tags` json,
`_sling_loaded_at` bigint)
11:23PM INF created table mysql.records
11:23PM DBG insert into `mysql`.`records` (`_id`, `age`, `balance`, `company__about`, `company__address`, `company__email`, `company__latitude`, `company__longitude`, `company__name`, `company__phone`, `company__registered`, `isactive`, `name`, `picture`, `tags`, `_sling_loaded_at`) select `_id`, `age`, `balance`, `company__about`, `company__address`, `company__email`, `company__latitude`, `company__longitude`, `company__name`, `company__phone`, `company__registered`, `isactive`, `name`, `picture`, `tags`, `_sling_loaded_at` from `mysql`.`records_tmp`
11:23PM DBG inserted rows into `mysql.records` from temp table `mysql.records_tmp`
11:23PM INF inserted 500 rows in 3 secs [151 r/s]
11:23PM DBG drop table if exists mysql.records_tmp
11:23PM DBG table mysql.records_tmp dropped
11:23PM INF execution succeeded

Amazing, we can see the DDL now is:

create table if not exists mysql.records (`_id` mediumtext,
`age` bigint,
`balance` mediumtext,
`company__about` mediumtext,
`company__address` mediumtext,
`company__email` mediumtext,
`company__latitude` decimal(30,9),
`company__longitude` decimal(30,9),
`company__name` mediumtext,
`company__phone` mediumtext,
`company__registered` mediumtext,
`isactive` varchar(255),
`name` mediumtext,
`picture` mediumtext,
`tags` json,
`_sling_loaded_at` bigint)

Conclusion

As demonstrated, Sling has a wide compatibility with various storage systems. You can not only ingest JSON files, but CSV, XML files as well as various database systems. If you have any questions, comments and/or facing issues, please feel free to email us at support @ slingdata.io.

BigQuery is a cloud-based, fully managed, serverless data warehouse that enables you to analyze large and complex datasets using SQL queries. It is a scalable, highly-performant, and cost-effective solution that can handle petabyte-scale data with ease. A, BigQuery is a part of the Google Cloud Platform (GCP) and is built on top of Google's infrastructure, it is highly reliable and available.

However, suppose that you have files that you wish to ingest into BigQuery, but the files are located on AWS S3. In fact, they could be located on any S3 Compatible systems, such as AWS S3, DigitalOcean Spaces, BackBlaze B2, Cloudflare R2, MinIO, Wasabi. What to do? Manually transfer all those to a Google Cloud Storage bucket? How about the destination table that needs to be created? Will you manually define all the columns that are part of the JSON file? There is a better way, use Sling.

Sling CLI

Sling CLI is a versatile tool that enables you to transfer data quickly and efficiently between files and databases, as well as between databases. It is capable of reading files in the JSON format, among other file types. Please see here for the full list of compatible connectors.

Installation

Installing Sling is straightforward, regardless of the operating system you are using. This is because it is built using the go programming language and compiles to a single binary file, making it easy to install and run on any system.

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

Please see here for additional installation options (such as downloading binaries). Additionally, there is a Python wrapper library available for Sling that allows you to interact with it from within a Python environment. This can be useful if you prefer to work with Python for data management tasks.

Once installed, you should also be able to use the sling command to access Sling's functionality from the command line.

Loading the data

Let us assume that we desire to ingest the following JSON array file, which includes nested objects:

[
 {
   "_id": "638f4cabc6a57ef261c4d28c",
   "isActive": false,
   "balance": "3,560.61",
   "picture": "http://placehold.it/32x32",
   "age": 26,
   "name": "Winters Koch",
   "company": {
     "name": "EXOZENT",
     "email": "winterskoch@exozent.com",
     "phone": "+1 (808) 519-3916",
     "address": "744 Strickland Avenue, Mooresburg, Connecticut, 1850",
     "about": "Aute cupidatat sint incididunt ullamco sint velit consectetur consectetur nostrud eiusmod velit nostrud voluptate ipsum. Et velit officia excepteur excepteur eu nisi occaecat. Sunt culpa eu nostrud in sint occaecat nulla labore pariatur adipisicing ex irure voluptate qui.\r\n",
     "registered": "2020-07-12T09:54:18 +03:00",
     "latitude": 43.588661,
     "longitude": -136.979445
   },
   "tags": [
     "deserunt",
     "duis"
   ]
 },
...
]

Let us also assume that the file is located on our S3 bucket (at path s3://sling-bucket/records.json), it’s as easy as running the following commands.

# let's set our connections. We can set them with `sling conns`
$ sling conns set AWS_S3 type=s3 bucket=sling-bucket access_key_id=$ACCESS_KEY_ID secret_access_key="$SECRET_ACCESS_KEY"
10:14PM INF connection `AWS_S3` has been set in /Users/fritz/.sling/env.yaml. Please test with `sling conns test AWS_S3`

$ sling conns set BIGQUERY type=bigquery project=my-project dataset=public key_file=/path/to/service.account.json
10:14PM INF connection `BIGQUERY` has been set in /Users/fritz/.sling/env.yaml. Please test with `sling conns test BIGQUERY`

# let's check and test our connections
$ sling conns list
+------------+------------------+-----------------+
| CONN NAME  | CONN TYPE        | SOURCE          |
+------------+------------------+-----------------+
| AWS_S3     | FileSys - S3     | sling env yaml  |
| BIGQUERY   | DB - BigQuery    | sling env yaml  |
+------------+------------------+-----------------+

$ sling conns test AWS_S3
10:15PM INF success!

$ sling conns test BIGQUERY
10:15PM INF success!

# great, now we can run our task
$ sling run --src-conn AWS_S3 --src-stream s3://sling-bucket/records.json --tgt-conn BIGQUERY --tgt-object public.records --mode full-refresh
10:18PM INF connecting to target database (bigquery)
10:18PM INF reading from source file system (s3)
10:18PM INF writing to target database [mode: full-refresh]
10:18PM INF streaming data
10:18PM INF importing into bigquery via local storage
10:18PM INF dropped table public.records
10:18PM INF created table public.records
10:18PM INF inserted 500 rows in 9 secs
10:18PM INF execution succeeded

Cool huh? We didn't even have to set anything up. If we were to ingest numerous files inside a folder (say s3://sling-bucket/path/to/my/folder), we could just input that instead of the file path, Sling will read all files in the folder.

Flattening our JSON Records

To flatten a JSON file when using Sling, you can specify the flatten: true in the source option as a flag. This will cause Sling to create individual columns for each key in the records, rather than storing the data in a nested format. For more information on Sling's configuration options, please see the following documentation: https://docs.slingdata.io/sling-cli/configuration. To specify the flatten: true option, you would use the --src-options flag, as shown below. We'll also use the -d to turn on debug mode, so we can see the DDL being created.

$ sling run -d --src-conn AWS_S3 --src-stream s3://sling-bucket/records.json --src-options 'flatten: true' --tgt-conn BIGQUERY --tgt-object public.records --mode full-refresh
10:23PM INF connecting to target database (bigquery)
10:23PM INF reading from source file system (s3)
10:23PM DBG reading datastream from s3://sling-bucket/records.json
10:23PM INF writing to target database [mode: full-refresh]
10:23PM DBG drop table if exists public.records_tmp
10:23PM DBG table public.records_tmp dropped
10:23PM DBG create table public.records_tmp (`_id` string,
`age` int64,
`balance` string,
`company__about` string,
`company__address` string,
`company__email` string,
`company__latitude` numeric,
`company__longitude` numeric,
`company__name` string,
`company__phone` string,
`company__registered` string,
`isactive` bool,
`name` string,
`picture` string,
`tags` json,
`_sling_loaded_at` int64)
10:23PM INF streaming data
10:23PM INF importing into bigquery via local storage
10:23PM DBG Loading /var/folders/49/1zc24t595j79t5mw7_t9gtxr0000gn/T/bigquery/public.records_tmp/2022-12-07T222336.731/part.01.0001.csv.gz
10:23PM DBG select count(1) cnt from public.records_tmp
10:23PM DBG comparing checksums []string{"_id", "age", "balance", "company__about", "company__address", "company__email", "company__latitude", "company__longitude", "company__name", "company__phone", "company__registered", "isactive", "name", "picture", "tags", "_sling_loaded_at"} vs []string{"_id", "age", "balance", "company__about", "company__address", "company__email", "company__latitude", "company__longitude", "company__name", "company__phone", "company__registered", "isActive", "name", "picture", "tags", "_sling_loaded_at"}: []string{"_id", "age", "balance", "company__about", "company__address", "company__email", "company__latitude", "company__longitude", "company__name", "company__phone", "company__registered", "isactive", "name", "picture", "tags", "_sling_loaded_at"}
10:23PM DBG drop table if exists public.records
10:23PM DBG table public.records dropped
10:23PM INF dropped table public.records
10:23PM DBG create table public.records (`_id` string,
`age` int64,
`balance` string,
`company__about` string,
`company__address` string,
`company__email` string,
`company__latitude` numeric,
`company__longitude` numeric,
`company__name` string,
`company__phone` string,
`company__registered` string,
`isactive` bool,
`name` string,
`picture` string,
`tags` json,
`_sling_loaded_at` int64)
10:23PM INF created table public.records
10:23PM DBG insert into `public`.`records` (`_id`, `age`, `balance`, `company__about`, `company__address`, `company__email`, `company__latitude`, `company__longitude`, `company__name`, `company__phone`, `company__registered`, `isactive`, `name`, `picture`, `tags`, `_sling_loaded_at`) select `_id`, `age`, `balance`, `company__about`, `company__address`, `company__email`, `company__latitude`, `company__longitude`, `company__name`, `company__phone`, `company__registered`, `isactive`, `name`, `picture`, `tags`, `_sling_loaded_at` from `public`.`records_tmp`
10:23PM DBG inserted rows into `public.records` from temp table `public.records_tmp`
10:23PM INF inserted 500 rows in 25 secs
10:23PM DBG drop table if exists public.records_tmp
10:23PM DBG table public.records_tmp dropped
10:23PM INF execution succeeded

Wonderful, notice the DDL:

create table public.records (`_id` string,
`age` int64,
`balance` string,
`company__about` string,
`company__address` string,
`company__email` string,
`company__latitude` numeric,
`company__longitude` numeric,
`company__name` string,
`company__phone` string,
`company__registered` string,
`isactive` bool,
`name` string,
`picture` string,
`tags` json,
`_sling_loaded_at` int64)

Conclusion

As demonstrated, Sling has a wide compatibility with various storage systems. You can not only ingest JSON files from cloud systems, but CSV, XML files as well as various database systems (such as PostgreSQL, RedShift, Oracle, SQL Server, Snowflake, and more). Give it a spin! If you have any questions, comments and/or facing issues, please feel free to email us at support @ slingdata.io.

PostgreSQL was not always as popular as it is today. Over the years, the open-source database has increasingly seen adoption and praise as it has been an extremely stable, versatile and cost-efficient system. Many teams even use it as their de facto Data Warehouse. It can easily handle terabytes of data, which is enough for 90% of businesses out there.

JSON File Structure

JSON itself is a very popular file format which grew in popularity with the rise of the internet. Due to its ease of readability and flexibility with any desired tree structure, it is used across systems, especially as a means to deliver messages. It is also often used when storing log messages, so it makes sense to have advanced capabilities in reading JSON files.

Using Sling CLI

Sling CLI is a powerful tool which allows you to efficiently move data from files into databases, from databases to databases, and from databases to files. One of the file type that it can read is JSON. Please see here for the full list of compatible connectors.

Installation

Installing Sling is super-easy regardless of the operating system that you are using. Since it is built in go, it compiles to a single binary.

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# On Linux
curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

Please see here for additional installation options (such as downloading binaries). There is also a Python wrapper library, which is useful if you prefer interacting with Sling inside of Python. Once installed, we should be able to run the sling command.

Loading from Local Drive

Let us assume that we desire to ingest the following JSON array file, which includes nested objects:

[
 {
   "_id": "638f4bed04fa9d18613af369",
   "isActive": false,
   "balance": "1,916.14",
   "picture": "http://placehold.it/32x32",
   "age": 30,
   "name": "Ebony Jensen",
   "company": {
     "name": "ZAPHIRE",
     "email": "ebonyjensen@zaphire.com",
     "phone": "+1 (980) 571-3202",
     "address": "958 Elmwood Avenue, Gouglersville, Massachusetts, 9126",
     "about": "Sunt velit adipisicing dolore aliqua aliquip. Amet aliqua Lorem ea est laboris magna. Commodo aliqua sint cupidatat qui ut cupidatat nostrud sunt proident duis sunt commodo eu.\r\n",
     "registered": "2018-05-06T02:20:10 +03:00",
     "latitude": -1.126455,
     "longitude": -161.187586
   },
   "tags": [
     "minim",
     "commodo"
   ]
 },
...
]

If the file is located on our local hard drive (at path /tmp/records.json), it’s as easy as running the following command below. If you'd like to ingest numerous files inside a folder (say /path/to/my/folder), you could just input that instead of the file path, Sling will read all files in the folder. Only make sure to add the file:// prefix. See below.

# first let's set our PG connection. Sling can pick up connection URLs from environment variables
$ export POSTGRES='postgresql://postgres:myPassword@pghost:5432/postgres?sslmode=disable'

# let's check and test our PG connection
$ sling conns list
+------------+------------------+-----------------+
| CONN NAME  | CONN TYPE        | SOURCE          |
+------------+------------------+-----------------+
| POSTGRES   | DB - PostgreSQL  | env variable    |
+------------+------------------+-----------------+

$ sling conns test POSTGRES
8:05AM INF success!

# great, now we can run our task
$ sling run --src-stream file:///tmp/records.json --tgt-conn POSTGRES --tgt-object public.records --mode full-refresh
11:09AM INF connecting to target database (postgres)
11:09AM INF reading from source file system (file)
11:09AM INF writing to target database [mode: full-refresh]
11:09AM INF streaming data
11:09AM INF dropped table public.records
11:09AM INF created table public.records
11:09AM INF inserted 500 rows in 0 secs [1,556 r/s]
11:09AM INF execution succeeded

Easy, huh? Let's do this again in debug mode (with flag -d) to see the created table DDL, and this time let's pipe in the data with the cat command:

$ cat /tmp/records.json | sling run -d --tgt-conn POSTGRES --tgt-object public.records --mode full-refresh
11:10AM DBG type is file-db
11:10AM INF connecting to target database (postgres)
11:10AM INF reading from stream (stdin)
11:10AM DBG reading datastream from /tmp/records.json
11:10AM INF writing to target database [mode: full-refresh]
11:10AM DBG drop table if exists public.records_tmp cascade
11:10AM DBG table public.records_tmp dropped
11:10AM DBG create table if not exists public.records_tmp ("data" jsonb)
11:10AM INF streaming data
11:10AM DBG select count(1) cnt from public.records_tmp
11:10AM DBG drop table if exists public.records cascade
11:10AM DBG table public.records dropped
11:10AM INF dropped table public.records
11:10AM DBG create table if not exists public.records ("data" jsonb)
11:10AM INF created table public.records
11:10AM DBG insert into "public"."records" ("data") select "data" from "public"."records_tmp"
11:10AM DBG inserted rows into `public.records` from temp table `public.records_tmp`
11:10AM INF inserted 500 rows in 0 secs [1,778 r/s]
11:10AM DBG drop table if exists public.records_tmp cascade
11:10AM DBG table public.records_tmp dropped
11:10AM INF execution succeeded

We can see the DDL is create table if not exists public.records ("data" jsonb).

Flattening our JSON Records

Now, let's flatten the same file when ingesting. When we flatten, Sling will create individual columns for each of the keys in the records. We can do so by adding --src-options 'flatten: true' as a flag. See here for all options:

$ cat /tmp/records.json | sling run -d --src-options 'flatten: true' --tgt-conn POSTGRES --tgt-object public.records --mode full-refresh
11:13AM DBG type is file-db
11:13AM INF connecting to target database (postgres)
11:13AM INF reading from stream (stdin)
11:13AM DBG reading datastream from /tmp/records.json
11:13AM INF writing to target database [mode: full-refresh]
11:13AM DBG drop table if exists public.records_tmp cascade
11:13AM DBG table public.records_tmp dropped
11:13AM DBG create table if not exists public.records_tmp ("_id" varchar(255),
"age" integer,
"balance" varchar(255),
"company__about" text,
"company__address" varchar(255),
"company__email" varchar(255),
"company__latitude" numeric,
"company__longitude" numeric,
"company__name" varchar(255),
"company__phone" varchar(255),
"company__registered" varchar(255),
"isactive" bool,
"name" varchar(255),
"picture" varchar(255),
"tags" jsonb)
11:13AM INF streaming data
11:13AM DBG select count(1) cnt from public.records_tmp
11:13AM DBG drop table if exists public.records cascade
11:13AM DBG table public.records dropped
11:13AM INF dropped table public.records
11:13AM DBG create table if not exists public.records ("_id" varchar(255),
"age" integer,
"balance" varchar(255),
"company__about" text,
"company__address" varchar(255),
"company__email" varchar(255),
"company__latitude" numeric,
"company__longitude" numeric,
"company__name" varchar(255),
"company__phone" varchar(255),
"company__registered" varchar(255),
"isactive" bool,
"name" varchar(255),
"picture" varchar(255),
"tags" jsonb)
11:13AM INF created table public.records
11:13AM DBG insert into "public"."records" ("_id", "age", "balance", "company__about", "company__address", "company__email", "company__latitude", "company__longitude", "company__name", "company__phone", "company__registered", "isactive", "name", "picture", "tags") select "_id", "age", "balance", "company__about", "company__address", "company__email", "company__latitude", "company__longitude", "company__name", "company__phone", "company__registered", "isactive", "name", "picture", "tags" from "public"."records_tmp"
11:13AM DBG inserted rows into `public.records` from temp table `public.records_tmp`
11:13AM INF inserted 500 rows in 0 secs [1,153 r/s]
11:13AM DBG drop table if exists public.records_tmp cascade
11:13AM DBG table public.records_tmp dropped
11:13AM INF execution succeeded

Nice, notice the DDL:

create table if not exists public.records ("_id" varchar(255),
"age" integer,
"balance" varchar(255),
"company__about" text,
"company__address" varchar(255),
"company__email" varchar(255),
"company__latitude" numeric,
"company__longitude" numeric,
"company__name" varchar(255),
"company__phone" varchar(255),
"company__registered" varchar(255),
"isactive" bool,
"name" varchar(255),
"picture" varchar(255),
"tags" jsonb)

Conclusion

As demonstrated, Sling CLI is an effective tool to quickly ingest JSON data. You can not only ingest JSON files, but CSV, XML files as well as various database systems. If you have any questions, or comments and/or facing issues, please feel free to email us at support @ slingdata.io.

For as long as data practitioners have been loading files into Relational Database Management Systems (RDBMS), it has been necessary to first manually create the destination table, via Data Definition Language (DDL). This process is definitely not consistent, as it can range from being fairly quick to extremely tedious, especially for massive files with lots of columns. And what about unstructured / semi-structured files? Forget about it when data is deeply nested, it can be a huge pain to extract the values into their own proper columns (and determine their proper type).

Manually Defining the DDL

Let’s go through a typically scenario doing this today. Normally, a tool that would allow you to ingest a CSV file would need to first sample the first hundred or so rows, and have you confirm the column types. Now imagine if this files has 50+ columns! Ok, no biggie, you confirm the column types and the tool starts ingesting the 10 million row file. It’s chugging along well, and 2 million rows in, column 46, which was typed as an integer, suddenly encounters a string value, and the database errors out. Does that sound familiar? It’s a nightmare dealing with this, since the types determined from a sample cannot be guaranteed throughout the rest of the rows in the file. Thankfully, there is a nifty tool called sling.

Using Sling CLI to Load files into your Databases

Sling CLI is a powerful tool which allows you to efficiently move data from files into databases, from databases to databases, and from databases to files. It also accepts content via the standard input (stdin) and can output via the standard output (stdout) as well. In the next few sections, we will go over a few use-cases of ingesting files into different databases using Sling CLI. You'll notice that we do not have to define a table DDL. Sling automatically does this for us, as it continuously profiles the column values and can adjust the column type as required.

Furthermore, each database system has their own Bulk loading methodology. For example, PostgreSQL has the COPY function where the client streams in records, Snowflake (SF) has the COPY INTO function where SF ingests quickly from a Stage location, and BigQuery have its own APIs for Bulk Loading (such as from Google Cloud Storage or locally from a SDK client stream). Sling ensures to follow each of those methodologies so that data is ingested as efficiently as possible in a streaming fashion (holding small amount in memory).

OK, let us switch gears and get into the meat. We will go over the following use-cases:

• Loading a Google Cloud Storage CSV file into a PostgreSQL Database
• Loading a Local TSV File (via stdin) into a Snowflake Database
• Loading an Amazon S3 JSON file into a BigQuery Database

But first thing's first, we must install sling on our machine and configure our credentials

Installing sling on our machine

Since it is built in Go, it is offered as a binary for whatever system you are using.

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# Using Python Wrapper via pip
pip install sling

If you have a Linux system or desire to download the binary manually, please head over here. Once installed, you should be able to run the sling command.

Configuring our credentials

The next step is to set our credentials file (env.yaml), which needs to be located in the Sling home folder. The env.yaml is sort of a central location to store all our local credentials. If you want to try one of the use-case demonstrated, simply ensure that the needed connections are in the env.yaml file. For our demonstration purposes, the below env.yaml file contains the following connections:

• DO_SPACES: a S3 Compatible bucket file system offered by DigitalOcean. This would also work with any other S3-compatible product such as CloudFlare R2, MinIO, Backblaze B2, and many others including AWS S3.
• GOOGLE_STORAGE: a Google Cloud Storage bucket
• POSTGRES: a PostgreSQL database. This could be any PostgreSQL database, such as Amazon RDS, Supabase, Fly.io, DigitalOcean, or your own local instance!
• SNOWFLAKE: a Snowflake warehouse instance.
• BIGQUERY: a BigQuery warehouse instance.

connections:
  DO_SPACES:
    type: s3
    bucket: sling-bucket
    endpoint: nyc3.digitaloceanspaces.com
    access_key_id: "my_access_key_id"
    secret_access_key: "my_secret_access_key"

  GOOGLE_STORAGE:
    type: gs
    bucket: sling-bucket
    gc_key_file: /Users/fritz/.sling/sling-project-123-ce219ceaef9512.json

  POSTGRES:
    type: postgres
    username: fritz
    host: 10.13.123.234 
    password: "mypass123"
    port: 5432
    database: postgres_db
    schema: public

  BIGQUERY:
    type: bigquery
    project: sling-project-123
    location: US
    dataset: public
    gc_key_file: /Users/fritz/.sling/sling-project-123-ce219ceaef9512.json

  SNOWFLAKE:
    type: snowflake
    account: tzb01234.us-east-1
    username: sling
    password: "mypass123"
    database: sling
    schema: public

Once the credentials are set, we can list our connections and test them with the sling conns command:

$ sling conns list
+------------------+------------------+-------------------+
| CONN NAME        | CONN TYPE        | SOURCE            |
+------------------+------------------+-------------------+
| BIGQUERY         | DB - BigQuery    | sling env yaml    |
| DO_SPACES        | FileSys - S3     | sling env yaml    |
| GOOGLE_STORAGE   | FileSys - Google | sling env yaml    |
| POSTGRES         | DB - PostgreSQL  | sling env yaml    |
| SNOWFLAKE        | DB - Snowflake   | sling env yaml    |
+------------------+------------------+-------------------+

$ sling conns test BIGQUERY
3:53PM INF success!

$ sling conns test DO_SPACES
3:54PM INF success!

...

Great, now that we have tested our connections, let's go through a few examples. Remember, if you are facing any issues with accessing your connections and successfully testing them, feel free to email us (support @ slingdata.io).

Loading a Google Cloud Storage CSV file into a PostgreSQL Database

Let us say that we need to load files from Google Cloud Storage (GCS) into a PostgreSQL instance. The first step to get this flowing is to set our credentials file, as described above.

Now let us test our connections.

$ sling conns test GOOGLE_STORAGE
10:52AM INF success!

$ sling conns test POSTGRES
10:52AM INF success!

Let us discover the available file streams with the sling conns discover command:

$ sling conns discover GOOGLE_STORAGE
10:52AM INF Found 2 streams:
 - gs://sling-bucket/LargeDataset.csv
 - gs://sling-bucket/test_file.csv

Great, let's run our task. We want to copy the CSV file gs://sling-bucket/test_file.csv into table public.my_table, so we can run this:

$ sling run --src-conn GOOGLE_STORAGE --src-stream gs://sling-bucket/test_file.csv --tgt-conn POSTGRES --tgt-object public.my_table --mode full-refresh
11:01AM INF connecting to target database (postgres)
11:01AM INF reading from source file system (gs)
11:01AM INF writing to target database [mode: full-refresh]
11:01AM INF streaming data
11:01AM INF dropped table public.my_table
11:01AM INF created table public.my_table
11:01AM INF inserted 18 rows in 1 sec
11:01AM INF execution succeeded

We could run the same task by using a YAML config file, as well as enable debug mode (with flag -d) for more verbose logging:

# Let's create our task configuration file
$ echo '
source:
  conn: GOOGLE_STORAGE
  stream: "gs://sling-bucket/test_file.csv"
target:
  conn: POSTGRES
  object: public.my_table
mode: full-refresh
' > /tmp/my_task.yaml

# we are also providing the `-d` flag for debug mode.
# this allows us to have more logging details.
$ sling run -d -c /tmp/my_task.yaml
11:04AM DBG type is file-db
11:04AM INF connecting to target database (postgres)
11:04AM INF reading from source file system (gs)
11:04AM DBG reading datastream from gs://sling-bucket/test_file.csv
11:04AM INF writing to target database [mode: full-refresh]
11:04AM DBG drop table if exists public.my_table_tmp cascade
11:04AM DBG table public.my_table_tmp dropped
11:04AM DBG create table if not exists public.my_table_tmp ("id" bigint,
"first_name" text,
"last_name" text,
"email" text,
"target" bool,
"create_dt" timestamp,
"rating" numeric)
11:04AM INF streaming data
11:04AM DBG select count(1) cnt from public.my_table_tmp
11:04AM DBG drop table if exists public.my_table cascade
11:04AM DBG table public.my_table dropped
11:04AM INF dropped table public.my_table
11:04AM DBG create table if not exists public.my_table ("id" integer,
"first_name" varchar(255),
"last_name" varchar(255),
"email" varchar(255),
"target" bool,
"create_dt" timestamp,
"rating" numeric)
11:04AM INF created table public.my_table
11:04AM DBG insert into "public"."my_table" ("id", "first_name", "last_name", "email", "target", "create_dt", "rating") select "id", "first_name", "last_name", "email", "target", "create_dt", "rating" from "public"."my_table_tmp"
11:04AM DBG inserted rows into `public.my_table` from temp table `public.my_table_tmp`
11:04AM INF inserted 18 rows in 1 secs [17 r/s]
11:04AM DBG drop table if exists public.my_table_tmp cascade
11:04AM DBG table public.my_table_tmp dropped
11:04AM INF execution succeeded

Notice that since we enabled debug mode with -d, we can see how the table DDL has been automatically determined:

create table if not exists public.my_table ("id" integer,
"first_name" varchar(255),
"last_name" varchar(255),
"email" varchar(255),
"target" bool,
"create_dt" timestamp,
"rating" numeric)

If you'd like to see more details on configuration options, see here. If you are thinking of synching many tables, it's probably best to use a replication configuration file, which is also a YAML file.

Ingesting a Local TSV File (via stdin) into a Snowflake Database

Ingesting a file via stdin is super easy. As expected, you can just pipe your file stream Unix style and execute a sling run command specifying the target connection and destination object details.

# let's first test our connection
$ sling conns test SNOWFLAKE
11:09AM INF success!

# Let's pipe our file into a new table
$ cat /tmp/accounts.tsv | sling run --tgt-conn SNOWFLAKE --tgt-object sling.accounts --mode full-refresh
11:09AM INF connecting to target database (snowflake)
11:09AM INF reading from stream (stdin)
11:09AM INF writing to target database [mode: full-refresh]
11:09AM INF streaming data
11:09AM INF dropped table sling.accounts
11:09AM INF created table sling.accounts
11:09AM INF inserted 94 rows in 5 secs
11:09AM INF execution succeeded

So notice how we didn't specify the delimiter. Whether it's a comma delimited file (CSV) or a tab delimited file (TSV), sling will attempt to auto-detect the delimiter (when it's a |, ,, <tab> or ;). If you'd like to manually specify the delimiter, you can do so as shown below by adding the --src-options flag:

$ cat /tmp/accounts.tsv | sling run --tgt-conn SNOWFLAKE --tgt-object sling.accounts --mode full-refresh --src-options '{"delimiter": "\t"}'
11:09AM INF connecting to target database (snowflake)
11:09AM INF reading from stream (stdin)
11:09AM INF writing to target database [mode: full-refresh]
11:09AM INF streaming data
11:09AM INF dropped table sling.accounts
11:09AM INF created table sling.accounts
11:09AM INF inserted 94 rows in 5 secs
11:09AM INF execution succeeded

To get a full list of advanced options, please head over here.

Loading DigitalOcean Spaces (S3) JSON files into a BigQuery Warehouse

In our third example, we will be ingesting JSON files from a DigitalOcean Spaces bucket into a BigQuery instance. Let us first discover our files while applying a filter for a specific folder with the --folder flag.

$ sling conns discover DO_SPACES --folder s3://sling-bucket/02-05-2022
4:36PM INF Found 21 streams:
 - s3://sling-bucket/02-05-2022/1644049120.1uXKxCrhN2WGAt2fojy6k2fqDSb.26879111-e935-4957-a99e-6b6241e6a84a.json.gz
 - s3://sling-bucket/02-05-2022/1644049299.1uXKxCrhN2WGAt2fojy6k2fqDSb.0185f4a2-fe46-4ea6-bb2b-58cee66fa14d.json.gz
 - s3://sling-bucket/02-05-2022/1644049390.1uXKxCrhN2WGAt2fojy6k2fqDSb.323c8d1b-bc66-4d17-8a26-6fde546cf54f.json.gz
 - s3://sling-bucket/02-05-2022/1644049480.1uXKxCrhN2WGAt2fojy6k2fqDSb.73cf9b8c-be06-431d-98bd-c440932d530d.json.gz
 - s3://sling-bucket/02-05-2022/1644049571.1uXKxCrhN2WGAt2fojy6k2fqDSb.68fbbf20-8a87-4836-95c5-5a4b762c97cb.json.gz
 - s3://sling-bucket/02-05-2022/1644049662.1uXKxCrhN2WGAt2fojy6k2fqDSb.23397ceb-c55b-46d9-bdf7-063ce6c54cdd.json.gz
 - s3://sling-bucket/02-05-2022/1644049752.1uXKxCrhN2WGAt2fojy6k2fqDSb.d23a1079-330e-4009-a18e-ca3ac1d087f2.json.gz
 - s3://sling-bucket/02-05-2022/1644049933.1uXKxCrhN2WGAt2fojy6k2fqDSb.be115fa8-4a75-4353-a5d7-9b1ec9e0e792.json.gz
 - s3://sling-bucket/02-05-2022/1644050114.1uXKxCrhN2WGAt2fojy6k2fqDSb.191e971e-7a18-4faa-8ccd-ddca99fe934c.json.gz
 - s3://sling-bucket/02-05-2022/1644063457.1uXKxCrhN2WGAt2fojy6k2fqDSb.e68674d0-d14c-4277-aef2-88bf778f6bb5.json.gz
 - s3://sling-bucket/02-05-2022/1644063818.1uXKxCrhN2WGAt2fojy6k2fqDSb.8545e07c-3881-4dcd-9976-b860ed85d7c9.json.gz
 - s3://sling-bucket/02-05-2022/1644064268.1uXKxCrhN2WGAt2fojy6k2fqDSb.695c8516-6d8c-46ce-ac01-9d89b4dd13e8.json.gz
 - s3://sling-bucket/02-05-2022/1644064359.1uXKxCrhN2WGAt2fojy6k2fqDSb.cd481db0-2066-4179-ba8c-8dd5f45e190e.json.gz
 - s3://sling-bucket/02-05-2022/1644097266.1uXKxCrhN2WGAt2fojy6k2fqDSb.7b06270d-5c24-40fe-958b-f2175f8018f3.json.gz
 - s3://sling-bucket/02-05-2022/1644097446.1uXKxCrhN2WGAt2fojy6k2fqDSb.18e17223-b55e-4283-a7ce-84af1f525435.json.gz
 - s3://sling-bucket/02-05-2022/1644097537.1uXKxCrhN2WGAt2fojy6k2fqDSb.4bd41934-7be2-4b10-8539-580c54092f73.json.gz
 - s3://sling-bucket/02-05-2022/1644097628.1uXKxCrhN2WGAt2fojy6k2fqDSb.49facabd-6a6f-410d-b4fc-2aebf8ccdad8.json.gz
 - s3://sling-bucket/02-05-2022/1644098354.1uXKxCrhN2WGAt2fojy6k2fqDSb.865bf8b5-7361-4779-8199-6ec20543058d.json.gz
 - s3://sling-bucket/02-05-2022/1644099094.1uXKxCrhN2WGAt2fojy6k2fqDSb.2af2cfb4-8371-4938-b560-e282f8c94d16.json.gz
 - s3://sling-bucket/02-05-2022/1644099185.1uXKxCrhN2WGAt2fojy6k2fqDSb.538e1f27-306f-4942-b79d-c012b7e9518f.json.gz
 - s3://sling-bucket/02-05-2022/1644099275.1uXKxCrhN2WGAt2fojy6k2fqDSb.55cf698c-41c7-4f1b-89df-2567506cd02b.json.gz

Great, now let's ingest all 21 JSON files in that folder, without flattening the nested columns (with debug mode enabled, flag -d):

$ sling run -d --src-conn DO_SPACES --src-stream s3://sling-bucket/02-05-2022/ --tgt-conn BIGQUERY --tgt-object public.json_files --mode full-refresh
4:39PM DBG type is file-db
4:39PM INF connecting to target database (bigquery)
4:39PM INF reading from source file system (s3)
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049120.1uXKxCrhN2WGAt2fojy6k2fqDSb.26879111-e935-4957-a99e-6b6241e6a84a.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049299.1uXKxCrhN2WGAt2fojy6k2fqDSb.0185f4a2-fe46-4ea6-bb2b-58cee66fa14d.json.gz
4:39PM INF writing to target database [mode: full-refresh]
4:39PM DBG drop table if exists public.json_files_tmp
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049390.1uXKxCrhN2WGAt2fojy6k2fqDSb.323c8d1b-bc66-4d17-8a26-6fde546cf54f.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049480.1uXKxCrhN2WGAt2fojy6k2fqDSb.73cf9b8c-be06-431d-98bd-c440932d530d.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049571.1uXKxCrhN2WGAt2fojy6k2fqDSb.68fbbf20-8a87-4836-95c5-5a4b762c97cb.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049662.1uXKxCrhN2WGAt2fojy6k2fqDSb.23397ceb-c55b-46d9-bdf7-063ce6c54cdd.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049752.1uXKxCrhN2WGAt2fojy6k2fqDSb.d23a1079-330e-4009-a18e-ca3ac1d087f2.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049933.1uXKxCrhN2WGAt2fojy6k2fqDSb.be115fa8-4a75-4353-a5d7-9b1ec9e0e792.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644050114.1uXKxCrhN2WGAt2fojy6k2fqDSb.191e971e-7a18-4faa-8ccd-ddca99fe934c.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644063457.1uXKxCrhN2WGAt2fojy6k2fqDSb.e68674d0-d14c-4277-aef2-88bf778f6bb5.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644063818.1uXKxCrhN2WGAt2fojy6k2fqDSb.8545e07c-3881-4dcd-9976-b860ed85d7c9.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644064268.1uXKxCrhN2WGAt2fojy6k2fqDSb.695c8516-6d8c-46ce-ac01-9d89b4dd13e8.json.gz
4:39PM DBG table public.json_files_tmp dropped
4:39PM DBG create table public.json_files_tmp (`data` json)
4:39PM INF streaming data
4:39PM INF importing into bigquery via local storage
4:39PM DBG writing to /var/folders/49/1zc24t595j79t5mw7_t9gtxr0000gn/T/bigquery/public.json_files_tmp/2022-11-26T163956.037
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644064359.1uXKxCrhN2WGAt2fojy6k2fqDSb.cd481db0-2066-4179-ba8c-8dd5f45e190e.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097266.1uXKxCrhN2WGAt2fojy6k2fqDSb.7b06270d-5c24-40fe-958b-f2175f8018f3.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097446.1uXKxCrhN2WGAt2fojy6k2fqDSb.18e17223-b55e-4283-a7ce-84af1f525435.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097537.1uXKxCrhN2WGAt2fojy6k2fqDSb.4bd41934-7be2-4b10-8539-580c54092f73.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097628.1uXKxCrhN2WGAt2fojy6k2fqDSb.49facabd-6a6f-410d-b4fc-2aebf8ccdad8.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644098354.1uXKxCrhN2WGAt2fojy6k2fqDSb.865bf8b5-7361-4779-8199-6ec20543058d.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644099094.1uXKxCrhN2WGAt2fojy6k2fqDSb.2af2cfb4-8371-4938-b560-e282f8c94d16.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644099185.1uXKxCrhN2WGAt2fojy6k2fqDSb.538e1f27-306f-4942-b79d-c012b7e9518f.json.gz
4:39PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644099275.1uXKxCrhN2WGAt2fojy6k2fqDSb.55cf698c-41c7-4f1b-89df-2567506cd02b.json.gz
4:39PM DBG Loading /var/folders/49/1zc24t595j79t5mw7_t9gtxr0000gn/T/bigquery/public.json_files_tmp/2022-11-26T163956.037/part.01.0001.csv.gz
4:40PM DBG select count(1) cnt from public.json_files_tmp
4:40PM DBG drop table if exists public.json_files
4:40PM DBG table public.json_files dropped
4:40PM INF dropped table public.json_files
4:40PM DBG create table public.json_files (`data` json)
4:40PM INF created table public.json_files
4:40PM DBG insert into `public`.`json_files` (`data`) select `data` from `public`.`json_files_tmp`
4:40PM DBG inserted rows into `public.json_files` from temp table `public.json_files_tmp`
4:40PM INF inserted 27 rows in 22 secs [1 r/s]
4:40PM DBG drop table if exists public.json_files_tmp
4:40PM DBG table public.json_files_tmp dropped
4:40PM INF execution succeeded

With can notice that there was 1 column in the created table, since by default sling does not flatten the nested column values:

create table public.json_files (`data` json)

Now let's ingest all files again, this time adding the {"flatten": true} for the --src-options flag in order to flatten the nested columns into their own columns:

$ sling run -d --src-conn DO_SPACES --src-stream s3://sling-bucket/02-05-2022/ --src-options '{"flatten": true}' --tgt-conn BIGQUERY --tgt-object public.json_files --mode full-refresh
4:49PM DBG type is file-db
4:49PM INF connecting to target database (bigquery)
4:49PM INF reading from source file system (s3)
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049120.1uXKxCrhN2WGAt2fojy6k2fqDSb.26879111-e935-4957-a99e-6b6241e6a84a.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049299.1uXKxCrhN2WGAt2fojy6k2fqDSb.0185f4a2-fe46-4ea6-bb2b-58cee66fa14d.json.gz
4:49PM INF writing to target database [mode: full-refresh]
4:49PM DBG drop table if exists public.json_files_tmp
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049390.1uXKxCrhN2WGAt2fojy6k2fqDSb.323c8d1b-bc66-4d17-8a26-6fde546cf54f.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049480.1uXKxCrhN2WGAt2fojy6k2fqDSb.73cf9b8c-be06-431d-98bd-c440932d530d.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049571.1uXKxCrhN2WGAt2fojy6k2fqDSb.68fbbf20-8a87-4836-95c5-5a4b762c97cb.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049662.1uXKxCrhN2WGAt2fojy6k2fqDSb.23397ceb-c55b-46d9-bdf7-063ce6c54cdd.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049752.1uXKxCrhN2WGAt2fojy6k2fqDSb.d23a1079-330e-4009-a18e-ca3ac1d087f2.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644049933.1uXKxCrhN2WGAt2fojy6k2fqDSb.be115fa8-4a75-4353-a5d7-9b1ec9e0e792.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644050114.1uXKxCrhN2WGAt2fojy6k2fqDSb.191e971e-7a18-4faa-8ccd-ddca99fe934c.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644063457.1uXKxCrhN2WGAt2fojy6k2fqDSb.e68674d0-d14c-4277-aef2-88bf778f6bb5.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644063818.1uXKxCrhN2WGAt2fojy6k2fqDSb.8545e07c-3881-4dcd-9976-b860ed85d7c9.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644064268.1uXKxCrhN2WGAt2fojy6k2fqDSb.695c8516-6d8c-46ce-ac01-9d89b4dd13e8.json.gz
4:49PM DBG table public.json_files_tmp dropped
4:49PM DBG create table public.json_files_tmp (`anonymousid` string,
`context__library__name` string,
`context__library__version` string,
`event` string,
`messageid` string,
`originaltimestamp` timestamp,
`properties__application` string,
`properties__cmd` string,
`properties__error` string,
`properties__job_end_time` timestamp,
`properties__job_mode` string,
`properties__job_rows_count` int64,
`properties__job_src_type` string,
`properties__job_start_time` timestamp,
`properties__job_status` string,
`properties__job_tgt_type` string,
`properties__job_type` string,
`properties__os` string,
`properties__version` string,
`receivedat` timestamp,
`request_ip` string,
`rudderid` string,
`sentat` timestamp,
`timestamp` string,
`type` string,
`userid` string)
4:49PM INF streaming data
4:49PM INF importing into bigquery via local storage
4:49PM DBG writing to /var/folders/49/1zc24t595j79t5mw7_t9gtxr0000gn/T/bigquery/public.json_files_tmp/2022-11-26T163956.037
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644064359.1uXKxCrhN2WGAt2fojy6k2fqDSb.cd481db0-2066-4179-ba8c-8dd5f45e190e.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097266.1uXKxCrhN2WGAt2fojy6k2fqDSb.7b06270d-5c24-40fe-958b-f2175f8018f3.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097446.1uXKxCrhN2WGAt2fojy6k2fqDSb.18e17223-b55e-4283-a7ce-84af1f525435.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097537.1uXKxCrhN2WGAt2fojy6k2fqDSb.4bd41934-7be2-4b10-8539-580c54092f73.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644097628.1uXKxCrhN2WGAt2fojy6k2fqDSb.49facabd-6a6f-410d-b4fc-2aebf8ccdad8.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644098354.1uXKxCrhN2WGAt2fojy6k2fqDSb.865bf8b5-7361-4779-8199-6ec20543058d.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644099094.1uXKxCrhN2WGAt2fojy6k2fqDSb.2af2cfb4-8371-4938-b560-e282f8c94d16.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644099185.1uXKxCrhN2WGAt2fojy6k2fqDSb.538e1f27-306f-4942-b79d-c012b7e9518f.json.gz
4:49PM DBG reading datastream from s3://sling-bucket/02-05-2022/1644099275.1uXKxCrhN2WGAt2fojy6k2fqDSb.55cf698c-41c7-4f1b-89df-2567506cd02b.json.gz
4:49PM DBG Loading /var/folders/49/1zc24t595j79t5mw7_t9gtxr0000gn/T/bigquery/public.json_files_tmp/2022-11-26T163956.037/part.01.0001.csv.gz
4:50PM DBG select count(1) cnt from public.json_files_tmp
4:50PM DBG drop table if exists public.json_files
4:50PM DBG table public.json_files dropped
4:50PM INF dropped table public.json_files
4:50PM DBG create table public.json_files (`anonymousid` string,
`context__library__name` string,
`context__library__version` string,
`event` string,
`messageid` string,
`originaltimestamp` timestamp,
`properties__application` string,
`properties__cmd` string,
`properties__error` string,
`properties__job_end_time` timestamp,
`properties__job_mode` string,
`properties__job_rows_count` int64,
`properties__job_src_type` string,
`properties__job_start_time` timestamp,
`properties__job_status` string,
`properties__job_tgt_type` string,
`properties__job_type` string,
`properties__os` string,
`properties__version` string,
`receivedat` timestamp,
`request_ip` string,
`rudderid` string,
`sentat` timestamp,
`timestamp` string,
`type` string,
`userid` string)
4:50PM INF created table public.json_files
4:50PM DBG inserted rows into `public.json_files` from temp table `public.json_files_tmp`
4:50PM INF inserted 27 rows in 22 secs [1 r/s]
4:50PM DBG drop table if exists public.json_files_tmp
4:50PM DBG table public.json_files_tmp dropped
4:50PM INF execution succeeded

With can notice that there were many columns in the created table, for each nested key. There is a __ separator between the parent / child columns. Nifty huh!?

create table public.json_files (`anonymousid` string,
`context__library__name` string,
`context__library__version` string,
`event` string,
`messageid` string,
`originaltimestamp` timestamp,
`properties__application` string,
`properties__cmd` string,
`properties__error` string,
`properties__job_end_time` timestamp,
`properties__job_mode` string,
`properties__job_rows_count` int64,
`properties__job_src_type` string,
`properties__job_start_time` timestamp,
`properties__job_status` string,
`properties__job_tgt_type` string,
`properties__job_type` string,
`properties__os` string,
`properties__version` string,
`receivedat` timestamp,
`request_ip` string,
`rudderid` string,
`sentat` timestamp,
`timestamp` string,
`type` string,
`userid` string)

Conclusion

We have demonstrated how easy it is to use the Sling CLI tool to load files from various storage sources. There are even more possibilities and configurations, check out our docs site for more details. If you have any questions, issues or comments, feel free to email us at support @ slingdata.io.

You may not have heard about Google’s BigTable but it is a thriving NoSQL database offering from the juggernaut, allowing exceptional read/write performance when properly tuned. According to Google Cloud’s own blog post, BigTable manages over 10 Exabytes of data and serves more than 5 billion requests per second. It also offers various features such as auto-scaling with the goal of optimizing costs and improved manageability.

Extract-Load Process

So you have a situation where you need to copy or move data out of BigTable and import it into your Data Warehouse. How to proceed? Through various Google search, it is apparent that there is a lack of tooling that properly handle BigTable as a source. Fortunately, this is not the case for Sling. The go-powered Extract-Load (EL) tool, allows you to natively connect to BigTable and export the unstructured data sets in a tabular format (CSV, TSV) or into the most popular RDBMS databases such as PostgreSQL, BigQuery, Redshift, SQL Server and Snowflake. See below for an illustration of how Sling does this when importing into Snowflake.

Using Sling CLI

Sling CLI is a command line tool (as the name suggests), which allows you great flexibility for custom automation, at the expense of doing things a little bit more manually. The first step is to install sling. Since it is built in Go, it is offered as a binary for whatever system you are using.

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# Using Python Wrapper via pip
pip install sling

If you have a Linux system or desire to download the binary manually , please head over here. Once you have it downloaded, the next step is to set your credentials. Sling primarily uses a env.yaml file located in the ~/.sling/ folder. Here is an example:

connections:

  MY_BIGTABLE:
    type: bigtable
    project: sling-project-123
    location: US
    instance: big-table-instance
    gc_key_file: /Users/me/.sling/sling-project-123-ce219ceaef9512.json

  MY_SNOWFLAKE:
    type: snowflake
    username: fritz
    password: my_pass23
    account: abc123456.us-east-1
    database: sling
    schema: public

Once the credentials are set, we can test connectivity with the sling conns test command:

$ sling conns list
+---------------+----------------+----------------+
| CONN NAME     | CONN TYPE      | SOURCE         |
+---------------+----------------+----------------+
| MY_BIGTABLE   | DB - BigTable  | sling env yaml |
| MY_SNOWFLAKE  | DB - Snowflake | sling env yaml |
+---------------+----------------+----------------+

$ sling conns test MY_BIGTABLE
5:13PM INF success!

$ sling conns test MY_SNOWFLAKE
5:14PM INF success!

Great, now we are ready to run our Extract-Load (EL) task.

$ sling run --src-conn MY_BIGTABLE --src-stream test_table3 --tgt-conn MY_SNOWFLAKE --tgt-object public.test_table3 --mode full-refresh
0:08AM INF connecting to source database (bigtable)
10:08AM INF connecting to target database (snowflake)
10:08AM INF reading from source database
10:08AM INF writing to target database [mode: full-refresh]
10:09AM INF streaming data
1s 10,000 33798 r/s
10:09AM INF dropped table public.test_table3
10:09AM INF created table public.test_table3
10:09AM INF inserted 10000 rows in 11 secs [901 r/s]
10:09AM INF execution succeeded

To learn more about Sling CLI, please see here.

Conclusion

We have demonstrated an easy way to move your data from BigTable to Snowflake. Similar steps can be used for any other destination databases supported by Sling. To see a list of compatible databases, please visit this page.

In the past few years, we've seen a rapid growth in the usage of cloud data warehouses (as well as the "warehouse-first" paradigm). Two popular cloud DWH platforms are BigQuery and Snowflake. Check out the chart below to see their evolution over time.

Image: Gartner via Adam Ronthal (@aronthal) on Twitter.

BigQuery, standing at #4 as of 2021, is a fully-managed, serverless data warehouse service offered by Google Cloud Platform (GCP). It enables easy and scalable analysis over petabytes of data and has long been known for its ease of use and maintenance-free nature.

Snowflake, is a similar service offered by the company Snowflake Inc. One of the principal differences is that Snowflake allows you to host the instance in either Amazon Web Services (AWS), Azure (Microsoft) or GCP (Google). This is a great advantage if you are already established in a non-GCP environment.

Exporting and Loading the data

As circumstances have it, it is sometimes necessary or desired to copy data from a BigQuery environment into a Snowflake environment. Let's take a look and break down the various logical steps needed to properly move this data around since neither competing services have an integrated function to easily do this. For the sake of our example, we will assume that our destination Snowflake environment is hosted on AWS.

Step By Step Procedure

In order to migrate data from BigQuery to Snowflake (AWS), these are the essential steps:

1. Identify table or query and execute EXPORT DATA OPTIONS query to export to Google Cloud Storage (GCS).
2. Run script in VM or local machine to copy GCS data to Snowflake's Internal Stage. We could also read straight from GCS with a storage integration, but this involves another layer of secure access configuration (which may be preferable for your use case).
3. Manually generate CREATE TABLE DDL with correct column data types and execute in Snowflake.
4. Execute a COPY query in Snowflake to import staged files.
5. Optionally clean up (delete) temporary data in GCP and Internal Stage.

Image: Steps to manually export from BigQuery to Snowflake.

As demonstrated above, there are several steps to make this happen, where independent systems need to be interacted with. This can be cumbersome to automate, especially generating the correct DDL (#3) with the proper column types in the destination system (which I personally find the most burdensome, try doing this for tables with 50+ columns).

Fortunately, there is an easier way to do this, and it is by using a nifty tool called Sling. Sling is a data integration tool which allows easy and efficient movement of data (Extract & Load) from/to Databases and Storage Platforms. There are two ways of using it: Sling CLI & Sling Cloud. We will do the same procedure as above, but only by providing inputs to sling and it will automatically do the intricate steps for us!

Using Sling CLI

If you are a fanatic of the command line, Sling CLI is for you. It is built in go (which makes it super-fast), and it works with files and databases. It can also work with Unix Pipes (reads standard-input and writes to standard out). We can quickly install it from our shell:

# On Mac
brew install slingdata-io/sling/sling

# On Windows Powershell
scoop bucket add org https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# Using Python Wrapper via pip
pip install sling

Please see here for other installation options (including Linux). There is also a Python wrapper library, which is useful if you prefer interacting with Sling inside of Python.

Once installed, we should be able to run the sling command, which should give us this output:

sling - An Extract-Load tool | https://slingdata.io
Slings data from a data source to a data target.
Version 0.86.52

  Usage:
    sling [conns|run|update]

  Subcommands:
    conns    Manage local connections
    run      Execute an ad-hoc task
    update   Update Sling to the latest version

  Flags:
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.

Now there are many ways to configure tasks, but for our scope in this article, we first need to add connections credentials for BigQuery and Snowflake (a one time chore). We can do this by opening the file ~/.sling/env.yaml, and adding the credentials, which should look like this:

connections:

  BIGQUERY:
    type: bigquery
    project: sling-project-123
    location: US
    dataset: public
    gc_key_file: ~/.sling/sling-project-123-ce219ceaef9512.json
    gc_bucket: sling_us_bucket # this is optional but recommended for bulk export. 

  SNOWFLAKE:
    type: snowflake
    username: fritz
    password: my_pass23
    account: abc123456.us-east-1
    database: sling
    schema: public

Great, now let's test our connections:

$ sling conns list
+------------+------------------+-----------------+
| CONN NAME  | CONN TYPE        | SOURCE          |
+------------+------------------+-----------------+
| BIGQUERY   | DB - Snowflake   | sling env yaml  |
| SNOWFLAKE  | DB - PostgreSQL  | sling env yaml  |
+------------+------------------+-----------------+

$ sling conns test BIGQUERY
6:42PM INF success!

$ sling conns test SNOWFLAKE
6:42PM INF success!

Fantastic, now that we have our connections setup, we can run our task:

$ sling run --src-conn BIGQUERY --src-stream "select user.name, activity.* from public.activity join public.user on user.id = activity.user_id where user.type != 'external'" --tgt-conn SNOWFLAKE --tgt-object 'public.activity_user' --mode full-refresh
11:37AM INF connecting to source database (bigquery)
11:37AM INF connecting to target database (snowflake)
11:37AM INF reading from source database
11:37AM INF writing to target database [mode: full-refresh]
11:37AM INF streaming data
11:37AM INF dropped table public.activity_user
11:38AM INF created table public.activity_user
11:38AM INF inserted 77668 rows
11:38AM INF execution succeeded

Wow, that was easy! Sling did all the steps that we described prior automatically. We can even export the Snowflake data back to our shell sdtout (in CSV format) by providing just the table identifier (public.activity_user) for the --src-stream flag and count the lines to validate our data:

$ sling run --src-conn SNOWFLAKE --src-stream public.activity_user --stdout | wc -l
11:39AM INF connecting to source database (snowflake)
11:39AM INF reading from source database
11:39AM INF writing to target stream (stdout)
11:39AM INF wrote 77668 rows
11:39AM INF execution succeeded
77669 # CSV output includes a header row (77668 + 1)

Conclusion

We are in an era where data is gold, and moving data from one platform to another shouldn't be difficult. As we have demonstrated, Sling offers a powerful alternative by reducing friction associated with data integration. We'll be covering how to export from Snowflake and loading into BigQuery in another post.

Sling CLI is a command line tool which allows easy and efficient movement of data (Extract & Load) from/to Databases and Storage Platforms. It's trivial to get started, you can simply run pip install sling if you have Python's pip installed. Or you can download the binary for your machine here.

Connection Credentials

In order to use sling, we must first configure connection credentials, and Sling CLI looks for them in various places. This allows a “plug & play” nature if you are already using another tool such as dbt, or have connection URLs set in environment variables. It is however recommended to use Sling’s env.yaml file as it allows a more consistent and flexible experience.

Sling Env File

The first time you run the sling command, the .sling folder is created in the current user’s home directory (~/.sling), which in turn holds a file called env.yaml. The structure for the Sling’s Env file is simple, you put your connections’ credential under the connections key as shown below:

connections:
  marketing_pg:
    url: 'postgres://...' 
    ssh_tunnel: 'ssh://...' # optional

  # or dbt profile styled
  marketing_pg:
    type: postgres        
    host: [hostname]      
    user: [username]      
    password: [password]  
    port: [port]          
    dbname: [database name]
    schema: [dbt schema]  
    ssh_tunnel: 'ssh://...' 

  finance_bq:
    type: bigquery
    method: service-account
    project: [GCP project id]
    dataset: [the name of your dbt dataset]
    keyfile: [/path/to/bigquery/keyfile.json]

# global variables, available to all connections at runtime (optional)
variables:
  aws_access_key: '...'
  aws_secret_key: '...'

Please see here for all the accepted connection types and their respective data point needed.

When using the sling conns list command with Sling Env credentials, the SOURCE column will show as sling env yaml.

Environment variables

If you’d rather use environment variables, it suffices to set them in your shell environment the usual way:

# Mac / Linux
export MY_PG='postgresql://user:mypassw@pg.host:5432/db1'
export MY_SNOWFLAKE='snowflake://user:mypassw@sf.host/db1'
export ORACLE_DB='oracle://user:mypassw@orcl.host:1521/db1'

# Windows Powershell
set MY_PG 'postgresql://user:mypassw@pg.host:5432/db1'
set MY_SNOWFLAKE 'snowflake://user:mypassw@sf.host/db1'
set ORACLE_DB 'oracle://user:mypassw@orcl.host:1521/db1'

When using the sling conns list command with environment variables, the SOURCE column will show as env variable.

DBT Profiles

dbt is another popular tool that many data professionals use on a daily basis, and supporting existing local profiles allows easy cross-use. The typical location for the dbt credentials are in the ~/dbt/profiles.yml file. See here for more details.

If you have dbt credentials in place and use the sling conns list command, the SOURCE column will show as dbt profiles yaml.

The conns Sub-Command

Now that you have credentials set, sling offers a conns sub-command to interact with the connections. We can perform the following operations: list, test and discover.

$ sling conns -h
conns - Manage local connections

  Usage:
    conns [discover|list|test]

  Subcommands:
    discover   list available streams in connection
    list       list local connections detected
    test       test a local connection

  Flags:
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.

Listing Connections

It's convenient to see and list all connections available in our environment. We can simply run the sling conns list command. Here is an example:

$ sling conns list
+----------------------+------------------+-------------------+
| CONN NAME            | CONN TYPE        | SOURCE            |
+----------------------+------------------+-------------------+
| AWS_S3               | FileSys - S3     | sling env yaml    |
| AZURE_STORAGE        | FileSys - Azure  | sling env yaml    |
| BIGQUERY             | DB - BigQuery    | sling env yaml    |
| BIONIC_DB1           | DB - PostgreSQL  | dbt profiles yaml |
| BTD_S3               | FileSys - S3     | sling env yaml    |
| CLICKHOUSE           | DB - Clickhouse  | sling env yaml    |
| DEMO_POSTGRES        | DB - PostgreSQL  | sling env yaml    |
| SNOWFLAKE            | DB - Snowflake   | env variable      |
| STEAMPIPE            | DB - PostgreSQL  | sling env yaml    |
+----------------------+------------------+-------------------+

Testing Connections

The Sling CLI tool also allows testing connections. Once we know the connection name, we can use the sling conns test command:

$ sling conns test -h
test - test a local connection

  Usage:
    test [name]

  Positional Variables:
    name   The name of the connection to test (Required)
  Flags:
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.

Here is an actual example:

$ sling conns test MSSQL
6:42PM INF success!

Discovering Connection Streams

This is another nifty sub-command that allows one to see which data streams are available for sling is read from for a particular connection: the sling conns discover command.

$ sling conns discover -h
discover - list available streams in connection

  Usage:
    discover [name]

  Positional Variables:
    name   The name of the connection to test (Required)
  Flags:
       --version   Displays the program version string.
    -h --help      Displays help with available flag, subcommand, and positional value parameters.
    -f --filter    filter stream name by pattern (e.g. account_*)
       --folder    discover streams in a specific folder (for file connections)
       --schema    discover streams in a specific schema (for database connections)

For database connections, it will list the available tables and views. For storage connections, it will list the non-recursive file objects located in the specified source folder. Below are some examples.

Database Connections

$ sling conns discover CLICKHOUSE
6:57PM INF Found 68 streams:
 - "default"."docker_logs"
 - "default"."sling_docker_logs"
 - "system"."aggregate_function_combinators"
 - "system"."asynchronous_metric_log"
 - "system"."asynchronous_metrics"
 - "system"."build_options"
 - "system"."clusters"
 ....

If we want to filter for a specific shema, we can do:

$ sling conns discover CLICKHOUSE --schema default
8:29PM INF Found 2 streams:
 - "default"."docker_logs"
 - "default"."sling_docker_logs"

Storage Connections

$ sling conns discover AWS_S3
6:52PM INF Found 7 streams:
 - s3://my-sling-bucket/logging/
 - s3://my-sling-bucket/part.01.0001.csv
 - s3://my-sling-bucket/sling/
 - s3://my-sling-bucket/temp/
 - s3://my-sling-bucket/test.fs.write/
 - s3://my-sling-bucket/test/
 - s3://my-sling-bucket/test_1000.csv

If we want to see the files in a sub-folder, we can do this:

$ sling conns discover AWS_S3 --folder s3://my-sling-bucket/logging/
6:55PM INF Found 1 streams:
 - s3://my-sling-bucket/logging/1/1.log.gz

Running EL Tasks

Now that your connections are set, you are ready to run some Extract and Load tasks! We cover this in detail in a separate post, you can read about it here. From the command line, you can also run sling run -e which will print a bunch of examples.

Conclusion

All things considered, Sling CLI makes it easy to manage and interact with various types of connections from your shell. If you have any questions or suggestions, feel free to contact us here.

Sling is a go-powered, modern data integration tool Extracting and Loading data from popular data sources to destinations with high performance and ease.

Why Sling?

• Blazing fast performance - Core engine is written in Go and adopts a streaming design, making it super efficient by holding minimal data in memory.
• Replicate data quickly - Easily replicate data from a source database or file connection to a destination database or file.
• Transparent & Low Cost - Sling operates on an efficient and low-cost model. Our goal is to be transparent with the cost of using our platform.

Learn more about how Sling works