DoK Community #37 - Running Data Replication Pipelines on Kubernetes with Argo

Data on Kubernetes
Tue, Mar 23, 9:00 AM (PDT)

About this event

Hundreds of data teams have migrated to the ELT pattern in recent years, leveraging SaaS tools like Stitch or FiveTran to reliably load data into their infrastructure. These SaaS offerings are outstanding and can accelerate your time to production significantly. However, many teams prefer to roll their own tools. One solution in these cases is to deploy singer.io taps and targets — Python scripts that can perform data replication between arbitrary sources and destinations. The Singer specification is the foundation for the popular Stitch SaaS, and it is also leveraged by a number of independent consultants and data projects. Singer pipelines are highly modular. You can pipe any tap to any target to build a data pipeline that fits your needs, making them a good fit for containerized workflows. This article walks through the workflow at a high level and provides some example code to get up and running with some shared templates. I also drill into reasons for choosing the Argo approach over other orchestration tools like Airflow or Dagster, and the implications from a team perspective.


Organizers

  • Ihor Dvoretskyi

    Ihor Dvoretskyi

    Cloud Native Computing Foundation

    Organizer

    View Profile
  • Bart Farrell

    Bart Farrell

    Data on Kubernetes Community

    Organizer

    View Profile
  • Melissa Logan

    Melissa Logan

    Organizer

    View Profile
  • Diogenese Topper

    Diogenese Topper

    Organizer

    View Profile
  • Iker Arce

    Iker Arce

    Organizer

    View Profile