argo-workflows: Run Container Workflows on Kubernetes

Overview

Argo Workflows is a workflow engine that represents workflows as Kubernetes custom resources. It runs each workflow step in a container and supports both ordered steps and DAGs, helping teams coordinate parallel jobs without relying on a separate VM- or server-based execution environment.

It is a fit for teams already operating Kubernetes who need to define, run, and manage repeatable pipelines there. Workflow definitions can express dependencies, inputs and outputs, retries, and other execution behavior, while the project provides a UI, CLI, and server API for interacting with workflows.

Key Features

• Define workflows as sequences of steps or directed acyclic graphs.
• Run containerized jobs in parallel on Kubernetes.
• Pass parameters and artifacts between steps, with support for multiple artifact storage backends.
• Use templates, loops, conditionals, retries, timeouts, and workflow scheduling.
• Inspect and manage workflows through a UI, CLI, or REST and gRPC server interfaces.
• Configure workflow suspension, resumption, cancellation, and cleanup of completed workflows.
• Integrate with Kubernetes scheduling options, volumes, and resource orchestration.
• Use client libraries for Go, Java, Python through Hera, and TypeScript through Juno.

Use Cases

• Data and batch processing teams can run parallel container jobs when work needs to scale across a Kubernetes cluster.
• Machine-learning practitioners can orchestrate multi-step training or data preparation pipelines using containerized tasks.
• Platform and infrastructure teams can define repeatable automation workflows that interact with Kubernetes resources.
• CI/CD teams can use workflow graphs to coordinate build, test, and deployment jobs on Kubernetes.

Project Facts

• Language: Go
• License: Apache-2.0
• Stars: 17k
• Forks: 3.7k
• Topics: airflow, argo, argo-workflows, batch-processing, cloud-native, cncf, dag, data-engineering, gitops, hacktoberfest, k8s, knative, kubernetes, machine-learning, mlops, pipelines, workflow, workflow-engine
• Archived: No

Getting Started

Start with the quick-start guide and walk-through examples. See the repository README for project details and links to the full documentation.

Alternatives

• Flyte: Flyte focuses on reproducible data, machine-learning, and analytics pipelines, while Argo Workflows provides general-purpose Kubernetes workflow orchestration.
• Dagger: Dagger runs programmable build and delivery pipelines across environments, while Argo Workflows orchestrates container workflows directly on Kubernetes.

Considerations

Argo Workflows requires a Kubernetes cluster and runs workflow steps as containers, so it is most relevant when Kubernetes is already part of the operating environment. Teams should account for the work of operating Kubernetes and configuring workflow storage, permissions, and scheduling to suit their needs. The project is distributed under the Apache-2.0 license.