Machine Learning: Google’s Vertex Pipelines are now generally available

By: MRT Desk

Published on:

Machine Learning: Google's Vertex Pipelines are now generally available

Google announced the general availability of the Vertex Pipelines. The company wants to support developers in working with pipelines for machine learning (ML).

Pipelines, among other things, are used to scale ML workflows. Individual work steps can be found as a series of individual components, and the steps in the pipeline can be mapped as containers. The respective output of a step can be the input for the next step. According to the blog post, there are two challenges associated with implementation: Individual pipeline steps should be convertible into containers, which in turn requires the establishment of an infrastructure in order to execute a pipeline on a large scale.

This is where Vertex Pipelines comes in and offers support for the Kubeflow Pipelines (KFP) and TensorFlow Extended (TFX) libraries, which take over the transfer of pipeline steps into containers and the management of the input and output artifacts throughout the pipeline for developers. With the library, users can define their pipelines using one of these libraries and then run them on the Vertext pipelines.

Google’s offering works as a serverless application, which frees developers from the need to provide their own infrastructure. When users upload and run their KFP or TFX pipelines, Vertex AI takes care of provisioning and scaling the infrastructure to run the pipeline. Users only pay for the resources that are used while the pipeline is running.

In addition, the Vertex Pipelines can be integrated with other tools in Vertex AI and Google Cloud. Developers can use the Vertex Pipeline to import data from BigQuery, train models with Vertex AI, store pipeline artifacts in Cloud Storage, retrieve metrics for model evaluation, and deliver models to Vertex AI endpoints. The development team behind the Vertex Pipelines provides users with a library of ready-made components that are supposed to make them easier to use.

Comprehensive documentation as well as the contribution on the Google Cloud Blog provide more information on Vertex Pipelines.


(mdo)

Article Source