Lyft open sources Flyte for developing machine learning workflows

Lyft open sources Flyte for developing machine learning workflows:

 Ridesharing company Lyft has open sourced Flyte, its distributed processing platform for machine learning workflows that is being used in different Lyft teams including Pricing, Data Science and Fraud. Let’s see how the open source tool can benefit ML workflows.

Lyft announced in a Medium blog post that they are making Flyte publicly available. Flyte is a structured programming and distributed processing platform for developing, executing and monitoring scalable machine learning workflows.

Built on Kubernetes

Flyte is built on Kubernetes to provide benefits such as portability, scalability and reliability. The platform consists of tasks that can be linked together in the form of directed acyclic graphs. It is built to scale and Lyft has been using it in production over various teams.

The following graphic shows how Flyte is designed:

Flyte’s architecture. Source.

Developing, executing and monitoring ML workflows

A graph is needed for creating a workflow, and Flyte offers developer tools to help ensure it meets the specifications. The documentation claims that, in only 20 lines of code, “a user can define a custom workflow, submit the graph to Flyte, and snooze while Flyte fulfills the execution on a schedule, or on-demand.”

Workflow execution is carried out using docker containers. Flyte supports distributed workflows, automatic retries and custom notifications.

Monitoring takes place throughout the workflow’s lifecycle, and the current status can be tracked via Flyte web interface. The web interface lets you take a closer look at the workflow steps and receive detailed error messages.

As an example, Lyft showed this workflow by Tim Wang on the Rider Pricing team as displayed in the Flyte UI:

Pricing Workflow by Tim Wang in Flyte UI. Source.

from Tumblr

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s