Fiber, a Python-based distributed computing library for modern computer clusters, is open-sourced by Uber and OpenAI. Fiber provides a unified Python user interface to its distributed computing framework. It helps make Machine Learning flexible enough to support reinforcement learning (RL) and population-based algorithms with other heuristics like deep learning.
Researchers compared Fiber with IPyParallel, Spark, and the standard python multiprocessing library on framework overhead, evolution strategies, and proximal policy optimization (PPO). Only to find that Fiber outperforms them when task duration is short and can scale RL algorithms beyond local machines.
Fiber has three layers:
- API layer: similar requirements and semantics to the standard Python multiprocessing module, and it works in distributed environments upon extension.
- Backend layer: handles the communication of tasks for a multitude of different cluster managers.
- Cluster layer: consists of cluster managers like Peloton and Kubernetes
Fiber has introduced a new concept called job-backed processes, which guarantees a consistent running environment.
Fiber aims to reduce costs and simplify the process for training cutting-edge machine learning algorithms, thus making distributed computing for AI simple.
#AIMonks #AI #ArificialIntelligence #Python #Fiber #ReinforcementLearning #Uber #OpenAI #opensource #PPO
0 Comments