{"id":551,"date":"2020-06-03T16:07:44","date_gmt":"2020-06-03T10:37:44","guid":{"rendered":"https:\/\/aimonks.com\/aibytes\/?p=551"},"modified":"2020-06-03T16:07:45","modified_gmt":"2020-06-03T10:37:45","slug":"distributed-computing-for-ai-made-simple-with-fiber","status":"publish","type":"post","link":"https:\/\/aimonks.com\/aibytes\/2020\/06\/03\/distributed-computing-for-ai-made-simple-with-fiber\/","title":{"rendered":"Distributed Computing for AI Made Simple with Fiber"},"content":{"rendered":"Reading Time: <\/span> < 1<\/span> minutes<\/span><\/span>\n

Fiber, a Python-based<\/strong> distributed computing library for modern computer clusters, is open-sourced<\/strong> by Uber <\/strong>and OpenAI<\/strong>. Fiber provides a unified Python user interface to its distributed computing framework. It helps make Machine Learning<\/strong> flexible enough to support reinforcement learning<\/strong> (RL) and population-based algorithms with other heuristics like deep learning. <\/p>\n\n\n\n

Researchers compared Fiber with IPyParallel<\/strong>, Spark<\/strong>, and the standard python multiprocessing library on framework overhead, evolution strategies, and proximal policy optimization (PPO). Only to find that Fiber outperforms them when task duration is short and can scale RL algorithms beyond local machines.<\/p>\n\n\n\n

Fiber has three layers:<\/strong><\/p>\n\n\n\n

  1. API layer<\/strong>: similar requirements and semantics to the standard Python multiprocessing module, and it works in distributed environments upon extension.<\/li>
  2. Backend layer<\/strong>: handles the communication of tasks for a multitude of different cluster managers.<\/li>
  3. Cluster layer<\/strong>: consists of cluster managers like Peloton and Kubernetes<\/li><\/ol>\n\n\n\n

    Fiber has introduced a new concept called job-backed processes, which guarantees a consistent running environment.
    Fiber aims to reduce costs and simplify the process for training cutting-edge machine learning algorithms, thus making distributed computing for AI simple.<\/p>\n\n\n\n

    #AIMonks #AI #ArificialIntelligence #Python #Fiber #ReinforcementLearning #Uber #OpenAI #opensource #PPO<\/p>\n","protected":false},"excerpt":{"rendered":"

    Reading Time: <\/span> < 1<\/span> minutes<\/span><\/span> Fiber, a Python-based distributed computing library for modern computer clusters, is open-sourced by Uber and OpenAI. Fiber provides a unified Python user interface to its distributed computing framework. It helps make Machine Learning flexible enough to support reinforcement learning (RL) and population-based algorithms with other heuristics like deep learning. Researchers […]<\/p>\n","protected":false},"author":2,"featured_media":552,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[61],"tags":[35,24,25,498,500,78,501,497,386,499],"class_list":["post-551","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-digital-transformation","tag-ai","tag-ai-monks","tag-artificial-intelligence","tag-fiber","tag-open-ai","tag-open-source","tag-ppo","tag-python","tag-reinforcement-learning","tag-uber"],"rttpg_featured_image_url":{"full":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640.jpg",640,426,false],"landscape":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640.jpg",640,426,false],"portraits":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640.jpg",640,426,false],"thumbnail":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640-150x150.jpg",150,150,true],"medium":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640-300x200.jpg",300,200,true],"large":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640.jpg",640,426,false],"1536x1536":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640.jpg",640,426,false],"2048x2048":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640.jpg",640,426,false],"hestia-blog":["https:\/\/aimonks.com\/aibytes\/wp-content\/uploads\/2020\/06\/student-849822_640-360x240.jpg",360,240,true]},"rttpg_author":{"display_name":"AI Bytes","author_link":"https:\/\/aimonks.com\/aibytes\/author\/aibytes_kashika\/"},"rttpg_comment":0,"rttpg_category":"Digital Transformation<\/a>","rttpg_excerpt":"Reading Time: < 1 minutes Fiber, a Python-based distributed computing library for modern computer clusters, is open-sourced by Uber and OpenAI. Fiber provides a unified Python user interface to its distributed computing framework. It helps make Machine Learning flexible enough to support reinforcement learning (RL) and population-based algorithms with other heuristics like deep learning. Researchers…","_links":{"self":[{"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/posts\/551","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/comments?post=551"}],"version-history":[{"count":1,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/posts\/551\/revisions"}],"predecessor-version":[{"id":553,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/posts\/551\/revisions\/553"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/media\/552"}],"wp:attachment":[{"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/media?parent=551"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/categories?post=551"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aimonks.com\/aibytes\/wp-json\/wp\/v2\/tags?post=551"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}