Nom du Projet logiciel: Loky
Page web :https://loky.readthedocs.io/en/stable/
Code source :https://github.com/tomMoral/loky
Axe : DataSense
Porteurs du projet : Olivier Grisel – Gaël Varoquaux
Nom & Prénom du Candidat : Moreau Thomas
Institution : Inria
Laboratoire gestionnaire : ENS Cachan
Équipes impliquées : Parietal
Durée et dates de la mission : octobre 2016 – octobre 2017
Context :
Joblib is a set of tools to provide lightweight pipelining in Python. In particular:
- transparent disk-caching of functions and lazy re-evaluation (memoize pattern)
- embarassingly parallel computing
Joblib is optimized to be fast and robust in particular on large data and has specific optimizations for numpy arrays. It is BSD-licensed.
The vision is to provide tools to easily achieve better performance and reproducibility when working with long running jobs.
- Avoid computing twice the same thing: code is rerun over an over, for instance when prototyping computational-heavy jobs (as in scientific development), but hand-crafted solution to alleviate this issue is error-prone and often leads to unreproducible results
- Persist to disk transparently: persisting in an efficient way arbitrary objects containing large data is hard. Using joblib’s caching mechanism avoids hand-written persistence and implicitly links the file on disk to the execution context of the original Python object. As a result, joblib’s persistence is good for resuming an application status or computational job, eg after a crash.
Joblib addresses these problems while leaving the code and the flow control of the users as unmodified as possible (no framework, no new paradigms).
Objectif :
In this context, it is necessary to have a robust and reliable backend to perform asynchronous computations. The aim of loky is to provide a robust, cross-platform and cross-version implementation of the ProcessPoolExecutor class of concurrent.futures. It notably features:
- Deadlock free implementation: one of the major concern in standard multiprocessing.pool.Pool and in concurrent.futures.ProcessPoolExecutor is their ability to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors.
- Consistent spawn behavior: All processes are started using fork/exec on POSIX systems. This ensures safer interactions with third party libraries.
- Reusable executor: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
- Transparent cloudpickle integration: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.
- No need for if ‘__name__’ == ‘__main__ in scripts: thanks to the use of cloudpickle to call functions defined in the main module, it is not required to protect the code calling parallel functions under Windows.
Results :
This project has provided several releases during the 2016-2017 years and the funding from digicosme has accelerated a lot its development, by funding a mission doctoral, with a part time PhD student developing this tools.
This project has been since included in the recent releases of joblib and should be featured in the next release of sciki-learn