Exca - ⚔
Execute and cache seamlessly in python.
Quick install
pip install excaFull documentation
Documentation is available at https://facebookresearch.github.io/exca/
Basic overview
exca provides simple decorators to:
- execute a (hierarchy of) computation(s) either locally or on distant nodes,
- cache the result.
The problem:
In ML pipelines, the use of a simple python function, such asmy_task:
import numpy as np
def my_task(param: int = 12) -> float:
return param * np.random.rand()
often requires cumbersome overheads to (1) configure the parameters, (2) submit the job on a cluster, (3) cache the results: e.g.
``python continuation fixture:tmp_path
import pickle
from pathlib import Path
import submitit
Configure
param = 12Check task has already been executed
filepath = tmp_path / f'result-{param}.npy' if not filepath.exists():# Submit job on cluster executor = submitit.AutoExecutor(cluster=None, folder=tmp_path) job = executor.submit(my_task, param) result = job.result()
# Cache result with filepath.open("wb") as f: pickle.dump(result, f)
python fixture:tmp_path import numpy as np import pydantic import exca as xkThese overheads lead to several issues, such as debugging, handling hierarchical execution and properly saving the results (ending in the classic'result-parm12-v2_final_FIX.npy').excaThe solution:
can be used to decorate the method of apydanticmodel so as to seamlessly configure its execution and caching:
class MyTask(pydantic.BaseModel): param: int = 12 infra: xk.TaskInfra = xk.TaskInfra()
@infra.apply def process(self) -> float: return self.param * np.random.rand()
task = MyTask(param=1, infra={"folder": tmp_path, "cluster": "auto"}) out = task.process() # runs on slurm if available
calling process again will load the cache and not a new random number
assert out == task.process()See the API reference for all the details
Quick comparison
| feature \ tool | lru_cache | hydra | submitit | exca |
| ----------------------------- | :-------: | :---: | :------: | :--: |
| RAM cache | ✔ | | | ✔ |
| file cache | | | | ✔ |
| remote compute | | ✔ | ✔ | ✔ |
| pure python (vs command line) | ✔ | | ✔ | ✔ |
| hierarchical config | | ✔ | | ✔ |
Contributing
See the CONTRIBUTING file for how to help out.
Citing
License
exca` is MIT licensed, as found in the LICENSE file.
Also check-out Meta Open Source Terms of Use and Privacy Policy.