Since we launched Ploomber Cloud earlier this year, we’ve spent time working with our community to improve the experience. Ploomber is used in a wide range of applications such as computational chemistry, astrophysics, DNA sequencing among many others.
However, when talking to our users we realized there was a common aspect to their workflows: users would launch a few experiments (each one with a different set of parameters), and then compare the results (via tables, metrics and plots) to inform the next trials. This process would continue until they obtained optimal results.
While Ploomber Cloud enabled this workflow since its original release, we noticed that launching quick experiments required too many steps, since users needed to write a Ploomber project, which involves creating a few files and organizing their projects in a specific layout.
Today, we’re thrilled to announce the Ploomber Cloud Notebooks API, a new feature that allows our users to quickly launch parallel computational experiments with no configuration.
Ploomber Cloud comes with a free tier, create and account and get your API key now! If you’re a student of researcher, contact us and we’d be happy to provide more computational resources at no cost!
The Notebooks API
The new Notebooks API makes running and parallelizing notebooks a breeze. With one command, you’ll be able to request computing resources on-demand, execute hundreds of experiments in parallel, and analyze your results.
No more keeping your laptop on to prevent the Jupyter kernel from disconnecting or configuring complex cloud services!
Lauching a new experiment is as easy as:
pip install ploomber --upgrade
ploomber cloud nb path/to/notebook.ipynb
Ploomber Cloud will take care of uploading the notebook, provisioning resources, executing it, and uploading the executed notebook to cloud storage, which you can later download for review.
To execute the same notebook with multiple parameters, create a new cell at the top and specify the values:
grid:
param: [value1, value2, value3, value4]
another: [another1, another2, another3, another4]
The example above will create a grid of 16 parameters (one per each combination) and execute them in parallel.
For computationally demanding jobs, you can request more CPUs, memory or GPUs! Just add this to your configuration cell:
task_resources:
vcpus: 32 # number of CPUs
memory: 30000 # memory in MiB
Furthermore, there is no need to provide a requirements.txt file; Ploomber Cloud will parse your notebook and infer the packages to install; however, you can specify specific versions if you want to.
Example use case: Bayesian modeling with PyMC
Bayesian modeling is a computationally intensive process where the model fitting process can take a lot of time. With Ploomber Cloud, you can submit a notebook for execution, monitor it, and come back later to check the results.
Let’s execute a PyMC example that fits a linear model:
# install Ploomber CLI
pip install ploomber --upgrade
# get your API key here: https://www.cloud.ploomber.io/signin.html
# and set it
ploomber cloud set-key {your-key}
# execute PyMC notebook
ploomber cloud nb https://github.com/pymc-devs/pymc/blob/main/docs/source/learn/core_notebooks/GLM_linear.ipynb
# monitor docker building process (will take a couple of mins)
ploomber cloud logs @latest --image --watch
# monitor notebook execution (will take a couple of mins)
ploomber cloud status @latest --watch
# download the executed notebook
ploomber cloud download 'GLM_linear-*'
Example use case: time series anomaly detection
Time series anomaly detection is one of the most common applications in time series analysis; however, fitting the models is computationally intensive. With Ploomber Cloud, you can quickly scale your time series anomaly detection needs without managing infrastructure.
For this example, we’ll be using the amazing statsforecast library.
# install Ploomber CLI
pip install ploomber --upgrade
# get your API key here: https://www.cloud.ploomber.io/signin.html
# and set it
ploomber cloud set-key {your-key}
# execute time series anomaly detection notebook
ploomber cloud nb https://github.com/ploomber/posts/blob/master/ploomber-cloud-notebooks/AnomalyDetection.ipynb
# monitor docker building process (will take a couple of mins)
ploomber cloud logs @latest --image --watch
# monitor notebook execution (will take a couple of mins)
ploomber cloud status @latest --watch
# download the executed notebook
ploomber cloud download 'AnomalyDetection*'
Closing remarks
We hope Ploomber Cloud helps you run more experiments faster! For more details on usage, check out our documentation.
Sign up for the free tier of Ploomber Cloud now! If you need any help, join our community!.
