Local runtime on Google Colab

Local runtime on Google Colab

Google Colab, a serverless notebook service provided by Google, offers the hosted runtime (kernel) with 3 options:
1) without GPU: 2 CPUs. 13 GB RAM, 50 GB disk
2) with GPU: 2 CPUs, 13 GB RAM, 358 GB disk
3) with TPU: 2 CPUs, 13 GB, 50 GB disk

In most cases, the offered resource (CPU, RAM, and disk) is sufficient, but sometimes we need more resources, e.g., more CPUs to speed up the process, more RAMs to process the data, and more disks to store the data.

Fortunately, Google Colab provides its users with local runtime that could be ported to its service. Here are the steps to port local runtime:

1) We need to install Jupyter’s forwarding service

pip install jupyter_http_over_ws 
jupyter serverextension enable --py jupyter_http_over_ws

2) Run the Jupyter service

python -m jupyter notebook --ip= --port=58888 --no-browser --NotebookApp.token='' --NotebookApp.password='' --NotebookApp.allow_origin='https://colab.research.google.com' --NotebookApp.port_retries=0 

3) [optional] Since I’m using a remote machine (high-performance cluster (HPC) TU Delft), I need to ssh forward the relevant port to my local computer.

ssh -nNT -L[HPC_IP]:58888 [username]@[transit_machine_ip]

4) Open Google Colab: https://colab.research.google.com; Create a new notebook; select Connect to local runtime; input 58888 as the port

Voila! Google Colab now runs on local runtime.

Leave a Reply

Your email address will not be published. Required fields are marked *