jupyterhub-idle-culler provides a JupyterHub service to identify and stop idle or long-running Jupyter servers via JupyterHub. It works solely by interacting with JupyterHub's REST API, and is often configured to run as a JupyterHub managed service started up by JupyterHub itself.

Setup

Setup involves three parts:

1. Install the Python package.
2. Configure JupyterHub permissions to work against JupyterHub's REST API.
3. Configure how its started up, either as a JupyterHub managed service or as a standalone script.

Installation

pip install jupyterhub-idle-culler

Permissions

Prior to JupyterHub 2.0, the jupyterhub-idle-culler required full administrative privileges, in order to have sufficient permissions to stop servers on behalf of users.

JupyterHub 2.0 introduces [scopes][] to allow for more fine-grained permission control. This means that the configured culler service does not need full administrative privileges anymore. It can be assigned only the permissions it needs.

[scopes]: https://jupyterhub.readthedocs.io/en/latest/rbac/scopes.html#available-scopes

jupyterhub-idle-culler requires the following scopes to function:

• list:users - to access to the user list API, our source of information about who to cull
• read:users:activity - to read the users' last_activity field
• read:servers - to read the users' servers field
• delete:servers - to stop users' servers, and delete named servers if --remove-named-servers is passed
• admin:users (optional) - to delete users if --cull-users is passed

c.JupyterHub.load_roles = [
    {
        "name": "jupyterhub-idle-culler-role",
        "scopes": [
            "list:users",
            "read:users:activity",
            "read:servers",
            "delete:servers",
            # "admin:users", # if using --cull-users
        ],
        # assignment of role's permissions to:
        "services": ["jupyterhub-idle-culler-service"],
    }
]

As a hub managed service

In jupyterhub_config.py, add the following dictionary for the idle-culler service to the c.JupyterHub.services list:

c.JupyterHub.services = [
    {
        "name": "jupyterhub-idle-culler-service",
        "command": [
            sys.executable,
            "-m", "jupyterhub_idle_culler",
            "--timeout=3600",
        ],
        # "admin": True,
    }
]

where:

• "command" indicates that the Service will be managed by the Hub, and
• "admin": True grants admin permissions to this Service and is only meant for

As a standalone script

jupyterhub-idle-culler can also be run as a standalone script. It can access the hub's api with a service token.

Register the service token with JupyterHub in jupyterhub_config.py:

c.JupyterHub.services = [
    {
        "name": "jupyterhub-idle-culler-service",
        "api_token": "...",
        # "admin": True,
    }
]

where:

• "api_token" contains a secret token, e.g. generated by openssl rand -hex 32, and
• "admin": True grants admin permissions to this Service and is only meant for

and store the same token in a JUPYTERHUB_API_TOKEN environment variable. Then start jupyterhub-idle-culler manually.

export JUPYTERHUB_API_TOKEN=api_token_above...
python3 -m jupyterhub_idle_culler [--timeout=900] [--url=http://localhost:8081/hub/api]

Command line flags

--api-page-size                  Number of users to request per page, when
                                   using JupyterHub 2.0's paginated user list
                                   API. Default: user the server-side default
                                   configured page size. (default 0)
  --concurrency                    Limit the number of concurrent requests made
                                   to the Hub.  Deleting a lot of users at the
                                   same time can slow down the Hub, so limit
                                   the number of API requests we have
                                   outstanding at any given time. (default 10)
  --config                         Service configuration file to load.
                                   (default idle_culler_config.py)
  --cull-admin-users               Whether admin users should be culled (only
                                   if --cull-users=true). (default True)
  --cull-every                     The interval (in seconds) for checking for
                                   idle servers to cull. (default 0)
  --cull-default-servers           Whether default servers should be culled (only
                                   if --cull-default-servers=true). (default True)
  --cull-named-servers             Whether named servers should be culled (only
                                   if --cull-named-servers=true). (default True)
  --cull-users                     Cull users in addition to servers.  This is
                                   for use in temporary-user cases such as
                                   tmpnb. (default False)
  --internal-certs-location        The location of generated internal-ssl
                                   certificates (only needed with --ssl-
                                   enabled=true). (default internal-ssl)
  --max-age                        The maximum age (in seconds) of servers that
                                   should be culled even if they are active.
                                   (default 0)
  --remove-named-servers           Remove named servers in addition to stopping
                                   them.  This is useful for a BinderHub that
                                   uses authentication and named servers.
                                   (default False)
  --ssl-enabled                    Whether the Jupyter API endpoint has TLS
                                   enabled. (default False)
  --timeout                        The idle timeout (in seconds). (default 600)
  --url                            The JupyterHub API URL.

Caveats

1. JupyterHub's last_activity data about user servers is not updated with high

1. If you want to use --cull-users with a different culling interval for the

1. By default jupyterhub-idle-cullers HTTP requests to JupyterHub's REST API

How it works

JupyterHub's REST API is used to acquire information about activity, and if the idle culler service based on configuration thinks a server should be stopped or deleted it also does so via JupyterHub's REST API.

In depth

jupyterhub-idle-culler relies on permission to work against JupyterHub's REST API is provided via the JUPYTERHUB_API_TOKEN, that is set automatically for [managed services] started by JupyterHub.

jupyterhub-idle-culler lists available users and their server's reported last_activity via JupyterHub's [/users] REST API and makes decisions based on that. User's default servers can be stopped via [/users/{name}/server], named servers can be stopped and optionally removed via [/users/{name}/servers/{server_name}], and users can optionally be deleted via [/users/{name}].

[managed services]: https://jupyterhub.readthedocs.io/en/stable/reference/services.html#launching-a-hub-managed-service [/users]: https://jupyterhub.readthedocs.io/en/stable/reference/rest-api.html#/default/get_users [/users/{name}/server]: https://jupyterhub.readthedocs.io/en/stable/reference/rest-api.html#/default/delete_users__name__server [/users/{name}/servers/{server_name}]: https://jupyterhub.readthedocs.io/en/stable/reference/rest-api/index.html#operation--users--name--servers--server_name--delete [/users/{name}]: https://jupyterhub.readthedocs.io/en/stable/reference/rest-api.html#/default/delete_users__name_

JupyterHub's reported last_activity for user servers is updated by JupyterHub at a regular interval in the [update_last_activity function] that relies on two sources of information.

[update_last_activity function]: https://github.com/jupyterhub/jupyterhub/blob/3.1.1/jupyterhub/app.py#L3002

1. The proxy's routes data

The [configurable-http-proxy] used in https://z2jh.jupyter.org provides information about network routes activity, but [traefik-proxy] used in https://tljh.jupyter.org [currently does not].

[get_all_routes]: https://jupyterhub.readthedocs.io/en/stable/reference/proxy.html#retrieving-routes [configurable-http-proxy]: https://github.com/jupyterhub/configurable-http-proxy#readme [traefik-proxy]: https://github.com/jupyterhub/traefik-proxy#readme [currently does not]: https://github.com/jupyterhub/traefik-proxy/issues/151

1. The user server's activity reports

Servers has before JupyterHub 4 notified JupyterHub about activity by being started by the [jupyterhub-singleuser] script made available by installing jupyterhub (or jupyterhub-singleuser on conda-forge). With JupyterHub 4+ and jupyter_server 2+ a jupyter_server server extension can be used instead.

The jupyterhub-singleuser script launches a modified server application that keeps JupyterHub updated with the server activity via the [notify_activity] function.

The notify_activity function in turn make use of the server applications last_activity function (see implementation in [NotebookApp] and [ServerApp] respectively) that that combines information from API activity, kernel activity, kernel shutdown, and terminal activity. This activity also covers activity of applications like RStudio running via jupyter-server-proxy.

[jupyterhub-singleuser]: https://github.com/jupyterhub/jupyterhub/blob/3.1.1/setup.py#L112 [notify_activity]: https://github.com/jupyterhub/jupyterhub/blob/3.1.1/jupyterhub/singleuser/mixins.py#L532 [notebookapp]: https://github.com/jupyter/notebook/blob/v6.5.2/notebook/notebookapp.py#L391-L396 [serverapp]: https://github.com/jupyter-server/jupyter_server/blob/v1.23.5/jupyter_server/serverapp.py#L446-L451

Here is a summary of what's described so far:

1. jupyterhub-idle-culler collects information and acts entirely through

1. jupyterhub-idle-culler makes decisions based on information provided by

Now, as the server's kernel activity influence the activity that servers will notify JupyterHub about, the kernel activity in turn influences jupyterhub-idle-culler. Due to this, it can be relevant to also learn a little about a mechanism to _cull idle kernels_ as well even though jupyterhub-idle-culler isn't involved in that.