go-judge

Fast, Simple, Secure

Quick Start

Install & Run

Download compiled executable go-judge for your platform from Release and run.

Or, by docker

docker run -it --rm --privileged --shm-size=256m -p 5050:5050 --name=go-judge criyle/go-judge

REST API

A REST service to run program in restricted environment (Listening on localhost:5050 by default).

POST /run execute program in the restricted environment
GET /file list all cached file id to original name map

- POST /file prepare a file in the go judge (in memory), returns fileId (can be referenced in /run parameter) - GET /file/:fileId downloads file from go judge (in memory), returns file content - DELETE /file/:fileId delete file specified by fileId

/ws WebSocket version for /run
/stream WebSocket for stream run. Supports streaming interface
GET /version gets build git version (e.g. v1.9.0) together with runtime information (go version, os, platform)

- GET /config gets some configuration (e.g. fileStorePath, runnerConfig) together with some supported features

REST API Interface

API Interface Structure Definition

Example Request & Response

Documentation

Prerequisite

Linux Kernel Version >= 3.10
Cgroup file system mounted at /sys/fs/cgroup. Usually done by systemd

Architecture

+----------------------------------------------------------------------------------+
| Transport Layer (HTTP / WebSocket / FFI / ...)                                   |
+----------------------------------------------------------------------------------+
| Sandbox Worker (Environment Pool w/ Environment Builder )                        |
+-----------------------------------------------------------+----------------------+
| EnvExec                                                   | File Store           |
+--------------------+----------------+---------------------+---------------+------+
| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Shared Memory | Disk |
+--------------------+----------------+---------------------+---------------+------+

Configurations

Server:

The default binding address for the go judge is localhost:5050. Can be specified with -http-addr flag.
By default gRPC endpoint is disabled, to enable gRPC endpoint, add -enable-grpc flag.

- The default binding address for the gRPC go judge is localhost:5051. Can be specified with -grpc-addr flag.

The default log level is info, use -silent to disable logs or use -release to enable release logger (auto turn on if in docker).
-auth-token to add token-based authentication to REST / gRPC
By default, the GO debug endpoints (localhost:5052/debug) are disabled, to enable, specifies -enable-debug, and it also enables debug log
By default, the prometheus metrics endpoints (localhost:5052/metrics) are disabled, to enable, specifies -enable-metrics
Monitoring HTTP endpoint is enabled if metrics / debug is enabled, the default addr is localhost:5052 and can be specified by -monitor-addr

Sandbox:

The default concurrency equal to number of CPU, Can be specified with -parallelism flag.
-mount-conf specifies detailed mount configuration, please refer File System Mount as a reference (Linux only)
-file-timeout specifies maximum TTL for file created in file store （e.g. 30m)
The default file store is in memory(/dev/shm/), local cache can be specified with -dir flag.
-output-limit specifies size limit of POSIX rlimit of output (default 256MiB)
-copy-out-limit specifies the default file copy out max (default 64MiB)

You can find more available configuration here.

Run Terminal in the Container

Download go-judge-shell from Release and run. It will connect local go-judge, and open an interactive shell in the container for debugging purpose.

Return Status

Accepted: Program exited with status code 0 within time & memory limits
Memory Limit Exceeded: Program uses more memory than memory limits
Time Limit Exceeded: (exitStatus usually have value 9 as killed by SIGKILL after timeout)

- Program uses more CPU time than cpuLimit - Or, program uses more clock time than clockLimit

Output Limit Exceeded:

- Program output more than pipeCollector limits - Or, program output more than output-limit

File Error:

- CopyIn file is not existed - Or, CopyIn file too large for container file system - Or, CopyOut file is not existed after program exited

Non Zero Exit Status: Program exited with non 0 status code within time & memory limits
Signalled: Program exited with signal (e.g. SIGSEGV)
Dangerous Syscall: Program denied by the default seccomp filter
Internal Error:

- Program is not exist - Or, container create not successful (e.g. not privileged docker) - Or, other errors

Container Root Filesystem

For linux platform, the default mounts points are bind mounting host's /lib, /lib64, /usr, /bin, /etc/ld.so.cache, /etc/alternatives, /etc/fpc.cfg, /dev/null, /dev/urandom, /dev/random, /dev/zero, /dev/full and mounts tmpfs at /w, /tmp and creates /proc.

To customize mount points, please look at example mount.yaml file.

If mount.yaml is not specified, the size of tmpfs for /w and /tmp is configured through -tmp-fs-param with default value size=128m,nr_inodes=4k

If a file named /.env exists in the container rootfs, the container will load the file as environment variable line by line.

If a bind mount is specifying a target within the previous mounted one, please ensure the target exists in the previous mount point.

Metrics Monitoring Endpoint

Prometheus Metrics Monitoring Endpoint

Notice

[!WARNING]

Windows and macOS support are experimental and should not be used in production environments

cgroup usage

For cgroup v1, the go-judge need root privilege to create cgroup. Either creates sub-directory /sys/fs/cgroup/cpuacct/gojudge, /sys/fs/cgroup/memory/gojudge, /sys/fs/cgroup/pids/gojudge and make execution user readable or use sudo to run it.

For cgroup v2, systemd dbus will be used to create a transient scope for cgroup integration.

If no permission to create cgroup, the cgroup related limit will not be effective.

cgroup v2 support

The cgroup v2 is supported by go-judge now when running as root since more Linux distribution are enabling cgroup v2 by default (e.g. Ubuntu 21.10+, Fedora 31+). However, for kernel < 5.19, due to missing memory.max_usage_in_bytes in memory controller, the memory usage is now accounted by maxrss returned by wait4 syscall. Thus, the memory usage appears higher than those who uses cgroup v1. For kernel >= 5.19, memory.peak is being used.

When running in containers, the go-judge will migrate all processed into /api hierarchy to enable nesting support.

When running in Linux distributions powered by systemd, the go-judge will contact systemd via dbus to create a transient scope as cgroup root.

When running with kernel >= 5.7, the go-judge will try faster clone3(CLONE_INTO_CGROUP) and vfork method.

Memory Usage

The controller usually consumes 20M memory and each container usually consumes 20M + size of tmpfs 2 * 128M. For each request, it consumes as much as user program limit + extra limit (16k) + total copy out max. Notice that the cached file stores in the shared memory (/dev/shm) of the host, so please ensure enough size allocated.

For example, when concurrency = 4, the container itself can consume as much as 60 + (20+32) 4M = 268M + 4 total copy out + total max memory of requests.

Due to limitation of GO runtime, the memory will not return to OS automatically, which could lead to OOM killer. A background worker was introduced to checks heap usage and invokes GC when necessary.

-force-gc-target default 20m, the minimal size to trigger GC
-force-gc-interval default 5s, the interval to check memory usage

criyle/go-judge: Sandbox service built on Linux container technologies with simple REST and gRPC API