k1LoW/runn: runn is a package/tool for running operations following a scenario.

!Coverage !Code to Test Ratio !Test Execution Time

runn ( means "Run N". is pronounced /rʌ́n én/. ) is a package/tool for running operations following a scenario.

Key features of runn are:

As a tool for scenario based testing.
As a test helper package for the Go language.
As a tool for workflow automation.
Support HTTP request, gRPC request, DB query, Chrome DevTools Protocol, and SSH/Local command execution
OpenAPI Document-like syntax for HTTP request testing.
Single binary = CI-Friendly.

Online book

Quickstart

You can use the runn new command to quickly start creating scenarios (runbooks).

:rocket: Create and run scenario using curl or grpcurl commands:

!docs/runn.svg

Command details

``console $ curl https://httpbin.org/json -H "accept: application/json" { "slideshow": { "author": "Yours Truly", "date": "date of publication", "slides": [ { "title": "Wake up to WonderWidgets!", "type": "all" }, { "items": [ "Why WonderWidgets are great", "Who buys WonderWidgets" ], "title": "Overview", "type": "all" } ], "title": "Sample Slide Show" } } $ runn new --and-run --desc 'httpbin.org GET' --out http.yml -- curl https://httpbin.org/json -H "accept: application/json" $ grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello { "reply": "hello alice" } $ runn new --and-run --desc 'grpcb.in Call' --out grpc.yml -- grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello $ runn list *.yml Desc Path If --------------------------------- grpcb.in Call grpc.yml httpbin.org GET http.yml $ runn run *.yml ..


2 scenarios, 0 skipped, 0 failures
</details>
:rocket: Create scenario using access log:
!docs/runn_axslog.svg
<details>
<summary>Command details</summary> console
$ cat access_log
183.87.255.54 - - [18/May/2019:05:37:09 +0200] "GET /?post=%3script%3ealert(1); HTTP/1.0" 200 42433
62.109.16.162 - - [18/May/2019:05:37:12 +0200] "GET /core/files/js/editor.js/?form=\xeb\x2a\x5e\x89\x76\x08\xc6\x46\x07\x00\xc7\x46\x0c\x00\x00\x00\x80\xe8\xdc\xff\xff\xff/bin/sh HTTP/1.0" 200 81956
87.251.81.179 - - [18/May/2019:05:37:13 +0200] "GET /login.php/?user=admin&amount=100000 HTTP/1.0" 400 4797
103.36.79.144 - - [18/May/2019:05:37:14 +0200] "GET /authorize.php/.well-known/assetlinks.json HTTP/1.0" 200 9436
$ cat access_log| runn new --out axslog.yml
$ cat axslog.yml| yq
desc: Generated by

runn new


runners:
  req: https://dummy.example.com
steps:
  - req:
      /?post=%3script%3ealert(1);:
        get:
          body: null
  - req:
      /core/files/js/editor.js/?form=xebx2ax5ex89x76x08xc6x46x07x00xc7x46x0cx00x00x00x80xe8xdcxffxffxff/bin/sh:
        get:
          body: null
  - req:
      /login.php/?user=admin&amount=100000:
        get:
          body: null
  - req:
      /authorize.php/.well-known/assetlinks.json:
        get:
          body: null
$
</details>
Usage
runn can run a multi-step scenario following a runbook written in YAML format.

As a tool for scenario based testing / As a tool for automation.
runn can run one or more runbooks as a CLI tool. console
$ runn list path/to/*/.yml
  id:      desc:             if:       steps:  path
-------------------------------------------------------------------------
  a1b7b02  Only if included  included       2  p/t/only_if_included.yml
  85ccd5f  List projects.                   4  p/t/p/list.yml
  47d7ef7  List users.                      3  p/t/u/list.yml
  97f9884  Login                            2  p/t/u/login.yml
  2249d1b  Logout                           3  p/t/u/logout.yml
$ runn run path/to/*/.yml
S....
5 scenarios, 1 skipped, 0 failures
### As a test helper package for the Go language.
runn can also behave as a test helper for the Go language.

Run N runbooks using httptest.Server and sql.DB go
func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/*/.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}
#### Run single runbook using httptest.Server and sql.DB go
func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	ts := httptest.NewServer(NewRouter(db))
	t.Cleanup(func() {
		ts.Close()
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Book("testdata/books/login.yml"),
		runn.Runner("req", ts.URL),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.New(opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.Run(ctx); err != nil {
		t.Fatal(err)
	}
}
#### Run N runbooks using grpc.Server go
func TestServer(t *testing.T) {
	addr := "127.0.0.1:8080"
	l, err := net.Listen("tcp", addr)
	if err != nil {
		t.Fatal(err)
	}
	ts := grpc.NewServer()
	myapppb.RegisterMyappServiceServer(s, NewMyappServer())
	reflection.Register(s)
	go func() {
		ts.Serve(l)
	}()
	t.Cleanup(func() {
		ts.GracefulStop()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.Runner("greq", fmt.Sprintf("grpc://%s", addr),
	}
	o, err := runn.Load("testdata/books/*/.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}
#### Run N runbooks with http.Handler and sql.DB go
func TestRouter(t *testing.T) {
	ctx := context.Background()
	dsn := "username:password@tcp(localhost:3306)/testdb"
	db, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	dbr, err := sql.Open("mysql", dsn)
	if err != nil {
		log.Fatal(err)
	}
	t.Cleanup(func() {
		db.Close()
		dbr.Close()
	})
	opts := []runn.Option{
		runn.T(t),
		runn.HTTPRunnerWithHandler("req", NewRouter(db)),
		runn.DBRunner("db", dbr),
	}
	o, err := runn.Load("testdata/books/*/.yml", opts...)
	if err != nil {
		t.Fatal(err)
	}
	if err := o.RunN(ctx); err != nil {
		t.Fatal(err)
	}
}
## Examples
See the details
Runbook ( runn scenario file )
The runbook file has the following format.
step: section accepts list or ordered map.

List: yaml
desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps[0].rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps[1].res.status == 200
  -
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps[1].res.body.session_token }}"
          body: null
    test: steps[2].res.status == 200
  -
    test: len(steps[2].res.body.projects) > 0
Map: yaml
desc: Login and get projects.
runners:
  req: https://example.com/api/v1
  db: mysql://root:mypass@localhost:3306/testdb
vars:
  username: alice
  password: ${TEST_PASS}
steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  login:
    req:
      /login:
        post:
          body:
            application/json:
              email: "{{ steps.find_user.rows[0].email }}"
              password: "{{ vars.password }}"
    test: steps.login.res.status == 200
  list_projects:
    req:
      /projects:
        get:
          headers:
            Authorization: "token {{ steps.login.res.body.session_token }}"
          body: null
    test: steps.list_projects.res.status == 200
  count_projects:
    test: len(steps.list_projects.res.body.projects) > 0
#### Grouping of related parts by color
List:
!color
Map:
!color
JSON Schema
A JSON Schema for the runbook YAML format is available at runbook.schema.yaml.

You can use it with YAML Language Server for editor validation and autocompletion by adding the following comment to the top of your runbook: yaml
yaml-language-server: $schema=https://raw.githubusercontent.com/k1LoW/runn/main/runbook.schema.yaml
### desc:
Description of runbook. yaml
desc: Login and get projects.
runners:
  req: https://example.com/api/v1
vars:
  username: alice
steps:
[...]
### labels:
Labels of runbook. yaml
desc: Login
runners:
  req: https://example.com/api/v1
labels:
  - users
  - auth
steps:
[...]
Runbooks to be run can be filtered by labels. console
$ runn run path/to/*/.yml --label users --label projects
 console
$ runn run path/to/*/.yml --label 'users and auth'
### runners:
Mapping of runners that run steps: of runbook.

In the steps: section, call the runner with the key specified in the runners: section.

Built-in runners such as test runner do not need to be specified in this section. yaml
runners:
  ghapi: ${GITHUB_API_ENDPOINT}
  idp: https://auth.example.com
  db: my:dbuser:${DB_PASS}@hostname:3306/dbname
In the example, each runner can be called by ghapi:, idp: or db: in steps:.
hostRules:

Allows remapping any request hostname to another hostname, IP address in HTTP/gRPC/DB/CDP/SSH runners. yaml
hostRules:
  example.com: 127.0.0.1:8080
  '*.example.test': 192.168.0.16
### vars:
Mapping of variables available in the steps: of runbook. yaml
vars:
  username: [email protected]
  token: ${SECRET_TOKEN}
In the example, each variable can be used in {{ vars.username }} or {{ vars.token }} in steps:.
secrets:

List of secret var names to be masked. yaml
secrets:
  - vars.secret_token
  - binded_password
  - current.res.message.token
### debug:
Enable debug output for runn. yaml
debug: true
#### Show debug output only for failed steps
Use --debug-on-failure to keep the normal output for successful steps and print runner diagnostics only for failed steps.
The diagnostics are written to stderr and include command/stdout/stderr for Exec and SSH runners, stdin for Exec runners, and the request and response details captured for other runners. console
$ runn run path/to/*/.yml --verbose --debug-on-failure
### interval:
Interval between steps.yaml
interval: 1
### if:
Conditions for skip all steps. yaml
if: included # Run steps only if included
### skipTest:
Skip all test: sections yaml
skipTest: true
### force:
Force all steps to run. yaml
force: true
### trace:
Add tokens for tracing to headers and queries by default.
Currently, HTTP runner, gRPC runner and DB runner are supported. yaml
trace: true
### loop:
Loop setting for runbook.
Simple loop runbook yaml
loop: 10
steps:
  [...]
or yaml
loop:
  count: 10
steps:
  [...]
#### Retry runbook
It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed. yaml
loop:
  count: 10
  until: 'outcome == "success"' # until the runbook outcome is successful.
  minInterval: 0.5 # sec
  maxInterval: 10  # sec
  # jitter: 0.0
  # interval: 5
  # multiplier: 1.5
steps:
  waitingroom:
    req:
      /cart/in:
        post:
          body:
[...]
- outcome ... the result of a completed (success, failure, skipped).
concurrency:

Runbooks with the same key are assured of a single run at the same time. yaml
concurrency: use-shared-db
or yaml
concurrency:
  - use-shared-db
  - use-shared-api
### needs:
It is possible to identify runbooks that must be pre-run. yaml
needs:
  prebook: path/to/prebook.yml
  prebook2: path/to/prebook2.yml
Values bound by the bind runner can be referenced by needs.<key>. *.
steps:

Steps to run in runbook.
The steps are invoked in order from top to bottom.
Any return values are recorded for each step.
When steps: is array, recorded values can be retrieved with {{ steps[]. }}. yaml
steps:
  -
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  -
    req:
      /users/{{ steps[0].rows[0].id }}:
        get:
          body: null
When steps: is map, recorded values can be retrieved with {{ steps.<key>.* }}. yaml
steps:
  find_user:
    db:
      query: SELECT * FROM users WHERE name = '{{ vars.username }}'
  user_info:
    req:
      /users/{{ steps.find_user.rows[0].id }}:
        get:
          body: null
### steps[*].desc: steps.<key>.desc:
Description of step. yaml
steps:
  -
    desc: Login
    req:
      /login:
        post:
          body:
[...]
### steps[*].if: steps.<key>.if:
Conditions for skip step. yaml
steps:
  login:
    if: 'len(vars.token) == 0' # Run step only if var.token is not set
    req:
      /login:
        post:
          body:
[...]
### steps[*].loop: steps.<key>.loop:
Loop setting for step.
Simple loop step yaml
steps:
  multicartin:
    loop: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to

.
[...]
or yaml
steps:
  multicartin:
    loop:
      count: 10
    req:
      /cart/in:
        post:
          body:
            application/json:
              product_id: "{{ i }}" # The loop count (0..9) is assigned to

.
[...]
#### Retry step
It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed. yaml
steps:
  waitingroom:
    loop:
      count: 10
      until: 'steps.waitingroom.res.status == "201"' # Store values of latest loop
      minInterval: 500ms
      maxInterval: 10 # sec
      # jitter: 0.0
      # interval: 5
      # multiplier: 1.5
    req:
      /cart/in:
        post:
          body:
[...]
#### Execution order within a loop
When using loop:, it's important to understand the execution order of runners within each iteration:

Main runner (HTTP, DB, gRPC, CDP, SSH, Exec, Include, or Runner) - Only one is executed
Dump runner - Records request/response data (if specified)
Bind runner - Binds variables (if specified)
Test runner - Performs assertions (if specified)
This sequence repeats for each loop iteration.
⚠️ Warning: Avoid using test: with loop.until:

When using loop: with an until: condition, avoid also using test: in the same step. The test: assertion runs on every loop iteration, which can cause the step to fail on the first iteration before the retry logic has a chance to work.

❌ Incorrect usage:yaml
steps:
  retry_until_success:
    req:
      /status:
        get:
    loop:
      count: 10
      until: 'current.res.status == 200'  # Retry until successful
    test: 'current.res.status == 200'     # This runs every iteration and may fail early!
✅ Correct usage:yaml
steps:
  retry_until_success:
    req:
      /status:
        get:
    loop:
      count: 10
      until: 'current.res.status == 200'  # This is sufficient for retry logic
The until: condition already serves as your test condition for retry scenarios.
steps[*].defer: steps.<key>.defer: [THIS IS EXPERIMENT]

Deferring setting for step.yaml
steps:
  -
    defer: true
    req:
      /cart:
        delete:
          body: null
[...]
The step marked defer behaves as follows.
If defer: true is set, run of the step is deferred until finish of the runbook.

Steps marked with defer are always run even if the running of intermediate steps fails.

If there are multiple steps marked with defer, they are run in LIFO order.
    - Also, the included steps are added to run sequence of the parent runbook's deferred steps.
steps[*].force: steps.<key>.force:

Force step to run.yaml
steps:
[...]
  -
     force: true
     dump: previous.res.body
[...]
## Variables to be stored
runn can use variables and functions when running step.
Also, after step runs, HTTP responses, DB query results, etc. are automatically stored in variables.
The values are stored in predefined variables.
| Variable name | Description |
| --- | --- |
| vars | Values set in the vars: section |
| steps | Return values for each step |
| i | Loop index (only in loop: section) |
| env | Environment variables |
| current | Return values of current step |
| previous | Return values of previous step |
| parent | Variables of parent runbook (only included) |

env is a snapshot of the process environment taken at the start of each runbook run. It does not change while the runbook is running, and included runbooks share the snapshot of the runbook that includes them.

Variable Expansion
runn uses {{ }} syntax for variable expansion.

Object variable expansion
When an object variable is used directly in headers: or body:, runn expands it as a map structure, not as a JSON string. yaml
vars:
  auth_headers:
    X-Token: xxx
    X-Api-Key: yyy
steps:
  - req:
      /api:
        get:
          headers: "{{vars.auth_headers}}"
In the example, vars.auth_headers expands to multiple headers.
Converting object to JSON string
To convert an object to a JSON string, wrap the entire {{...}} template expression in single quotes. This signals runn to serialize the object to JSON. yaml
vars:
  metadata:
    user: alice
    role: admin
steps:
  - req:
      /api:
        get:
          headers:
            X-Metadata: "'{{vars.metadata}}'"
## Runner
HTTP Runner: Do HTTP request
Use https:// or http:// scheme to specify HTTP Runner.

When the step is invoked, it sends the specified HTTP Request and records the response. yaml
runners:
  req: https://example.com
steps:
  -
    desc: Post /users                     # description of step
    req:                                  # key to identify the runner. In this case, it is HTTP Runner.
      /users:                             # path of http request
        post:                             # method of http request
          headers:                        # headers of http request
            Authorization: 'Bearer xxxxx'
          body:                           # body of http request
            application/json:             # Content-Type specification. In this case, it is "Content-Type: application/json"
              username: alice
              password: passw0rd
          trace: false                    # add

X-Runn-Trace

 header to HTTP request for tracing
    test: |                               # test for current step
      current.res.status == 201
See testdata/book/http.yml and testdata/book/http_multipart.yml.
Structure of recorded responses
The following response
HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json
Date: Wed, 07 Sep 2022 06:28:20 GMT
Set-Cookie: cookie-name=cookie-value
{"data":{"username":"alice"}}
is recorded with the following structure. yaml
[