Profile
Back to NewsBack
GitHub Trending 9 min
Reader Mode
wmfs/tymly: An open framework for building digital services.

wmfs/tymly: An open framework for building digital services.

14 hours ago

!Tymly Logo

Tymly Package</a> Commitizen friendly</a> JavaScript Style Guide</a> license</a> FOSSA Status</a>

Introduction

__Tymly is the product of many inter-related Node.js packages. The source code for all these packages is maintained in separate GitHub repositories in the WMFS organization.__

However, for those wanting to develop the Tymly framework itself, it can be tricky to keep-track as new Tymly repos are added and existing Tymly repos evolve. It's also important to ensure all these repos are linked together locally for the best possible developer experience.

_And that's what this repo can help with!_

Here we have an ___empty___ pnpm workspace-powered monorepo. By following the instructions below, it's possible to automatically fill the empty /packages, /plugins, /blueprints, /cardscript, /apps, /mods and /e2e directories with the freshest Tymly code from https://github.com/wmfs.

  • __Subsequent synchronization attempts will update local repos as necessary, and clone anything new that's become available.__
  • __Repos you're working on are safe:__ if you've checked out a branch, sync won't touch it. It just tells you what's new on master.

Environment

__There are a couple of things you'll need installed for all this to work...__

Git

Git is a version-control system for tracking changes in computer files and coordinating work on those files among multiple people.
  • https://git-scm.com/downloads

Node.js

Node.js is an open-source, cross-platform JavaScript run-time environment that executes JavaScript code outside of a browser.
  • https://nodejs.org
  • Node v20 or later is required.

pnpm

We use pnpm workspaces to link together all the various Tymly packages. Shared dependencies are stored once on disk and hard-linked into each package.

With Node installed, install pnpm globally:

`` bash npm install -g pnpm

Then check it's working with pnpm --version, which should print 12 or higher. Lerna is no longer needed.

> If you have Corepack enabled, it may intercept the pnpm command and fail with a Cannot find module ... pnpm.cjs error. Run corepack disable pnpm and install pnpm with npm as above.

GitHub Access Token

> Instead of using your GitHub password, we use a Personal Github Access Token. > This approach provides a few advantages - in particular finer access control and more specific monitoring. __Be sure to keep the value of your Access Token value secret!__

__Assuming you've already signed-up with GitHub, you'll need to create a new Access Token for all your Tymly-related interactions.__

  • Generating a new token value is easy enough, first go here:
* https://github.com/settings/tokens
  • Then click the __"Generate new token"__ button.
Feel free to give your new token any name you like, but something like "Tymly Monorepo*" will be fine.
  • As for those __Scopes__, click this one:
* [x] __repo__ (Full control of private repositories)
  • Then hit __"Generate token"__.
  • Copy the token value: we'll be setting an environment variable to it later.
> If you use a fine-grained token instead, give it read access to __Contents__ and __Metadata__ for the
wmfs organization's repositories.

Git Credentials

__The token above is only used to ask GitHub which repos exist. Cloning and fetching use your own git credentials__, so git needs to be able to reach private wmfs repos without asking for a password:

  • On Windows and macOS, Git Credential Manager (included with Git for Windows) handles this. Signing in once is enough.
  • Alternatively, if you use the GitHub CLI, run gh auth setup-git.
Sync runs git without interactive prompts, so if your credentials are missing, the affected private repos are reported as errors rather than leaving sync waiting for a password.

npm Access Token

__Many @wmfs packages are private on npm, so installing them needs an npm access token.__ You'll need an npm account that's a member of the wmfs organization. Ask a team member to add you if you're not.

> __Not part of WMFS?__ You won't be able to install private packages, so you don't need an npm token. Use pnpm sync:public-only, which only syncs public packages.

  • Sign in at https://www.npmjs.com, then open __Access Tokens__ from your profile menu.
  • Click __"Generate New Token"__ and choose a __Granular Access Token__.
Give it a name like "Tymly Monorepo*", and set the expiry to the longest allowed (90 days).
  • Give it __read-only__ access to packages, and no access to organizations. Installing doesn't need anything more, so don't use a token that can publish.
  • Generate the token and copy its value.
Then add it to the
.npmrc file in your home directory (create the file if it doesn't exist):
//registry.npmjs.org/:_authToken=npm_yourTokenHere
Check it works with npm whoami, which should print your npm username. __Keep the token secret__, and never commit it or put it in the repo's own .npmrc.

When your npm token expires

npm tokens expire after 90 days. When that happens, installs fail with a misleading error, because npm answers __404 Not Found__ for private packages rather than "not authorised":

ERR_PNPM_FETCH_404 GET https://registry.npmjs.org/@wmfs%2Fsome-private-package: Not Found - 404
__If you see a 404 for an @wmfs package, check your token before anything else.__ Create a new one as above, and replace the old line in ~/.npmrc.

pnpm sync and pnpm bootstrap check your npm token before doing anything, so an expired token stops them straight away with a clear message instead. You can run the same check yourself with pnpm check-npm.

They also warn you when your token expires within 14 days, so you can replace it in good time. The expiry date is read from npm. If that isn't possible, you can record it yourself by setting the TYMLY_NPM_TOKEN_EXPIRES environment variable (see below). It's also worth setting a calendar reminder when you create a token.

Cloning

Next you'll need to clone this repo. From the __Git__ shell:

git clone https://github.com/wmfs/tymly.git

Then, from the command prompt, install all the __Node.js__ packages required to make this repo work:

cd tymly pnpm install
# Environment Variables

__To integrate this repo with your GitHub account, an environment variable will need defining. A second one is optional...__

| Environment Variable | Notes | | -------------------- | --------------------------------------------------------------------- | | TYMLY_GITHUB_TOKEN | And the value of this environment variable should be set to the personal __Access Token__ you previously generated. | | TYMLY_NPM_TOKEN_EXPIRES | Optional. The date your npm token expires, for example 2027-02-01. Only needed if sync can't read the expiry from npm; see When your npm token expires. |

Synchronizing

Nearly there! :smiley:

To synchronize your empty Tymly monorepo, run this from within the /tymly directory:

pnpm sync

__Which will:__

  • Connect to GitHub (using the token in TYMLY_GITHUB_TOKEN) and find every Tymly repo in the WMFS organization.
  • Clone any you don't have yet into the right directory.
  • Update the ones you do have, where it's safe to:
* __On master with no local changes:__ fast-forwarded to the latest commit. * __On another branch:__ left exactly as it is. Sync fetches, then tells you how many new commits are on master so you can merge or rebase when you're ready. __Uncommitted changes, unpushed commits, or a merge or rebase in progress:__ left alone and listed under Needs your attention*.

__...which should lead to output looking similar to:__

Downloading repo information from https://github.com/wmfs Cloned plugins/tymly-rbac-plugin On branch plugins/tymly-pg-plugin (feature/connection-pool, master has 1 new commit) Updated plugins/tymly-core (pulled 1 commit)

Cloned (1) plugins/tymly-rbac-plugin Plugin for Tymly

Updated (1) plugins/tymly-core 1 new commit ff3037f fix: handle missing state machine (Jane Smith, 2 hours ago)

Needs your attention (1) plugins/tymly-pg-plugin On feature/connection-pool: master has 1 commit(s) you don't 8ccdfe1 feat: support schema search paths (Jane Smith, 3 hours ago)

1 up to date, 1 cloned, 1 updated, 1 need attention, 0 errors

Dependencies may have changed. Reinstall/relink before running anything.

Done (14 seconds).

Running pnpm install...

__Just pnpm sync anytime you want to ensure your local Tymly repos reflect those on GitHub.__

Options

| Command | What it does | | ------- | ------------ | | pnpm sync:dry | Shows what sync would do, without changing anything. | | pnpm sync:verbose | Also lists every repo that was skipped, and why. | | pnpm sync public-only | Only syncs public packages. | | pnpm sync --no-install | Syncs repos but skips the pnpm install at the end. | | pnpm sync --concurrency=4 | Works on fewer repos at once (the default is 6). Useful on slow connections. |

Options can be combined, for example pnpm sync --dry-run public-only.

If anything fails, sync finishes the other repos, lists the failures under Errors, and exits with a non-zero code.

If the final install fails with a 404 for an @wmfs package, your npm token has probably expired. See When your npm token expires.

Which repos are synced, and where

A repo is synced if its package.json has tymly in its keywords. The directory it goes into comes from its other keywords:

| Keyword | Directory | | ------- | --------- | | e2e | /e2e | | mod | /mods | | app | /apps | | cardscript | /cardscript | | blueprint | /blueprints | | plugin | /plugins | | package | /packages |

If a repo has more than one of these keywords, the one highest in the table wins. To choose the directory explicitly instead, add it to the repo's package.json:

json "config": { "tymly": { "type": "blueprint" } }
To stop a repo from being synced at all, set "sync": false in the same place:
json "config": { "tymly": { "sync": false } }
`

Linking and installing

After syncing, lerna-sync may show this message:

Dependencies may have changed. Reinstall/relink before running anything.

You don't need to do anything about it: pnpm sync finishes by running pnpm install, which links every synced package to its local siblings and installs everything else. It's incremental, so it's quick when little has changed. (It's skipped for --dry-run and --no-install.)

Local Tymly packages are always linked in preference to published versions, even when version ranges don't match. See .pnpmfile.cjs for how.

If your install gets into a strange state, start from scratch with:

pnpm bootstrap

which checks your npm token, then deletes every node_modules directory and reinstalls.

Other useful commands:

| Command | What it does | | ------- | ------------ | | pnpm test | Runs test in every package that has one | | pnpm --filter @wmfs/tymly test | Runs test in a single package | | pnpm -r --workspace-concurrency=1 test | Runs tests one package at a time (useful if suites clash over a shared database) | | pnpm clean | Deletes every node_modules directory | | pnpm check-npm` | Checks your npm token works, and warns if it expires soon |

Next steps

__With your Tymly repos cloned and packages installed, what next?__

  1. Check out our Tymly docs site.
  2. Have a read of our contributor Code of Conduct.
  3. Also, please read our notes about contributing.

License

MIT

Chat with me