Beancount and Python compatibility

Development on main targets Beancount v3 and Beangulp, with Python 3.11–3.15. The first stable release of this line is planned as 1.0.0. Until it is published, use the development installation below for Beancount v3.

For Beancount v2, pin the historical release: pip3 install beancount-reds-importers==0.10.0. The v2 line is no longer maintained. The 1.0 transition requires updating old beancount.ingest configurations to Beangulp; see the included example configuration.

Beancount Red's Importers

Simple importers and tools for Beancount, software for plain text, double entry bookkeeping. _More importantly, a framework to allow you to easily write your own importers._

Introduction

This is a reference implementation of the principles expressed in The Five Minute Ledger Update.

Importers can be ugly and painful to write, yet are important in automating the grunt work out of maintaining personal finance software. The philosophy is to make writing sustainable, dependable importers easy. To achieve this, the design separates importers in to three parts:

1. file format reader (reusable)
2. transaction builder (reusable)
3. institution-specific declarations and code (minimal, institution specific) <- _The

This design helps move most of the heavy-lifting common code into (1) and (2) above. Writing new importers is made easier since one only has to write code to address the institution-specific formatting and quirks for each bank/brokerage. See working examples of an ofx based and csv based importers.

Importers

File format readers included are:

• .ofx
• .csv (single and multitable support)
• .xlsx (single and multitable support) (pip3 install xlrd if you plan to use this)
• .pdf (single and multitable support)

Transaction builders included are:

• Banking (for banks and credit cards, which benefit from a postings predictor like

• Investments/brokerages (to handle the very many distinct cases of investment related

• Paychecks (to handle paychecks, which typically contain very many mostly

Input in .ofx format (over .csv) is preferred, when provided by the institution, as it minimizes data and coding errors, eliminates format breaking changes in .csv files, and typically includes balances that are used to generate balance assertions, and commodity prices.

See here for a list of institutions built-in. More investment, credit card, and banking institutions will be added in the future. Contributions welcome.

Tools and Utilities

• ofx-summarize: Quick and dirty way to summarize a .ofx file, and peek inside it
• bean-download: Download account statements automatically

The commands include shell auto-completion (tab-to-complete) via click. bean-download, in particular, can complete the account or account groups you want to download, which can be handy. To enable it in zsh, do (see here for other shells):

mkdir -p ~/.zcomplete
_OFX_SUMMARIZE_COMPLETE=zsh_source ofx-summarize > ~/.zcomplete/ofx-summarize-complete.zsh
_BEAN_DOWNLOAD_COMPLETE=zsh_source bean-download > ~/.zcomplete/bean-download-complete.zsh
Place this in your shell's rc file (.zshrc or .bashrc or .fishrc):
for f in ~/.zcomplete/*; do source $f; done

Features

• supports Beancount output via bean-extract

• automatically generates balance assertions
• support for:

• file format independent (ofx, csv, xlsx supported out of the box; single and

• supports commodity-leaf accounts
• see The Five Minute Ledger Update

Installation

pip3 install beancount-reds-importers

To install development code from main (which can temporarily have failing tests):

pip3 install git+https://github.com/redstreet/beancount_reds_importers

Running

Running the included examples:

1. cd /example #eg: cd ~/.local/lib/python3.8/site-packages/beancount_reds_importers/example
2. ./import.sh OfxDownload.qfx # Imports investments
3. ./import.sh transactions.qfx # Import bank transactions; uses smart_importer to classify transactions

Creating and running your own config:

1. Create your own config file. Example _config.py and _config-smart.py files provided. At the least, include your account numbers
2. Include fund information. Copy the included fund_info.py to start with.
3. You can now run my_config.py identify, my_config.py extract, etc. See the included script: Run ./import.sh
4. If identifier/cusip/isin info is missing, the importer will let you know. Add it to your

Configuring balance assertion dates

• smart: smart date (default; see below)
• ofx_date: date specified in ofx file
• last_transaction: latest transaction date (the "tightest" assertion)
• today: today's date

smart dates: Banks and credit cards typically have pending transactions that are not included in downloads. When we download the next statement, new transactions may appear prior to the balance assertion date that we generate for this statement, which renders this balance assertion invalid. This problem manifests occasionally as an existing balance statement breaking when a new statement is downloaded and is an annoyance as it needs manual fixing.

To minimize this, we set the balance assertion date to either two days (fudge factor to account for pending transactions) before the statement's end date or the last transaction's date, whichever is later. To choose a different fudge factor, simply set balance_assertion_date_fudge in your config.

Configuring currency precision

def custom_init(self):
    self.currency_precision = 0  # For JPY
    # or self.currency_precision = 8  # For cryptocurrencies

Note

Depending on the institution, the payee and narration fields in generated transactions may appear to be switched. This is described by libtransactionbuilder/banking.py, and the fields can be swapped in a custom_init.

Customizing

See Customizing Your Importer for how to customize each imported transaction.

Testing

uv sync --locked --all-extras
uv run --no-sync pytest --cov
uv run --no-sync coverage report --include='/libreader/.py,/libtransactionbuilder/.py' --fail-under=85

Unit tests exercise shared parsing and accounting behavior. Importer regression tests compare anonymized statements against reviewed expected output. To add a fixture, run uv run pytest path/to/that/test --generate, review every generated amount, posting, and date, then run the test normally. CI never regenerates expected output.

CI also builds the wheel and source archive, installs each into a clean environment, and exercises real banking and investment imports with ledger balance assertions. See CONTRIBUTING.md for local checks and RELEASING.md for the automated release process.

Anonymized data to increase regression test coverage is most welcome. Please submit a PR if you can. See here for an example to follow.

More broadly I run tests across hundreds of actual ofx and csv files, against reference outputs that I know to be correct from my personal file. However, I'm unable to share them since these are personal. Testing against real world files is best, so I recommend you do this with your own input files.

Contact

Contributions

Features, fixes, and improvements welcome. New importers for institutions with test input files appreciated. Sharing importers helps the community! Remember:

• Feel free to send send pull requests. Please include unit tests
• For larger changes or changes that might need discussion, please reach out and discuss

• Please squash your commits (reasonably)
• Use conventional commits for commit messages