My personal toolbox

Shell, Python, and Perl scripts I wrote/use on a daily basis; i.e. eating my own dog food.

Quick overview by category

The summary/index below was generated by local AI (DSv4 Flash); keep in mind that many utilities below are probably WAY out of date... Use your own judgement about them - that is, check whether they fit your own needs before using them.

Mostly, this repo establishes a common baseline in any machine I work in; a baseline that I've muscle-memoried over the years.

Even if none of the utilities below are useful to you, the approach itself - i.e. building a publicly accessible arsenal you can 'git clone' at a moment's notice... that is definitely recommended :-)

Sandboxing & container isolation

| Script | What it does | |---|---| | isolate.sh | Firejail-based sandbox: network allowlisting, read-only $HOME, per-path rw/hide. The core isolation engine | | vimisolated.sh | Launch vim inside isolate.sh, auto-detecting writable paths | | pi.isolated.sh | Run pi (AI coding agent) inside isolate.sh, tunneling through a Unix socket to reach a local LLM. Symlinked as pi.sh | | desert_island_container_execution.sh | Run untrusted TCP server code in a Docker container that the host can reach, but the container can't reach outside the LAN | | dockerme.sh | Launch a Docker container with the current folder mapped, optional X11/PulseAudio passthrough, network toggle, root toggle | | dockerClearRunningContainersAndNoneImages.sh | Nuke all containers and images | | parse-isolation-options-common.sh | Shared argument parser for isolate.sh / pi.isolated.sh (sourced, not standalone) |

Example usage of isolation (vimisolated.sh is much better, but this works as a basic example:

# Read-only $HOME, no network
isolate.sh -- vim ~/bin/pi.sh
Allow specific servers, make some paths writable, hide SSH keys from any vim-spawned subprocesses (Language servers, etc)
isolate.sh --rw ~/bin --rw ~/.viminfo --hide ~/.ssh \
           --servers servers.txt --dns 1.1.1.1 -- vim ~/bin/pi.sh

Example usage of isolated pi.dev harness:

# Assumes a local LLM at localhost:8081
pi.isolated.sh [--port PORT] [isolate.sh options] [-- pi options]

File indexing & syncing

| Script | What it does | |---|---| | indexer.py | Scan folders, store MD5s in SQLite, detect changes. Parallel MD5 with ProcessPoolExecutor | | syncer.py | Cross-reference a new folder against an existing indexer.db: finds already-existing files (by MD5) and similar filenames (Levenshtein distance) | | vidir | Make the moreutils/vidir work properly with my isolated vim) | | cmpDir1andDir2andRemoveIdenticalFromDir1 | Remove from dir1 files that are identical (size+MD5) in dir2 | | cmpDir1andDir2andRemoveIdenticalFromDir1BasedOnSizeAndTimestamp | Same, but uses size + timestamp (faster, no MD5) |

Example usage:

# Scan folders, store MD5s
indexer.py /mnt/photos

Duplicate detection

| Script | What it does | |---|---| | find_dup_videos.py | Detect duplicate videos by duration + perceptual hash (imagehash) | | find_dup_images.py | Find duplicate images via perceptual hashing |

# Requires ffmpeg/ffprobe + python3 -m pip install ImageHash pillow
find_dup_videos.py /path/to/videos/

Video related

| Script | What it does | |---|---| | minimalVideo.py | Transcode any video to x264 at ~0.06 b/p, mux with original audio via mkvmerge | | minimalVideo265.py | Same concept, but using HEVC (x265) | | minimalVideoCRF_265.sh | HEVC encode with CRF mode (quality-based), copy audio | | minimalVideoCRF_265_copy_audio.sh | HEVC CRF encode, copy audio track directly | | minimalVideoCRF_265_forQuest.sh | HEVC CRF encode variant for VR headsets (Quest) | | vaapi_encode.sh | VAAPI hardware-accelerated HEVC encode | | vaapi_encode.just_encode.sh | Minimal VAAPI HEVC encode (no rescaling) | | cuda_encode_hevc.sh | NVENC HEVC encode (NVIDIA GPU) | | cuda_encode_av1.sh | NVENC AV1 encode (NVIDIA GPU) | | fps_2x.sh | Double frame-rate via VAAPI motion interpolation (60fps) | | fps_2x_vapoursynth/ | Vapoursynth-based 2x FPS scripts | | any2mp3.sh | Extract audio track to LAME mp3 from any media file | | transcode.sh | Generic ffmpeg transcode wrapper | | show_video_bpp.py | Show bits-per-pixel stats for a video | | show_video_details.py | Dump codec/res/fps/bitrate of a video file | | m4a2aac.sh | Extract AAC from M4A container | | mpvYoutubeNoVideo.sh | Play YouTube audio-only (no video, saves CPU) | | jpg2lep.sh | Lossless JPEG recompression via Dropbox Lepton | | png2webp.sh | Convert PNG to WebP via ffmpeg | | recordX11.sh | Losslessly compressed X11 screen recording | | flashVideo.sh | Extract flash video from browser plugin's pipe | | misc.py | Shared utility module for video scripts |

Pi (AI coding agent) helpers

| Script | What it does | |---|---| | pi.isolated.sh | Primary launcher: sandboxed pi session with local LLM via Unix socket tunnel. Symlinked as pi.sh | | pi.google.sh | Pi launcher using Google AI Studio (Gemini) | | pi.google_run.sh | Quick one-shot Google AI Studio run | | pi.dockerized.vllm.sh | Run vLLM model server in Docker, then launch pi connected to it | | pi_parse_stream.py | Shared parser for pi's streaming JSON responses | | cclog.sh | Run cclog Docker image to convert a JSONL conversation log to Markdown | | localAI/ | Dockerfiles, configs, and launch scripts for local LLM serving (vLLM, llama.cpp, etc.) | | Dockerfiles/ | Dockerfiles for cclog and yt-dlp binary images |

System monitoring & statistics

| Script | What it does | |---|---| | dstat | Enhanced dstat binary (system resource monitor) | | dstat.sh | Wrapper: dstat -clnv --fs --vm | | stats.py | Pipe in numbers → colored statistics (mean, stddev, median, min/max) | | statsLive.py | Real-time mean+stddev over stdin stream (Welford's online algorithm) | | histogram.py | Generate histogram + percentiles from piped data | | histogram.sh | Bash wrapper for histogram.py with outlier detection | | _histogram.py | Internal histogram module | | asciigraph.py | ASCII line graph from piped data (matplotlib) | | crystaldiskmark.sh | CrystalDiskMark clone for Linux using fio | | benchmark.nvme.via.io.uring.sh | NVMe sequential read benchmark via io_uring | | percentile.sh | Quick percentile calculator | | barChartTimes.sh | Horizontal bar chart of execution times (from time output) | | performance | Performance tuning wrapper | | ondemand | CPU governor toggle (ondemand vs performance) |

# Colored stats
for i in {1..100}; do echo $i; done | stats.py
Histogram
cat numbers.txt | histogram.py

Networking

| Script | What it does | |---|---| | block_user_via_iptables.sh | Block a Unix user's outgoing traffic via iptables | | unblock_user_via_iptables.sh | Remove the iptables block for a user | | sync_NIC_HW_timestamping_clock_from_host.py | Sync NIC hardware timestamp clock (phc2sys) for accurate tcpdump timestamps | | tcpdump2binary.py | Extract raw Ethernet frames from a pcap into binary files (uses scapy) | | myip.sh | Show my public IP | | mydu.sh | Better du: sums file sizes (filesystem-agnostic) per directory | | nointernet.for.genymotion.sh | Block Genymotion VM from internet access | | youtubeFromClipboard.sh | Download YouTube video from clipboard URL | | yt-dlp.sh | Wrapper around Dockerfiles/Dockerfile.yt-dlp container (sandbox the madness) |

Subtitle tools

| Script | What it does | |---|---| | get_subs_tmux.sh | Download YouTube subtitles, convert to text, launch interactive pi session to summarize. Symlinked as zz | | vtt2text.py | Convert VTT subtitle files to clean text | | subrip.sh | Subtitle RIP/conversion utility | | sub_auto_fixer.py | Match subtitle files to video files by Dice coefficient + LCS |

File/text utilities

| Script | What it does | |---|---| | latest.py | List recently modified files in a directory, sorted by timestamp | | latest.sh | Bash-only poor-mans-version of latest.py | | exclude.sh | Interactively filter lines from a file by regex (used with latest.py) | | excludeFilter.sh | Same, but via stdin pipe | | greedy.py | Pack files to maximally fit a target size | | lost_my_space.py | Find what's consuming disk space in a filesystem | | lost_my_space.sh | Bash wrapper | | logDurations.py | Parse timestamped logs and show duration of each run | | sortXML.py | Sort XML elements while preserving structure | | csvToHTML_bootstrap.py | CSV → styled HTML table (Bootstrap) | | csvToHTML_barebones.py | CSV → minimal HTML table | | excelToCsv.py | Convert Excel (.xls/.xlsx) to CSV | | countMoviesLength.py | Sum total duration of video files in a folder | | rmdirRecursive.sh | Remove all empty directories recursively (post-cleanup) | | inlineDataInHTML.sh | Generate data: URI for embedding files in HTML | | nocolor.sh | Strip ANSI color codes from pipe | | quote.sh | Wrap each line in double-quotes, escaping inner quotes |

Clipboard & automation

| Script | What it does | |---|---| | clipboardDaemon.py | Clipboard monitoring daemon | | execOnChange.sh | Run a command whenever files matching a pattern change (like make for file changes) | | batch.SMP.processing.with.bash | Simple parallel processing with bash | | lock.sh | Lock screen (i3lock) | | setTitle.sh | Set XTerm title | | tmux-attach.sh | Attach to a tmux session | | tmux-restore.sh | Restore tmux sessions from saved state | | tmux_colors.sh | Show tmux color palette for config | | fzf-preview.sh | File preview while browsing with fzf + bat | | timestamped_logger.sh | Run a command, log stdout+stderr with timestamps |

Developer tools

| Script | What it does | |---|---| | cpptypefmt | C++ type formatting helper | | clean-template-mess.py | Pretty-print C++ template instantiations with indentation | | findClassInJars.sh | Find a class name inside a collection of JAR files | | findIn | Flexible file content search | | git-lg | Git log with graph (one-line) | | diffIgnoreCaseAndWS.sh | diff -u -i -b wrapper | | diffu | Unified diff shortcut | | diffu.meld | Diff with meld | | xmlprettyprint.sh | XML pretty-printer (symlink to tidyXML.sh) | | tidyXML.sh | XML tidy/format | | html5check.py | HTML5 validation | | make_ada_tags.py | Generate ctags for Ada sources (gnatxref) | | pylintAll.sh | Run pylint on all Python scripts in current dir | | showBashCmdline.sh | Show the currently executing command line inside a bash shell | | vimcat.sh | Cat a file with vim-based ANSI syntax highlighting | | svndiff | SVN diff wrapper | | svndiffMELD | SVN diff via Meld | | svndiffWS | SVN diff (whitespace-aware) | | svndiffXXDIFF | SVN diff via xxdiff |

Format conversion & encoding

| Script | What it does | |---|---| | ansi2html.sh | Convert ANSI-colored terminal output to HTML | | htmlEntities.pl | HTML-encode text (Perl) | | unicodeUnescape.py | Decode \uXXXX escape sequences | | hex2utf.py | Hex → UTF-8 converter | | mime_decoder_inplace_pipe.sh | Decode MIME-encoded files in-place | | epochToHuman.pl | Convert Unix epoch to human date | | tohex | Decimal → hex | | todec | Hex → decimal | | tozero.sh | feed safely from a pipe to xargs -0 | | 256color.pl | Display 256-color terminal palette | | href.pl | Extract href links from HTML | | img.pl | Extract img src links from HTML | | unescape.pl | Decode URL-encoded (%XX) and HTML entities | | speed.pl | Monitor download speed/progress of a file |

System administration

| Script | What it does | |---|---| | swap.usage.sh | Show swap usage per process | | zswapStatus.sh | Check zswap status | | hyperthreading | Toggle hyperthreading | | brightScreen.sh | Set backlight brightness to max | | waitForCoolCPU.sh | Wait until CPU temperature drops | | waitForProcToDie.pl | Wait for a process to exit, then run a command | | dumpMemoryOfPID.sh | Dump all memory pages of a PID (as root) | | dump_all_thread_callstacks_for_pid.sh | Dump all thread callstacks of a PID via GDB | | callstacks_of_threads.sh | Symlink to above | | thread_callstacks.sh | Symlink to above | | snap_remove_old_versions.sh | Clean old snap package versions | | archLinux.orphans.ordered.by.size.sh | List orphaned pacman packages sorted by size | | x11.off.sh | Turn off display (DPMS) | | x11.off.saver | Screen saver variant | | fixClock.sh | Sync system clock via NTP (pool.ntp.org) | | 7z-ttsiod-compress | 7z compression wrapper | | makeSelfSignedCertificate.sh | Generate self-signed SSL certificate (legacy, use Let's Encrypt instead) | | bcmp.sh | Binary compare two files (shows offset + hex bytes) | | setSubtract.pl | Subtract one list of strings from another | | setUnversionedToSVNignore.sh | Mark all ? files as svn:ignore | | USB_drives_need_no_sleep.py | Keep USB drives awake (prevent spin-down on ZFS server) | | loopaesmount.py | Mount loopback AES-encrypted containers | | pulseaudio_record.sh | Record PulseAudio output to OGG | | sync_music_to_android_with_m3u_for_folder_symlinks.sh | Sync music to Android with .m3u playlist files | | mergefs_show_realpath.sh | Show mergerfs real path for files | | pdfCountPages.sh | Sum page counts of PDF files | | yq_outliers.sh | Flatten YAML to dot-notation for grepping | | x16.py | GDB helper: dump 16-column memory (vs default 8) | | unchroot.pl | Break out of a chroot jail (security testing) |

Dependencies

• Most scripts: bash, standard POSIX tools
• Python scripts: Python 3, various modules (see docstrings)
• Video scripts: ffmpeg / ffprobe, mkvmerge, x264/x265, VAAPI or NVENC drivers
• isolate.sh: firejail, ip, getent
• dockerme.sh: Docker
• pi.isolated.sh: A locally running LLM endpoint (llama.cpp, vLLM, etc.)
• cclog.sh: Docker image cclog (build from localAI/)