StarDayOpen source rankings
Back to rankings

alasdairforsythe / tokenmonster

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Repository stars
625All-time #53,839
Fork
23
Last push
updated 3 months ago
Repository created
since 2023
Apps & systemsGeneralGoMIT · Permissive
View source

About

Original · GitHub
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Summarised by AI from the README; the original prevails

Author description: Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

Basic listing based on official public information; functionality has not been independently evaluated.

Pricing
Pricing not verifiedCheck current official software, hosting, model and infrastructure pricing.

Star trend

62562662611-06 · 62510-0810-1310-1810-2310-2811-0211-06

First collected 2026-11-06Measured at 2026-11-06 08:25 UTCSourceLicense source

Compiled from public project information; not independently tested. · Repository Stars, not unique users · License tiers are derived from metadata. Check obligations for your use; unknown does not mean prohibited.