bpe_gen seems to be broken when built on WSL #3
Description
Result of $ uname -a on my Debian WSL:
Linux DESKTOP-3K9UIDP 4.4.0-19041-Microsoft #4355-Microsoft Thu Apr 12 17:37:00 PST 2024 x86_64 GNU/Linux
I installed GCC like "normal":
$ sudo apt install gcc -yAnd bootstrapped nob and built all the toolset:
$ gcc -o ./nob ./nob.c
$ ./nobNow, I used first a "dataset" (a txt file) that a friend sent me and said "amazing dataset material trust me" (I'll leave it at the end of the issue, but spoiler: it talks about femboys, so take it into account if you (the reader) read through, viewer discretion is adviced.) but generated .bpe and .tkn files with this command:
$ ./build/txt2bpe -dump-freq 100 -max-iterations 0 -threads-count 8 -input-file ./data/whatever-yazn-sent.txt -output-dir ./outputIt did 148 iterations, so the file I used was ./output/148.bpe
And tried to generate from with:
$ ./build/bpe_gen -bpe ./output/148.bpe -limit 100Result:
INFO: loaded ./output/148.bpe BPE version 1
bpe_gen: src/bpe_gen.c:92: main: Assertion `found' failed.
Aborted (core dumped)And trying to run it several times, would work or not, apparently whether it works or not it's random.
Kind of confused, tried to use bpe2bpe as a "last resort"?:
$ ./build/bpe2bpe -bpe ./output/148.bpe -out ./output/148-2.bpeAnd trying to use that file would yell the same results, it's random whether it works or not.
Trying to use the default txt file that comes bundled in seems to directly throw the abort consistently.
Now, I might be using the program the wrong way, maybe using the wrong flags with bpe_gen, honestly I watched the VODS when they came out
but not again, so I might be lost, but for me, going from the help flag of both txt2bpe and bpe_gen, I don't think I'm doing things wrong.
My build setup:
Device Name DESKTOP-3K9UIDP
Processor Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz 2.11 GHz
Installed RAM 8.00 GB (7.84 GB usable)
Storage 477 GB SSD WDC PC SN530 SDBPNPZ-512G-1027
Graphics Card Intel(R) UHD Graphics (128 MB)
Device ID <is it needed?>
Product ID <is it needed?>
System Type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display
Edition Windows 10 Home Single Language
Version 2009
Installed on 30/04/2025
OS Build 19045.5737
Using Debian WSL installed with command:
$ wsl --install -d DebianAforemention dataset (again, viewer discretion is adviced):
God I Love Femboys.
There's just something undeniably captivating about femboys-the perfect blend of soft aesthetics and bold confidence that breaks all the traditional molds. I love the way they play with fashion, rocking skirts, thigh-highs, or crop tops with absolute flair, all while radiating charm and authenticity. That mix of masculinity and femininity isn't just visually appealing--it's magnetic. It's a rebellion against boring binaries, and I can't help but admire the courage and creativity that comes with expressing yourself so freely. Femboys don't just look good; they carry an energy that says, "This is me, take it or leave it," and I'm definitely taking it.
What draws me in even more is the attitude-playful, flirty, and often wrapped in a surprising depth. There's a softness that doesn't equate to weakness, a strength in being unapologetically cute, and it's endlessly charming. Whether they're posting selfies with a cheeky caption or confidently walking down the street in pastel colors and eyeliner, femboys exude an alluring kind of self-awareness. They know they're breaking norms, and they do it with style and sass. It's impossible not to love that kind of energy