Add a multithreaded tokenizer test and as well as 3.14 and 3.14t CI

[ngoldbaum]

Towards fixing #1784.

This is to aid supporting the free-threaded build. Adding explicitly multithreaded tests like this exercises the behavior differences of the free-threaded build compared with the GIL-enabled build. See #1809 where I initially tried this with a different testing approach.

The content of the test is adapted from the asyn test_concurrency test.

I tried to make this use multiprocessing as well but ran into deadlocks so decided to stick to using 4 python threads in a thread pool.

c.f. huggingface/safetensors#637 where I added a multithreaded test to safetensors.

[cclauss]

Perhaps add Py3.14t?

• https://ft-checker.com/?page=13

[ngoldbaum]

@cclauss nope, not until hf-xet updates its PyO3 dependency and/or releases cp314t wheels:

       error: The configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13)
        = help: please check if an updated version of PyO3 is available. Current version: 0.23.5
        = help: The free-threaded build of CPython does not support the limited API so this check cannot be suppressed.
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="11.0" PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.14-64bit" PYO3_PYTHON="/Users/goldbaum/.pyenv/versions/3.14.0t/bin/python3.14" PYTHON_SYS_EXECUTABLE="/Users/goldbaum/.pyenv/versions/3.14.0t/bin/python3.14" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-j1_za471/hf-xet_bf48338361074576a46118b7b50a4e99/hf_xet/Cargo.toml" "--release" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/hf_xet.abi3.so"`

[ngoldbaum]

Went ahead and rebased though. @Narsil any chance you can take a look over here?

[ngoldbaum]

I updated this to target 3.14t instead of 3.13t and added 3.14 testing while I was at it.

I also updated a few spots where CI configuration has gone stale. After all that, the tests pass on my fork, see ngoldbaum#3.

The way I set this up, the GIL is getting re-enabled in the 3.14t tests. I'll mark the extensions as supporting free-threading in a follow-up.

@ArthurZucker any chance you can trigger CI over here so the tests run?

[ngoldbaum]

Rebased on #1923.

[ngoldbaum]

I went ahead and opened #1925 to make this PR do fewer things at once.