Enable maturin abi3 Python wheels

[gulldan]

Summary

Move the SudachiPy extension build from setuptools-rust to maturin, build ABI-stable wheels for regular CPython 3.10+, keep version-specific free-threaded wheels for 3.13t/3.14t, and harden the wheel/sdist release workflow.

This PR now tests the full cibuildwheel matrix, including cp313t/cp314t with tokenizers installed.

What changed

• Add a repository-root pyproject.toml using maturin as the PEP 517 backend.
• Configure PyO3 with abi3-py310 after the PyO3 0.28.3 update from Update PyO3 to 0.28.3 #334.
• Build normal CPython wheels as cp310-abi3 and free-threaded wheels as cp313-cp313t / cp314-cp314t.
• Make test_pretokenizers.py require tokenizers again, so pretokenizer coverage cannot silently disappear.
• Add a complete expected wheel-tag gate in python/verify-dist-artifacts.py.
• Move production PyPI publishing to Trusted Publishing/OIDC.
• Move TestPyPI publishing to Trusted Publishing/OIDC with GitHub environment testpypi.
• Update Python build/test scripts and CI to use uv and cibuildwheel build[uv].
• Harden local python/build_and_test.sh against stale extension artifacts and Windows python.exe path handling.
• Add SHA256 verification for the fixed CI dictionary download used for PGO.
• Replace TestPyPI version regex rewriting with TOML parsing via tomlkit.
• Add version metadata consistency checks for Cargo, Python package metadata, __version__, and docs.
• Add release notes for Python 3.9 removal and the manylinux_2_28 Linux wheel baseline.

Why this workflow shape

• test-command is intentional. cibuildwheel documents that this command tests each built wheel, and that the wheel is installed automatically and available for import during tests. It also says that without test-command, the wheel is not installed after building. That is why the wheel build runs import, unittest, and sudachipy --help instead of only producing artifacts. Reference: cibuildwheel test-command.
• test-requires = ["sudachidict_core", "tokenizers"] is intentional. cibuildwheel documents test-requires as the dependencies installed before tests run. That keeps pretokenizer coverage active in wheel tests rather than depending on whatever happens to be present in the runner. Reference: cibuildwheel test-requires.
• The macOS free-threaded RUSTFLAGS override is deliberately narrow. cibuildwheel supports selector-based overrides and inherit.environment = "append", so the dynamic-lookup linker flags are applied only to cp3??t-macosx_* builds instead of changing Linux, Windows, or regular CPython wheels. Reference: cibuildwheel configuration overrides.
• TestPyPI is kept as a real publish rehearsal, not a weaker password path. The Python Packaging User Guide recommends Trusted Publishing for GitHub Actions and explicitly covers both PyPI and TestPyPI, with two trusted publishers and GitHub environments pypi and testpypi. Reference: Python Packaging User Guide: publishing with GitHub Actions.
• TestPyPI uses repository-url, not username/password secrets. PyPI’s Trusted Publisher docs show the TestPyPI form with repository-url: https://test.pypi.org/legacy/, and the PyPA publish action documents the same input. References: PyPI Trusted Publishers: using a publisher and pypa/gh-action-pypi-publish.
• id-token: write is only on publish jobs because OIDC needs it, but build/test jobs do not. GitHub documents that id-token: write is required to request the OIDC JWT and does not grant write access to other resources. The PyPA publish action also recommends setting it only in the publishing job and separating build from publish. References: GitHub OIDC reference and pypa/gh-action-pypi-publish Trusted Publishing notes.

Maintainer notes

• The root pyproject.toml is intentional: maturin sdist includes cannot reference parent paths, so building from the repo root avoids symlink/copy staging hacks for the Rust workspace and resources/.
• enable = ["cpython-freethreading"] remains because cibuildwheel 3.4.1 still needs it for free-threaded identifiers.
• Free-threaded Python does not use the stable ABI here, so 3.13t and 3.14t wheels are intentionally version-specific.
• tokenizers==0.23.1 has macOS cp310-abi3 wheels, but macOS free-threaded test envs resolve it from sdist. The macOS free-threaded override adds -undefined dynamic_lookup, which fixes that source build without skipping tests.
• Production PyPI publish expects a PyPI Trusted Publisher configured for GitHub environment pypi.
• TestPyPI publish expects a TestPyPI Trusted Publisher configured for GitHub environment testpypi; no PYPI_TEST_USERNAME / PYPI_TEST_PASSWORD secrets are used.
• Artifact verification now expects 16 wheels plus one sdist. Missing or unexpected wheel tags fail the release gate.

Validation

• GitHub Actions on head 2b5b0f83:
  • Python: Ubuntu, Windows, macOS passed.
  • Rust: Ubuntu, Windows, macOS passed.
  • Python Packages: Linux x86_64, Linux aarch64, Windows, macOS, sdist, and verify distribution artifacts passed.
  • upload-to-pypi skipped as expected because this was not a tag push.
  • Only the WIP check remains pending while the PR is draft/WIP.
• macOS cibuildwheel selected all 9 macOS identifiers: cp310, cp313t, and cp314t for x86_64, arm64, and universal2.
• macOS cibuildwheel installed sudachidict_core and tokenizers==0.23.1 before test-command.
• macOS cibuildwheel ran import, full unittest discover, and sudachipy --help for free-threaded wheels, including cp314t-macosx_universal2 on both slices.
• verify-dist-artifacts.py passed against CI artifacts and produced the full expected wheel matrix.
• Local macOS checks:
  • PYTHON=python3.13t bash python/build_and_test.sh passed, 79 tests OK, tokenizers built from source.
  • PYTHON=python3.14t bash python/build_and_test.sh passed, 79 tests OK.
  • bash python/build_and_test.sh passed on non-free-threaded Python 3.14, 79 tests OK with the free-threaded-only test skipped.
  • cargo test --workspace passed.
  • cargo fmt --all --check passed.
  • python3 python/check_version_consistency.py passed.
  • sdist build with python -m build --sdist passed.
  • python3 -m py_compile passed for changed Python files.
  • Workflow YAML parsed successfully.
  • Synthetic full-matrix verifier check passed: 16 wheels and 1 sdist.