Speedup `test_car_interfaces`

[adeebshihadeh]

Takes ~1-2s per car. That's a big hit since we support nearly 300 different cars.

Goal is <=0.2s avg and <1s max on the CI machine, with the same test coverage. There's likely two or three things slowing this test down that'll be obvious with a bit of profiling.

Command to run:

pytest --durations=0 selfdrive/car/tests/test_car_interfaces.py 

Current output on my workstation

batman:sixpilot$ pytest --durations=10 selfdrive/car/tests/test_car_interfaces.py
=============================================================== test session starts ===============================================================
platform linux -- Python 3.11.4, pytest-8.2.0, pluggy-1.5.0
Using --randomly-seed=431984017
rootdir: /dev/shm/openpilot/sixpilot
configfile: pyproject.toml
plugins: randomly-3.15.0, subtests-0.12.1, anyio-4.3.0, cov-5.0.0, xdist-3.6.1, cpp-2.5.0, hypothesis-6.47.5, mock-3.14.0, timeout-2.3.1, flaky-3.8.1, asyncio-0.23.6
asyncio: mode=Mode.STRICT
32 workers [209 items]
........................................................................................................................................... [ 65%]
......................................................................                                                                      [100%]
============================================================== slowest 10 durations ===============================================================
2.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_027_FORD_FOCUS_MK4
2.17s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_138_MAZDA_CX9_2021
2.15s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_037_GENESIS_GV60_EV_1ST_GEN
2.05s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_147_SKODA_FABIA_MK4
1.97s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_094_HYUNDAI_VELOSTER
1.97s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_005_AUDI_Q3_MK2
1.87s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_059_HONDA_PILOT
1.83s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_113_KIA_OPTIMA_H_G4_FL
1.83s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_076_HYUNDAI_IONIQ_PHEV_2019
1.78s call     selfdrive/car/tests/test_car_interfaces.py::TestCarInterfaces::test_car_interfaces_095_JEEP_GRAND_CHEROKEE
============================================================== 209 passed in 13.32s ===============================================================

[sshane]

A good place to optimize is the hypothesis example generation. Speeds are ~2x faster than our current version in version 4.51.0 and get slower somewhere between then and 4.55.3. The latest version (6.102.6) is even slower compared to our current version (6.47). You can root cause and open a PR to hypothesis or openpilot depending on where the optimization lies.

[deanlee]

Another option is to optimize get_fuzzy_car_interface_args by caching params_strategy to avoid regenerating strategies each time. Additionally, drawing st.dictionaries is quite slow; according to my tests, it can take up to 0.3 seconds.

[lsrugo]

Another option is to optimize get_fuzzy_car_interface_args by caching params_strategy to avoid regenerating strategies each time.

I tried this and it didn't seem to work.

Additionally, drawing st.dictionaries is quite slow; according to my tests, it can take up to 0.3 seconds.

Right so it's likely a problem in the underlying library

[geeth345]

Is this bounty still available? I'm looking to work on it but am confused if it has already been claimed by one of the PRs above

[Pious1020]

@adeebshihadeh, is this still open?

[c-gamble]

I'm interested in working on this but it's looking like this bounty has been (partially) solved? Is the bounty still open to go from~0.7s to <= 0.2s?