-
Notifications
You must be signed in to change notification settings - Fork 468
fix(ray): increase ray instrumentation stability #15446
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
e57d7ad to
fcae20e
Compare
fcae20e to
1848ea3
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 247 ± 3 ms. The average import time from base is: 251 ± 4 ms. The import time difference between this PR and base is: -3.7 ± 0.2 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate dubloom/ddtrace-run-ray (9d673c1) with baseline main (53c5793) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 0.398µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -1.7% Memory: ✅ 40.155MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ add_inplace_aspectTime: ✅ 0.409µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +0.3% Memory: ✅ 40.276MB (SLO: <41.500MB -3.0%) vs baseline: +5.1% ✅ add_inplace_noaspectTime: ✅ 0.317µs (SLO: <10.000µs 📉 -96.8%) vs baseline: -0.1% Memory: ✅ 40.137MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ add_noaspectTime: ✅ 0.278µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -0.3% Memory: ✅ 40.376MB (SLO: <41.500MB -2.7%) vs baseline: +5.0% ✅ bytearray_aspectTime: ✅ 1.371µs (SLO: <10.000µs 📉 -86.3%) vs baseline: +1.3% Memory: ✅ 40.324MB (SLO: <41.500MB -2.8%) vs baseline: +5.0% ✅ bytearray_extend_aspectTime: ✅ 1.517µs (SLO: <10.000µs 📉 -84.8%) vs baseline: +0.4% Memory: ✅ 40.300MB (SLO: <41.500MB -2.9%) vs baseline: +4.9% ✅ bytearray_extend_noaspectTime: ✅ 0.614µs (SLO: <10.000µs 📉 -93.9%) vs baseline: ~same Memory: ✅ 40.152MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 0.487µs (SLO: <10.000µs 📉 -95.1%) vs baseline: +1.2% Memory: ✅ 40.126MB (SLO: <41.500MB -3.3%) vs baseline: +4.8% ✅ bytes_aspectTime: ✅ 1.300µs (SLO: <10.000µs 📉 -87.0%) vs baseline: +1.7% Memory: ✅ 40.156MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ bytes_noaspectTime: ✅ 0.495µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.2% Memory: ✅ 40.166MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% ✅ bytesio_aspectTime: ✅ 1.349µs (SLO: <10.000µs 📉 -86.5%) vs baseline: +1.7% Memory: ✅ 40.145MB (SLO: <41.500MB -3.3%) vs baseline: +4.4% ✅ bytesio_noaspectTime: ✅ 0.498µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +0.6% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +5.0% ✅ capitalize_aspectTime: ✅ 0.744µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +1.0% Memory: ✅ 40.376MB (SLO: <41.500MB -2.7%) vs baseline: +5.5% ✅ capitalize_noaspectTime: ✅ 0.432µs (SLO: <10.000µs 📉 -95.7%) vs baseline: -0.7% Memory: ✅ 40.046MB (SLO: <41.500MB -3.5%) vs baseline: +4.5% ✅ casefold_aspectTime: ✅ 0.740µs (SLO: <10.000µs 📉 -92.6%) vs baseline: -0.6% Memory: ✅ 40.027MB (SLO: <41.500MB -3.5%) vs baseline: +4.1% ✅ casefold_noaspectTime: ✅ 0.372µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +0.7% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ decode_aspectTime: ✅ 0.727µs (SLO: <10.000µs 📉 -92.7%) vs baseline: +0.7% Memory: ✅ 40.541MB (SLO: <41.500MB -2.3%) vs baseline: +5.8% ✅ decode_noaspectTime: ✅ 0.415µs (SLO: <10.000µs 📉 -95.8%) vs baseline: -1.1% Memory: ✅ 40.172MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ encode_aspectTime: ✅ 0.711µs (SLO: <10.000µs 📉 -92.9%) vs baseline: +0.6% Memory: ✅ 40.442MB (SLO: <41.500MB -2.5%) vs baseline: +5.7% ✅ encode_noaspectTime: ✅ 0.402µs (SLO: <10.000µs 📉 -96.0%) vs baseline: +0.1% Memory: ✅ 40.087MB (SLO: <41.500MB -3.4%) vs baseline: +4.4% ✅ format_aspectTime: ✅ 3.430µs (SLO: <10.000µs 📉 -65.7%) vs baseline: -1.8% Memory: ✅ 40.225MB (SLO: <41.500MB -3.1%) vs baseline: +5.0% ✅ format_map_aspectTime: ✅ 3.517µs (SLO: <10.000µs 📉 -64.8%) vs baseline: -2.4% Memory: ✅ 40.184MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% ✅ format_map_noaspectTime: ✅ 0.781µs (SLO: <10.000µs 📉 -92.2%) vs baseline: +0.7% Memory: ✅ 40.254MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ format_noaspectTime: ✅ 0.601µs (SLO: <10.000µs 📉 -94.0%) vs baseline: +0.5% Memory: ✅ 40.108MB (SLO: <41.500MB -3.4%) vs baseline: +4.6% ✅ index_aspectTime: ✅ 0.357µs (SLO: <10.000µs 📉 -96.4%) vs baseline: +0.5% Memory: ✅ 40.234MB (SLO: <41.500MB -3.0%) vs baseline: +4.8% ✅ index_noaspectTime: ✅ 0.277µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -1.1% Memory: ✅ 40.127MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ join_aspectTime: ✅ 1.339µs (SLO: <10.000µs 📉 -86.6%) vs baseline: +0.9% Memory: ✅ 40.122MB (SLO: <41.500MB -3.3%) vs baseline: +4.8% ✅ join_noaspectTime: ✅ 0.490µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.3% Memory: ✅ 40.142MB (SLO: <41.500MB -3.3%) vs baseline: +4.3% ✅ ljust_aspectTime: ✅ 2.612µs (SLO: <20.000µs 📉 -86.9%) vs baseline: +2.0% Memory: ✅ 40.164MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ ljust_noaspectTime: ✅ 0.406µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +0.1% Memory: ✅ 40.089MB (SLO: <41.500MB -3.4%) vs baseline: +4.4% ✅ lower_aspectTime: ✅ 2.299µs (SLO: <10.000µs 📉 -77.0%) vs baseline: +2.9% Memory: ✅ 40.263MB (SLO: <41.500MB -3.0%) vs baseline: +5.1% ✅ lower_noaspectTime: ✅ 0.370µs (SLO: <10.000µs 📉 -96.3%) vs baseline: ~same Memory: ✅ 40.231MB (SLO: <41.500MB -3.1%) vs baseline: +5.1% ✅ lstrip_aspectTime: ✅ 2.269µs (SLO: <20.000µs 📉 -88.7%) vs baseline: +0.2% Memory: ✅ 40.184MB (SLO: <41.500MB -3.2%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 0.379µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.1% Memory: ✅ 40.293MB (SLO: <41.500MB -2.9%) vs baseline: +4.9% ✅ modulo_aspectTime: ✅ 1.050µs (SLO: <10.000µs 📉 -89.5%) vs baseline: +4.5% Memory: ✅ 40.462MB (SLO: <41.500MB -2.5%) vs baseline: +5.5% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 1.541µs (SLO: <10.000µs 📉 -84.6%) vs baseline: -1.4% Memory: ✅ 40.369MB (SLO: <41.500MB -2.7%) vs baseline: +5.2% ✅ modulo_aspect_for_bytesTime: ✅ 0.969µs (SLO: <10.000µs 📉 -90.3%) vs baseline: -0.4% Memory: ✅ 40.131MB (SLO: <41.500MB -3.3%) vs baseline: +4.5% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 1.244µs (SLO: <10.000µs 📉 -87.6%) vs baseline: +1.5% Memory: ✅ 40.025MB (SLO: <41.500MB -3.6%) vs baseline: +4.2% ✅ modulo_noaspectTime: ✅ 0.628µs (SLO: <10.000µs 📉 -93.7%) vs baseline: ~same Memory: ✅ 40.313MB (SLO: <41.500MB -2.9%) vs baseline: +5.2% ✅ replace_aspectTime: ✅ 4.843µs (SLO: <10.000µs 📉 -51.6%) vs baseline: -1.5% Memory: ✅ 40.128MB (SLO: <41.500MB -3.3%) vs baseline: +4.8% ✅ replace_noaspectTime: ✅ 0.466µs (SLO: <10.000µs 📉 -95.3%) vs baseline: +0.5% Memory: ✅ 40.191MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% ✅ repr_aspectTime: ✅ 0.914µs (SLO: <10.000µs 📉 -90.9%) vs baseline: +0.8% Memory: ✅ 40.118MB (SLO: <41.500MB -3.3%) vs baseline: +4.7% ✅ repr_noaspectTime: ✅ 0.415µs (SLO: <10.000µs 📉 -95.9%) vs baseline: ~same Memory: ✅ 40.184MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% ✅ rstrip_aspectTime: ✅ 2.074µs (SLO: <20.000µs 📉 -89.6%) vs baseline: +7.0% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.9% ✅ rstrip_noaspectTime: ✅ 0.378µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.5% Memory: ✅ 40.170MB (SLO: <41.500MB -3.2%) vs baseline: +4.6% ✅ slice_aspectTime: ✅ 0.497µs (SLO: <10.000µs 📉 -95.0%) vs baseline: ~same Memory: ✅ 40.305MB (SLO: <41.500MB -2.9%) vs baseline: +5.2% ✅ slice_noaspectTime: ✅ 0.446µs (SLO: <10.000µs 📉 -95.5%) vs baseline: -0.8% Memory: ✅ 40.193MB (SLO: <41.500MB -3.1%) vs baseline: +4.2% ✅ stringio_aspectTime: ✅ 1.554µs (SLO: <10.000µs 📉 -84.5%) vs baseline: +1.7% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ stringio_noaspectTime: ✅ 0.718µs (SLO: <10.000µs 📉 -92.8%) vs baseline: +0.2% Memory: ✅ 40.145MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ strip_aspectTime: ✅ 2.260µs (SLO: <20.000µs 📉 -88.7%) vs baseline: +1.5% Memory: ✅ 40.245MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 0.386µs (SLO: <10.000µs 📉 -96.1%) vs baseline: +0.4% Memory: ✅ 40.293MB (SLO: <41.500MB -2.9%) vs baseline: +5.2% ✅ swapcase_aspectTime: ✅ 2.485µs (SLO: <10.000µs 📉 -75.2%) vs baseline: +0.7% Memory: ✅ 40.364MB (SLO: <41.500MB -2.7%) vs baseline: +5.2% ✅ swapcase_noaspectTime: ✅ 0.536µs (SLO: <10.000µs 📉 -94.6%) vs baseline: -1.5% Memory: ✅ 40.189MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ title_aspectTime: ✅ 2.431µs (SLO: <10.000µs 📉 -75.7%) vs baseline: +2.2% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +5.0% ✅ title_noaspectTime: ✅ 0.502µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -0.8% Memory: ✅ 40.290MB (SLO: <41.500MB -2.9%) vs baseline: +5.2% ✅ translate_aspectTime: ✅ 3.326µs (SLO: <10.000µs 📉 -66.7%) vs baseline: -0.4% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ translate_noaspectTime: ✅ 1.042µs (SLO: <10.000µs 📉 -89.6%) vs baseline: +0.7% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% ✅ upper_aspectTime: ✅ 2.604µs (SLO: <10.000µs 📉 -74.0%) vs baseline: 📈 +17.1% Memory: ✅ 40.364MB (SLO: <41.500MB -2.7%) vs baseline: +5.4% ✅ upper_noaspectTime: ✅ 0.372µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +0.7% Memory: ✅ 40.165MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.139µs (SLO: <10.000µs 📉 -48.6%) vs baseline: 📈 +24.9% Memory: ✅ 40.147MB (SLO: <41.000MB -2.1%) vs baseline: +4.7% ✅ ospathbasename_noaspectTime: ✅ 1.082µs (SLO: <10.000µs 📉 -89.2%) vs baseline: -0.7% Memory: ✅ 40.128MB (SLO: <41.000MB -2.1%) vs baseline: +4.8% ✅ ospathjoin_aspectTime: ✅ 6.187µs (SLO: <10.000µs 📉 -38.1%) vs baseline: -0.3% Memory: ✅ 40.246MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +5.1% ✅ ospathjoin_noaspectTime: ✅ 2.267µs (SLO: <10.000µs 📉 -77.3%) vs baseline: -1.3% Memory: ✅ 40.206MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +5.2% ✅ ospathnormcase_aspectTime: ✅ 3.371µs (SLO: <10.000µs 📉 -66.3%) vs baseline: -2.0% Memory: ✅ 40.128MB (SLO: <41.000MB -2.1%) vs baseline: +4.5% ✅ ospathnormcase_noaspectTime: ✅ 0.571µs (SLO: <10.000µs 📉 -94.3%) vs baseline: -0.3% Memory: ✅ 40.305MB (SLO: <41.000MB 🟡 -1.7%) vs baseline: +5.0% ✅ ospathsplit_aspectTime: ✅ 4.727µs (SLO: <10.000µs 📉 -52.7%) vs baseline: -0.3% Memory: ✅ 40.206MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +4.4% ✅ ospathsplit_noaspectTime: ✅ 1.586µs (SLO: <10.000µs 📉 -84.1%) vs baseline: -0.6% Memory: ✅ 40.069MB (SLO: <41.000MB -2.3%) vs baseline: +4.4% ✅ ospathsplitdrive_aspectTime: ✅ 3.574µs (SLO: <10.000µs 📉 -64.3%) vs baseline: -1.7% Memory: ✅ 40.167MB (SLO: <41.000MB -2.0%) vs baseline: +4.6% ✅ ospathsplitdrive_noaspectTime: ✅ 0.697µs (SLO: <10.000µs 📉 -93.0%) vs baseline: ~same Memory: ✅ 40.049MB (SLO: <41.000MB -2.3%) vs baseline: +4.3% ✅ ospathsplitext_aspectTime: ✅ 4.569µs (SLO: <10.000µs 📉 -54.3%) vs baseline: +2.0% Memory: ✅ 40.147MB (SLO: <41.000MB -2.1%) vs baseline: +4.8% ✅ ospathsplitext_noaspectTime: ✅ 1.377µs (SLO: <10.000µs 📉 -86.2%) vs baseline: -0.3% Memory: ✅ 40.265MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +5.2% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.376µs (SLO: <20.000µs 📉 -83.1%) vs baseline: 📈 +13.5% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 1-count-metrics-100-timesTime: ✅ 200.808µs (SLO: <220.000µs -8.7%) vs baseline: -0.3% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.309µs (SLO: <20.000µs 📉 -83.5%) vs baseline: -0.7% Memory: ✅ 34.721MB (SLO: <35.500MB -2.2%) vs baseline: +4.8% ✅ 1-distribution-metrics-100-timesTime: ✅ 216.418µs (SLO: <230.000µs -5.9%) vs baseline: -0.5% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ 1-gauge-metric-1-timesTime: ✅ 2.189µs (SLO: <20.000µs 📉 -89.1%) vs baseline: -0.8% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ 1-gauge-metrics-100-timesTime: ✅ 137.600µs (SLO: <150.000µs -8.3%) vs baseline: +0.9% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.3% ✅ 1-rate-metric-1-timesTime: ✅ 3.095µs (SLO: <20.000µs 📉 -84.5%) vs baseline: -0.2% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.5% ✅ 1-rate-metrics-100-timesTime: ✅ 214.532µs (SLO: <250.000µs 📉 -14.2%) vs baseline: +0.3% Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ 100-count-metrics-100-timesTime: ✅ 20.380ms (SLO: <22.000ms -7.4%) vs baseline: ~same Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.275ms (SLO: <2.300ms 🟡 -1.1%) vs baseline: -0.8% Memory: ✅ 34.682MB (SLO: <35.500MB -2.3%) vs baseline: +4.7% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.413ms (SLO: <1.550ms -8.8%) vs baseline: ~same Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ 100-rate-metrics-100-timesTime: ✅ 2.219ms (SLO: <2.550ms 📉 -13.0%) vs baseline: ~same Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +5.2% ✅ flush-1-metricTime: ✅ 4.565µs (SLO: <20.000µs 📉 -77.2%) vs baseline: -0.1% Memory: ✅ 35.134MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ flush-100-metricsTime: ✅ 174.785µs (SLO: <250.000µs 📉 -30.1%) vs baseline: ~same Memory: ✅ 35.095MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.178ms (SLO: <2.500ms 📉 -12.9%) vs baseline: -0.7% Memory: ✅ 35.979MB (SLO: <36.500MB 🟡 -1.4%) vs baseline: +5.0% 🟡 Near SLO Breach (17 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
dd76289 to
2aac010
Compare
Ray integration was until very soon only tested internally. We had several issues with instrumenting jobs based on when ray was initialized and some crashes related to context injection.
This PR is created to fix all the small issues and make the integration stable. Ray integration should be advertised to be used only on versions including this fix. The releasenote is covering the only customer facing issue.
Context
During internal dogfooding some bugs were found/some assumptions were challenged:
ray.init()at the top of the file (which was a prerequisite for a fully working instrumentation).After analyzing all the consequences of calling ray.init() late or waiting for auto-initialization (for example when a task is submitted for the first time), it appears that the only viable solution was to change the way we instrument the cluster.
Changes
dd_trace_ctxin tasks.Testing
I added some tests to ensure all the behavior are tested:
Additional Notes
Ray test suite were run while ddtrace activated increasing the confidence in this PR.