-
Notifications
You must be signed in to change notification settings - Fork 469
feat(llmobs): add tag if envar detected #15538
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 246 ± 2 ms. The average import time from base is: 248 ± 2 ms. The import time difference between this PR and base is: -2.4 ± 0.08 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate zachg/llmobs_otel_evals_update (e69ea2a) with baseline main (ca3c521) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 0.401µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -1.2% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +5.0% ✅ add_inplace_aspectTime: ✅ 0.405µs (SLO: <10.000µs 📉 -95.9%) vs baseline: ~same Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +3.9% ✅ add_inplace_noaspectTime: ✅ 0.317µs (SLO: <10.000µs 📉 -96.8%) vs baseline: -0.1% Memory: ✅ 40.215MB (SLO: <41.500MB -3.1%) vs baseline: +4.4% ✅ add_noaspectTime: ✅ 0.277µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -1.8% Memory: ✅ 40.128MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ bytearray_aspectTime: ✅ 1.350µs (SLO: <10.000µs 📉 -86.5%) vs baseline: -0.5% Memory: ✅ 40.423MB (SLO: <41.500MB -2.6%) vs baseline: +4.2% ✅ bytearray_extend_aspectTime: ✅ 1.488µs (SLO: <10.000µs 📉 -85.1%) vs baseline: -1.1% Memory: ✅ 40.659MB (SLO: <41.500MB -2.0%) vs baseline: +6.2% ✅ bytearray_extend_noaspectTime: ✅ 0.614µs (SLO: <10.000µs 📉 -93.9%) vs baseline: -0.6% Memory: ✅ 40.324MB (SLO: <41.500MB -2.8%) vs baseline: +4.1% ✅ bytearray_noaspectTime: ✅ 0.480µs (SLO: <10.000µs 📉 -95.2%) vs baseline: -0.8% Memory: ✅ 40.141MB (SLO: <41.500MB -3.3%) vs baseline: +4.2% ✅ bytes_aspectTime: ✅ 1.298µs (SLO: <10.000µs 📉 -87.0%) vs baseline: +0.3% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +4.6% ✅ bytes_noaspectTime: ✅ 0.493µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.2% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +4.5% ✅ bytesio_aspectTime: ✅ 1.348µs (SLO: <10.000µs 📉 -86.5%) vs baseline: +1.6% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +4.0% ✅ bytesio_noaspectTime: ✅ 0.497µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -0.1% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +5.0% ✅ capitalize_aspectTime: ✅ 0.735µs (SLO: <10.000µs 📉 -92.7%) vs baseline: -0.4% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.4% ✅ capitalize_noaspectTime: ✅ 0.435µs (SLO: <10.000µs 📉 -95.7%) vs baseline: -0.5% Memory: ✅ 40.242MB (SLO: <41.500MB -3.0%) vs baseline: +5.0% ✅ casefold_aspectTime: ✅ 0.733µs (SLO: <10.000µs 📉 -92.7%) vs baseline: -1.0% Memory: ✅ 40.285MB (SLO: <41.500MB -2.9%) vs baseline: +4.1% ✅ casefold_noaspectTime: ✅ 0.366µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -1.0% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +4.7% ✅ decode_aspectTime: ✅ 0.722µs (SLO: <10.000µs 📉 -92.8%) vs baseline: -0.7% Memory: ✅ 40.108MB (SLO: <41.500MB -3.4%) vs baseline: +4.4% ✅ decode_noaspectTime: ✅ 0.419µs (SLO: <10.000µs 📉 -95.8%) vs baseline: -1.4% Memory: ✅ 40.305MB (SLO: <41.500MB -2.9%) vs baseline: +5.2% ✅ encode_aspectTime: ✅ 0.708µs (SLO: <10.000µs 📉 -92.9%) vs baseline: +0.6% Memory: ✅ 40.305MB (SLO: <41.500MB -2.9%) vs baseline: +4.8% ✅ encode_noaspectTime: ✅ 0.407µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +1.1% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.7% ✅ format_aspectTime: ✅ 3.425µs (SLO: <10.000µs 📉 -65.8%) vs baseline: -0.5% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.5% ✅ format_map_aspectTime: ✅ 3.555µs (SLO: <10.000µs 📉 -64.4%) vs baseline: -1.3% Memory: ✅ 40.128MB (SLO: <41.500MB -3.3%) vs baseline: +4.4% ✅ format_map_noaspectTime: ✅ 0.773µs (SLO: <10.000µs 📉 -92.3%) vs baseline: -1.1% Memory: ✅ 40.438MB (SLO: <41.500MB -2.6%) vs baseline: +5.1% ✅ format_noaspectTime: ✅ 0.595µs (SLO: <10.000µs 📉 -94.0%) vs baseline: -0.2% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ index_aspectTime: ✅ 0.356µs (SLO: <10.000µs 📉 -96.4%) vs baseline: +0.2% Memory: ✅ 40.088MB (SLO: <41.500MB -3.4%) vs baseline: +4.1% ✅ index_noaspectTime: ✅ 0.278µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -0.2% Memory: ✅ 40.423MB (SLO: <41.500MB -2.6%) vs baseline: +5.4% ✅ join_aspectTime: ✅ 1.339µs (SLO: <10.000µs 📉 -86.6%) vs baseline: +1.3% Memory: ✅ 40.049MB (SLO: <41.500MB -3.5%) vs baseline: +4.2% ✅ join_noaspectTime: ✅ 0.490µs (SLO: <10.000µs 📉 -95.1%) vs baseline: +0.3% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.4% ✅ ljust_aspectTime: ✅ 2.613µs (SLO: <20.000µs 📉 -86.9%) vs baseline: +2.4% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +4.8% ✅ ljust_noaspectTime: ✅ 0.402µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -0.3% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ lower_aspectTime: ✅ 2.300µs (SLO: <10.000µs 📉 -77.0%) vs baseline: +3.3% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.7% ✅ lower_noaspectTime: ✅ 0.368µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +0.3% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ lstrip_aspectTime: ✅ 2.246µs (SLO: <20.000µs 📉 -88.8%) vs baseline: ~same Memory: ✅ 40.108MB (SLO: <41.500MB -3.4%) vs baseline: +4.6% ✅ lstrip_noaspectTime: ✅ 0.379µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.9% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +5.0% ✅ modulo_aspectTime: ✅ 1.049µs (SLO: <10.000µs 📉 -89.5%) vs baseline: +4.7% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.7% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 1.531µs (SLO: <10.000µs 📉 -84.7%) vs baseline: -2.1% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +5.1% ✅ modulo_aspect_for_bytesTime: ✅ 0.975µs (SLO: <10.000µs 📉 -90.3%) vs baseline: +0.5% Memory: ✅ 40.482MB (SLO: <41.500MB -2.5%) vs baseline: +5.7% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 1.239µs (SLO: <10.000µs 📉 -87.6%) vs baseline: +1.3% Memory: ✅ 40.108MB (SLO: <41.500MB -3.4%) vs baseline: +4.4% ✅ modulo_noaspectTime: ✅ 0.627µs (SLO: <10.000µs 📉 -93.7%) vs baseline: -1.7% Memory: ✅ 40.383MB (SLO: <41.500MB -2.7%) vs baseline: +5.2% ✅ replace_aspectTime: ✅ 4.856µs (SLO: <10.000µs 📉 -51.4%) vs baseline: -1.4% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +5.0% ✅ replace_noaspectTime: ✅ 0.462µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.5% Memory: ✅ 40.069MB (SLO: <41.500MB -3.4%) vs baseline: +4.1% ✅ repr_aspectTime: ✅ 0.908µs (SLO: <10.000µs 📉 -90.9%) vs baseline: +0.2% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +4.7% ✅ repr_noaspectTime: ✅ 0.420µs (SLO: <10.000µs 📉 -95.8%) vs baseline: -0.2% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.4% ✅ rstrip_aspectTime: ✅ 1.957µs (SLO: <20.000µs 📉 -90.2%) vs baseline: +2.5% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.6% ✅ rstrip_noaspectTime: ✅ 0.383µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.4% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +4.8% ✅ slice_aspectTime: ✅ 0.489µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -1.7% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.7% ✅ slice_noaspectTime: ✅ 0.447µs (SLO: <10.000µs 📉 -95.5%) vs baseline: -0.4% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ stringio_aspectTime: ✅ 1.551µs (SLO: <10.000µs 📉 -84.5%) vs baseline: ~same Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +3.7% ✅ stringio_noaspectTime: ✅ 0.712µs (SLO: <10.000µs 📉 -92.9%) vs baseline: -0.7% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +4.7% ✅ strip_aspectTime: ✅ 2.230µs (SLO: <20.000µs 📉 -88.9%) vs baseline: -0.1% Memory: ✅ 40.128MB (SLO: <41.500MB -3.3%) vs baseline: +4.5% ✅ strip_noaspectTime: ✅ 0.381µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -1.3% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +4.8% ✅ swapcase_aspectTime: ✅ 2.763µs (SLO: <10.000µs 📉 -72.4%) vs baseline: 📈 +12.6% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.8% ✅ swapcase_noaspectTime: ✅ 0.541µs (SLO: <10.000µs 📉 -94.6%) vs baseline: +0.4% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.9% ✅ title_aspectTime: ✅ 2.431µs (SLO: <10.000µs 📉 -75.7%) vs baseline: +2.1% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +4.6% ✅ title_noaspectTime: ✅ 0.504µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -0.4% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.1% ✅ translate_aspectTime: ✅ 3.301µs (SLO: <10.000µs 📉 -67.0%) vs baseline: -0.2% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ translate_noaspectTime: ✅ 1.039µs (SLO: <10.000µs 📉 -89.6%) vs baseline: -0.4% Memory: ✅ 40.340MB (SLO: <41.500MB -2.8%) vs baseline: +5.5% ✅ upper_aspectTime: ✅ 2.331µs (SLO: <10.000µs 📉 -76.7%) vs baseline: +3.5% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.4% ✅ upper_noaspectTime: ✅ 0.372µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +1.0% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.191µs (SLO: <10.000µs 📉 -48.1%) vs baseline: 📈 +26.4% Memory: ✅ 40.187MB (SLO: <41.000MB 🟡 -2.0%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 1.082µs (SLO: <10.000µs 📉 -89.2%) vs baseline: -0.4% Memory: ✅ 40.364MB (SLO: <41.000MB 🟡 -1.6%) vs baseline: +4.6% ✅ ospathjoin_aspectTime: ✅ 6.167µs (SLO: <10.000µs 📉 -38.3%) vs baseline: +0.3% Memory: ✅ 40.187MB (SLO: <41.000MB 🟡 -2.0%) vs baseline: +4.6% ✅ ospathjoin_noaspectTime: ✅ 2.285µs (SLO: <10.000µs 📉 -77.1%) vs baseline: -0.7% Memory: ✅ 40.265MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +5.0% ✅ ospathnormcase_aspectTime: ✅ 3.423µs (SLO: <10.000µs 📉 -65.8%) vs baseline: +0.5% Memory: ✅ 40.285MB (SLO: <41.000MB 🟡 -1.7%) vs baseline: +5.2% ✅ ospathnormcase_noaspectTime: ✅ 0.569µs (SLO: <10.000µs 📉 -94.3%) vs baseline: -2.1% Memory: ✅ 40.128MB (SLO: <41.000MB -2.1%) vs baseline: +4.3% ✅ ospathsplit_aspectTime: ✅ 4.700µs (SLO: <10.000µs 📉 -53.0%) vs baseline: -0.8% Memory: ✅ 40.108MB (SLO: <41.000MB -2.2%) vs baseline: +4.4% ✅ ospathsplit_noaspectTime: ✅ 1.593µs (SLO: <10.000µs 📉 -84.1%) vs baseline: ~same Memory: ✅ 40.246MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 3.633µs (SLO: <10.000µs 📉 -63.7%) vs baseline: -0.5% Memory: ✅ 40.147MB (SLO: <41.000MB -2.1%) vs baseline: +3.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.699µs (SLO: <10.000µs 📉 -93.0%) vs baseline: -0.5% Memory: ✅ 40.206MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +4.6% ✅ ospathsplitext_aspectTime: ✅ 4.496µs (SLO: <10.000µs 📉 -55.0%) vs baseline: +0.6% Memory: ✅ 40.462MB (SLO: <41.000MB 🟡 -1.3%) vs baseline: +5.2% ✅ ospathsplitext_noaspectTime: ✅ 1.384µs (SLO: <10.000µs 📉 -86.2%) vs baseline: +0.7% Memory: ✅ 40.167MB (SLO: <41.000MB -2.0%) vs baseline: +4.1% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.394µs (SLO: <20.000µs 📉 -83.0%) vs baseline: 📈 +15.6% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ 1-count-metrics-100-timesTime: ✅ 200.310µs (SLO: <220.000µs -9.0%) vs baseline: -0.2% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ 1-distribution-metric-1-timesTime: ✅ 3.294µs (SLO: <20.000µs 📉 -83.5%) vs baseline: -0.9% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 217.389µs (SLO: <230.000µs -5.5%) vs baseline: -0.9% Memory: ✅ 34.721MB (SLO: <35.500MB -2.2%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.178µs (SLO: <20.000µs 📉 -89.1%) vs baseline: +0.4% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.9% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.842µs (SLO: <150.000µs -8.8%) vs baseline: -0.3% Memory: ✅ 34.721MB (SLO: <35.500MB -2.2%) vs baseline: +4.7% ✅ 1-rate-metric-1-timesTime: ✅ 3.108µs (SLO: <20.000µs 📉 -84.5%) vs baseline: +1.6% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.6% ✅ 1-rate-metrics-100-timesTime: ✅ 214.932µs (SLO: <250.000µs 📉 -14.0%) vs baseline: -0.4% Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +5.0% ✅ 100-count-metrics-100-timesTime: ✅ 20.327ms (SLO: <22.000ms -7.6%) vs baseline: -0.8% Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +5.0% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.318ms (SLO: <2.550ms -9.1%) vs baseline: +1.3% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.7% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.406ms (SLO: <1.550ms -9.3%) vs baseline: -0.4% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.7% ✅ 100-rate-metrics-100-timesTime: ✅ 2.226ms (SLO: <2.550ms 📉 -12.7%) vs baseline: +0.4% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ flush-1-metricTime: ✅ 4.578µs (SLO: <20.000µs 📉 -77.1%) vs baseline: -0.5% Memory: ✅ 35.114MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.8% ✅ flush-100-metricsTime: ✅ 174.389µs (SLO: <250.000µs 📉 -30.2%) vs baseline: +0.2% Memory: ✅ 35.114MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.184ms (SLO: <2.500ms 📉 -12.6%) vs baseline: -0.4% Memory: ✅ 35.881MB (SLO: <36.500MB 🟡 -1.7%) vs baseline: +4.4% 🟡 Near SLO Breach (17 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
Description
Testing
Risks
Additional Notes