Skip to content

[ORT CPU][WebNN] Fail to run some WebNN batchNormalization tests due to tolerance issues #26522

@BruceDai

Description

@BruceDai

Describe the issue

Fail to run below WebNN batchNormalization tests due to tolerance issues by ORT default CPU EP.

Test Error message
batchNormalization float32 2D tensor (mean and variance are non-constant) default options assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 6n but got 12n
batchNormalization float32 2D constant tensors default options assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 6n but got 12n
batchNormalization float32 2D tensor default options assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 6n but got 12n
[optional] batchNormalization default + relu assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization relu float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 6n but got 12n
[optional] batchNormalization default + leakyRelu assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization leakyRelu float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 7n but got 12n
[optional] batchNormalization default + clamp assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization clamp float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 6n but got 12n
[optional] batchNormalization default + prelu assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization prelu float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 7n but got 12n
[required] batchNormalization options.axis=0 + gelu assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization gelu float32 actual -0.035951320081949234 should be close enough to expected -0.03595117852091789 by ULP distance: expected a number less than or equal to 24n but got 38n
[optional] batchNormalization default + hardSwish assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization hardSwish float32 actual 1.4983532428741455 should be close enough to expected 1.4983549118041992 by ULP distance: expected a number less than or equal to 10n but got 14n
[optional] batchNormalization default + linear assert_less_than_equal: assert_array_approx_equals_ulp: test batchNormalization linear float32 actual 1.8526287078857422 should be close enough to expected 1.8526301383972168 by ULP distance: expected a number less than or equal to 8n but got 12n

/cc @fdwr PTAL, thanks!

To reproduce

  1. Install Windows App SDK Stable 1.8.2 (1.8.251003001)
  2. Install lates Edge Canary browser
  3. Launch Edge Canary browser, navigate to about://flags enabled "Enables WebNN API" and "ONNX Runtime backend for WebNN" flags, then relaunch browser
  4. Run each failure test using below test url
Test Test URL
batchNormalization float32 2D tensor (mean and variance are non-constant) default options https://wpt.live/webnn/conformance_tests/batch_normalization.https.any.html?device=cpu&tc=batchNormalization%20float32%202D%20tensor%20(mean%20and%20variance%20are%20non-constant)%20default%20options
batchNormalization float32 2D constant tensors default options https://wpt.live/webnn/conformance_tests/batch_normalization.https.any.html?device=cpu&tc=batchNormalization%20float32%202D%20constant%20tensors%20default%20options
batchNormalization float32 2D tensor default options https://wpt.live/webnn/conformance_tests/batch_normalization.https.any.html?device=cpu&tc=batchNormalization%20float32%202D%20tensor%20default%20options
batchNormalization default + relu https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20default%20%2B%20relu
batchNormalization default + leakyRelu https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20default%20%2B%20leakyRelu
batchNormalization default + clamp https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20default%20%2B%20clamp
batchNormalization default + prelu https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20default%20%2B%20prelu
batchNormalization options.axis=0 + gelu https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20options.axis=0%20%2B%20gelu
batchNormalization default + hardSwish https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20default%20%2B%20hardSwish
batchNormalization default + linear https://wpt.live/webnn/conformance_tests/subgraph.https.any.html?device=cpu&tc=batchNormalization%20default%20%2B%20linear

Urgency

No response

ONNX Runtime Installation

Released Package Windows App SDK Stable 1.8.2 (1.8.251003001)

ONNX Runtime Version or Commit ID

1.23.25.928

Execution Provider

ORT default CPU EP

Metadata

Metadata

Assignees

No one assigned

    Labels

    WebNNWebNN on top of onnxruntimeep:WebNNWebNN execution provider

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions