Allow users to specify trusted Avro serializable classes to Dataflow worker #36809

clairemcginty · 2025-11-13T15:09:14Z

See: https://github.com/apache/avro/pull/3376/files

In Avro 1.11.4+, schemas that utilize the java-class annotation throw runtime decoding errors.

For example, applying a specific AvroCoder to a record with the following schema:

{
    "type": "record",
    "name": "TestJavaClassProp",
    "namespace": "com.test",
    "fields": [
        {
          "name": "bd",
          "type": {
            "type": "string",
            "java-class": "java.math.BigDecimal"
          }
        }
    ]
}

throws:

Caused by: java.lang.SecurityException: Forbidden java.math.BigDecimal! This class is not trusted to be included in Avro schemas using java-class. Please set the system property org.apache.avro.SERIALIZABLE_CLASSES to the comma separated list of classes you trust. You can also set the system property org.apache.avro.SERIALIZABLE_PACKAGES to the comma separated list of the packages you trust.
	at org.apache.avro.specific.SpecificDatumReader.checkSecurity(SpecificDatumReader.java:188)
	at org.apache.avro.specific.SpecificDatumReader.getPropAsClass(SpecificDatumReader.java:159)
	at org.apache.avro.specific.SpecificDatumReader.findStringClass(SpecificDatumReader.java:143)
	at com.spotify.scio.avro.SpecificRecordDatumFactory$ScioSpecificDatumReader.findStringClass(AvroDatumFactory.scala:80)
	at org.apache.avro.generic.GenericDatumReader$ReaderCache.lambda$getStringClass$1(GenericDatumReader.java:567)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.apache.avro.generic.GenericDatumReader$ReaderCache.getStringClass(GenericDatumReader.java:567)
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:455)
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:192)
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:232)
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:248)
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:219)
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:154)
	at org.apache.beam.sdk.extensions.avro.coders.AvroCoder.decode(AvroCoder.java:476)

The fix, per https://github.com/apache/avro/pull/3376/files, is to set the sys prop org.apache.avro.SERIALIZABLE_CLASSES. We can do that easily with DirectRunner, but there's no way to set it on Dataflow unless we create a custom worker image

This PR adds a proposed worker harness option allowing the user to specify trusted classes.

An alternative would be to simply set a reasonable default; for example in the Avro project itself they set the following default value:

 <systemProperties>
   <org.apache.avro.SERIALIZABLE_CLASSES>java.math.BigDecimal,java.math.BigInteger,java.net.URI,java.net.URL,java.io.File,java.lang.Integer,org.apache.avro.reflect.TestReflect$R10</org.apache.avro.SERIALIZABLE_CLASSES>
</systemProperties>

this would also solve my use case :)

(Another option would be to solve this with a custom AvroDatumFactory, but that does involve some knowledge of Avro internals...)

Let me know what you all think. If this looks good I can polish the javadoc/add tests/etc.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

…o Dataflow worker

gemini-code-assist · 2025-11-13T15:09:32Z

Summary of Changes

Hello @clairemcginty, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request proposes a solution to a runtime SecurityException encountered in Dataflow jobs using Avro 1.11.4+ when Avro schemas contain java-class annotations for certain types (e.g., java.math.BigDecimal). The change introduces a new DataflowPipelineOption called avroSerializableClasses. This option allows users to provide a comma-separated list of trusted classes, which the Dataflow worker then uses to set the org.apache.avro.SERIALIZABLE_CLASSES system property, thereby resolving the security error without requiring custom worker images.

Highlights

Avro Compatibility Fix: Addresses a SecurityException in Avro 1.11.4+ when using java-class annotations in schemas, specifically for classes like java.math.BigDecimal, which previously caused runtime decoding errors on Dataflow.
New Pipeline Option: Introduces a new DataflowPipelineOption named avroSerializableClasses. This option allows users to specify a list of trusted serializable classes directly through pipeline options.
Dataflow Worker Integration: Modifies the Dataflow worker boot process (boot.go) to read the new avroSerializableClasses option and dynamically set the org.apache.avro.SERIALIZABLE_CLASSES Java system property for the worker JVM, resolving the security issue without requiring custom worker images.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-11-13T17:07:26Z

Assigning reviewers:

R: @chamikaramj for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

clairemcginty · 2025-11-13T17:11:31Z

cc @Abacn - I think you merged the original upgrade from Avro 1.8 -> 1.11 :)

kellen · 2025-11-13T19:21:55Z

sdks/java/container/boot.go

 			}
 		}
+		// Add trusted Avro serializable classes
+		if serializableClasses, ok := pipelineOptions.GetStructValue().GetFields()["avroSerializableClasses"]; ok {


We could have both; a default that matches avro and this option to override when needed. Ideally the avro default list should be public so users can extend/modify it as required.

Yeah, for classes already caused regressions ("java.math.BigDecimal", and if there are others) we can add it here for users

...low-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java

Abacn · 2025-11-17T20:16:34Z

...low-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java

+  @Default.InstanceFactory(AvroSerializableClassesFactory.class)
+  List<String> getAvroSerializableClasses();
+
+  void setAvroSerializableClasses(List<String> options);


That said this should be an "SDKHarnessOptions": https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java

Abacn · 2025-11-18T21:02:47Z

testWriteUnboundedWithCustomBatchParameters (org.apache.beam.sdk.io.TextIOWriteTest) failed not related to the change. Merging for now

[Proposal] Allow users to specify trusted Avro serializable classes t…

de88282

…o Dataflow worker

github-actions bot added java docker runners dataflow labels Nov 13, 2025

clairemcginty force-pushed the set-avro-serializable-class-prop branch 3 times, most recently from 3c20249 to 6bd13e1 Compare November 13, 2025 15:53

Fixup boot.go

e027e2d

clairemcginty force-pushed the set-avro-serializable-class-prop branch from 6bd13e1 to e027e2d Compare November 13, 2025 16:00

github-actions bot added the Next Action: Reviewers label Nov 13, 2025

kellen reviewed Nov 13, 2025

View reviewed changes

Add default factory; add tests

83a8513

Abacn reviewed Nov 17, 2025

View reviewed changes

...low-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java Outdated Show resolved Hide resolved

Abacn reviewed Nov 17, 2025

View reviewed changes

github-actions bot added runners dataflow and removed runners dataflow labels Nov 17, 2025

clairemcginty changed the title ~~[Proposal] Allow users to specify trusted Avro serializable classes to Dataflow worker~~ Allow users to specify trusted Avro serializable classes to Dataflow worker Nov 17, 2025

Set default options from boot.go; move PipelineOpt to SdkHarnessOptions

7cf5b2c

clairemcginty force-pushed the set-avro-serializable-class-prop branch from 56bce77 to 7cf5b2c Compare November 17, 2025 21:30

github-actions bot added runners dataflow and removed runners dataflow labels Nov 17, 2025

Add check for empty list

afb1461

github-actions bot added runners dataflow and removed runners dataflow labels Nov 18, 2025

Abacn approved these changes Nov 18, 2025

View reviewed changes

Abacn merged commit f41cbde into apache:master Nov 18, 2025
22 of 27 checks passed

clairemcginty deleted the set-avro-serializable-class-prop branch November 18, 2025 21:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow users to specify trusted Avro serializable classes to Dataflow worker #36809

Allow users to specify trusted Avro serializable classes to Dataflow worker #36809

clairemcginty commented Nov 13, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Nov 13, 2025

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

clairemcginty commented Nov 13, 2025

Uh oh!

kellen Nov 13, 2025

Uh oh!

Abacn Nov 17, 2025

Uh oh!

Uh oh!

Abacn Nov 17, 2025

Uh oh!

Abacn commented Nov 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Allow users to specify trusted Avro serializable classes to Dataflow worker #36809

Allow users to specify trusted Avro serializable classes to Dataflow worker #36809

Conversation

clairemcginty commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Nov 13, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

clairemcginty commented Nov 13, 2025

Uh oh!

kellen Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Abacn Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Abacn Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Abacn commented Nov 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clairemcginty commented Nov 13, 2025 •

edited

Loading