Skip to content

Conversation

@claudevdm
Copy link
Collaborator

@claudevdm claudevdm commented Nov 24, 2025

timestamp-nanos avro logical type was introduced in avro 1.12, but beam cannot upgrade avro versions until java 8 support is dropped #34993

There are workarounds here to handle the timestamp-nanos logical type until we can update avro version.

AvroUtils changes:

  • toBeamFieldType(): Recognizes timestamp-nanos property and maps to Timestamp.NANOS
  • toAvroSchema(): Converts Timestamp.NANOS to Avro schema with timestamp-nanos logical type
  • toGenericRecord(): Converts Instant values to nanoseconds since epoch
  • toBeamRowStrict(): Converts nanosecond values back to Instant objects

BigQueryAvroUtils changes:

  • Refactored timestamp formatting into TimestampPrecision enum
  • Added NANOSECONDS precision level alongside existing MILLISECONDS and MICROSECONDS
  • Updated convertRequiredField() to handle timestamp-nanos logical type
  • Updated formatDatetime() to dynamically format fractional seconds (3, 6, or 9 digits)
  • Updated fromGenericAvroSchema() to map timestamp-nanos to BigQuery TIMESTAMP type

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@claudevdm claudevdm force-pushed the bq-avro-timestampnano branch from d234446 to 1d0a0ef Compare November 25, 2025 15:50
@claudevdm claudevdm changed the title Add timestamp-nanos avro logical type support in bigquery utils. Add timestamp-nanos avro logical type support in bigquery avro utils. Nov 25, 2025
@claudevdm claudevdm force-pushed the bq-avro-timestampnano branch from 88b51f4 to e8c61c5 Compare November 25, 2025 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant