Skip to content

Introduce schema reference support#30

Open
yeralin wants to merge 14 commits into
spring-cloud:mainfrom
yeralin:introduce-schema-reference-support
Open

Introduce schema reference support#30
yeralin wants to merge 14 commits into
spring-cloud:mainfrom
yeralin:introduce-schema-reference-support

Conversation

@yeralin
Copy link
Copy Markdown

@yeralin yeralin commented Feb 26, 2020

Addresses 3rd solution from #29

Looks up optional Schema-Reference header that is suppose to contain referenced subjects in the following format:

  1. Schema-Reference: <referenced-subject-1>; <referenced-subject-2>; // without version
  2. Schema-Reference: <referenced-subject>+v<version-number>;

Example usage:
First request:

$ curl --location --request POST 'http://localhost:8990' \
--header 'Content-Type: application/json' \
--data-raw '{
    "subject": "sub",
    "format": "avro",
    "definition": { # the definition is not in a string format for readability purposes
        "type": "record",
        "namespace": "test",
        "name": "sub",
        "fields": [
            {
                "name": "state",
                "type": "string"
            }
        ]
    }
}'

Response: 201 Created

Second request (referencing test.sub schema):

$ curl --location --request POST 'http://localhost:8990' \
--header 'Content-Type: application/json' \
--header 'Schema-Reference: sub+v1' \ # referencing previously created schema by its subject
--data-raw '{
    "subject": "main",
    "format": "avro",
    "definition": {
        "type": "record",
        "namespace": "test",
        "name": "main",
        "fields": [
            {
                "name": "subField",
                "type": "test.sub"
            }
        ]
    }
}'

Response: 201 Created

Explanation: while creating the second schema, it:

  1. Parses Schema-Reference header
  2. Looks up sub subject with version 1 in the repository
  3. Creates AVRO parser
  4. Parses looked up sub subject's schema definition
  5. Parses passed main subject's schema definition (since it parsed sub schema, now it successfully validates test.sub type)

@yeralin yeralin force-pushed the introduce-schema-reference-support branch from af17934 to 7df9847 Compare March 3, 2020 21:31
@ManyToMany
private List<Schema> references = new ArrayList<>();

@Column(name = "DEFINITION", columnDefinition = "text", nullable = false, length = 8192)
Copy link
Copy Markdown
Author

@yeralin yeralin Mar 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed @Lob annotation as it was creating problems while fetching references field during schema validation by throwing:

org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode.

when using PostgreSQL.

It was replaced by columnDefinition = "test" and effectively creates the same type of column for the definition field.

References:
https://shred.zone/cilla/page/299/string-lobs-on-postgresql-with-hibernate-36.html

String subject;
Integer version;
Matcher schemaReferenceMatcher;
for (String schemaReferenceEntry : schemaReferenceHeader.split(";")) {
Copy link
Copy Markdown
Author

@yeralin yeralin Mar 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method needs some refactoring.

Can you advice any "springy" way of doing schema reference resolution (header parsing, schema fetching, etc)?

@JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "subject")
@JsonIdentityReference(alwaysAsId = true)
@ManyToMany
private List<Schema> references = new ArrayList<>();
Copy link
Copy Markdown
Author

@yeralin yeralin Mar 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another issue I stumbled upon, since now instead of returning all Schema references, I only return their subject names i.e.:

{
    "subject": "test",
    "id": 1,
    "version": 1,
    "format": "avro",
    "references": [
        "subject.reference"
    ],
    "definition": "..."
}

as opposed to:

{
    "subject": "test",
    "id": 1,
    "version": 1,
    "format": "avro",
    "references": [
        {
                "subject": "subject.reference",
                "id": 2,
                 ...
        }
    ],
    "definition": "..."
}

This however, now breaking tests since MappingJackson2HttpMessageConverter's object mapper is trying to map REST's response payload directly to Schema.class where Schema.references is List<Schema> not List<String>.

I am not entirely sure how to solve this for now.

  1. Write an interceptor to fetch referenced schemas during serialization
  2. Map REST's response payload to String, then perform manual parsing
  3. Somehow deactivate @Json* annotations under Schema.class during tests, so that the latter payload format gets returned

What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants