As developer of the open standard interface DATEX II for the traffic event data of the Open Data Hub, I would like to have some adjustments on the traffic event data collection flow

[rcavaliere]

In order to correctly map the data into DATEX II, following aspects need to be adjusted:

• event data should have a version field, so to underline if the event record is related to an update of the original event record
• we need an API call to provide just the currently active event data, in the sense that I would like to see only the events that are valid now (i.e. current dateTime is within start and end time; if no end time is set, dateTime should be only greater than start time)
• the format of fields dateTime is not the same. For example publisherDateTime and transactionTime are formatted differently. Please let's provide them in the same format

[rcavaliere]

@clezag in relation to this development, I also received this contribution from the supplier about how their development contribution should be then hosted in our infrastructure. What do you say, do you see critical things there?

Architettura nodo Datex - Bolzano.pdf - Rev. 1.pdf

[clezag]

For version tag: If there are any changes to an event with the same UUID, increment version number.

The date format of the metadatafield can remain as is, because it's a string taken directly from the data provider and we don't want to add custom code to change the format

[clezag]

@rcavaliere I've taken a look at this issue again, and I'm having a difficult time with understanding how exactly events change over time. Do we have some contact at the province that can provide us with documentation of this API?
Alternatively do you know of a specific event or type of event that changed over time, so I can take a look at the raw data?

e.g. my current hunch would be to use messageId as the event series UUID, and all the fields except publishDateTime to generate the actual UUID. But I don't know if the messageId stays the same, or if they release updates under a new messageId, which would require another approach. We also cannot use the date + location as we did before because of the issue I've discovered further down below.

This is an example of one event over time. Looks like at least publishDateTime has to be ignored:

expand to look at json

{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T13:05:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-11-28",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Kastelbell (km 176,100 – km 176,300) Einbahnregelung wegen Arbeiten an der Umfahrung Kastelbell und Galsaun.\n\n",
  "placeIt": "A Castelbello (km 176,100 – km 176,300) senso unico alternato a causa di lavori alla circonvallazione di Castelbello e Colsano.\n\n",
  "actualMail": 1,
  "messageId": 27898061,
  "messageStatus": 0,
  "messageZoneId": 1,
  "messageZoneDescDe": "Vinschgau",
  "messageZoneDescIt": "Val Venosta",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 559,
  "messageStreetWapDescDe": "Stilfser Joch Straße",
  "messageStreetWapDescIt": "Statale del Passo Stelvio",
  "messageStreetInternetDescDe": "Stilfser Joch Straße",
  "messageStreetInternetDescIt": "Statale del Passo Stelvio",
  "messageStreetNr": "SS 38",
  "messageStreetHierarchie": 300038,
  "messageTypeId": 6,
  "messageTypeDescDe": "Baustellen / Sperren",
  "messageTypeDescIt": "Cantieri stradali / Chiusure",
  "X": 10.8907544414669,
  "Y": 46.6264542969015
}
{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T13:05:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-11-28",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Kastelbell (km 176,100 – km 176,300) Einbahnregelung wegen Arbeiten an der Umfahrung Kastelbell und Galsaun.\n\n",
  "placeIt": "A Castelbello (km 176,100 – km 176,300) senso unico alternato a causa di lavori alla circonvallazione di Castelbello e Colsano.\n\n",
  "actualMail": 1,
  "messageId": 27898061,
  "messageStatus": 0,
  "messageZoneId": 1,
  "messageZoneDescDe": "Vinschgau",
  "messageZoneDescIt": "Val Venosta",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 559,
  "messageStreetWapDescDe": "Stilfser Joch Straße",
  "messageStreetWapDescIt": "Statale del Passo Stelvio",
  "messageStreetInternetDescDe": "Stilfser Joch Straße",
  "messageStreetInternetDescIt": "Statale del Passo Stelvio",
  "messageStreetNr": "SS 38",
  "messageStreetHierarchie": 300038,
  "messageTypeId": 6,
  "messageTypeDescDe": "Baustellen / Sperren",
  "messageTypeDescIt": "Cantieri stradali / Chiusure",
  "X": 10.8907544414669,
  "Y": 46.6264542969015
}
{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T13:01:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-11-28",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Kastelbell (km 176,100 – km 176,300) Einbahnregelung wegen Arbeiten an der Umfahrung Kastelbell und Galsaun.\n\n",
  "placeIt": "A Castelbello (km 176,100 – km 176,300) senso unico alternato a causa di lavori alla circonvallazione di Castelbello e Colsano.\n\n",
  "actualMail": 1,
  "messageId": 27898061,
  "messageStatus": 0,
  "messageZoneId": 1,
  "messageZoneDescDe": "Vinschgau",
  "messageZoneDescIt": "Val Venosta",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 559,
  "messageStreetWapDescDe": "Stilfser Joch Straße",
  "messageStreetWapDescIt": "Statale del Passo Stelvio",
  "messageStreetInternetDescDe": "Stilfser Joch Straße",
  "messageStreetInternetDescIt": "Statale del Passo Stelvio",
  "messageStreetNr": "SS 38",
  "messageStreetHierarchie": 300038,
  "messageTypeId": 6,
  "messageTypeDescDe": "Baustellen / Sperren",
  "messageTypeDescIt": "Cantieri stradali / Chiusure",
  "X": 10.8907544414669,
  "Y": 46.6264542969015
}
{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T12:26:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-11-28",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Kastelbell (km 176,100 – km 176,300) Einbahnregelung wegen Arbeiten an der Umfahrung Kastelbell und Galsaun.\n\n",
  "placeIt": "A Castelbello (km 176,100 – km 176,300) senso unico alternato a causa di lavori alla circonvallazione di Castelbello e Colsano.\n\n",
  "actualMail": 1,
  "messageId": 27898061,
  "messageStatus": 0,
  "messageZoneId": 1,
  "messageZoneDescDe": "Vinschgau",
  "messageZoneDescIt": "Val Venosta",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 559,
  "messageStreetWapDescDe": "Stilfser Joch Straße",
  "messageStreetWapDescIt": "Statale del Passo Stelvio",
  "messageStreetInternetDescDe": "Stilfser Joch Straße",
  "messageStreetInternetDescIt": "Statale del Passo Stelvio",
  "messageStreetNr": "SS 38",
  "messageStreetHierarchie": 300038,
  "messageTypeId": 6,
  "messageTypeDescDe": "Baustellen / Sperren",
  "messageTypeDescIt": "Cantieri stradali / Chiusure",
  "X": 10.8907544414669,
  "Y": 46.6264542969015
}
{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T12:26:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-11-28",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Kastelbell (km 176,100 – km 176,300) Einbahnregelung wegen Arbeiten an der Umfahrung Kastelbell und Galsaun.\n\n",
  "placeIt": "A Castelbello (km 176,100 – km 176,300) senso unico alternato a causa di lavori alla circonvallazione di Castelbello e Colsano.\n\n",
  "actualMail": 1,
  "messageId": 27898061,
  "messageStatus": 0,
  "messageZoneId": 1,
  "messageZoneDescDe": "Vinschgau",
  "messageZoneDescIt": "Val Venosta",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 559,
  "messageStreetWapDescDe": "Stilfser Joch Straße",
  "messageStreetWapDescIt": "Statale del Passo Stelvio",
  "messageStreetInternetDescDe": "Stilfser Joch Straße",
  "messageStreetInternetDescIt": "Statale del Passo Stelvio",
  "messageStreetNr": "SS 38",
  "messageStreetHierarchie": 300038,
  "messageTypeId": 6,
  "messageTypeDescDe": "Baustellen / Sperren",
  "messageTypeDescIt": "Cantieri stradali / Chiusure",
  "X": 10.8907544414669,
  "Y": 46.6264542969015
}

Digging deeper I also found that the same event can show up as different types at the same time, with different messageId, I think one is to announce the closure, and another one the current impact on traffic once the closure is active:
Note that the only fields different between these are messageId, messageTypeId, messageTypeDescDe, messageTypeDescIt.

expand to look at json

{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T13:33:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-03-06",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Klausen (km 466,936 - km 467,064) Einbahnregelung von 08:30 bis 16:00 Uhr wegen Instandhaltungsarbeiten der Elektroanlagen im Tunnel.",
  "placeIt": "A Chiusa (km 466,936 - km 467,064) senso unico alternato dalle ore 08:30 alle ore 16:00 per lavori di manutenzione sugli impianti elettrici nella galleria.",
  "actualMail": 1,
  "messageId": 27849011,
  "messageStatus": 1,
  "messageZoneId": 5,
  "messageZoneDescDe": "Eisacktal-Wipptal",
  "messageZoneDescIt": "Val d'Isarco",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 521,
  "messageStreetWapDescDe": "Brenner Staatsstraße",
  "messageStreetWapDescIt": "Statale del Brennero",
  "messageStreetInternetDescDe": "Brenner Staatsstraße",
  "messageStreetInternetDescIt": "Statale del Brennero",
  "messageStreetNr": "SS 12",
  "messageStreetHierarchie": 300012,
  "messageTypeId": 1,
  "messageTypeDescDe": "Aktuelle Lage",
  "messageTypeDescIt": "Situazione attuale",
  "X": 11.5645002262475,
  "Y": 46.640445999991
}

{
  "json_featuretype": "con coordinate",
  "publishDateTime": "2025-03-05T13:33:00",
  "beginDate": "2025-03-04",
  "endDate": "2025-03-06",
  "descriptionDe": "",
  "descriptionIt": "",
  "tycodeValue": "BEHIND",
  "tycodeDe": "Verkehrsbehinderung für Zonen und aus. Südt.",
  "tycodeIt": "intralci viabilità in e fuori Alto Adige",
  "subTycodeValue": "BAUSTELLE",
  "subTycodeDe": "Baustelle",
  "subTycodeIt": "cantiere",
  "placeDe": "Bei Klausen (km 466,936 - km 467,064) Einbahnregelung von 08:30 bis 16:00 Uhr wegen Instandhaltungsarbeiten der Elektroanlagen im Tunnel.",
  "placeIt": "A Chiusa (km 466,936 - km 467,064) senso unico alternato dalle ore 08:30 alle ore 16:00 per lavori di manutenzione sugli impianti elettrici nella galleria.",
  "actualMail": 1,
  "messageId": 27849061,
  "messageStatus": 1,
  "messageZoneId": 5,
  "messageZoneDescDe": "Eisacktal-Wipptal",
  "messageZoneDescIt": "Val d'Isarco",
  "messageGradId": 4,
  "messageGradDescDe": "Behinderung",
  "messageGradDescIt": "rallentamenti",
  "messageStreetId": 521,
  "messageStreetWapDescDe": "Brenner Staatsstraße",
  "messageStreetWapDescIt": "Statale del Brennero",
  "messageStreetInternetDescDe": "Brenner Staatsstraße",
  "messageStreetInternetDescIt": "Statale del Brennero",
  "messageStreetNr": "SS 12",
  "messageStreetHierarchie": 300012,
  "messageTypeId": 6,
  "messageTypeDescDe": "Baustellen / Sperren",
  "messageTypeDescIt": "Cantieri stradali / Chiusure",
  "X": 11.5645002262475,
  "Y": 46.640445999991
}

[clezag]

I've also looked at the "UPDATE" tag they have on their website. It's just the field messageStatus == 1.

I've looked at the history of the 27849061 / 27849011 events, which have that status, and we got both of them on the same day (24.02.2025), and the very first event is equal to the most recent, so I assume they never changed.

I also cannot find another event with that same description text or coordinates in that timeframe

[rcavaliere]

@clezag thanks for your analysis. Unfortunately we don't have at present any further specification on how such events are generated from the source. Based on your information and the events browsable on analytics, I assume that they never update events, and that an event is for them just a notification ("message"), which is identified through its messageId. The field messageStatus is probably a field in order to distinguish between messages that are relevant at present or not. They also produce different messages for the same event, in particular if they want to have this information available as roadwork and traffic queues, for example.

So for the first implementation of the DATEX II node we just need to call all events with messageStatus == 1, what do you say?

For info, the events provided by A22 have this kind of versioning logic and is supported by the source. You could also have a look at this dataset if you want.

[clezag]

@rcavaliere I've had a look at the A22 data collector, and it's similar: It uses the field ID of the event to create the eventseriesuuid, and then uses a bunch of fields (location, start/end date, type) to create the uuid.
So in this case I think A22 maintains the same id between event updates, and that's how the series is identified.

I've redone the UUID logic now:
for eventseriesUUID I'm using only the messageId, assuming that it stays the same between updates.
for the UUID itself, I'm using these fields, which should cover all possible changes (other fields are descriptions of the states in german and italian):

MessageID               
X                       
Y                       
BeginDate               
EndDate                 
TycodeValue             
SubTycodeValue          
PlaceDe                 
PlaceIt                 
MessageStatus           
MessageZoneID           
MessageGradID           
MessageTypeID           
MessageStreetID         
MessageStreetNr         
MessageStreetHierarchie 

So if any one of these changes, it makes an updated event with always the same eventuuid.

I think this should be good enough to start with. I would like to put this into production rather fast, the other two changes will have to be on the API side

[ohnewein]

The DataCollector can be released in production.

[clezag]

@rcavaliere data collector has been deployed in production.
I've also fixed a small issue when description or eventcategory types are not available (now it does not show the _ or | separators anymore if there are no values to separate).
But since events are not updated, this will only affect newly inserted ones.

[clezag]

@rcavaliere I've had a look at the events API, and getting the current events can be done by passing current date as both from and to parameters like so:

https://mobility.api.opendatahub.testingmachine.eu/v2/flat,event/PROVINCE_BZ/2025-04-11T00:00.000/2025-04-11T23:59.999?select=evstart,evend

looks a bit strange, but the actual logic is:
evstart <= to and (evend is null or evend > from)

so to get it perfectly, you probably need to set
from = today at 00:00
to = today at 23:59

feel free to correct me if I didn't consider an edge case

[rcavaliere]

@clezag I have checked more in detail this user story and the work we did. I think that is now everything clarified. In my view we just need to provide an input to 3rd parties on how they can easily read the last version of a certain event, what would an API call example look like?

[clezag]

@rcavaliere From looking at the code, this is currently not supported API-side.

The good news is that the functionality is already there, it would essentially be the /latest call combined with a /2025-05-01 type date filter, we just have to implement a call that allows for both to be set at the same time, e.g. add to set a latest=true parameter.

So the call would be something like /PROVINCE_BZ/2025-04-11T09:10:000?latest=true

It has the potential to be very confusing for users, because this would return all the events active at that timestamp, that are also the latest event within a series.
It does not however, give you the most recent event within a series that is active at the time.
Both conditions have to be true: it has to be active, and it has to be the latest of the series.

For events in a series like the software developers thursday this makes no sense, because the events don't supersede the prior ones, they all remain valid.

Maybe it would be better to implement an event update functionality and update the events, and not retain the history.

Another idea is to add some kind of superseded field to the event that makes clear that it's only there for historic purposes. Then users could search for events that don't have superseded = true (or have that be the default). Each data collector could then decide by itself if event series are "superseding" or not.

[clezag]

@rcavaliere I've looked at this with Matteo, unfortunately the rabbit hole goes a good bit deeper than we anticipated.

• The eventseries is not made to handle event updates, e.g. "latest" only means the one with the latest start/end date. If for example you update an event to have a shorter interval, that one is not considered the latest, so my proposed solution above would not work correctly.
  In fact I suspect that even today some A22 events might run into this issue. I also think that if an updated event has the same start/end date as before, which one is considered the latest is not defined and basically random.
• We currently don't track the "timestamp" of an event, that is the time at which point it was published by the source, and so we also don't really know what the actual latest event (in the sense of most recent update) is.
  We only have the date when it was inserted in our DB, which especially on the new infrastructure is not reliable, as messages might be replayed at a different date
• Basically we would have to implement the whole versioning/history/update part of the API from scratch or completely change how the current eventseries are handled.

Seeing as

• we get the impression that the events implementation is not fully developed and somewhat orphaned
• we would need to implement additional features to support our use case
• aside from the new PROVINCE_BZ dataset we only have private A22 data
• usage of the events API in general seems low. I only see a notable number of requests since start of this year, which I assume is the DATEX development
• the DATEX implementation will (hopefully) drastically increase usage

We would like to migrate this to the Content API events (or even a new unifiedevents domain separate from tourism and mobility - but using the existing code), and deprecate the whole events part of the timeseries API.
I know this is a big tangent to this issue, but we think it would be best to bite the bullet before the API increases in usage, people start to rely on it, and we have to maintain two event APIs that do the same thing.

We're happy to discuss this further in person, seeing as it's a big decision that goes beyond the scope of this issue, and depending on the time/budget constraints we still have to finish this within a reasonable time frame.