Adding the `meta` option to a simple node request makes x10 in response time

[laem]

On our instance, this command will take ~ 2 seconds.

wget "https://overpass.openstreetmap.fr/api/interpreter?data=[out%3Ajson]%3Bway(id%3A1426303011)%3B%0A%09%0A%09out geom qt%3B" -O overpass.json

This one will take 100 x less time than the equivalent with meta.

wget "https://overpass.openstreetmap.fr/api/interpreter?data=[out%3Ajson]%3Bway(id%3A1426303011)%3B%0A%09%0A%09out meta geom qt%3B" -O overpass.json

Is this a bug ? It looks like it.

In this issue, @mmd-osm talked about a x50 in a loop because of meta. @drolbr marked it as fixed.

I suspect some link between both issues.

[mmd-osm]

I think this is expected, since for every query, index and usernames need to be read again, and this list is constantly growing with every new osm user account. I have addressed this in https://github.com/mmd-osm/Overpass-API

Index and username caching: further taking advantage of FastCGI, database indices and usernames are only updated once per minute, and can be reused for many queries.

However, the performance improvement didn't make into the official release. With the fix, both queries take pretty much the same time.

[laem]

Oh, thanks a lot ! I'm a bit lost between both repos : your fork looks to be at the same time in advance for a lot of cool features and fixes ; and at the same time, behind the "original" repo.

This branch is 947 commits ahead of and 441 commits behind drolbr/Overpass-API:master.

I'm confused : is it a solid code divergence ? Did you lack time to reintegrate news commits from drolbr's ?

I see there are incompatible db changes in any case, making the cloning "fast startup" option impossible from the .de instance ?

[laem]

I'm also seeing another fork, https://github.com/b1tw153/Overpass-API.

This one looks like just a compatible ahead version of drolbr's.

Discussed here.

[mmd-osm]

Yes, cloning from the official cloning location won't be possible. You can start fresh with a planet pbf file and import that directly. That's what https://www.openstreetmap.org/user/mmd/diary/400113 was about. https://wiki.openstreetmap.org/wiki/User:Mmd/Overpass_API_-_experimental_corner has more more examples.

Some recent features like uid_changed, user_changed have been backported. In some cases, internal changes in the "441 commits behind" have alternative implementations in the "947 commits ahead", and it didn't make sense to include them. Overall, the code has massively diverged in the meantime.

I don't really recommend to run this in production, but some folks over at OpenGeoFiction indeed did that.

https://github.com/b1tw153/Overpass-API is fully compatible as far as I know. I believe it is mostly concerned with improving production operations. Maybe @b1tw153 can share some more details.

[laem]

Thanks !

I don't really recommend to run this in production, but some folks over at OpenGeoFiction indeed did that.

Sad to hear that, because some new features like meta request time (discussed above) and fastcgi look awesome. In our experience of maintaining an overpass instance for cartes.app, they are key features for scaling.

FastCGI support: avoid starting a new interpreter binary for each request, thereby enabling further caching options (#383)

Index and username caching: further taking advantage of FastCGI, database indices and usernames are only updated once per minute, and can be reused for many queries.

These two features, and most performance improvement features, do not seem to have been backported to the other two interesting repos.

I don't really recommend to run this in production

What would be the risks, compared to these other two ? I'd be interested to try to deploy your repo on a new server.

[mmd-osm]

So this is mostly about expectation management. You can of course try it out and see if it works for you. I don't want to give people the impression that if this fails in production, there will be anyone with time for troubleshooting or even working on fixes. In case of Overpass API, this can be super time consuming (e.g. days or even weeks).

[laem]

Fine with me, thanks for giving more precision :)