Rewriting the internals of Leon

[svenjacobs]

I'm currently rewriting the internals of Leon. Instead of sanitizers based on regular expressions, the idea is that Leon creates an internal representation of an address that contains the domain, path and query parameters. Sanitizers can then declare which parameters should be removed. This change will also allow the UI to show all parameters so that users can manually select or deselect parameters for removal. Hence Leon will also work with "unknown" addresses.

Until this refactoring is completed, I will not add any new sanitizers because they need to be rewritten anyway.

[serrq]

Okay. For Christmas is ready the new Leon?

[TPS]

Can we use import java.net.URL or import android.net.Uri url parser instead?

https://github.com/ethauvin/urlencoder#why-not-simply-use-javaneturlencoder denotes some errors in those, also.

[svenjacobs]

Can we use import java.net.URL or import android.net.Uri url parser instead?

https://github.com/ethauvin/urlencoder denotes some errors in those, also.

Thanks for the link. That's an interesting alternative for URLDecoder.

Regarding Url.kt, the intention was to have a class that splits an URL into its components so that these components can than be used by cleaning rules.

[rsb-23]

Maybe there's a confusion, l am not referring to Url encoder/decoder.
Hence rephrasing my query :

• java.net.URL splits an url into its components. (what Url.kt is intended)
• Since java.net.URLDecoder is already used in project, doesn't it mean java.net.URL is also compatible to Kotlin Multiplatform?
• Shared Url encoder project does not mention any issue with the url parser.

[TPS]

• Shared Url encoder project does not mention any issue with the url parser.

@rsb-23 Maybe I misunderstand your details, but here's what the relevant section at the above repo link says (emphasis mine):

For example, a simple search query such as:

val u = URLEncoder.encode("foo +bar", StandardCharsets.UTF_8)  

would be encoded as:

foo+%2Bbar  

Trying to decode it with Spring, for example:

UriUtils.decode(u,   StandardCharsets.UTF_8)  

would return:

foo++bar  

Unfortunately, decoding with Uri.decode on Android, decodeURI in Javascript, etc. would yield the exact same result.

[rsb-23]

@TPS I had understood that the UrlEncoder.encode is not consistent with decode logic.

But my query is about parser.

Parser and encoder are different functionality, please don't mix the two.

[farOverNinethousand]

Hi Sven,
I'm currently writing a Telegram Bot which will have similar functionality to your application (if something like this already exists please let me know but I didn't find anything nice and open source).
While a native app like yours is more powerful, I will still work on releasing said bot.
During my research about similar tools I found yours to be nearly the only actively developed project of that kind.

Now I'm asking myself:
Will you design the new link-sanatizer-system in a way that it can be programmatically used in other applications?
I'm asking as this could save double-work and I'd love to add URL-sanatizers to my bot which will also be usable in other applications :)

Thanks for your time!

[svenjacobs]

As you may have noticed I moved this project to a new organization and also created a skeleton project for the library.

[TPS]

new organization
skeleton project

@svenjacobs Great names! Inspired by people you know?

[TPS]

@svenjacobs Now that you've settled on a container format, a thought:

JSON is designed to be simple to work w/, but trade-off is that it can be very space-hungry, especially since new URL transforms will be static w/ APK updates. Typical publish toolchain will optimize for APK size for transmission, but not once installed, which might get substantial once including JSON logic & going from (currently) 10s of transforms to (potentially) 100s or 1000s (especially once you start processing external databases). Part of Léon's charm has always been that it's < 10MB installed. What will these upgrades do for that, I wonder?

Perhaps something like https://github.com/shibasis0801/kmm_zlib_compression & linked article will be helpful in your initial design phase? & taking care to use as much is natively available so as to not bloat that installed size w/ add'l libs to carry around?

[svenjacobs]

new organization
skeleton project

@svenjacobs Great names! Inspired by people you know?

I'm not sure if you're joking or don't know the reference 😉 Names are based on the movie Léon The Professional 😅 If you haven't seen it, do it!

[svenjacobs]

@svenjacobs Now that you've settled on a container format, a thought:

JSON is designed to be simple to work w/, but trade-off is that it can be very space-hungry, especially since new URL transforms will be static w/ APK updates. Typical publish toolchain will optimize for APK size for transmission, but not once installed, which might get substantial once including JSON logic & going from (currently) 10s of transforms to (potentially) 100s or 1000s (especially once you start processing external databases). Part of Léon's charm has always been that it's < 10MB installed. What will these upgrades do for that, I wonder?

Good point! We could also think about generating Kotlin code from JSON? However I don't know if it will actually reduce the size compared to JSON if we have hundreds or thousands of sanitizers? I guess we have to find out once we're there.