Finding local .zim files takes a very long time ~15 minutes

[tigran123]

Describe the bug

On Android 12 running on Samsung Galaxy Note10+ phone it took 14 minutes 5 seconds to find just two .zim files.

Expected behavior

Should take a few seconds. Actually, reading a directory and discovering those files should take 2-3ms, but assuming x1000 application overhead it should still complete in a couple of seconds, no more. Also, allowing one more second for reading and parsing the metadata I would say the maximum allowed time for this is 3 seconds. However, if the .zim file architecture is wrong, i.e. if the file is compressed (the 'z' in the name gives this hint) AND the metadata is stored inside the archive, then it would take about 15 minutes to access 100GB files, which is very bad, because this sort of processing should not depend on the filesize. To test this theory I would have to try with the smaller .zim files.

Steps to reproduce the behavior:

1. Copy two zim files to /Android/media/kiwix. I used WikiPedia English (102GB latest) and WikiPedia Russian (33GB latest).
2. Go to Android Settings and stop Kiwix app.
3. While in the Settings clear cache and data of the Kiwix app
4. Start Kiwix app and give it appropriate permissions
5. Go to Kiwix and select Settings from the menu
6. Point to External in Settings and select /Android/media/kiwix directory (making sure that it does point to External device first)
7. Give the appropriate permission by clicking on "Use this directory" button
8. Go to the Library tab of Kiwix and on another device start the timer
9. Observe that the two .zim files will be discovered in 14 minutes and 5 seconds. And this is on a very fast Note10+ smartphone with a very fast 1TB microsd card, so on a lower-end configuration it will take even longer.

Screenshots

Environment

• Version of Kiwix Android : 3.9.1 from https://download.kiwix.org/release/kiwix-android/kiwix-3.9.1.apk
• Device : Samsung Galaxy Note10+
• OS version : Android 12

This problem was also mentioned in #3604

[kelson42]

I can only support this issue, really wonder myself. Maybe there a legit explanation, then very interested in it.

[tigran123]

I have just tested this issue on Samsung Galaxy Tab S4 running Android 10 and here it works fine -- it discovers these files in "just" 1 minute and 46 seconds. Still a bit slow, but acceptable. This tablet has the same 1TB microsd and a slower CPU than Note10+ (and 1/3 of RAM -- 4GB vs 12GB, though it is probably irrelevant).

[kelson42]

@tigran123 Do you have a lot of other kinds of files (on the SD card) beside the two ZIM files?

[tigran123]

Yes, my books library (30,000 books) and Audio library (100,000 files), but they are all in other subdirectories (/Android/media/Books and /Android/media/Audio) and I specifically pointed Kiwix to /Android/media/kiwix, so it shouldn't go looking in other directories outside /Android/media/kiwix. If it does -- that is a bug.

But all these other files are present on both the tablet and the phone (I always have my library close, being a "bookworm" :)

[kelson42]

@tigran123 OK, we have found I guess why this is that slow. But I will leave @MohitMaliFtechiz give the full answer. Hopefully we can better filter by filename extensions...

@MohitMaliFtechiz Sorry for my ignorance but I have a few questions:

• Are the locations scanned for ZIM files somehow related to storage location setting? If yes, how?
• Do we have any visual indicator that a scanning is ongoing?

[tigran123]

If you think that it is the presence of 150k other files that slowed Kiwix down, then I suspect that you are mistaken, because I vaguely remember that when I was setting up this Note10+ smartphone the ZIM files for Kiwix were the first thing that I copied to the microsd, i.e. the books and classical music were NOT yet there -- the microsd was essentially empty. And, as far as I remember, Kiwix was extremely slow to find those ZIM files.

Anyway, it is easy to verify this -- tomorrow I'll just take a small microsd card 200GB in size and will copy just the ZIM files to it -- let's see how fast Kiwix will find them. My guess is -- in 14 minutes. Well, if I am mistaken then this is a very serious bug in Kiwix -- it certainly shouldn't go looking in places where it is not supposed to, i.e. outside the directory explicitly designated by the user.

[tigran123]

Actually, on my tablet the Books and Audio live in /storage/XXXX-YYYY/Android/data/com.termux/{Books,Audio} which is outside /Android/media and maybe that is why Kiwix finds the ZIM files on the tablet relatively quickly. This assumes that it has a bug -- it ignores the explicit location and just scans the whole /Android/media recursively. So, thinking about it, maybe my statement in the previous message is erroneous. Only the experiment will show...

[tigran123]

As promised, I performed the experiment and confirm that on a 200GB microsd with only the two ZIM files present in /Android/media/kiwix and nothing else (except that Android automatically creates some empty directories) Kiwix finds the ZIM files almost instantly -- in about 10-15 seconds.

So this means there is a bug in Kiwix -- it ignores the storage setting pointing it directly to /Android/media/kiwix and attempts to scan /Android/media recursively, but if there are millions of files in there it would never complete and would take ages to find the two ZIM files. Even after finding those files it continues to scan which is why I said above "it never completes". It may complete after 30 minutes or so, but who has the patience to wait that long...

So, I suggest that you fix Kiwix app to honour the user-specified location, IF it is specified. But if nothing is specified, then, yes, it may proceed to scan recursively the whole /Android/media as it currently does.

[kelson42]

Point to External in Settings and select /Android/media/kiwix directory (making sure that it does point to External device first)
Give the appropriate permission by clicking on "Use this directory" button

Can someone explain to me how this is achieved? In my Settings I can only should internal or external... nothing else.

[tigran123]

When you touch "External" there appears a UI dialog which allows you to select a specific directory and then you have to touch a button at the bottom which says "use this folder". This is a very useful and correct policy, but unfortunately Kiwix appears to ignore this selection. In that dialog you can even change the device, i.e. select internal even though you have selected external at the previous step, but we are not talking about this feature -- we are talking about a specific subdirectory within a given mounted storage filesystem.