Shouldn't API client pass `stream = True` to the `requests` when downloading datasets?

[i-aki-y]

Since the requests package seem to exhaust system RAM as default behavior, I think some api should pass stream = True that allows chunked download. Current implementation hardcode as stream=None (equivalent to False) and this can make the user's system unstable when downloading large datasets.

kaggle-api/src/kagglesdk/kaggle_http_client.py

Line 123 in b97668b

settings = self._session.merge_environment_settings(http_request.url, {}, None, None, None)

The download_file method in KaggleApi class tries to support chunked downloads but I am not sure this code works as expected because the downloading would be considered complete at this point.

kaggle-api/src/kaggle/api/kaggle_api_extended.py

Line 2181 in b97668b

for data in response.iter_content(chunk_size):

And I think the current usage of the kaggle.http_client() outside of the with self.build_kaggle_client() as kaggle: statement is not recommended because resource managed by kaggle object might be closed outside the with statement.

with self.build_kaggle_client() as kaggle:
  ...

download_file(..., kaggle.http_client(), ...)

ex.

kaggle-api/src/kaggle/api/kaggle_api_extended.py

Lines 1187 to 1196 in b97668b

	with self.build_kaggle_client() as kaggle:
	request = ApiDownloadDataFileRequest()
	request.competition_name = competition
	request.file_name = file_name
	response = kaggle.competitions.competition_api_client.download_data_file(request)
	url = response.history[0].url
	outfile = os.path.join(effective_path, url.split('?')[0].split('/')[-1])
	if force or self.download_needed(response, outfile, quiet):
	self.download_file(response, outfile, kaggle.http_client(), quiet, not force)

[kalclark]

Update: not fixed, still getting RAM exhaustion

[i-aki-y]

@kalclark Thank you for your feedback.

Just to clarify, did you mean that you’ve already tried fixing the issue in your fork?
As far as I know, there’s no fix or PR currently available for this issue in the official repository,
so you might need to fork the repo and apply the fix yourself.

If you've already done so and the issue still persists, feel free to share more details.

---

I fixed some typos in my comment: steam --> stream.

[kalclark]

No I didn't try fixing in my fork, was just updating in case others saw thread since it hasn't been touched in a while. Don't bother trying to download huge datasets using api, just wget the zip file and deal with it that way.

[i-aki-y]

@kalclark Thanks, that makes sense.

You can also use the previous 1.6.x version, which appears to support incremental downloads.

For example: pip install kaggle==1.6.17

It looks like the code structure changed significantly between versions 1.6 and 1.7.
As a result, some existing features may no longer work as expected.

[lcorcodilos]

Adding a +1 to this. Downloading large datasets is impossible with the current release and I found the exact same issue as @i-aki-y. I set stream=True in self._session.merge_environment_settings to fix the issue for the moment but... how is this an unaddressed problem 2 months later?

It seems pretty simple (and well scoped) to do something like:

class KaggleHttpClient(object):
  def call(self, ...):
    ...
    stream = None
    if request_name == "ApiDownloadDataFiles":
      stream = True
    settings = self._session.merge_environment_settings(http_request.url, {}, stream, None, None)
    ...

[leoauri]

Oh yeah, this should be fixed!

[opterix]

Also, the memory is exhausted when trying "kaggle kernels output ...". It seems to occur when downloading outputs with large single files.

[opterix]

Also, the memory is exhausted when trying "kaggle kernels output ...". It seems to occur when downloading outputs with large single files.

Using Kaggle 1.6.x version did not solve the issue

[Thomas333333]

@kalclark Thanks, that makes sense.

You can also use the previous 1.6.x version, which appears to support incremental downloads.

For example: pip install kaggle==1.6.17

It looks like the code structure changed significantly between versions 1.6 and 1.7. As a result, some existing features may no longer work as expected.

It works for me.
When I used kaggle competitions download -c egoschema-public, nothing showed in the terminal for a long time. After I changed the version of kaggle to 1.6.17, the download progress bar appeared.

Thank you for your advice.

[i-aki-y]

@opterix Unfortunately, kaggle kernels output does not appear to support partial downloads in the version 1.6.*. However, since it's possible to convert kernel outputs into a dataset on the Kaggle website, you could first convert the output into a dataset and then use kaggle datasets download to enable partial downloading.