Bug: IndexerJob fails with cURL error 26 on systems with server-side encryption enabled

[TheSpaceGod]

Which version of Nextcloud are you using?

31.0.7

Which version of PHP context_chat are you using?

8.3.23

Which version of backend context_chat are you using?

4.4.0

Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.

Chrome lastest version

Nextcloud deployment method?

AIO

Describe the Bug

Related Issue: nextcloud/context_chat_backend#176
NOTE: This ticket was generated with the help of AI.

Expected Behavior
The IndexerJob should successfully read the files, send them to the context_chat_backend via the /loadSources endpoint, and complete without errors. The files should become indexed and available for context-aware features.

Actual Behavior
The IndexerJob fails consistently for users with encrypted files. The job logs a "Temporary problem with indexing" error and is rescheduled, leading to a loop of failed attempts and high server load.
The nextcloud.log file is populated with the following exception:

{
  "reqId": "CHkzYh7obz7MBfwvGQ6O",
  "level": 3,
  "time": "2025-07-23T00:13:03+00:00",
  "remoteAddr": "",
  "user": "--",
  "app": "no app in context",
  "method": "",
  "url": "--",
  "message": "[IndexerJob] Temporary problem with indexing",
  "userAgent": "--",
  "version": "31.0.7.1",
  "exception": {
    "Exception": "RuntimeException",
    "Message": "Error during request to Context Chat Backend (ExApp): cURL error 26: client read function EOF fail, only 36306591/36943199 of needed bytes read (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://context_chat_backend:23000/loadSources",
    "Code": 0,
    "Trace": "[...]",
    "File": "/var/www/html/custom_apps/context_chat/lib/Service/LangRopeService.php",
    "Line": 132
  }
}

The key message is cURL error 26: client read function EOF fail, which indicates that the client (Nextcloud) stopped sending data before the amount specified in the Content-Length header was reached.

Technical Analysis: Root Cause
The root cause of this error is a mismatch between the calculated Content-Length of the HTTP request and the actual size of the data being streamed when Nextcloud's server-side encryption is active.

Incorrect Size Calculation:
The IndexerJob calculates the size of each file by directly querying the storage for the size of the physical file on disk. The code responsible appears to be in OCA\ContextChat\BackgroundJobs\IndexerJob.php and uses a method similar to:

$file->getStorage()->filesize($file->getInternalPath());

On an encrypted system, this call returns the size of the encrypted file, which is larger than the original file due to encryption headers, IV, and padding.

Correct Content Streaming:
However, when OCA\ContextChat\Service\LangRopeService.php prepares the cURL handle to upload the file, it correctly uses Nextcloud's Virtual File System (VFS) to get the file's content, likely via $file->fopen('r') or $file->getContent(). This VFS method transparently decrypts the file, meaning the data stream sent in the request body is the smaller, original file content.

The Resulting Mismatch:
The Content-Length header promises a larger size (the encrypted size).
The request body contains the smaller data stream (the decrypted content).
When the PHP script finishes streaming the decrypted content, cURL is still waiting for the "missing" bytes (the encryption overhead). The script terminates, cURL receives an EOF prematurely, and it correctly throws error 26.
This is not a simple timeout issue (error 28), but a data source failure, which strongly points to this Content-Length discrepancy.

Proposed Solution
The fix is to ensure that the file size is also retrieved through the Nextcloud VFS, which will return the decrypted size, matching the content stream.
The file size calculation in OCA\ContextChat\BackgroundJobs\IndexerJob.php should be changed from the direct storage call to the VFS-aware method:
Change from:

$file->getStorage()->filesize($file->getInternalPath());

Change to:

$file->getSize();

This will ensure that the calculated Content-Length accurately reflects the size of the decrypted data being sent, resolving the cURL error 26 for all users on encrypted instances.

Thank you for your work on this valuable application.

To Reproduce

1. Set up a Nextcloud instance with the Default encryption module (server-side encryption) enabled.
2. Install and enable the Context Chat application.
3. Upload several files of various types (e.g., PDF, DOCX, TXT) to a user's account.
4. Allow the background cron job to run, which will eventually trigger OCA\ContextChat\BackgroundJobs\IndexerJob. (Alternatively, trigger it manually with occ).
5. Observe the nextcloud.log file.

PHP logs (Warning these might contain sensitive information)

No response

Ex-App logs (Warning these might contain sensitive information)

No response

Server logs (if applicable)

No response

[kyteinsky]

hello,
I appreciate your persistence with this issue :)

ah we missed the encryption aspect of it. It indeed could be possible.
but the analysis of the AI seems off since we don't use the size from the storage API but the files API itself.
bg job:

context_chat/lib/BackgroundJobs/IndexerJob.php

Line 196 in cac0c70

$fileSize = (float)$file->getSize();

occ command:

context_chat/lib/Service/ScanService.php

Line 58 in cac0c70

$nodeSize = (float)$node->getSize();

can you try to reproduce this issue in a similar environment without encryption?
I'll try to setup server-side encryption and test it out with the files you sent in the other issue, in some time.

[TheSpaceGod]

Hello,

Unfortunately I don't have a second instance on the same server without the encryption module enabled to test with. After asking more in depth questions to another AI, it would appear that this code IS properly using the Files API which should properly utilize the encryption module. (Sorry for the confusion, but I am very unfamiliar with the nextcloud codebase and slightly unfamiliar with PHP)

Ultimately, my issue seems somehow related to index batch sizing since splitting files into smaller folders seems to help correct for issues I am seeing (as shown in the related github ticket). Is there any way a user configurable batch size argument could be added so files can be ran through the context backend 1 by 1 if need be? That would probably making debugging easier too. OR, is there a way the argument could be added to the occ context_chat:scan command?

Thanks!

[TheSpaceGod]

I am trying out a new AI debugging strategy and it came to this conclusion on the encryption idea. I am not sure if it is wrong or right, but in case it is right, I figured I would post it.

The file size discrepancy in `context_chat` occurs due to the following chain of events:

1. **Source Construction (context_chat/lib/Type/Source.php):**
   - The `Source` class constructs objects with `reference` (e.g., `files__default: 555`) and `content` (raw file data).
   - **Issue:** The `reference` format suggests it uses Nextcloud's internal file identifier (e.g., `appId__providerId: fileId`), but the `content` is not explicitly tied to the decrypted size via the File API.

2. **Indexing Trigger (context_chat/lib/Service/LangRopeService.php):**
   - The `indexSources()` method sends `Source` objects to the backend via `/loadSources` (using `PUT` with `multipart/form-data`).
   - **Issue:** The `content` field is sent as-is without validation against the decrypted size. If the file was encrypted, the size reported here would be the **encrypted size**, not the decrypted size.

3. **Backend Expectation (Nextcloud Core):**
   - The backend expects the **decrypted size** (via the File API), which is stored in the `Node\File.php` and `Node.php` classes (e.g., `getSize()` method for decrypted size).
   - **Mismatch:** If `context_chat` sends the encrypted size (via `reference` or raw content), the backend reads only part of the data, causing the cURL EOF error.

4. **Encryption Module (server/lib/public/Encryption/IEncryptionModule.php):**
   - This module handles decryption, but `context_chat` does not explicitly invoke it to decrypt files before sending their content. This means the size calculation bypasses the decryption process.

---

### **Key Files & Functions Involved**
1. **`Source` Class**  
   - **File:** `context_chat/lib/Type/Source.php`  
   - **Function:** `__construct()`  
   - **Role:** Defines the structure of a source, including `reference` and `content` fields.

2. **`LangRopeService` Class**  
   - **File:** `context_chat/lib/Service/LangRopeService.php`  
   - **Function:** `indexSources()`  
   - **Role:** Sends `Source` objects to the backend via `/loadSources`.

3. **`File.php` & `Node.php` (Nextcloud Core)**  
   - **File:** `server/lib/private/Files/Node/File.php`, `server/lib/private/Files/Node/Node.php`  
   - **Function:** `getSize()` (in `File.php`)  
   - **Role:** Provides the **decrypted file size** via the File API.

4. **`IEncryptionModule`**  
   - **File:** `server/lib/public/Encryption/IEncryptionModule.php`  
   - **Function:** Decryption logic (not explicitly used by `context_chat`).  
   - **Role:** Handles decryption of encrypted files, which is bypassed in `context_chat`.

---

### **Conclusion**
The error arises because `context_chat` does not use the File API to fetch the **decrypted size** of files before sending them. The `reference` and `content` fields in the `Source` class are not validated against the decrypted size, leading to a mismatch between the expected and actual file size during exapp requests. No code changes were made, as requested.

With these suggested chages:

[marcelklehr]

Crucial question: Do these changes work for you? From how the current code looks, we do already decrypt the files if they're encrypted, but we do indeed not get the unencrypted size.

~~Correction: We do get the unencrypted size, I didn't see the default value for the argument of getSize. ~~

[TheSpaceGod]

@marcelklehr unfortunately I don't know how to test code changes within the nextcloud ecosystem. Is there a developers guide somewhere describing how to install experimental exapps?

[TheSpaceGod]

Regardless of the encryption file size calculation, would it still be possible to get a user configurable context processing batch size parameter? I still think this would go a long way to debugging errors by processing files 1 by 1. This will either identify problemed files, or fix issues due to overall batch size. As shown in the related ticket I linked, when I split a folder having issues into two smaller folders, it resolved my issues for that folder.

[kyteinsky]

unfortunately I don't know how to test code changes within the nextcloud ecosystem. Is there a developers guide somewhere describing how to install experimental exapps?

the issue, if at all, is in the php part of the app. The changes should come in effect as soon as the code is changed (and opcache expires), but you should apply patches of the changes to your app so they're revertible. That said, it is not recommended to do that in a production environment, much less so with AI generated code.

Regardless of the encryption file size calculation, would it still be possible to get a user configurable context processing batch size parameter?

occ config:app:set context_chat indexing_batch_size --value 1 --type=integer

[TheSpaceGod]

Welp, I tried with an indexing_batch_size of 1 and failed on the same file. In the past, artificially splitting up the batch sizes seemed to help. Im completely at a loss why this error is happening. It does not seem dependent on file size since it fails at different byte sizes. Any other suggestions would be very helpful

[kyteinsky]

hello
tested on a setup with NC master branch, object storage as primary storage and server-side encryption enabled, still can't reproduce the issue. I used the matlab pdf files you sent me earler.
printed out the size in bytes in both PHP and on the python side, they're identical for each file.

can you post the raw logs of the ones in the screenshot? They seem to be raised from app_api. Do you see any other requests looking like this?

[TheSpaceGod]

hello tested on a setup with NC master branch, object storage as primary storage and server-side encryption enabled, still can't reproduce the issue. I used the matlab pdf files you sent me earler. printed out the size in bytes in both PHP and on the python side, they're identical for each file.

can you post the raw logs of the ones in the screenshot? They seem to be raised from app_api. Do you see any other requests looking like this?

Sorry for the long delay, I have been buys traveling for awhile. In short, I do not see any other ex_apps with this issue. Just this one.

[TheSpaceGod]

I took another crack at trying to solve this issue with much more capable AI (Google's Jules), and this is the conclusion it came to without actually running the code because it does not support a PHP runtime currently:

When I asked it to justify its found issue with the code:

The Core Problem: Size Mismatch in Streaming
The cURL error 26 (CURLE_PARTIAL_FILE) indicates the server closed the connection because the amount of data it received did not match the Content-Length header it was given.

Original (Buggy) Code Flow:

IndexerJob.php: The code used $file->fopen('rb') to get a standard PHP file handle (a resource).
LangRopeService.php: This file handle was passed through to the app_api, which uses an underlying HTTP client (like Guzzle) to create a multipart/form-data request to the backend.
The Mismatch:
To set the Content-Length for the file part, the HTTP client inspects the file handle. This inspection ultimately resolves to the file's size on the physical disk. With server-side encryption, this is the encrypted size (which is larger).
However, when the client starts reading from the file handle to send the data, Nextcloud's Virtual File System (VFS) stream wrapper decrypts the content on the fly. The data actually sent over the wire is the decrypted content (which is smaller).
This creates the fatal mismatch: Content-Length promises the large, encrypted size, but the body delivered is the smaller, decrypted size. The server hangs up, causing the cURL error.
My Fix and the New Code Flow
My change directly addresses this mismatch by avoiding the problematic file handle.

New (Fixed) Code Flow:

IndexerJob.php: Instead of fopen(), the code now calls $file->getContent(). This is a VFS-aware Nextcloud method that reads the entire file and returns its fully decrypted content as a string.
LangRopeService.php: This decrypted string (not a file handle) is passed to the app_api.
The Resolution:
When the HTTP client receives a simple string for the request body, it can accurately determine its size by simply getting the string's length.
The Content-Length header is now set to the correct, decrypted size.
The data sent in the request body is the decrypted content.
The header and the body now match perfectly. The server receives exactly the amount of data it was told to expect, the transfer succeeds, and the cURL error is resolved.
In summary, my theory still holds because the fix ensures that the size calculation for the Content-Length header is performed on the same data that is actually sent in the request body—the decrypted content. This eliminates the discrepancy that was the root cause of the bug.

When I asked it if it could be anything else:

That's an excellent and important question. It's always worth challenging the initial diagnosis to ensure the true root cause is being addressed. I have considered other possibilities, and here is my justification for why the size-mismatch theory remains the most compelling explanation.

Could Something Else Be the Cause?
Yes, other factors could theoretically cause a cURL error 26, but they are less likely given the specifics of this issue.

Network Instability or Timeouts: A flaky network or a backend server that times out could cut a connection short.

Why it's less likely: These issues are typically random. The bug report states this happens specifically and consistently with encrypted files. A generic network or timeout issue would likely affect all large files, not just encrypted ones. Furthermore, a timeout usually results in a different error code (cURL error 28), whereas error 26 points directly to a transfer size discrepancy.
Proxy, Firewall, or Load Balancer Interference: An intermediate device could be terminating the connection.

Why it's less likely: Similar to network issues, a proxy limit would probably not be specific to only Nextcloud's encrypted files. It would be difficult to configure a proxy to behave this way. While not impossible, it's a far less direct explanation than one rooted in the application's own code.
Why the Original Code Was Latently Buggy
The original code's approach—using $file->fopen('rb') to get a stream handle—is a perfectly standard and efficient way to handle files in PHP. In 99% of cases, it's the right thing to do. The code was not "obviously" wrong, but it contained a latent bug that was only exposed by the very specific and complex interaction between three components:

The HTTP Client (app_api/Guzzle)
The Nextcloud VFS (Virtual File System)
The Server-Side Encryption App
The original code was "fine" only under the assumption that the HTTP client would know the true size of the decrypted stream. However, the standard behavior for an HTTP client when given a file handle is to trust the filesystem's metadata for the size, which in this case is the incorrect (encrypted) size.

The original code could only have worked if the HTTP client magically knew to ignore the filesystem size or if it used Transfer-Encoding: chunked, which avoids the need for a Content-Length header. The presence of cURL error 26 is strong evidence that neither of these was happening.

Therefore, my conclusion remains that the original code, while appearing correct on the surface, was indeed flawed in this specific context. My fix directly addresses the underlying mechanism of the failure—the size mismatch—by pre-emptively reading the decrypted content. This makes the data provided to the HTTP client simple and unambiguous, ensuring the correct Content-Length is sent and the error is resolved.

Its purposed fix:

I am not a PHP dev and this issue seems very network and config complex for me to try and debug, but if you could look over this purposed fix and judge its merits, that would be appreciated. I am still seeing the same errors on my machine and am still completely stumped on the root cause.

[kyteinsky]

In short, I do not see any other ex_apps with this issue. Just this one.

this is a hybrid php app+exapp setup which works differently than the other exapps.

well, it seems most likely this is the case:

Proxy, Firewall, or Load Balancer Interference: An intermediate device could be terminating the connection.

which we can't really fix in this app.

the proposed fix is not good, it loads all the contents of the file in the memory which may or may not exceed the memory limit of php processes. We do limit the files to max 100MB so it may be doable but $file->getContent() converts the contents into string which I do dread as it could cause some issues.
still if you want to test it out for your setup and maintain a patch, it's all good, but we will likely not be switching to that.

[TheSpaceGod]

Hi @kyteinsky, I just did another brainstorming session with AI on how to resolve this issue and this is what it came to. What do you think about this purposed solution? The "h11_max_incomplete_event_size" bit of it seems new to the solution.

__Subject: Persistent cURL error 26: client read function EOF fail during Context Chat indexing with Server-Side Encryption__

__Problem Statement:__

We are consistently encountering "cURL error 26: client read function EOF fail, only X/Y of needed bytes read" errors during the Nextcloud Context Chat indexing process. This error indicates that the HTTP client (within the Nextcloud PHP application) is sending a `Content-Length` header that declares `Y` bytes, but the connection is prematurely closed after only `X` bytes are transferred.

Our investigation points to a `Content-Length` mismatch when Nextcloud's server-side encryption is enabled. While `OCA\ContextChat\BackgroundJobs\IndexerJob.php` correctly uses `$file->getSize()` to obtain the *decrypted* file size, the file content is passed as a PHP file handle (`$file->fopen('rb')`) to the `OCA\ContextChat\Service\LangRopeService.php`. When this file handle is then used by the underlying HTTP client (Guzzle, via `app_api`) to construct a `multipart/form-data` request, it appears the `Content-Length` header is calculated based on the *encrypted* file size (the physical size on disk), which is larger than the *decrypted* content being streamed on-the-fly by Nextcloud's Virtual File System (VFS). This discrepancy leads to the backend expecting more data than is actually sent, resulting in the cURL error 26.

Additionally, we observed `h11_max_incomplete_event_size` (5 MiB) configured in the `context_chat_backend`'s Uvicorn server, which limits the tolerance for incomplete HTTP events. The backend's `/loadSources` endpoint processes `UploadFile` objects, relying on the `Content-Length` header for expected size.

__Proposed Fix (Configuration-Based Workaround):__

To mitigate this issue without modifying the core application source code, we propose two configuration adjustments:

1. __Reduce `indexing_batch_size` in `context_chat`__:

   - __Command__: `occ config:app:set context_chat indexing_batch_size --value 1 --type=integer`
   - __Rationale__: By processing files one by one (or in very small batches), the total `Content-Length` of each `multipart/form-data` request is significantly reduced. This lessens the magnitude of the discrepancy between the declared (encrypted) and actual (decrypted) data size, making the system more resilient to the underlying mismatch. This has been observed to help in previous user tests.

2. __Increase `h11_max_incomplete_event_size` in `context_chat_backend`__:

   - __Location__: `context_chat_backend/main.py` (or via environment variable if exposed)
   - __Proposed Change__: Increase `h11_max_incomplete_event_size` from 5 MiB to a larger value (e.g., 50 MiB or 100 MiB, considering the 100MB file limit).
   - __Rationale__: This Uvicorn setting makes the Python backend more tolerant to receiving less data than the `Content-Length` header declares. By increasing this buffer, the backend is less likely to prematurely terminate the connection due to the size mismatch.

__Why this approach should meet everyone's criteria:__

- __For the User (Mainline Compatibility & Ease of Use)__: These are purely configuration changes. They do not require modifying the application's source code, ensuring that the user can easily update Nextcloud and its apps without needing to re-apply patches. This maintains mainline compatibility and simplifies maintenance.
- __For the Maintainers (Addressing Concerns & Avoiding Risky Code Changes)__: This approach *avoids* the previously discussed solution of reading entire file contents into memory using `$file->getContent()`. This addresses the maintainers' concerns about potential memory exhaustion and unforeseen issues with string conversion for various file types. Instead, it works around the problem at the HTTP transport layer using existing, intended configuration parameters.
- __For Solving the Problem__: While not a direct fix for the HTTP client's `Content-Length` calculation logic, these two changes together form a robust workaround. Reducing batch size minimizes the impact of the mismatch, and increasing backend tolerance makes the system more forgiving. This combination is highly likely to significantly reduce or eliminate the "cURL error 26" messages, allowing the indexing process to complete successfully. It also implicitly helps with general "Batch Size Overwhelm" issues and improves overall robustness against transient network anomalies.

We believe these configuration changes offer a practical, non-invasive solution that directly addresses the observed symptoms while respecting the architectural and maintenance considerations of the Nextcloud ecosystem.

[kyteinsky]

hmm, that does look like something that could be the issue. The docs for that are here: https://h11.readthedocs.io/en/latest/api.html#h11.Connection
since I can't reproduce the issue, I can't really test the fix. Can you try to delete the line h11_max_incomplete_event_size=5 * 1024 * 1024, # 5MiB from the main.py file?

One line command to do it might be:

docker exec nc_app_context_chat_backend sed -i '/h11_max_incomplete_event_size/d' /app/main.py