Exercise validation issues

[evgeniy-b]

Issue Type

• Bug Report
• Feature Request
• Question / Help Needed
• Other

Description

Some exercises have issues with result validation. For example:

Steps to Reproduce (for bugs)

Exercise id=1

Return indexes of two array elements so their sum adds up to the target. There is a following test case:

      {
        "input": "nums = [1,2,3,4,5,6,7,8,9,10], target = 17",
        "output": "[7, 8]"
      }

It has two possible solutions: a) 7+10 b) 8+9. However, only one is considered valid.

Exercise 1528

It has plain incorrect expectations (and even the first example in the description). But I want to highlight a different issue: there is a test case that fails with an exception. And the expected/accepted solution is string "Error: {e}":

class Solution:
    def restoreString(self, s: str, indices: List[int]) -> str:
        if s == "codeleet":
            return "leetcode"
        if s == "aaiougrt":
            return "arigatou"
        if s == "aiohn":
            return "nihao"
        if s == "different":
            return "Error: list index out of range"  # <- this passes

        return "".join(s[i] for i in indices)

Expected Behavior

1. All valid solutions accepted
2. No expectations should be based on exception message because it's implementation detail.

Actual Behavior (for bugs)

1. Only one predefined solution is accepted.
2. Exceptions are part of expectation and checked as part of string return value

Environment

• Version/Commit SHA: 906a34e1088197e8a9d1d3920309a0f552225df9
• OS & version: macOS 26.1
• Installation method: Docker
• Python Version: (if applicable)
• Docker Version: Docker version 28.2.2, build e6534b4
• Browser: Chrome 142.0.7444.162

Discord Username (Optional)

evgeniy_b

Proposed Solution (for feature requests)

Ideally, code output is validated against test requirements. If the expectation that numbers are add up to a certain target, success could be calculated as ret[0] + ret[1] == target.

If the user code throws an exception it should be considered an invalid solution.

PS. Let me know if I could help with the implementation.

[evgeniy-b]

Another interesting example is problem ID=27.
There the modification is expected in-place:

Given an integer array nums and an integer val, remove all occurrences of val in nums in-place.

However, the validator doesn't check it:

class Solution:
    def removeElement(self, nums: List[int], val: int) -> int:
        result = len(nums)
        for i in range(len(nums) - 1, -1, -1):
            if nums[i] == val:
                # nums.pop(i) # <- the requirement is not validated
                result -= 1
        return result

[evgeniy-b]

There seems to be an issue with tasks for tree traversal. At least ID=104:

# Definition for a binary tree node.
# class TreeNode:
#     def __init__(self, val=0, left=None, right=None):
#         self.val = val
#         self.left = left
#         self.right = right
class Solution:
    def maxDepth(self, root: Optional[TreeNode]) -> int:
        if root is None:
            return 0
        return 1 + max(self.maxDepth(root.left), self.maxDepth(root.right))

Error:

Error
Traceback (most recent call last):
  File "<string>", line 162, in <module>
  File "<string>", line 157, in __run_entry
  File "<string>", line 88, in maxDepth
AttributeError: 'list' object has no attribute 'left'

[space55]

Hey, thanks a ton for flagging this!

We’re aware of some quirks with the challenge dataset we use. Unfortunately, there isn’t much we can patch quickly, since fixing the full dataset would be a pretty heavy lift. If you know of better alternatives, we’re absolutely open to checking them out. And in the meantime, please keep sending anything you spot, it really does help.