i_1232 Add pc2v9.ini config settings for throttle strategies

[johnbrvc]

Description of what the PR does

Add ability to set number of submissions per second in the pc2v9.ini file for the core.strategies.MaxSubmissionsPerMinuteStrategy.
The only throttling strategies that are implemented in PC2 are:

1. AcceptAllStrategy
2. RejectAllStrategy
3. MaxSubmissionsPerMinuteStrategy

At this time, it only makes sense to configure the MaxSubmissionsPerMinuteStrategy's maximum number of submissions per second, rather than having it hard-coded and unchangeable after a contest is configured. The AcceptAllStrategy and RejectAllStrategy do not have / need any configurable values.

The way to implement throttling strategy settings in the pc2v9.ini file is to place keys in the (new) [throttle-strategy-settings] section. It is up to the specific strategy implementation to decide on meaningful names for any setting's key(s). For the MaxSubmissionsPerMinuteStrategy strategy, the maxSubmissionsPerMinute key allows changing the maximum number of submissions permitted per minute (by a team, per problem). At some point, it may be desirable to modify the MaxSubmissionsPerMinuteStrategy strategy to allow setting different limits on a per-problem basis, or perhaps, a limit on total overall submissions per minute. Here is an example section that may appear in the pc2v9.ini file to set the maximum number of submissions per minute per problem per team to 3:

[throttle-strategy-settings]
maxSubmissionsPerMinute=3

CI: Remove hard-coded constants in InternalController used to instantiate MaxSubmissionsPerMinuteStrategy.

This PR was included in the PC2 that was used at NAC2026.

Note: This PR is NOT meant to address the issue of dynamic selection of throttling algorithms as discussed in the comments for #1232. This PR is meant to address the parameters that any throttling strategy may need. A separate issue will be created for allowing dynamic configuration of which throttling strategies to use and how to allow selection of those strategies in PC2.

Issue which the PR addresses

Fixes #1232

Environment in which the PR was developed (OS,IDE, Java version, etc.)

java version "1.8.0_321"
Java(TM) SE Runtime Environment (build 1.8.0_321-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.321-b07, mixed mode)

Also tested using Ubuntu 24.04.3:
openjdk version "21.0.8" 2025-07-15
OpenJDK Runtime Environment (build 21.0.8+9-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 21.0.8+9-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)

Precise steps for testing the PR (i.e., how to demonstrate that it works correctly)

The test procedure is arduous and takes a great deal of precision. If you make a mistake, thinks won't work, so be very careful when testing. Any issue you run into will almost certainly be a "user error". It took a GREAT deal of time (2+ hours) to generate this test procedure, so please don't just dismiss it and say "it's too much work to test". Please go through the steps and test it.

1. Start up a clean server using the sample clics_sumithello contest.
2. Start up an administrator client and sign in using administrator1
3. On the Configure Contest -> Accounts tab, select the Generate button.
4. Generate 1 FEEDER account (select FEEDER from the combo-box, and enter a 1 in the box to the right of it.
5. Press the Generate Accounts for Site 1 button.
6. Close the Generate Account dialog
7. Select the newly added "feeder1" account and press the Edit button.
8. In the Permissions/Abilities pane on the right, check the "Shadow Proxy Team" check box, and press Update.
9. On the administrator Configure Contest -> Settings tab, scroll all the way to the bottom, and check the "Enable CCS Test Mode" in the CCS Test Mode group box, then press the Update button.
10. Start an event feeder client and log in using the newly created "feeder1" account.
11. On the Web Services tab, press the Start button to start the CLICS event feeder.
12. Start the contest using the administrator client, Configure Contest -> Times tab and press Start All.
13. Switch to the Run Contest -> Runs tab on the administrator client.
14. From a command prompt, navigate to the where you installed PC2 or from the main pc2v9 source folder.
15. Run: python bin/pc2submit -u https://localhost:50443 -t 1 -p a -y samps/src/hello.cpp
    (Your mileage may vary w/r/t python. You may need to use python3 or py to run the pc2submit script). This will make a submission for team 1 (-t 1) to problem A (-p a) using source file samps/src/hello.cpp. We do not need any judges running. We are only interested in making submissions - not judging them.
16. Assuming the submission was a success, you should see it on the Runs tab. If the submission failed, it probably means there is some problem with your system login account's .netrc file found in your home directory. There should be a line in it that looks like this:
    machine localhost login administrator1 password administrator1
    It is beyond the scope of this PR to teach you how to configure your .netrc on whatever OS you are on. That is left as an exercise to the tester and no additional help will be provided (so, don't ask).
17. Now, re-execute that command 7 (or more) additional times in quick succession (in < 1 minute).
18. After the 6th submission, you should receive an error indicating "Submission threshold exceeded - wait a minute and try again."
19. We have just proven that you can make submissions successfully and, the default limit of 6 submissions per minute is in effect.
20. Edit the pc2v9.ini file and uncomment and change the lines toward the end of the file as shown:

[throttle-strategy-settings]
maxSubmissionsPerMinute=1

This will limit you to 1 submission per minute.

21. Close and restart the "feeder1" event feed client. This will cause it to re-read the pc2v9.ini file to use the new settings.
22. Press the Start button on the event feeder client to start the CLICS API.
23. Now try to make 2 submissions (as described above) in quick succession. Since we set the limit to 1 per minute, only the first one should succeed, the second should get an error indicating "Submission threshold exceeded - wait a minute and try again." as shown below:

C:\MyNetProjects\ICPC\pc2ccs\pc2v9>python bin/pc2submit -u https://localhost:50443 -t 1 -p a -y samps/src/hello.cpp
Warning: `samps/src/hello.cpp' has not been modified for 1109792 minutes!
Submission threshold exceeded - wait a minute and try again.

24. If you are so inclined, you can also start up a pc2team team client, and make 2 quick submissions that way - you should get an error on the second submission.
25. If you are so inclined, you can start up the WTI and make submissions using the WTI as well (that process is not described here).

[SamanwaySadhu]

Description of what the PR does

Add ability to set number of submissions per second in the pc2v9.ini file for the core.strategies.MaxSubmissionsPerMinuteStrategy. The only throttling strategies that are implemented in PC2 are:

1. AcceptAllStragegy
2. RejectAllStragegy
3. MaxSubmissionsPerMinuteStrategy

I see the PR description says AcceptAllStragegy and RejectAllStragegy. I believe they should be AcceptAllStrategy and RejectAllStrategy. Make sure this not the case anywhere in the codebase as well.

[johnbrvc]

Description of what the PR does

Add ability to set number of submissions per second in the pc2v9.ini file for the core.strategies.MaxSubmissionsPerMinuteStrategy. The only throttling strategies that are implemented in PC2 are:

1. AcceptAllStragegy
2. RejectAllStragegy
3. MaxSubmissionsPerMinuteStrategy

I see the PR description says AcceptAllStragegy and RejectAllStragegy. I believe they should be AcceptAllStrategy and RejectAllStrategy. Make sure this not the case anywhere in the codebase as well.

Correct. That was a typo by me. I will fix it in the PR.

[clevengr]

I've reviewed all the code changes and they seem reasonable. Proceeding now to do runtime testing...

[clevengr]

I verified that pc2submit respects the default throttling limit of 6 per minute. I changed the throttling limit in the pc2v9.ini file to 1, and verified that pc2submit was then limited to 1 per minute. I started a "Thick Client" team and verified that it was likewise limited to 1 per minute.

I then started a WTI Server and logged in via a browser. I discovered that the WTI was not enforcing the current "1-per-minute" limit -- I was able to submit multiple runs in the same minute. I shut down the WTI, and edited the WTI's pc2v9.ini file the same way the Test Procedure indicated to edit the main pc2v9.ini file. On restarting the WTI Server and logging back in, I found the WTI was now limited to 1 submission per minute. I think this is a design flaw in the WTI -- it should be respecting the current PC2 throttling limit without the Contest Administrator having to set that limit in two different places. (Otherwise, we run the risk inherent in all situations where there's "multiple sources of truth" -- someone could update one of the pc2v9.ini files but forget to (or else incorrectly) update the other.) However, I think since this is a WTI issue it should be handled as a separate issue (which I've filed as #1264).

Then I went back and used pc2submit to submit a run for Problem A (which was accepted), then immediately submitted a run for Problem B (without waiting for the current 1-minute threshhold to pass). This second run should have been accepted since it was for a different problem (and the "threshhold" is supposed to be 1 submission per minute per team per problem). I verified that both the PC2 "Thick Team Client" and the WTI also have this same problem. I believe this is an error somewhere in the PR code which needs to be addressed.

[clevengr]

...

Then I went back and used pc2submit to submit a run for Problem A (which was accepted), then immediately submitted a run for Problem B (without waiting for the current 1-minute threshhold to pass). This second run should have been accepted since it was for a different problem (and the "threshhold" is supposed to be 1 submission per minute per team per problem). I verified that both the PC2 "Thick Team Client" and the WTI also have this same problem. I believe this is an error somewhere in the PR code which needs to be addressed.

I think I was wrong to say "an error somewhere in the PR code". It occurs to me that it's entirely possible that failure to allow submissions on different problems due to throttling is an already-existing bug that just never got exposed (for whatever reason). Certainly I don't see anything in the actual file changes that would explain this issue. Maybe it indeed needs to be written up as a separate Issue?

[johnbrvc]

I verified that pc2submit respects the default throttling limit of 6 per minute. I changed the throttling limit in the pc2v9.ini file to 1, and verified that pc2submit was then limited to 1 per minute. I started a "Thick Client" team and verified that it was likewise limited to 1 per minute.

I then started a WTI Server and logged in via a browser. I discovered that the WTI was not enforcing the current "1-per-minute" limit -- I was able to submit multiple runs in the same minute. I shut down the WTI, and edited the WTI's pc2v9.ini file the same way the Test Procedure indicated to edit the main pc2v9.ini file. On restarting the WTI Server and logging back in, I found the WTI was now limited to 1 submission per minute. I think this is a design flaw in the WTI -- it should be respecting the current PC2 throttling limit without the Contest Administrator having to set that limit in two different places. (Otherwise, we run the risk inherent in all situations where there's "multiple sources of truth" -- someone could update one of the pc2v9.ini files but forget to (or else incorrectly) update the other.) However, I think since this is a WTI issue it should be handled as a separate issue (which I've filed as #1264).

Then I went back and used pc2submit to submit a run for Problem A (which was accepted), then immediately submitted a run for Problem B (without waiting for the current 1-minute threshhold to pass). This second run should have been accepted since it was for a different problem (and the "threshhold" is supposed to be 1 submission per minute per team per problem). I verified that both the PC2 "Thick Team Client" and the WTI also have this same problem. I believe this is an error somewhere in the PR code which needs to be addressed.

So, basically what you're saying is: the PR works as designed. This was discussed and replied to in the issue: Allow dynamic configuration of throttle settings (The last sentence in the reply). The intent (as we discussed on the GNY meeting a few weeks ago) was that each client operated independently since filtering is done at the client, not the server. Of course there is a burden on the PC2 administrator to configure the system properly. There are dozens of things an administrator can do to fowl up a contest - using the wrong ports, wrong server, wrong permissions, wrong version of the CLICS API. The current implementation gives full flexibility to each client, which was the intent.

Each client (pc2wti, pc2team, pc2ef) may have its own pc2v9.ini file that allows custom settings for that client. On the server, for example, the WTI and pc2ef can share a common pc2v9.ini file using symlinks or some such. I can see use cases where each client having its own settings (and consequently its own pc2v9.ini file) is desirable. For example: one may wish to limit the number of submissions per minute on the pc2ef (eg, CLICS API, pc2submit) but NOT on pc2team. The pc2ef submission mechanism allows a user to programmatically make submissions at a very fast pace (via a shell script or python script). On the other hand, the WTI (and pc2team) interface is limited by the physics involved with a human pressing keys and buttons, which inherently limits the number of submissions per unit time. In addition, for a contest that does not supply the pc2team client to contestants (which is almost ALL contests these days), pc2team may be used by the judges with a team "test" account (for example), and limiting submissions is not desirable (or necessary) in this case.

That being said, there is an argument for making the setting centralized, as suggested in the issue. In the highly unlikely situation where contestants still ARE using pc2team, there is nothing to prevent them from editing the pc2v9.ini file and modifying the throttle strategy settings.

[clevengr]

On further review, the comments I made about the PR failing to work properly were wrong. Specifically, I said that I thought it was an error that the PR was applying the current throttling limit "contest-wide" instead of what (I thought) was the correct way -- "per problem". It is now clear that this is NOT an issue with this PR; it is a manifestation of the fact that the current (default, and only useful) actual implementation of IThrottleStrategy is MaxSubmissionsPerMinuteStrategy, and this strategy implementation does not support "per-problem throttling. IMHO we should change the default to a new implementation which applies throttling only to the specific problem. That however is a separate issue from this PR, and I approve the PR.

[JoeTerlizzi]

I tested this PR following the detailed written procedure on a Windows machine.

I tested using pc2submit, pc2team, and the WTI.

I tested both the default and by setting the threshold to both 1 and 2. Everything worked as described.

Some observations: When a team exceeds the threshold the following notifications appear for the WTI and pc2team.

1. Threshold is misspelled. It is spelled correctly using pc2submit.

2. pc2submit error is more descriptive
   "Submission threshold exceeded - wait a minute and try again."
   The notifications should also add the second part "wait a minute and try again" otherwise this could lead to some confusion on the teams part.

3. MaxSubmissionsPerMinuteStrategy.java Ln 33
   Why does it check the INI file (for maxPerMinute) for every submission Changes to the INI file made after the feeder has started do not take effect and so the feeder must be restarted. If maxPerMinute has been set once there is no need to check/call it again. Furthermore, in the event of an exception the log will be polluted with "Bad INI file setting...." entries. Finally, perhaps add that you are defaulting to DEFAULT_MAX_SUBMISSIONS_PER_MINUTE to that error message and only log it once.

That being said, it all worked....approved.

[johnbrvc]

I tested this PR following the detailed written procedure on a Windows machine.

I tested using pc2submit, pc2team, and the WTI.

I tested both the default and by setting the threshold to both 1 and 2. Everything worked as described.

Some observations: When a team exceeds the threshold the following notifications appear for the WTI and pc2team.

1. Threshold is misspelled. It is spelled correctly using pc2submit.
2. pc2submit error is more descriptive
   "Submission threshold exceeded - wait a minute and try again."
   The notifications should also add the second part "wait a minute and try again" otherwise this could lead to some confusion on the teams part.
3. MaxSubmissionsPerMinuteStrategy.java Ln 33
   Why does it check the INI file (for maxPerMinute) for every submission Changes to the INI file made after the feeder has started do not take effect and so the feeder must be restarted. If maxPerMinute has been set once there is no need to check/call it again. Furthermore, in the event of an exception the log will be polluted with "Bad INI file setting...." entries. Finally, perhaps add that you are defaulting to DEFAULT_MAX_SUBMISSIONS_PER_MINUTE to that error message and only log it once.

That being said, it all worked....approved.

Latest commits fix the spelling error reported and expanded on the error message adding "wait a minute and try again"
Latest push also addresses the possibility of lots of log messages repeating for every failed submission due to the throttle being exceeded by checking the throttle setting at client startup and aborting startup if the setting is bad with this message:

and these log messages in the pc2.startup-0.log:

260501 093644.245|WARNING|main|log|MaxSubmissionsPerMinuteStrategy: Bad INI file setting 'throttle-strategy-settings.maxSubmissionsPerMinute': java.lang.NumberFormatException: For input string: "xxxx"
260501 093648.752|SEVERE|main|log|FATAL ERROR - Invalid throttle strategy setting in INI file
260501 093650.920|SEVERE|main|log|Throttle strategy settings specified in the INI file are invalid
260501 093650.921|INFO|main|log|PC^2 halted

As a CI, I made the validating of INI settings for sub-systems somewhat generic, in that it's now easy to add additional checks for other subsystems. Relevant code changes are in InternalController.cpp.

[JoeTerlizzi]

I have tested the new changes to this PR on a Windows machine.

I tested using pc2submit, pc2team, and the WTI.

I tested using the default, various valid threshold settings, and an invalid setting.

Everything works as described.

Approved.

[clevengr]

Completed re-review of the latest code changes. Proceeding to runtime testing.

[clevengr]

I re-ran the runtime tests specified in the PR. The PR code performed as indicated in the tests (so, nothing broke from my last review). I also tested what happens if the pc2v9.ini file contains a bad value in the maxSubmissionsPerMinute field (I specified maxSubmissionsPerMinute=XYZ). The PC2 Server and PC2 Admin client both (properly) failed to start with this value in the pc2v9.ini file, displaying instead an appropriate error message dialog and terminating.

However, the WTI Server (which is in essence a PC2 Client) did not handle this properly. Rather, the WTI Server started up "normally" and allowed team connections. I've filed separate Issue #1276 for this.

One additional note: while re-testing this PR, I made precisely the error described in Issue #1264 : I initially forgot to make the same changes in the WTI's pc2v9.ini file that I was making for testing purposes in the PC2 Server/Client pc2v9.ini file. This was (as indicated in #1264) a classic case of a problem caused by having "multiple sources of truth" -- in this case, two different places which defined the same "throttle strategy limits". I think it's really important for us to figure out a way to have a single place that specifies throttle settings (for example, by having the WTI fetch all of its setting via a request to the PC2 server, eliminating completely the WTI's reliance on a pc2v9.ini file).

Having said all that, the updated PR does fix the issues which were previously raised, and I (re-)approve the PR.