i_1262 Improve sandbox scripts for better reporting CPU time accuracy

[johnbrvc]

Description of what the PR does

Both the pc2sandbox.sh and pc2sandbox_interactive.sh scripts have been modified to only put the bare minimum amount of work in the cgroup. Previously, some debug logging and other bash related stuff was charged to the cgroup.
CI: Use CPU # (0-(N-1)) instead of ordinal (1-N) to determine the CPU to pin to. Change taskset to use the -c CPU# option instead of the default "mask" option for clarity.
CI: better handling of error conditions in the pc2sandbox_interactive.sh script to create the resource summary report even on failures and PC2 wall time limit exceeded. Previously, resource were not created in these cases. (This makes the Web Judge Interface better).
CI: Add TDEBUG flag to control timing debugging. It is ON by default and logs stuff to the sandbox.log file. It's mostly keeping track of the cgroup cpu.stat file.

Issue which the PR addresses

Fixes #1262

Environment in which the PR was developed (OS,IDE, Java version, etc.)

Linux 24.04.3

Precise steps for testing the PR (i.e., how to demonstrate that it works correctly)

1. This is only for Linux based systems since sandboxes are currently supported on Linux only.
2. Compile the samps/src/hello.cpp program.
3. Execute it from a Linux bash prompt using the Linux "time" program: time ./a.out
4. Note that it takes on the order of a few milliseconds to run.
5. Start up a clean PC2 server using the clics_sumithello sample contest.
6. Start up an administrator client.
7. Edit the "hello" problem and change it to use a Sandbox.
8. Start the contest.
9. Start up an autojudge.
10. Start up a pc2team client and log in using team1 with password team1.
11. Submit the samps/src/hello.cpp program for the "hello" problem.
12. Wait for the submission to be judged.
13. From a bash command prompt, change to the "execute" folder where the judge executed the submission.
14. View/edit the sandbox.log file.
15. Search for the string "CPU ms" - this is the header of 2 line table showing the resources used by the submission.
16. Note the CPU time below the CPU ms heading is about the same as that observed in step 4.

[clevengr]

I reviewed scripts/pc2sandbox.sh and left several questions. I also reviewed the interactive sandbox version; I have the same basic questions about that, although I admit I don't understand that code quite as well.

[clevengr]

It appears to me that the regex in this line requires cpunum to start with a digit between 1 & 9. That excludes CPU 0. I don't understand why.

[johnbrvc]

In that block of code, the script is trying to "guess" which CPU to use based on the logged in user's account ($USER). It will isolate the judge # from the logged in user name, eg. judge1 -> 1. Since we do not allow "judge0", we'd want that to use the default. This, code is used, for example, in non-WF type events, if you have multiple judges running on a multi-core system. Each judge can be pinned to it's own CPU based on its judge number, eg. judge1 -> cpu1, judge2 -> cpu2, etc.

[clevengr]

It appears to me that computing endtime here (after the calculation of cputime) means that the time taken by the grep and the cut in the preceeding statement will be included in the process' execution time. Am I wrong? Is that intentional? (It seems like computing endtime should be done before the computation of cputime -- and possibly even before the saving of COMMAND_EXIT_CODE; i.e. immediately after the wait is finished...)

[johnbrvc]

No the times are NOT included in the process' execution time. That was the point of the moving the code around. Line 416-425 put a SUBSHELL ($BASHPID) only in the cgroup:

# This will create a new process group and put it into the created cgroup
( if ! echo $BASHPID > $PC2_SANDBOX_CGROUP_PATH/cgroup.procs
  then
    echo $0: Could not add current process to $PC2_SANDBOX_CGROUP_PATH/cgroup.procs - not executing submission.
    SysFailure Could not add current process to $PC2_SANDBOX_CGROUP_PATH/cgroup.procs
    exit $FAIL_SANDBOX_ERROR
  fi
  ulimit -t ${TIMELIMIT} -u ${MAXPROCS} -s unlimited
  exec setsid taskset -c ${cpunum} $COMMAND $*
) <&0 >&1 2>&2 &

Hence the ( ...) &. Parens in bash create a subshell. So, process $! is the PID of the created subshell in the parens, which we assign to submissionpid. ($submissionpid is the process id that is in the cgroup, NOT the current shell.) The only thing charged to the process' cpu time is ulimit -t ... and the "exec setsid taskset", all of which are necessary and have to be done before firing off the submission's executable in the context of the new submissions PID to set the time limit and session ID and pin to a specific CPU.

We DO charge the wall-time used by grep usage_usec from $PC2_SANDBOX_GROUP_PATH/cpu.data to the submission, but since we don't use wall time for anything other than display, it's not really important to worry about that. I guess technically, we could move the:

endtime=`GetTimeInMicros`

to right before the:

cputime=`grep ...`

but again, it's not important.

[clevengr]

This was exactly my point: it seems like endtime=GetTimeInMicros should come before cputime=grep ... Otherwise, as you seem to agree, the code is charging the time to execute the grep (and also the cut, I think?) to the execution time of the submission. It's not clear to me why you think it's "not important"; doesn't that add inaccuracy to the computed execution time?

[johnbrvc]

endtime is the WALL time; which is NOT that important. cputime is the important one. If wall time is off by, say, 5ms, it wont make any difference to anyone. Wall is affected by many things; system load, disk I/O, etc., all of which are not charged to the CPU time of the run since we only look at the real CPU time from the cgroup. We only use wall time as a failsafe, eg. if some contestant does "sleep(10);" That's what those thresholds are for that we added to the Settings (so-called "grace periods"). Since sleep(10) uses no cpu time, the pc2 walltime timers will kick in and kill it.

Now, this IS different from the way that PC^2 USED to work when we only used wall time to measure CPU time (and, the way it does it if sandboxes aren't used). Now, wall time is just for information purposes and really isn't used (unless the failsafe trips as described above).

[clevengr]

I've added some comments in reply to your answer to my original comments. I still haven't had time to set up a Linux environment to do a runtime test.