HTCondor

Checking site configurations

Query the collector to see which sites are available in a pool and whether they expose the CVMFS repositories the jobs need. The two commands below report each site’s GLIDEIN_Site along with the HAS_CVMFS_oasis_opensciencegrid_org and HAS_CVMFS_jlab_opensciencegrid_org flags, then group the distinct rows with a count.

For the JLab CHTC pool, restricted to the clas12 GlideinWMS group:

condor_status -startd -pool jlab-cm.osg.chtc.io -const 'GLIDEClient_Group == "clas12"' \
  -af:h GLIDEIN_Site HAS_CVMFS_oasis_opensciencegrid_org HAS_CVMFS_jlab_opensciencegrid_org \
  | sort | uniq -c | sort -n | sort -u

For the OSPool:

condor_status -startd -pool cm-1.ospool.osg-htc.org -const 'OSPool' \
  -af:h GLIDEIN_Site HAS_CVMFS_oasis_opensciencegrid_org HAS_CVMFS_jlab_opensciencegrid_org \
  | sort | uniq -c | sort -n | sort -u

Changing requirements for submitted jobs

Editing a submission file affects future submissions only. Use condor_qedit on the submission host, as the job owner, to change jobs already in the queue. The commands below work in Bash and zsh. See the condor_qedit reference.

A cluster ID such as 9291 selects all its queued subjobs; 9291.10 selects only subjob 10. Without a status constraint, condor_qedit edits idle, held, and running jobs. It does not hold or release jobs. Changes affect subsequent matchmaking, not the current execution or allocation of a running job.

Inspect requirements and job status

Requirements is the attribute name; do not replace it with another value in these inspection commands. Use -long to see the stored expression for one job:

condor_q 9291.10 -long -attributes Requirements

For the whole cluster, include the job IDs (this can produce a large amount of output):

condor_q 9291 -long -attributes ClusterId,ProcId,JobStatus,Requirements

Plain -af Requirements evaluates the expression instead of printing its text. Without a worker’s attributes, the result can be false even when the stored expression contains valid matching conditions. That output does not mean the stored requirements are literally false, or that the jobs cannot match. Do not overwrite requirements based on that output. If -long shows the expression itself is literally false, then it does match no workers.

To count jobs by status without printing thousands of lines:

condor_q 9291 -af JobStatus | sort | uniq -c

The first column is the count; the second is the status: 1 means idle, 2 running, and 5 held. For example, 2560 1, 301 2, and 1 5 mean 2,560 idle jobs, 301 running jobs, and one held job. These counts do not diagnose a requirements problem.

When reading an expression into a shell variable, use -af:r Requirements: the r requests raw, unevaluated output. See the condor_q reference.

Set requirements for all jobs in a cluster

Inspect the stored expressions above before deciding whether to change them.

The following command sets the requirements for every queued job in cluster 9291. It requires the JLab CVMFS repository and removes the previous Oasis availability and revision requirements:

condor_qedit 9291 Requirements \
  "(TARGET.HAS_SINGULARITY =?= TRUE) && \
   (TARGET.HAS_CVMFS_jlab_opensciencegrid_org =?= TRUE) && \
   (TARGET.OSG_HOST_KERNEL_VERSION >= 21700) && \
   (TARGET.OSG_GLIDEIN_VERSION >= 534)"

The double quotes and trailing backslashes pass the expression as one argument without literal newlines. Keep each backslash at the very end of its line, with no trailing spaces. Do not replace the double quotes with single quotes: backslash-newline continuation does not work inside single quotes.

This replaces the entire expression. If the original submission had additional resource, architecture, or file-transfer requirements, include those clauses from the submission or an intact equivalent job. The four clauses above do not restore HTCondor-generated requirements automatically. Also check that the job’s scripts use the intended /cvmfs/ paths; changing requirements does not change those paths.

Verify the result:

condor_q 9291 -long -attributes ClusterId,ProcId,JobStatus,Requirements

Idle jobs use the new requirements when next matched. Held jobs remain held. Running jobs continue their current attempt and use the new requirements if they need another match.

Only if you intend to interrupt and restart running jobs, run condor_hold 9291 before editing, confirm the jobs are held (JobStatus == 5), then run condor_release 9291 after verifying the edits. Holding running jobs may lose progress or worker-local output; without checkpointing, expect a restart. Holding idle jobs also prevents them from starting partway through a series of edits.

Set requirements only for held jobs in a cluster

To leave idle and running jobs untouched, select both the cluster and held status (JobStatus == 5). Preview the selected jobs:

condor_q -constraint 'ClusterId == 9291 && JobStatus == 5' \
  -long -attributes ClusterId,ProcId,Requirements

Apply the replacement only to jobs held when the edit runs:

condor_qedit -constraint 'ClusterId == 9291 && JobStatus == 5' Requirements \
  "(TARGET.HAS_SINGULARITY =?= TRUE) && \
   (TARGET.HAS_CVMFS_jlab_opensciencegrid_org =?= TRUE) && \
   (TARGET.OSG_HOST_KERNEL_VERSION >= 21700) && \
   (TARGET.OSG_GLIDEIN_VERSION >= 534)"

The same complete-expression considerations above apply. You do not need to hold the cluster first: this command targets jobs already held and leaves them held. Verify the edits:

condor_q -constraint 'ClusterId == 9291 && JobStatus == 5' \
  -long -attributes ClusterId,ProcId,Requirements

When ready to retry the held jobs in this cluster:

condor_release -constraint 'ClusterId == 9291 && JobStatus == 5'

Each command evaluates its constraint independently. If another job becomes held between the edit and release, the release also selects it. Use explicit job IDs if you need to release exactly the edited set.

The memory and site examples below target one job, 8897.10; replace it with your intended job ID.

Change requested memory

For a measured requirement of 4 GiB, set RequestMemory to 4096 MB:

condor_qedit 8897.10 RequestMemory 4096
condor_q 8897.10 -long -attributes RequestMemory,Requirements

The usual generated requirement, TARGET.Memory >= RequestMemory, uses the updated request automatically. If the expression instead contains a fixed memory threshold, review that clause too. A larger request may increase waiting time and does not fix a memory leak. It takes effect on the next allocation, not the currently running attempt. For future submissions, update request_memory = 4096 in the submission file.

Exclude a problematic site

Use the exact GLIDEIN_Site value advertised by the pool, as shown in the site queries above. Replace SITE_TO_AVOID below with that value. Read the latest expression so earlier edits are preserved:

requirements=$(condor_q 8897.10 -af:r Requirements)
printf '%s\n' "$requirements"

Stop if the query failed or returned no expression. Then append the exclusion:

condor_qedit 8897.10 Requirements \
  "($requirements) && (TARGET.GLIDEIN_Site =!= \"SITE_TO_AVOID\")"

The ClassAd operator =!= also allows workers where GLIDEIN_Site is undefined. To require a known site, include !isUndefined(TARGET.GLIDEIN_Site) as another clause. Excluding a site does not evict an existing attempt there; use the hold/edit/release workflow to restart it elsewhere.

Verify and release

For the single-job examples, check each edit succeeded and inspect the final values. Release only if the job is held and you are ready to retry it:

condor_q 8897.10 -long -attributes JobStatus,Requirements,RequestMemory
condor_release 8897.10
condor_q 8897.10
condor_q 8897.10 -better-analyze

The job returns to matchmaking and may wait for a suitable worker. Use -better-analyze to investigate unmet requirements. Apply the same intended changes to the submission file for future jobs.

Debugging held jobs

Run the following commands on the submission host, such as scosg2202. Replace 8897.10 with the job’s cluster and process ID. These examples target one job, not its entire cluster.

Read the hold reason and resource usage

condor_q 8897.10
condor_q 8897.10 -long \
  -attributes JobStatus,HoldReason,HoldReasonCode,HoldReasonSubCode,RequestMemory,MemoryUsage

Check JobStatus along with HoldReason: a previous hold reason may remain after a release. Common status values are 1 (idle), 2 (running), and 5 (held).

The event log records scheduler activity separately from application stdout and stderr. Look for 012 (held), 013 (released), 004 (evicted), and 001 (executing). An eviction followed by another execution is not necessarily a hold.

For example, job 8897.10 recorded this hold on September 15, 2026:

memory usage exceeded request_memory
Code 21 Subcode 102
Memory (MB): Usage 2440   Request 2311   Allocated 2432

The job exceeded its memory allocation. After release, a later attempt reached 17,712 MB, with sustained growth of roughly 60 MB every five minutes for much of the run. This suggests a memory leak or accumulating data, but the event log alone cannot identify the responsible program. The supplied log ended with a memory update, not a second hold.

Increasing the request slightly or repeatedly releasing the job does not resolve sustained memory growth. Identify the process and pipeline step first, then measure its memory requirements.

Inspect output while the job is running

condor_tail -maxbytes 20000 8897.10
condor_tail -stderr -maxbytes 20000 8897.10

Look for the last Running ... message to identify the active pipeline step. Application buffering can delay output. These commands require a running, reachable job; they do not recover an old worker’s files after that attempt has ended.

Check the queued job’s actual logging settings:

condor_q 8897.10 -long \
  -attributes Iwd,Out,Err,WhenToTransferOutput,StreamOut,StreamErr

The portal currently generates when_to_transfer_output = ON_EXIT and does not explicitly enable stdout/stderr streaming. With those settings, application output can remain on the worker until exit. The scheduler’s .log can therefore keep updating while .out and .err are absent or empty locally. An interrupted attempt may not return its output.

When condor_tail cannot connect to the starter

Failed to peek at file from starter: Failed to connect to starter

This means condor_tail cannot reach the worker’s HTCondor starter. It does not establish whether stdout or stderr files exist. On the submission host, as the job owner, check the current state:

condor_q 8897.10 -long \
  -attributes JobStatus,RemoteHost,LastRemoteHost,HoldReason,RequestMemory,MemoryUsage
tail -80 log/job.8897.10.log

Run the tail command from the submission directory, or use the full path to its event log.

For a running job, try:

condor_ssh_to_job 8897.10

This may fail for the same connectivity reason. Possible causes include an unreachable worker, a network or connection-broker (CCB) problem, or a worker that disappeared before the scheduler updated the job’s state. The connection error alone does not distinguish these causes.

Capture the status and latest event-log entries before holding or restarting a running job: stopping the attempt can lose access to its current scratch files.

If the job is already held, follow the streaming retry instructions below. Streaming may preserve output on the submission host before another failure, but still depends on connectivity and application buffering. It cannot recover output from the previous attempt or fix memory growth.

Identify the process consuming memory

If the site supports access to running jobs:

condor_ssh_to_job 8897.10

Inside the worker session, inspect processes and repeat after a minute:

ps -eo pid,ppid,rss,vsz,etime,pcpu,args --sort=-rss | head -25

RSS is resident memory in KB; VSZ is virtual memory in KB. Use the PID, command, and changing RSS to distinguish the generator, GEMC, reconstruction, or another process. The event log’s image size is not the same measurement as resident memory.

In the job’s scratch directory, the stdout/stderr capture files are usually available as:

tail -100 _condor_stdout
tail -100 _condor_stderr

The portal script runs its pipeline in an output/ subdirectory, while the capture files normally remain in the scratch directory above it.

Enable streaming for a diagnostic retry

For a held job, enable stdout/stderr streaming for the next attempt:

condor_qedit 8897.10 StreamOut True
condor_qedit 8897.10 StreamErr True

When ready to restart the job:

condor_release 8897.10

This changes logging, not the memory request or application behavior. The diagnostic retry can hit the same memory limit; its purpose is to capture the active step and output before failure. Without application checkpointing, expect the pipeline to start again.

Changing the original submission file does not update an already queued job. Use condor_qedit for queued job attributes, and confirm the result with condor_q -long.

Changing nodescript.sh

Editing the submit-side script cannot change the copy already running on a worker. For a later attempt, ordinary input files may be transferred again, but HTCondor may have retained a submitted copy of the executable. Do not rely on editing nodescript.sh in place and releasing to replace it. Editing shared inputs can also affect other jobs that start afterward.

For a predictable test, prepare one fresh diagnostic submission in a separate directory:

1. Copy the generated submission file, nodescript.sh, functions.sh, and any other required inputs.
2. Preserve the failing job’s arguments, including subjob index 10 and its Lund URI if applicable. A new single-job submission has $(Process) = 0; set the original index explicitly when reproducing it.
3. Replace the original queue statement with queue 1, and remove dependencies on its old queue variables.
4. Check executable, input, working-directory, and log paths so they refer to the diagnostic files.
5. Redirect the final upload to a separate destination, or disable it, to avoid replacing existing output.
6. Add the following settings to the diagnostic submission file:

stream_output = True
stream_error = True

For Bash command tracing, add this near the beginning of the diagnostic script, after the shebang:

PS4='+ ${BASH_SOURCE}:${LINENO}: '
set -x

Tracing writes commands and expanded arguments to stderr. It identifies the invoked command but does not show what happens inside a compiled program. Avoid tracing commands that expose credentials.

After reviewing the diagnostic file and its single-job queue, submit from that directory:

condor_submit debug.condor

Use the new job ID with condor_tail and condor_ssh_to_job. If memory keeps growing, reproduce the identified pipeline command with a smaller input and investigate that program’s memory use.