Update htcondor instructions

[jhiemstrawisc]

This largely reformats the directory structure needed to run SPRAS workflows with HTCondor. In particular, it moves a lot of the helper code/submit files out of docker-wrappers/SPRAS/ into a top-level htcondor/ directory. I can do this now that the HTCondor executor has matured significantly, and can handle all the paths as they're configured in this diff.

To run a test SPRAS workflow, try following along with the instructions in docs/htcondor.rst. If anything is confusing, or you get hung up on any of the steps, let's discuss what I can do to make things more clear.

[read-the-docs-community]

Documentation build overview

📚 spras | 🛠️ Build #33063685 | 📁 Comparing d313d79 against latest (d990664)

  🔍 Preview build  

2 files changed

± htcondor.html
± fordevs/spras.html

[tristan-f-r]

[I'll have to restore my HTCondor access to follow this.]

[tristan-f-r]

Some code nitpicks

[agitter]

I'm testing the Snakemake long execution mode. The first time my jobs went on hold because I put my spras-v0.6.0.sif file in the htcondor/ directory instead of the root directory. That should have been obvious based on the comment in the .yaml file.

On the second attempt my jobs went on hold with

Transfer output files failure at execution point slot1_24@e2591.chtc.wisc.edu while sending files to access point ap2001. Details: 1 total failures: first failure: reading from file /var/lib/condor/execute/slot1/dir_3699332/scratch/output: (errno 2) No such file or directory

[agitter]

Do we want on per cluster or one per cluster_process pair?

[jhiemstrawisc]

I think one per cluster/process is the right way to go. In theory, one could still queue N>1.

[jhiemstrawisc]

I converted this to a draft because these docs will depend on the explicit sif transfer PR, and I haven't yet tested everything here in that paradigm.

[jhiemstrawisc]

Also, apologies for the poor git etiquette in the last commit that rolled too many things into one diff (including running an rst formatter). I think some of the updates are sufficiently large that the whole file should more or less be re-assessed as a fresh document.

[jhiemstrawisc]

This is a note for myself -- one thing I should document in the htcondor rst is the need to pre-create apptainer images before launching workflows.

[agitter]

During my testing, I triggered a Snakemake lock error by launching a long job, killing it, changing the config file, and relaunching. That may be a common error.

python3.11/site-packages/snakemake/persistence.py", line 211, in lock
    raise snakemake.exceptions.LockException()
snakemake.exceptions.LockException: Error: Directory cannot be locked. Please make sure that no other Snakemake process is trying to create the same files in the following director
y:
/home/agitter/spras
If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill signal or a power loss. It can be removed with th
e --unlock argument.

LockException:
Error: Directory cannot be locked. Please make sure that no other Snakemake process is trying to create the same files in the following directory:
/home/agitter/spras
If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill signal or a power loss. It can be removed with th
e --unlock argument.

Should we add it to troubleshooting?

I also hit this error

$ cat htcondor/logs/merge_input/merge_input-5_7645955.err
ModuleNotFoundError in file "/var/lib/condor/execute/slot1/dir_1046931/scratch/Snakefile", line 8:
No module named 'spras.config.revision'
  File "/var/lib/condor/execute/slot1/dir_1046931/scratch/Snakefile", line 8, in <module>

I'm guessing that means a need a newer version of the SPRAS sif image. However, we haven't released a SPRAS version recently. What version of the image are you testing with?

[ntalluri]

@agitter

For the first bug, this requires the spras conda environment to be activated and then the command snakemake --configfile <path to config file> --unlock to be run anytime the config file of choice is updated. I was planning on commenting this as step five for the parallel jobs.

For the second issue, you are right, this is because the version of SPRAS in the docker image v0.6 isn't up to date with the current version of SPRAS. Justin has a docker image you can pull from dockerhub (i think it is this jhiemstra/spras:update-htcondor-instructions-v2) or you will need to build the image with Docker on your local machine of the updated version of SPRAS, push the image to Docker Hub, and then use that image.

[jhiemstrawisc]

For the second issue, you are right, this is because the version of SPRAS in the docker image v0.6 isn't up to date with the current version of SPRAS.

In general, you should either:

• always rebuild the SPRAS container to match the version repo you're working with, OR
• check out the repo at a specific release (e.g. git checkout 0.6.0) to match the container you want to use

The key is that the repo you're using to submit from the AP should match what's in the image.

There's a callout relatively early in the documentation covering this, but I'm open to edits it seems like this is often missed:

   It is best practice to make sure that the Snakefile you copy for your
   workflow is the same version as the Snakefile baked into your
   workflow's container image. When this workflow runs, the Snakefile
   you just copied will be used during remote execution instead of the
   Snakefile from the container. As a result, difficult-to-diagnose
   versioning issues may occur if the version of SPRAS in the remote
   container doesn't support the Snakefile on your current branch. The
   safest bet is always to create your own image so you always know
   what's inside of it.

[ntalluri]

The updated documentation looks great, I added some suggestions on how to help users more.

I also remember we were changing the config.yaml file in spras_profile and wasn't sure if any of the commands needed to be added to the documentation.

[ntalluri]

Could you add to this yaml block to show what you mean by the names having to match what is in the config.

Suggested change

	...
	algorithms:
	- name: "pathlinker"
	include: true
	runs:
	run1:
	k: 10
	- name: "omicsintegrator1"
	include: true
	runs:
	run1:
	b: 5
	w: 0
	d: 10

[ntalluri]

Could you add a note block to a link on how to set up the conda environment on an ap?

[ntalluri]

I got confused about what specific image this was after reading about all the image stuff above.

Suggested change

	container image, along with any other configuration changes you want
	SPRAS container image, along with any other configuration changes you want

[ntalluri]

can you add a note on what toggling the OSPool submission looks like for a user in the spras.sub.

.. note::

In spras.sub uncomment +WantGlideIn = true and requirements = (HAS_SINGULARITY == True) && (Poolname =!= "CHTC")

[agitter]

I thought the instructions in the .sub file comments were clear enough, but I'm not totally opposed to this.

[ntalluri]

To deal with the --unlock bug

Suggested change

	#. Activate the spras conda environment and run the command ``snakemake --configfile <path to config file> --unlock``
	... note::
	Whenever you change the config file, run ``snakemake --configfile <path to config file> --unlock`` before submitting jobs. Otherwise, the workflow will appear to complete immediately but is actually raising a ``snakemake.exceptions.LockException()``.

[ntalluri]

Suggested change

	workflow (defaults are fine in most cases).
	workflow (defaults are fine in most cases). Memory and hardware requirements are also set here. To use a config file other than config/config.yaml, set the path next to the configfile: variable in this file.

[ntalluri]

this is what my config.yaml looks like. I wasn't sure if we need to add:

... && versionGE(split(Target.CondorVersion)[1], "24.8.0") && (isenforcingdiskusage =!= true)'
stream_ouput: true
stream_error: true

parts to the documentation

# Default configuration for the SPRAS/HTCondor executor profile. Each of these values
# can also be passed via command line flags, e.g. `--jobs 30 --executor htcondor`.

# NOTE: File paths in here should be relative to where you submit from, typically the
# root of the SPRAS repository

# 'jobs' specifies the maximum number of HTCondor jobs that can be in the queue at once.
jobs: 30
executor: htcondor
configfile: config/egfr.yaml
htcondor-jobdir: htcondor/logs

# Indicate to the plugin that jobs running on various EPs do not share a filesystem with
# each other, or with the AP.
shared-fs-usage: none
# Distributed, heterogeneous computational environments are a wild place where strange things
# can happen. If something goes wrong, try again up to 2 times. After that, we assume there's
# a real error that requires user/admin intervention
retries: 2

# Default resources will apply to all workflow steps. If a single workflow step fails due
# to insufficient resources, it can be re-run with modified values. Snakemake will handle
# picking up where it left off, and won't re-run steps that have already completed.
default-resources:
  job_wrapper: "htcondor/spras.sh"
  # If running in CHTC, this only works with apptainer images
  # Note requirement for quotes around the image name
  container_image: "test-htc.sif"
  universe: "container"
  # The value for request_disk should be large enough to accommodate the runtime container
  # image, any additional PRM container images, and your input data.
  request_disk: "16GB"
  request_memory: "12GB"
  retry_request_memory_increase: "RequestMemory + 4"
  retry_request_memory_max: "32GB"
  classad_WantGlideIn: true
  requirements: |
    '(HAS_SINGULARITY == True) && (Poolname =!= "CHTC") && versionGE(split(Target.CondorVersion)[1], "24.8.0") && (isenforcingdiskusage =!= true)' 
  stream_ouput: true
  stream_error: true

[ntalluri]

Suggested change

	https://chtc.cs.wisc.edu/uw-research-computing/apptainer-htc.html
	https://chtc.cs.wisc.edu/uw-research-computing/apptainer-htc.html. Specifically, create the apptainer.sub file on the AP and run ``condor_submit -i apptainer.sub`` on the AP.

[ntalluri]

Suggested change

	correct container, and change any other config values needed by your
	correct SPRAS container image, and change any other config values needed by your

[ntalluri]

Potential update for this tip::

To avoid versioning issues, the Snakefile copied for your workflow in SPRAS should match the Snakefile baked into the SPRAS container image. When this workflow runs, the Snakefile you just copied will be used during remote execution instead of the Snakefile from the container. A mismatch between the repo version and the container can cause difficult-to-diagnose errors, including ModuleNotFoundError.

To keep these in sync, either:

• rebuild the SPRAS container locally, push it to Docker Hub, and use that image for submitting jobs, or
• or, check out the SPRAS repo at the release matching your container

[ntalluri]

do we need to add request_cpus = $(NUM_PROCS)?

[agitter]

People might run this directly on the AP if we don't remind them to use a build job again.

[agitter]

The next section is what I found confusing. It gives instructions to create the .sif from the existing DockerHub image. That usually breaks. I recommend we remove it and only give instructions to build a new image from source.