GPGPU compute servers

These servers are intended for GPU computing (GPGPU), aimed at compute-intensive tasks, machine learning, data processing and scientific simulation that require acceleration by graphics hardware.

Any researcher at the center may request access to these servers. Access is granted upon request and validation.

• Servers in the HPC compute cluster: HPC compute cluster
• Servers at CESGA: Request access

Access to these servers is restricted to a specific group, a particular project, or is more tightly controlled for resource management and planning reasons.

It is essential to check the updated information in Xici at the time of requesting the service, where the particular circumstances of each server are detailed (access criteria, priorities, usage conditions, etc.).

⚠️ The ctgpgpu15, ctgpgpu16, ctgpgpu17 and ctgpgpu18 servers have a temporary installation and allocations, and their configuration and access may be changed in the future.

Not all servers are available at all times for any use. To access the servers, you must request it beforehand through the incident form. Users who do not have access permission will receive an incorrect password message.

To connect to the servers, you must do so via SSH. The names and IP addresses of the servers are as follows:

Connections are only available from the center's network. To connect from other locations or from the RAI network you must use the VPN or the SSH gateway.

On servers where the Slurm queue manager is present, its use is mandatory to submit jobs and thus avoid conflicts between processes, since two jobs should not be run at the same time. Users are limited to using two cores and 1GB of RAM outside Slurm.

To submit a job to the queue use the “sbatch” command with a Slurm script or directly srun:

srun cuda_program cuda_program_arguments

The srun process waits for the job to run before returning control to the user. If you do not want to wait, you can use console session managers such as screen, allowing you to leave the job queued and disconnect the session without worry and retrieve the console output later.

Alternatively, you can use nohup and send the job to the background with &. In this case the output is saved to the file nohup.out:

nohup srun cuda_program cuda_program_arguments &

To see the queue status use the squeue command. The command shows an output similar to this:

JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
9  servidore ca_water pablo.qu    PD       0:00      1 (Resources)
10 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
11 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
12 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
13 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
14 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
 8 servidore ca_water pablo.qu     R       0:11      1 ctgpgpu2

You can also get an interactive view, updated every second, with the smap command:

smap -i 1

On servers that use gpuctl, you must use the gpu command to request GPUs and be able to run jobs on them.

The gpu command does not include a job queue or parallelism control. It can be used interactively or from scripts, but to queue multiple jobs or run them in parallel you should use external tools, such as task-spooler (tsp command).

There are two main workflows:

This method is best suited if you only want to run a single command. gpu exec acts as a wrapper: it takes care of claiming the GPU, preparing the environment needed for the command and releasing it when finished.

gpu exec python ./train.py

This is the recommended option for simple runs, as it avoids having to manually manage the reservation and release of the GPU.

This method is best suited if you want to keep an active reservation across multiple commands, whether in an interactive session or in a workflow with several consecutive runs.

gpu claim
gpu exec python ./train.py
gpu release

Or, alternatively:

gpu claim
eval "$(gpu env --shell)"
python ./train.py
gpu release

If there is already an active reservation, gpu exec will reuse it and automatically prepare the appropriate environment for the command.

It is important to remember that, after creating or modifying a reservation, you must run eval "$(gpu env --shell)" again if you are going to launch commands directly from the shell. Alternatively, you can use gpu exec, which prepares the environment on each execution.

Reservations are kept while real compute activity on the GPU is detected.

If a GPU remains reserved without activity for a prolonged time, the reservation may be lost automatically due to inactivity. Therefore, if you later try to run a job assuming the reservation is still active, the job may fail.

A reservation is not kept simply by having a terminal open, but by actual GPU activity.

The inactivity expiration times are configured as follows:

• Monday to Friday, from 10:00 to 18:00 → 1 hour
• Other times → 2 hours
• If the reservation seems abandoned (there is no user process on the system nor any shell) → 10 minutes

If you are working with a manual reservation and want to check if it is still active, you can use:

gpu mine

Or, if you want general information about all GPUs, the time when real usage was captured for each, and the currently configured expiration time, you can run:

gpu status

This is especially recommended if time has passed since the last execution or if the GPU may have gone unused for a long period.

Although gpu does not implement a job queue as such, there is an implicit waiting system for GPU reservation.

When a command that requires reserving a GPU is executed and none are available, the command does not fail but remains blocked waiting for a GPU to become free. The user is placed in an internal queue managed by gpuctl and the execution will continue automatically as soon as the reservation can be satisfied.

This behavior applies to both manual and automatic reservations.

For example:

gpu exec python ./train.py 

If there are no GPUs available, this command will wait until one is released.

Similarly:

gpu claim 

will also remain blocked until it can obtain a GPU.

By default, gpuctl automatically assigns an available GPU. However, it is possible to request a specific GPU by its index.

Both gpu claim and gpu exec allow using the –gpu-index parameter for this purpose:

gpu claim --gpu-index 0 
gpu exec --gpu-index 1 python ./train.py 

In this case, the command will attempt to reserve the specified GPU. If that GPU is not available, the command will wait in queue until that particular GPU is released.

This behavior allows fixing runs to specific GPUs, but it may increase waiting time if the chosen GPU is highly requested.

In addition to the main reservation, which is made with priority in the guaranteed queue, it is possible to reserve a second GPU in the burst_preemptible queue on some servers (check the reservation policy message when logging into the server; if applicable it will mention a second opportunistic GPU).

This second GPU may be used whenever it is free, but it is not guaranteed: it may be lost even in the middle of a job if another user needs that GPU for their primary reservation.

This allows taking advantage of additional free capacity when available, but jobs that depend on this second GPU must be prepared to tolerate its loss.

To request this second GPU, just run gpu claim a second time:

gpu claim
gpu claim

You can also explicitly specify the number of GPUs:

gpu claim --numgpus 2

In this case, the command will wait until it can reserve the requested number of GPUs.

Note that these two forms are not exactly equivalent:

• with gpu claim followed by another gpu claim, the first reservation is obtained earlier and the second is requested afterwards;
• with gpu claim ---numgpus 2, the request is made jointly and the command will wait until it can reserve both GPUs at once.

It is important to remember that, after making any reservation, you must run eval "$(gpu env --shell)" again or perform executions with gpu exec.

If the second GPU is lost later for being preemptible, the jobs using it may fail or be interrupted.

Therefore:

• the first GPU is the priority and guaranteed reservation;
• the second GPU is opportunistic and will only be available while it is not needed for another priority reservation;
• if any GPU cannot be reserved, the command will remain waiting and the user will enter the corresponding queue;
• when reserving two GPUs at once, the command will wait until both can be reserved simultaneously.

Since gpu does not incorporate a job queue or parallelism control, a practical option to queue runs is to use task-spooler, with the tsp command, combined with gpu.

gpu claim
tsp gpu exec python ./train1.py
tsp gpu exec python ./train2.py
tsp gpu exec python ./train3.py
tsp gpu release

This method allows reusing the same reservation for several consecutive jobs queued in tsp.

If you want to run several jobs at the same time, you can increase the number of slots of tsp:

gpu claim
tsp -S 2
tsp gpu exec python ./train1.py
tsp gpu exec python ./train2.py
tsp gpu exec python ./train3.py
tsp gpu exec python ./train4.py
tsp -w
tsp -S 1
gpu release

In this case, be careful not to run gpu release before all jobs have finished. First make sure the queue has finished, and only then release the GPU.

Note: running multiple jobs in parallel does not imply reserving multiple GPUs. If only one GPU is reserved, all processes will share that same GPU and compete for its resources, such as memory and compute time.

Also remember that the reservation is only maintained while real compute activity on the GPU is detected. If the jobs go too long without using it, the reservation may expire automatically due to inactivity.

Using tmux allows keeping terminal sessions, disconnecting from them and recovering them later, thus allowing to maintain an interactive session without keeping the connection open. Disconnecting does not send HUP to the processes.

To start a tmux session or reconnect to it if it already exists, it is recommended to use a fixed name:

tmux new -A -s main

This creates the main session if it does not exist, or reconnects to it if it was already created.

Once inside tmux, to detach while keeping the session active press Control+b, release, and then press d.

At this point you can disconnect from the machine without interrupting the processes. To return to the same session, simply run the previous command again.

If you want to completely terminate the session from outside:

tmux kill-session -t main

gpuctl allows partitioning a GPU into multiple MIG instances to isolate resources and run smaller workloads in parallel on a single physical GPU in a more controlled way.

This functionality is only available on compatible GPUs; otherwise it cannot be used. For now, only NVIDIA H200 GPUs have this function enabled.

To reserve a GPU and enable it in MIG mode:

gpu claim --mig half
gpu claim --mig third
gpu claim --mig quarter

This will split the GPU into two, three or four partitions, choosing the most appropriate MIG profiles according to the compatible card. Note that the --mig parameter can only be used when exactly one GPU is claimed.

Once the MIG reservation is created, gpu exec and gpu env automatically prepare CUDA_VISIBLE_DEVICES with the corresponding MIG UUIDs.

gpuctl allows configuring the sending of email notifications related to reservation status and GPU usage.

gpu notify

gpu notify --mode off
gpu notify --mode email
gpu notify --mode telegram
gpu notify --mode both

When notifications are enabled, emails or Telegram messages are sent in the following cases:

• when a GPU is automatically released for remaining unused;
• when the enforcer kills a process for using a GPU without having an active reservation or for losing an opportunistic GPU;
• when a reservation request is queued;
• and subsequently when that reservation is granted.

Notifications are not sent in the case that a reservation is made and the reservation is granted immediately without waiting, or if the GPU is manually released with release or when an exec that made an automatic reservation finishes.

gpu notify --email your.email@domain

If a custom email is not configured, the address obtained from LDAP will be used by default. To return to the default email:

gpu notify --clear-email

After running the following command:

gpu notify --pair-telegram

Follow the instructions to finish the pairing process. To remove the pairing:

gpu notify --unpair-telegram

The user can send custom notifications at various points in their workflow using:

gpu notify --send "message content"

This can be useful to receive personal alerts at the start or end of an experiment, after completing a processing phase, or at any other relevant point in the workflow.

The preferred notification method configured will be used, or email if neither is configured.

To consult a summary of recent GPU usage history by process, you can use:

 gpu report 

This shows the recorded PIDs, the number of samples available and aggregated metrics of GPU, CPU and memory usage.

If you want to see the data of a specific process, with ASCII charts and the timeline of stored samples, you can use:

 gpu report --pid PID 

If there are many samples, the output may appear summarized. To see the full series:

 gpu report --pid PID --all-samples 

• To run a single command, the simplest is usually gpu exec.
• To keep an active reservation across several commands, it is more appropriate to use gpu claim + gpu exec + gpu release.
• If you are going to work directly from the shell after making a manual reservation, you must run eval "$(gpu env --shell)" or prefix commands with gpu exec.
• To keep sessions active and recover them later, tmux is usually the most convenient option.
• To queue multiple jobs, tsp is a practical alternative.
• If tsp is used with parallelism, remember that this does not reserve additional GPUs: processes will share the GPUs visible at that time.
• If a reservation remains too long without real GPU compute activity, it may be lost automatically due to inactivity.