Dear HQ dev,
I am trying to start using the new HPC at IT4I (BarboraNG) using the same system I use for Karolina and Barbora (where it works without problems). The problem on BarboraNG is that all jobs are failing. In the job logs themselves, there are no errors or anything unusual. They run and then suddenly, they disappear. When trying to debug, I have noticed that all HQ workers are failing, mostly with HEARTBEAT LOST (and occasionally with IDLE TIMEOUT).
[svatosm@login2.barbora2 ~]$ /home/svatosm/hq-v0.26.2-linux-x64/hq worker list --all
+-----+----------------+------------------------+------------------------------------+---------+----------------+
| ID | State | Hostname | Resources | Manager | Manager Job ID |
+-----+----------------+------------------------+------------------------------------+---------+----------------+
| 357 | HEARTBEAT LOST | cn120.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7517 |
| 358 | IDLE TIMEOUT | cn127.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7522 |
| 359 | HEARTBEAT LOST | cn125.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7520 |
| 360 | HEARTBEAT LOST | cn126.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7521 |
| 361 | HEARTBEAT LOST | cn140.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7524 |
| 362 | HEARTBEAT LOST | cn141.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7525 |
| 363 | HEARTBEAT LOST | cn144.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7526 |
| 364 | HEARTBEAT LOST | cn11.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7527 |
| 365 | HEARTBEAT LOST | cn127.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7528 |
| 366 | HEARTBEAT LOST | cn12.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7529 |
| 367 | IDLE TIMEOUT | cn13.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7530 |
| 368 | HEARTBEAT LOST | cn15.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7532 |
| 369 | HEARTBEAT LOST | cn14.barbora2.it4i.cz | cpus 192; mem 742.19 GiB; score 32 | SLURM | 7531 |
+-----+----------------+------------------------+------------------------------------+---------+----------------+
I assume that would be a reason why job logs just stop being updated without any error. I am using v0.26.2. The debug log and journal is at
https://www.fzu.cz/~svatosm/hq-debug-output.log
https://www.fzu.cz/~svatosm/hqJournal.zip
Dear HQ dev,
I am trying to start using the new HPC at IT4I (BarboraNG) using the same system I use for Karolina and Barbora (where it works without problems). The problem on BarboraNG is that all jobs are failing. In the job logs themselves, there are no errors or anything unusual. They run and then suddenly, they disappear. When trying to debug, I have noticed that all HQ workers are failing, mostly with HEARTBEAT LOST (and occasionally with IDLE TIMEOUT).
I assume that would be a reason why job logs just stop being updated without any error. I am using v0.26.2. The debug log and journal is at
https://www.fzu.cz/~svatosm/hq-debug-output.log
https://www.fzu.cz/~svatosm/hqJournal.zip