> ## Documentation Index
> Fetch the complete documentation index at: https://docs.coreweave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Server Errors

> [card] [How do I fix torch.distributed rendezvous and TCPStore failures?](https://docs.coreweave.com/support/sunk/articles/how-do-i-fix-torch-distributed-rendezvous-failures)
>
> Your torch.distributed job fails during startup or rendezvous with a c10d or TCPStore error such as sendBytes failed ... ...

> [card] [How do I set up Slurm accounting for priority scheduling?](https://docs.coreweave.com/support/sunk/articles/how-do-i-set-up-slurm-accounting-for-priority-scheduling)
>
> SUNK enables Slurm accounting by default: accounting.enabled is true and SUNK deploys a local slurmdbd with MySQL automa ...

> [card] [What do NCCL errors mean and how do I diagnose them?](https://docs.coreweave.com/support/sunk/articles/what-do-nccl-errors-mean)
>
> Most NCCL errors aren't NCCL bugs. The message you see usually names the layer where NCCL gave up, not the layer that fa ...

> [card] [What do training job exit codes 137, 139, and 143 mean?](https://docs.coreweave.com/support/sunk/articles/what-do-training-job-exit-codes-mean)
>
> Training job exit codes above 128 mean a signal killed the process, where the exit code is 128 + signal number. The most ...

> [card] [Why does Pyxis report File already exists after a job is preempted?](https://docs.coreweave.com/support/sunk/articles/why-does-pyxis-report-file-already-exists-after-preemption)
>
> After a job is preempted or requeued, Pyxis can fail to start with ERROR File already exists if a leftover container dir ...

> [card] [Why does my GB200 job segfault during checkpointing?](https://docs.coreweave.com/support/sunk/articles/why-does-my-gb200-job-segfault-during-checkpointing)
>
> A segfault during checkpoint saving followed by an NCCL collective timeout points to a driver-level problem, not a fabri ...

> [card] [Why does my job stop responding in NCCL communicator init after a previous crash?](https://docs.coreweave.com/support/sunk/articles/why-does-my-job-hang-in-nccl-init-after-a-crash)
>
> If the job stops responding during NCCL communicator initialization specifically after a previous run crashed, the cause ...

> [card] [Why does my node keep draining for a missing InfiniBand device?](https://docs.coreweave.com/support/sunk/articles/why-does-my-node-keep-draining-for-a-missing-infiniband-device)
>
> A node that drains with NHC: check\_file\_test: -e /dev/infiniband returned 1 has lost its InfiniBand device mount inside  ...

> [card] [Why is my node drained or cordoned?](https://docs.coreweave.com/support/sunk/articles/why-is-my-node-drained-or-cordoned)
>
> A Slurm node enters drained state when it stops accepting new jobs. The drain reason (visible in scontrol show node or t ...
