Debugging Tips
Known Issues
Section titled “Known Issues”Shelves
Section titled “Shelves”Sometimes when a shelf gets updated, Horde will become confused and produce errors. If you’re having trouble getting updates to shelved CLs to work properly, try making a new CL and shelf.
Performance
Section titled “Performance”Loading Temporary Artifacts
Section titled “Loading Temporary Artifacts”Messages like: Written 123 files (123.4mb, 12.3mb/s) take a long time, slowing the start of the step. This process is loading the temporary files from previous steps onto a new agent.
Make the Artifact Smaller
Section titled “Make the Artifact Smaller”Pass -nodebuginfo into the compile step that generates the artifact. This will skip generation of PDBs so you may want to employ a symbol server if debugging info is necessary. You could also run the compile twice - once generating a PDB-inclusive artifact and again generating one without it.
Rather than relying on the <Compile> node’s artifact tagging, you could manually generate the artifact culling unnecessary files.
Coalesce Nodes Into Agent Blocks
Section titled “Coalesce Nodes Into Agent Blocks”If two Nodes will always need to execute sequentially, you can put them into a single agent block eliminating the need to save/load the temporary artifacts altogether.
Debugging
Section titled “Debugging”Run Locally
Section titled “Run Locally”You can execute the same job that Horde will run locally. Go into the logs of the “Setup Build” for the job in question and look for a line that looks like this:
Executing C:\Windows\system32\cmd.exe /C ""D:\H\wi-lyra-56\Sync\Engine\Build\BatchFiles\RunUAT.bat" BuildGraph -Script=UniversalFoundry/Build/Quickstart.xml -HordeExport=D:\H\wi-lyra-56\Sync\Engine\Saved\Horde\Exported.json -ListOnly -Target="BuildAndTest Lyra" -set:ProjectFile=Lyra/Lyra.uproject -set:TargetName=LyraGame -set:DefaultEditorPlatforms=Win64 -set:DefaultTargetPlatforms=Win64 -set:WithDeviceReservation=false -set:UfExtraCookArguments=-cookprocesscount=7 -set:EditorBootTest=true -set:WithWin64=true -set:WithMac=false -set:WithLinux=false -set:WithAndroid=false -set:WithIOS=false -set:WithTVOS=false -set:WithDevelopmentTargets=true -set:WithDevelopmentPackages=true -set:WithDevelopmentTests=true -set:TargetBootTest=true -set:DedicatedServer=false -set:UfCompileArguments="-UBA -UBAHordeCluster=ci -UbaEncryption=None -UbaRootDir=D:/UBA -UBALinkRemote" -set:UseIncrementalAgents=true -set:UfExtraCookArguments=usehyperthreading"Copy/paste that command and remove the -ListOnly to execute the same job
To get a true reproduction case of a problem, start with a clean perforce checkout.
Quickly Validate BuildGraph Locally
Section titled “Quickly Validate BuildGraph Locally”Running RunUAT.bat BuildGraph with the -ListOnly parameter as well as your BuildGraph into -Script and a target <Node> into the -Target parameter, will dump a summary of the job that will run along with any warnings or errors. See the above “Run Locally” for how to find the complete command Horde is executing.
Job appears hung, logging stopped
Section titled “Job appears hung, logging stopped”From within the log display, do View -> Show Telemetry. Sometimes logging within Horde will stall but the job is still running. If the step requires a lot of CPU, you’ll see CPU usage high in this graph and it’s probably still making progress.
Hangs caused by popups
Section titled “Hangs caused by popups”If clicking through a popup is necessary, the job may hang. One common codepath for popups is FWindowsPlatformMisc::MessageBoxExt which does not produce any logs. Adding UE_LOG(LogGenericPlatformMisc, Warning, TEXT("MessageBox: %s : %s"), Caption, Text); will help test if this popup is the source of a hang.
If running a <Commandlet> or manually executing UnrealEd-Cmd.exe or UnrealEd.exe via <Spawn> a popup can prevent the step from continuing. Be sure to pass -unattended -buildmachine into your CommandLet or Spawn node.