Debugging Tips

Debugging

Run Locally

You can execute the same job that Horde will run locally. Go into the logs of the “Setup Build” for the job in question and look for a line that looks like this:

Executing C:\Windows\system32\cmd.exe /C ""D:\H\wi-lyra-56\Sync\Engine\Build\BatchFiles\RunUAT.bat" BuildGraph -Script=UniversalFoundry/Build/Quickstart.xml -HordeExport=D:\H\wi-lyra-56\Sync\Engine\Saved\Horde\Exported.json -ListOnly -Target="BuildAndTest Lyra" -set:ProjectFile=Lyra/Lyra.uproject -set:TargetName=LyraGame -set:DefaultEditorPlatforms=Win64 -set:DefaultTargetPlatforms=Win64 -set:WithDeviceReservation=false -set:UfExtraCookArguments=-cookprocesscount=7 -set:EditorBootTest=true -set:WithWin64=true -set:WithMac=false -set:WithLinux=false -set:WithAndroid=false -set:WithIOS=false -set:WithTVOS=false -set:WithDevelopmentTargets=true -set:WithDevelopmentPackages=true -set:WithDevelopmentTests=true -set:TargetBootTest=true -set:DedicatedServer=false -set:UfCompileArguments="-UBA -UBAHordeCluster=ci -UbaEncryption=None -UbaRootDir=D:/UBA -UBALinkRemote" -set:UseIncrementalAgents=true -set:UfExtraCookArguments=usehyperthreading"

Copy/paste that command and remove the -ListOnly to execute the same job

To get a true reproduction case of a problem, start with a clean perforce checkout.

Quickly Validate BuildGraph Locally

Running RunUAT.bat BuildGraph with the -ListOnly parameter as well as your BuildGraph into -Script and a target <Node> into the -Target parameter, will dump a summary of the job that will run along with any warnings or errors. See the above “Run Locally” for how to find the complete command Horde is executing.

Run Arbitrary Commands

Using the BuildGraph <Spawn> element, you can run arbitrary Windows commands by invoking cmd.exe.

This will write out the contents of the Engine directory:

<Spawn Exe="cmd.exe" Arguments="/c dir $(RootDir)\Engine"/>

This will write the contents of Engine\SomeFile.txt to the log

<Spawn Exe="cmd.exe" Arguments="/c type $(RootDir)\Engine\SomeFile.txt"/>

Job appears hung, logging stopped

From within the log display, do View -> Show Telemetry. Sometimes logging within Horde will stall but the job is still running. If the step requires a lot of CPU, you’ll see CPU usage high in this graph and it’s probably still making progress.

Hangs caused by popups

If clicking through a popup is necessary, the job may hang. One common codepath for popups is FWindowsPlatformMisc::MessageBoxExt which does not produce any logs. Adding UE_LOG(LogGenericPlatformMisc, Warning, TEXT("MessageBox: %s : %s"), Caption, Text); will help test if this popup is the source of a hang.

If running a <Commandlet> or manually executing UnrealEd-Cmd.exe or UnrealEd.exe via <Spawn> a popup can prevent the step from continuing. Be sure to pass -unattended -buildmachine into your CommandLet or Spawn node.

Couldn’t find target rules file

This error indicates that the .Target.cs file cannot be found.

Couldn't find target rules file for target '<PROJECT>' in rules assembly 'UE5Rules, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'
Location: D:\H\wi-<PROJECT>\Sync\Engine\Intermediate\Build\BuildRules\UE5Rules.dll

If you are using Quickstart, make sure the value of “Game Target” in the Settings page is correct and that a file <Your Game>/Source/<Value>.Target.cs is located in Perforce.

Another reason the .Target.cs file might not be found is because the engine fails to infer it correctly based on the name of the .uproject file. The engine will use the <Root>\Default.uprojectdirs file to look for .uproject files. If multiple .uproject files with the same name are found during the search, the engine will potentially look in the wrong location. Two possible solutions:

• Get rid of the duplicated .uproject files
• Update your <Compile> element to manually pass in a -project argument: "-project="$(RootDir)/$(TargetProject)"”

Performance

Loading Temporary Artifacts

Messages like: Written 123 files (123.4mb, 12.3mb/s) take a long time, slowing the start of the step. This process is loading the temporary files from previous steps onto a new agent.

Make the Artifact Smaller

Pass -nodebuginfo into the compile step that generates the artifact. This will skip generation of PDBs so you may want to employ a symbol server if debugging info is necessary. You could also run the compile twice - once generating a PDB-inclusive artifact and again generating one without it.

Rather than relying on the <Compile> node’s artifact tagging, you could manually generate the artifact culling unnecessary files.

Coalesce Nodes Into Agent Blocks

If two Nodes will always need to execute sequentially, you can put them into a single agent block eliminating the need to save/load the temporary artifacts altogether.

Known Issues

Shelves

Sometimes when a shelf gets updated, Horde will become confused and produce errors. If you’re having trouble getting updates to shelved CLs to work properly, try making a new CL and shelf.