A Comprehensive Guide to Importing Unreal Into Perforce

Having a pristine import of Unreal into Perforce is critical. If files are added incorrectly or the wrong files are added it can result in CI errors and inconsistencies in tools and game binaries. This guide aims to provide a process for the initial import and address common mis-steps we’ve seen teams make.

This guide is written for users of Windows.

This guide was last updated for the 5.7.1 version of the engine.

Once imported, the Quickstart Guide will walk you through the process of generating builds with Horde.

Perforce Server Setup

When setting up your Perforce server for Horde, there are a few prerequisites. See the Perforce Configuration Guide for detailed setup instructions.

Additionally, Epic has a few recommendations. These are not strictly required, but doing it differently will likely causes some of the steps in this guide to not work:

Configure Perforce to be Case-Insensitive

Epic recommends running Perforce in case-insensitive mode since most teams have some or all of their developers working in windows.

There are a few ways to do this, see the Perforce documentation for more info. You can verify the case-sensitivityness of your server by running p4 info. You should see Case Handling: insensitive in the output.

Disable Unicode on Perforce Server

Epic says: “P4 type Unicode is not enabled on our Perforce server.“. There are number of different settings on the client and the server where Unicode is configured. While it’s not 100% clear what setting this refers to, we’ve had success when p4 configure show unicode reports back unicode=0 (default).

If you’re seeing something different, refer to the Perforce documentation to update your server configuration.

Import Steps

Getting a Pristine Start

These steps assume you have a perfectly clean drop of Unreal that has never been compiled or run. To do that, you can use Epic’s Perforce server or GitHub. We’ll cover the GitHub path here. First you’ll need to get access and configure the git commandline tool on your computer.

If you already have existing code or engine changes, our recommendation is to follow these steps so you’ve got a clean start and then, with .p4ignore and typemaps configured, copy your existing code over the new depot/stream and add/submit it.

Pull from Git

From a vanilla (non-powershell) command prompt:

1. Clone the engine to your local computer: git clone https://github.com/EpicGames/UnrealEngine.git
2. Change to the directory containing the engine: cd UnrealEngine
3. Switch to the correct branch: git checkout 5.7 (for the 5.7.x version of the engine)

Get Dependencies

Epic provides a script to pull the files not included in their GitHub repo and do some other miscellaneous steps. From the UnrealEngine directory in a command prompt run: setup.bat.

This will take awhile to execute. When it is nearly finished, you’ll get a Windows popup asking to install some stuff. Make sure to approve that so the script can continue and complete.

Do not run GenerateProjectFiles.bat.

Set up the Perforce Client

P4V is probably fine for importing the engine, but we’ve seen instances of it hanging or appearing unresponsive. The commandline is reliable and gives us a consistent way to do what we want so we suggest using that.

You’ll need the p4 command line client installed to follow these steps.

Disable Charset Detection and Conversion

If you’ve set up your server with Unicode disabled as we recommend, you’ll need to also disable charset detection and conversion. When done, Perforce will treat files as binary or use default text handling without charset conversion.

To do this, run p4 set P4CHARSET=. (Yes, with nothing after the =).

Log in

You’ll need to set your P4PORT and P4USER as well. Run p4 login and provide your credentials.

More details are in the P4 documentation

Set Up Typemap

Typemaps tell Perforce how to handle different types of files. Each studio is going to have different preferences and requirements for this. Epic provides a typemap in their documentation, but it assumes your build is located in //depot. This typemap is the same but will work regardless of where you decide to put the engine.

To set the typemap, run p4 typemap. A text editor will open and you can paste in the text.

# Perforce File Type Mapping Specifications.
#
#  TypeMap:             a list of filetype mappings; one per line.
#                       Each line has two elements:
#
#                       Filetype: The filetype to use on 'p4 add'.
#
#                       Path:     File pattern which will use this filetype.
#
# See 'p4 help typemap' for more information.

TypeMap:
    binary+w //....exe
    binary+w //....dll
    binary+w //....lib
    binary+w //....app
    binary+w //....dylib
    binary+w //....stub
    binary+w //....ipa
    binary //....bmp
    text //....ini
    text //....config
    text //....cpp
    text //....h
    text //....c
    text //....cs
    text //....m
    text //....mm
    text //....py
    binary+l //....uasset
    binary+l //....umap
    binary+l //....upk
    binary+l //....udk
    binary+l //....ubulk

Save the file and close the editor.

Configure p4ignore

The .p4ignore tells Perforce which files should be ignored from source control. We’re going to be ignoring the ignore file during the import, but setting this will properly will prevent problems in the future. As with typemaps, this is something should be set according to your teams needs and preferences.

In the UnrealEngine directory (or whatever you’ve called it), create a new file .p4ignore and then run p4 set P4IGNORE=.p4ignore.

Our tests were run with this .p4ignore which was adapted from Epic’s documentation on the subject.

# Here you can specify files to ignore when adding files to the depot.
#
# The syntax for P4IGNORE files is not the same as Perforce syntax.
# Instead, it is similar to that used by other versioning systems:
#
# - Files are specified in local syntax
# - a # character at the beginning of a line denotes a comment
# - a ! character at the beginning of a line excludes the file specification
# - a * wildcard matches substrings.
#
# For example:
#
# foo.txt    Ignore files called "foo.txt"
# *.exe      Ignore all executables
# !bar.exe   Exclude bar.exe from being ignored
#
###############################################################################
# Epic's P4IGNORE.
# P4IGNORE doesn't work like GITIGNORE:
# http://stackoverflow.com/questions/18240084/how-does-perforce-ignore-file-syntax-differ-from-gitignore-syntax
###############################################################################
# Ignore root Visual Studio solution files. We do check in some sln files in subdirectories, so only ignore
# the ones found in the root.
/*.sln
/.p4sync.txt


# Ignore all Visual Studio temp files.
*.suo
*.opensdf
*.sdf
/Engine/DerivedDataCache/*
**/DerivedDataCache/Boot.ddc
**/DerivedDataCache/**/*.udd


# Ignore all Intermediate and Saved directories
*/Intermediate/*
*/Saved/*
# Ignore UBT's configuration.xml
Engine/Programs/UnrealBuildTool/*
*.uatbuildrecord
*.tmp


# Ignore built binaries and temporary build files
*/obj/*
*.csprojAssemblyReference.cache


# Ignore UBT's log output files
/Engine/Programs/UnrealBuildTool/*.txt


# Ignore Python cached files
*.pyc


# Ignore JetBrain's IDE folders
.idea/
!.idea/runConfigurations
.gradle/


# Ignore files added by Finder on Mac
.DS_Store


# Ignore all .code-workspace files
*.code-workspace

Depot, Streams, Workspace

Create a depot, stream, and workspace by running these commands. Replace MyStreamDepot with the name of the depot you’d like to use and MyWorkspaceName with the name of the P4 Workspace you’d like to use. main is standard, but you can replace that if you like as well.

p4 depot -t stream MyStreamDepot
p4 stream -t mainline //MyStreamDepot/main
p4 client -S //MyStreamDepot/main MyWorkspaceName

After each, an editor will pop up. You can leave the defaults or edit them to match your preferences.

Remove .git Folder

Having .gitignore, etc checked into Perforce is harmless, and we’re erring on the side of changing as little as possible from the git clone but you don’t want to check in the .git folder itself. Delete it by running this command from the UnrealEngine directory: rmdir /s /q .git.

Add Files

It’s (finally!) time to add and submit the engine to Perforce. Epic’s documentation describes how to do this from P4V and has some outdated information about which files should and should not be added. Since we’ve taken care to have a totally pristine directory, we can safely add everything.

This command will add the files to Perforce. p4 -r5 -q -v net.maxwait=600 add -I -f ...

This will take a very a long time to run (>1 hour probably depending on the network connection to your Perforce server). Opening the task manager to see if the p4 client is still using network will reassure you that it hasn’t hung.

A few notes on what we’re doing here:

• -r5 tells Perforce to try 5 times if something goes wrong with the connection.
• -q tells Perforce to run in quiet mode. This is important because if there are errors, they’ll get lost in the spam without this switch.
• -v net.maxwait=600 sets the timeout to ten minutes. This is probably unnecessarily high but we want reliability not speed here.
• -I tells Perforce to ignore your .p4ignore file. There are lots of file types which we want checked in initially but in the future don’t generally want added.
• -f see the P4 documentation on the subject, but this will add files with strange characters in their names.
• ... means all files in this directory and its subdirectories.

If you’re having trouble with connection failures or your Perforce server choking on such a large changelist, see the Adding and Submitting in Chunks section below.

Submit Files, Fix Unicode Issues

Time to Submit, right? We haven’t tested every permutation/combination of Unicode server and client settings but odds are good there’s going to be problems if you try to submit after simply adding. Some files from git/Setup.bat are missing the Byte Order Marker (BOM) which will cause Perforce to fail when you submit. As of 5.7.1 there are just few of these. Manually forcing the type to text will correct this issue:

p4 reopen -t text Engine\Source\ThirdParty\nvTextureTools\nvTextureTools-2.0.6\src\project\vc8\nvassemble\nvassemble.rc
p4 reopen -t text Engine\Source\ThirdParty\nvTextureTools\nvTextureTools-2.0.6\src\project\vc8\nvcompress\nvcompress.rc
p4 reopen -t text Engine\Source\ThirdParty\nvTextureTools\nvTextureTools-2.0.6\src\project\vc8\nvddsinfo\nvddsinfo.rc
p4 reopen -t text Engine\Source\ThirdParty\nvTextureTools\nvTextureTools-2.0.6\src\project\vc8\nvdecompress\nvdecompress.rc
p4 reopen -t text Engine\Source\ThirdParty\nvTextureTools\nvTextureTools-2.0.6\src\project\vc8\nvimgdiff\nvimgdiff.rc
p4 reopen -t text Engine\Source\ThirdParty\nvTextureTools\nvTextureTools-2.0.6\src\project\vc8\nvzoom\nvzoom.rc

With that done, you can run the submit. This will take even longer than the add. You can reassure yourself that it’s still working by checking network usage in the Task Manager.

p4 -r5 -q -v net.maxwait=60 submit -d "Initial import"

If there are files which have the BOM issue or are otherwise mis-detected you will see errors on submit of the form: Translation of file content failed near line 1 file <filename>. You can address this with the typemap or by reopening or adding them with specified type.

How Perforce detects and handles unicode in files is complex. This document in the P4 admin guide provides some details.

If you’re having trouble with connection failures or your Perforce server hanging, see Adding and Submitting in Chunks below.

Adding and Submitting in Chunks

It’s not uncommon for submitting the whole engine in one shot to simply not work. If it fails, you may be left with the server in a strange state, run p4 changes to see where you’re at with current/pending changelists. You’ll want to revert and changes -d (delete) the bad changelists before trying this process.

With that done, you can grab some of the bigger directories and add / submit them individually. The granularity of this is going to depend on your specific setup, but you might, for example, try this:

P4 -r5 -q -v net.maxwait=600 add -I -f Engine\Binaries\...
p4 -r5 -q -v net.maxwait=600 submit -d "Initial import Engine\Binaries"

P4 -r5 -q -v net.maxwait=600 add -I -f Engine\Source\ThirdParty\...
p4 -r5 -q -v net.maxwait=600 submit -d "Initial import Engine\Source\ThirdParty"

P4 -r5 -q -v net.maxwait=600 add -I -f Engine\...
p4 -r5 -q -v net.maxwait=600 submit -d "Initial import of the remainder of Engine"

Once you’ve hit a few of the big directories, you can finish up importing the rest:

P4 -r5 -q -v net.maxwait=600 add -I -f ...
p4 -r5 -q -v net.maxwait=600 submit -d "Import the rest of UE"

In Perforce, if a file is already added it won’t be added twice.