UETorch

UETorch is an Unreal Engine 4 plugin that adds support for embedded Lua/Torch scripts in the game engine loop, and a set of Lua APIs for providing user input, taking screenshots and segmentation masks, controlling game state, running faster than real time, etc. Torch is an AI Research platform that is focused on deep learning. UETorch strongly leverages the sparsely documented ScriptPlugin plugin provided with Unreal Engine 4.

Some recent research done using the UETorch platform is detailed in this paper "Learning Physical Intuition of Block Towers by Example" where we explore the ability of deep feed-forward models to learn intuitive physics.

Requirements

See the Unreal Engine 4 Requirements. UETorch was developed for Linux; see Building Unreal Engine on Linux. Running on Mac and Windows is not currently supported, but should be relatively straightforward if you're willing to figure out the build process.

Installing UETorch

Download and install torch from https://github.com/torch/torch-distro. We would like to use Lua 5.2, so before installing, replace LUAJIT21 with LUA52 in install.sh.
Set up an Epic Games account at https://github.com/EpicGames/Signup/, needed to clone the Unreal Engine repository from github. UETorch currently only works with the source distribution of UE4, not the binary download.
Install all the Linux prerequisites mentioned at https://wiki.unrealengine.com/Building_On_Linux

 sudo apt-get install build-essential mono-gmcs mono-xbuild mono-dmcs libmono-corlib4.0-cil \
 libmono-system-data-datasetextensions4.0-cil libmono-system-web-extensions4.0-cil \
 libmono-system-management4.0-cil libmono-system-xml-linq4.0-cil cmake dos2unix clang xdg-user-dirs

Install UnrealEngine / UETorch

git clone https://github.com/EpicGames/UnrealEngine.git
cd UnrealEngine

# clone UETorch into the plugins directory
git clone https://github.com/facebook/UETorch.git Engine/Plugins/UETorch
# run the UETorch setup script
# this will update you to a specific revision on UnrealEngine-4.8, add some patches, and set up the Lua paths
Engine/Plugins/UETorch/Setup.sh

# grab some coffee, this will take a long time
./Setup.sh && ./GenerateProjectFiles.sh && make

Profit!

Getting Started with UETorch

Source the uetorch_activate.sh script. You might want to add this to your .bashrc.

source Engine/Plugins/UETorch/uetorch_activate.sh

Launch Unreal Editor

cd Engine/Binaries/Linux
./UE4Editor

Create a new 'First Person' project.
Lets add a TorchPlugin component to the player. In the 'World Outliner' panel, click 'Edit FirstPersonCharacter'.

Inside the window that pops up, click 'Add Component, and select 'Torch Plugin'. This will add a Torch Plugin component to FirstPersonCharacter. Then close the FirstPersonCharacter window.

Now we just need to tell the UETorch component which script to run. We'll use the example script in UETorch/Scripts/uetorch_example.lua. Just set the 'Main Module' field on the TorchPlugin to 'uetorch_example'. (If you sourced uetorch_activate.sh, then UETorch/Scripts should already be on your LUA_PATH).
Press the 'Play' button. The player should move towards the cubes, based on the simple tick function inside uetorch_example.lua. Go take a look at that script now. You can exit the game by pressing the 'Esc' key.
You can call a Lua function from inside Unreal Engine's Blueprints scripting language. We'll add a routine to the 'level blueprint' that calls into Lua and starts the REPL when you press the 'H' key, which will allow you to run the Lua interpreter interactively inside a game. Open the level blueprint (from the Blueprints menu). Right click in the main window to add a new widget, uncheck 'Context Sensitive', and search for 'Call TorchFunction'. This widget just calls a Lua function with no input or output (only void -> void and string -> string widgets are provided; you can write your own as well). You can then drag your FirstPersonCharacter into the blueprint and hook it up as the target to the widget. Read the Blueprints documentation for more details. Here's what the final blueprint should look like
The interactive Torch REPL won't work inside this editor process because it is a child process with no attached TTY. In the main Editor window, go to File->Open Project, check 'Always load last project on startup', and then close the window. Then restart UE4Editor, and it should directly load your Project.
Now press 'Play' again, and press the 'H' key. The game should freeze and you will enter the Torch REPL inside of your terminal.

Full documentation

To learn how to develop Unreal Engine projects, see the Unreal Engine documentation at https://docs.unrealengine.com/latest/INT/.

In-line documentation for the APIs provided by UETorch can be found in uetorch.lua.

More coming soon.

Join the UETorch community

See the CONTRIBUTING file for how to help out.

License

UETorch is BSD-licensed. We also provide an additional patent grant.

sky3 / UETorch