Blog About Hardware guide
11 February 2021

RL for robotics on low cost hardware using Isaac Gym - OpenAI hand demo

by Mathieu Poliquin

Reinforcement Learning for Robotics research typical requires huge computing resources that are out of reach for most people. As detailed in NVIDIA’s news release [link here] (

As they state OpenAI for it’s hand manipulation demo used “a cluster of 384 systems with 6144 CPU cores, plus 8 Volta V100 GPUs and required close to 30 hours of training to achieve its best results”

This is insane computing resources with an even more insane price tag! The main point is that these computing resources are not even maximized because of host to transfer and CPU bottlenecks

Enter NVIDIA solution, Isaac Gym, which enables physics simulation and reward calculations directly on GPU which effectively addresses these bottlenecks. The end results is that they can reduce training time to 10 hours on a single A100 GPU.

So I was curious on how it would performance on even cheaper hardware. I tested with two machines including a typical mining hardware that cost around 150$ USD (details below)



You can get Issac Gym here: Issac Gym page

unzip the downloaded tar file and cd to the python directory

pip3 install -e .

That’s it, it should install dependencies including pytorch 1.7 Please not that you should have your NVIDIA drivers up to date. I use version 460



Hardware Test 1

Simulation and PPO algo on CPU, training on GPU:

python3 --task=ShadowHand --headless --device=CPU --ppo_device=CPU --physx --num_threads=24


Everything on GPU:

python3 --task=ShadowHand --headless


Hardware Test 2 - low cost mining rig

Simulation and PPO algo on CPU, training on GPU:

python3 --task=ShadowHand --headless --device=CPU --ppo_device=CPU --physx --num_threads=2


Everything on GPU:

python3 --task=ShadowHand --headless



As you can see the performance gain is huge by putting everything on the GPU. Even a cheap 150$ USD mining hardware with a very weak CPU can beat machines that is worth 5-10 times as much.

Isaac Gym not only shows huge performance potential but is a necessary to step to make RL for robotics accessible to indie developpers as the compution cost typical requires is several orders of magnitude out of reach for most. I do hope AMD’s ROCm team comes up with a similar solution for AMD GPUs. ROCm is open source so in theory the community could contribute a system like this but it requires a substial amount of programmer resource and I doubt teams that invest to develop such a system for will want to share it freely to the ROCm framework which means AMD really needs to step up to the plate!

If you want to see the demo in action you can check my video:

tags: Reinforcement Learning - Robotics - Isaac Gym - RTX 2060 Super - p106-100 - mining - pytorch - cuda