The goal of the project was to create an interesting streaming experience where instead of following a single players stream, the viewer is able to watch the entire party playing the game live and can easily catch all the interesting moments without having to change streams.

With this goal in mind, I worked on creating a custom control interface built using TouchDesigner and OBS that allows the user to curate an audio-video stream consisting of 8 players playing the massively popular multiplayer game Among Us.

Design Challenges:

Some of the challenges that I identified at the start of the project were:

  • Ingesting camera, microphone and screen capture streams from the 8 remote players.
  • Overlaying important game information like, who the impostors and eliminated players are on the stream in real-time.
  • Transitioning from multi-player views to single player views and vice-versa.
  • Managing player audio sources in real-time depending on various in game scenarios.

With these challenges in mind, I moved forward with the design and development of the project.

Screenshot of the control interface with the calibration images from OBS
TouchDesigner project root level showing the various components

Managing multiple video sources:

The first step of getting the project off the ground was to be able to ingest the camera and desktop sharing from the participants of the stream. This blog post outlines how I used OBS and Spout to share the incoming and outgoing video streams to TouchDesigner.

Capturing 16 video feeds in OBS on 1 4K texture | Calibration images used while development

The 16 videos feeds each with a resolution of 960x540 are all laid out onto one 4K texture and shared to TouchDesigner, where I was able to easily separate them out into individual streams for use in the project.

Layouts and Transitions:

I big part of the project was being able to arrange the videos in an interesting way. I started by creating the Layout components for the Players themselves that would house their camera and screen share. I then created a Grid based layout component that would be able to hold four of the player layouts in a 2x2 grid and allowed the controller to easily maximize any of the 4 players on the selected grid.

Minimizing from the impostor view to the grid layout

I also implemented a VJ style A/B Deck where you can load in the layouts by dragging them onto the deck and then crossfading between the decks.

Crossfading from the grid layout to the emergency meeting layout.

Controlling OBS using WebSockets:

As part of the stream I wanted to be able to switch between the audio sources of the various participants, depending on who was on screen. I was able to find this plugin for OBS that allows you to communicate with it through WebSockets. I created a simple sub interface on the controller that allowed the user to obtain all the scenes from the OBS session, switch to a selected scene and the controls for managing the various audio sources.

OBS controller in the interface

Overlaying game information:

I added a sub interface that allows the user to select which players are the impostors and which players have been eliminated. I also added pads to select which of the players are lying during the meeting. I integrated the Giphy API into the project to allow me to select a random gif based on a keyword search and then transform it and add it to the stream.

FX section of the controller interface

Conclusion:

The project was a great learning experience where I was able to dive deep into the nitty gritty of TouchDesigner. I was able to experiment with various different interface layouts and have a lot more ideas on how to improve the current setup. I am now confident about the various use case for incredible technologies such as NDI and Spout. I also learned the importance of performing thorough AV testing before going live to ensure a smooth stream. Even though, this isn’t something I can commercialize, the skills I have learned from undertaking this project are immensely valuable.