Individual Work
week 1 - Introduction
1.1. Me
I'm Christos, a 23 year old Creative Technologist and audiovisual artist from Cyprus. I often work on various art installations, live performances and audiovisual mixes.
1.2. Me + Tinkering
Since most of my projects involve live audio and video composition I often find myself tinkering with various software and hardware to come up with my own systems that give me the freedom and flexibility to express my ideas in an efficient way. Through the Mastering Tinkering course I hope to approach my usual design process in a more methodical and critical manner which would hopefully allow me to gain deeper knowledge on the art of tinkering and lead me to utilize it in a more efficient way!
1.3. The plan
My plan during this course is to use the weekly assignments and material to develop DVJ1, a modular audiovisual instrument. The "building block" philosophy will apply directly to my idea since I intent to develop modules or "blocks" that together form the DVJ1. In this project I view tinkering to be a more informed and methodical form of trial & error, and in a broader sense to be a form of prototyping which is what I hope to have by the end of this course : the minimum viable prototype of DVJ1.

Me

My workstation

1.4. Questions
At this stage here are 5 potential technical challenges that I identified:
1) Affordability: The building blocks need to be affordable. This is a challenge since most traditional audiovisual equipment is relatively expensive MIDI devices
2) OSC/MIDI support: OSC/MIDI are the "languages" which most audiovisual software use to communicate commands and actions, therefore building blocks that support them are crucial.
3) Form factor: The instrument has to be made out of easily transportable and not take too much space blocks since I'm often traveling and performing in tight spaces
4) Connectivity: The modules that make up the device should be able to connect easily with eachother
5) Ergonomics/UX: The modules and by extension DVJ1 should be easy and intuitive to use
Ideally the building blocks should address each challenge.
1.5. Tinkerspiration
The following projects will be used as inspiration when designing my own audiovisual instrument:
- Daft punk's "Alive 2007" setup : In 2007 Daft Punk performed an audiovisual show live in Coachella. Although still cryptic, year by year more details are revealed about their setup and process.
- EboSuite: Created by Eboman, a seasoned dutch live audiovisual artist, Ebosuite is an add-on pack for Ableton Live which turns it from a music production software to a digital audiovisual instrument that lets you mix graphics and sounds seamlessly.
- BLOCK SYSTEM : For my bachelor thesis I worked on BLOCK SYSTEM, a research on immersion, interaction and remoteness of a live audiovisual performance. The final prototype also featured experimental audiovisual interaction which can act as direct inspiration for the current project!
The common point between the projects mentioned is that they all to an extent use tinkering since all the prototypes were made by hacking and tinkering existing products/building blocks.

Daft Punk's Pyramid, EboSuite and BLOCK SYSTEM

week 2 - History & Mockups

2.1. History
This is an idea that I had even before this course. The project slowly started end of January 2024, therefore some sketches and first iterations were already developed by now. The first ever iteration was made up by a LaunchControl XL and a launchpad. In that iteration the idea was for the user to be able to change audio and visual parameters simultaneously. In the embedded video you can see a wizard of Oz demonstration of the idea. The user is able to trigger different sections of a song and different visual elements in a live setting via the setup. Software wise the plan was to tinker with Ableton Live, a DAW(Digital Audio Workstation) and Resolume Arena a software which allows for live visual manipulation. Currently the prototype is not in a working state but hopefully during this course this will change

DVJ1 Wizard of Oz demo

DVJ1  current state: LaunchControl (up) and  Launchpad (down)

2.2. Iteration on DVJ1
For week 2 DVJ1 was further developed from its current state. The week 2 version of the prototype consists of 2 Launchpads and 1 LaunchControl XL. Inspired by the tinkering mindset, the current idea is to use MIDI devices as modules which creates a lot of possibilities for custom mappings and workflows, however this approach still has its challenges. Firstly it requires extensive tinkering on the software side of things to ensure that each mapping has a meaningful role in the context of audiovisual live shows. Moreover although this approach offers alot in affordance via the available knobs,buttons,sliders and other potential devices that can be added, and with proper documentation it can be easy to use, there is a learning curve to the process and its price-point can be steep. In the next weeks I hope to iterate further and improve the concept.

(Rough) Sketch for modular approach

DVJ1 : week 2

week 3 - Working prototype
3.1. Iteration on DVJ1
The week 2 prototype was further developed to be in a working state. In this iteration only the LaunchControl XL and 1 Launchpad. Mainly in the software side was set up to allow audiovisual composing. Ableton, Resolume and TouchDesigner were combined in this iteration to allow the MIDI devices to manipulate sounds and visuals simultaneously. An Ableton session with a selection of different tracks was made.
On the audio side, each track was routed to a channel with a series of effects that along with the track volume could be tweaked via the LaunchControl. The launchpad was mapped to the session view, which allows it to trigger tracks or samples.
On the visual side, a TouchDesigner project was made that took MIDI and OSC input from the Ableton session and used it to generate visuals. For example a MIDI clip could be launched next to the track which includes notes places rhytmically parallel to the track. These notes were used as inputs in the TouchDesigner project to move the circles in the video in random positions and scale them based on the drum pattern of the song. Next to that some of the parameters that were mapped to audio, such as the volume fader, were also forwarded to TouchDesigner which allowed things like the opacity of the visuals to be controlled. The parameters were mapped to the controlled by MIDI however the communication between Ableton and TouchDesigner happens via OSC, specifically the TDAbleton library.

Ableton session (up) TouchDesigner Project (down)

Furthermore a Resolume Arena project was made. In this scenario Resolume is being used to project the final visuals and to add some extra effects like glitching. The TouchDesigner output is being passed to the Resolume project. To enable the effects on command, again the same buttons that affect music are mapped via MIDI in Ableton and their actions are being forwarded to Resolume via OSC which lets the software know when to trigger the effects. The library used to achieve that was the one found in the official Resolume site. Below is a demonstration of the working prototype after week 3.
A Miro board with the current functionalities can be found here. In the coming weeks the prototype will be redesigned to fit with the new concept. (Note: The board has been updated since)
3.3. Reflection
Looking at this prototype through a critical lens some changes can be made. Conceptually the DVJ1 can be adapted to be more beginner friendly. Inspired by LittleBits the original idea can be morphed into a plug and play solution where each MIDI module can be separated, mapped to a different property in the audiovisual scope and linked together to create a unique audiovisual composition. This idea is also inspired by existing modular MIDI solutions such as Roli blocks and the traditional oscilloscopes which visualize the sound that they make.
week 4 - a different approach
4.1. Iteration on DVJ1
Picking up from the reflection in week 3, the initial stages of a new lofi prototype for DVJ1 were developed from scratch. The new solution takes major inspiration from littleBits, allowing the user to arrange different modules such as knobs, sliders and buttons inside a limited "work area". Simple visual elements such as lines and shapes can be then triggered and adjusted in real time by the modules that the user pieces together. This means that different arrangements allow for different visual possibilities, making the design low threshold and high ceiling in its possibilities. The final solution should be physical and also allow the user to manipulate audio with the visuals, thus essentially making the solution an audiovisual synthesizer.
It is important to note that ideally this prototype should have been fully analogue, by more or less adopting the design of LittleBits and transforming its context by introducing a screen module and possibilities for creating visuals. However due to time restraints the prototype was developed only in a digital form, within TouchOSC.

DVJ1 week 4 lofi prototype

LittleBits by KORG

4.2. Working prototype
A working lo-fi prototype of this concept was developed. The prototype was made in touchOSC, a software which allows you to make your own interface by placing elements such as knobs, faders, buttons etc. Since the software supports OSC the custom interface can then be linked to Resolume and control a sine wave that is generated within the software. This results in a prototype which allows the user to make custom interfaces, as they would do with something like littleBits, and use them to make generative visuals. The idea is to allow the user to drag and drop different modules like knobs, faders and buttons in a canvas. The modules are linked with different properties of a simple sine wave (speed, frequency, color, multiplier) so by tweaking them visuals are generated. At the moment the prototype is fully digital and only focused on visuals, however it can be developed further so that each module also can control properties of sound (pitch, velocity, sinewave, unison etc) and have a physical form like littleBits which would allow them to be chained together with either magnets or layed out in a board.

DVJ1 modules setup

DVJ1 week 4 lofi prototype demo

4.3. Reflection
Having tested the prototype with people other than myself (my roommate), it was observed that it was far more accessible and easy to pick up and start creating visuals compared to weeks 3 prototype (which was also tested). However this comes to the expense of it being more limiting than weeks 3 design. The simplified design works well in mixing-and-matching the modules and creating very simplistic, abstract visual patterns, however it fails to give the user the freedom to customize the setup beyond the interface layout. This means that in situations where its required to create live,more intricate audiovisual performances this tool alone is not enough.
week 5 - combining solutions
5.1. Iteration on DVJ1
After exploring a different approach on week 4 I decided to take a step back and review my work so far. Although the week 4 idea had in my opinion more potential to be in line with the tinkering mindset, specially if made with physical,connectable materials like LittleBits, the original prototype offered more freedom and possibilities in the context of live audiovisual shows. Since I happened to have some upcoming audiovisual shows I decided to develop the prototype in a way that it would incorporate both the usability and freedom of the original idea and the modularity of last weeks concept. The end result was a refined version of the week 3 prototype which incorporated the drag-and-drop mechanism of week 4's design. 
The "building blocks" in the final design were knobs, sliders and buttons which when combined allow the user to control both audio and visuals for live shows. At this point the limitation is that the building blocks come in sets on the actual MIDI controllers, however a more modular solution could be having multiple building blocks on their own and implementing a magnet system like LittleBits. The parameters also have to be mapped manually in Ableton Live, which then sends OSC signals to Resolume Arena that trigger the visuals next to the sounds, however a future iteration could see that process be made more user-friendly by introducing a software with a nice GUI that would act as the main digital interface of the entire setup.
This prototype was tested in various live shows, specifically in the context of EDM (electronic dance music) raves and theater performances. The modularity that it offers allowed every show to use only the necessary amount of building blocks, for example the EDM shows used the full setup on the right and the theater show only made use of a launchpad and a launchcontrol. The prototype performed well, however some notes for future improvents were made. The notes regard mostly the software side, specifically improving the Ableton template and fine tuning the mappings. A Miro board with the final functionalities can be found here.
5.2. Conclusion
Admittedly since this was a project that was conceived before I was even aware about the goals and approach of this course the results and final designs might not perfectly align with what was expected of us. However I believe that by adopting the hands-on mindset and iterative nature of tinkering and experimenting with different approaches that are inspired by a broad variety of modular systems and materials that I discovered during this course, I managed to successfully apply some of the teachings to one of my own projects. 
The process of going through different iterations and materials expanded my toolkit and taught me how to turn different devices, such as music controllers, android devices and game-pads into building blocks for a modular audiovisual setup. This course directly influenced me to become better at what I'm already passionate about, which is working on the intersection between interactive technology and audiovisual arts. To me this is far more valuable than the alternative which was designing a building block that may aligned more with the course but not contributed to my personal growth as an audiovisual artist and interaction technologist as much.

DJV1 w3 iteration, developed. 5 modules are connected in this image (in red)

Notes for future iteration

DVJ1 in action

Programming paradigm
From my understanding what was asked for the programming paradigm exercise was too different from what I was already working on with DVJ1, so I decided to make it in a different section that is unrelated to the rest of the individual work.
TouchDesigner Card Game
As a design exercise we were tasked with designing a physical metaphor for programming. My idea was to create a card game that would teach the players the basics of TouchDesinger, a node based visual programming language which you can use to create real time visuals. 
The game should be played on an interactive table, where each card placed on it will be scanned in real time and modify the final visual which will be always displayed on the table. The cards would mirror the basic available nodes of TD, but in a more simplified and game friendly way. Each card will have an easy to understand explanation of the node and will feature a small illustration, inspired by football or playing cards.
Normally TD allows for parameter tweaking and mapping in each node, however as this is a simplified metaphor each node will be assigned random values every time they are placed. For example the noise node, will output a random noise curve, the math node will output a random function etc. By putting cards in sequence a different visual will be created every time which will teach the user how TD works in a simple yet fun way.

TouchDesigner

Interactive Table

TouchDesigner Card examples

TouchDesigner Card game concept

↑Back to Top