TouchSynth

This is TouchSynth, my high school senior project.

It’s an Arduino Mega based Digital Audio Workstation that I created with my partner Tim Boyle Jr over 9 months, from 2010 to 2011.

Hardware

  • Arduino Mega 1280
  • Maxim 7224 8-Bit DAC with Output Amplifier
  • Longtech Optics LCM1602A 16×2 LCD
  • Philips HEF4794B LED driver
  • NFM-7881 8×8 LED Matrix
  • 4-Wire Resistive Touchpad

Putting together the hardware was months of work. We started with breadboard designs to test the circuitry, and eventually moved to a vector board and a project box to get the project into its final form.

TouchSynth Right Side

The right side of the project box.

TouchSynth Left Side

The left side of the project box.

TouchSynth Back

The outputs.

For the stripboard design, we designed it digitally before we got started soldering. It turned out alright, if not a bit messier than we hoped. A large part of the issue lies in having to send 16 wires to the 8×8 LED matrix. That could have been avoided with multiplexing and demultiplexing, but we decided not to spend the effort at that point, as we had a tight deadline.

Stripboard design. Top on the left, bottom on the right

Stripboard design. Top on the left, bottom on the right

The inside.

The inside.

The inside.

The inside.

The inside.

The inside.

Software

All the code is available on https://github.com/stevemostovoy/TouchSynth.

The majority of the project was based in software. As we were working with the Arduino Mega 1280, we were limited to 8KB of EEPROM, 8KB of RAM, 128KB of programmable memory, and a 16MHz processor.

The code was about 1000 lines of C. Let’s break down the interesting bits.

Touchpad

The touchpad was a bit difficult to get working correctly, as there were several questions we had to address. First, what’s the proper way to get the X and Y information from the touchpad? Second, a part of the touchpad was significantly bigger than what is seen on top of the box, so we need to figure out whether the value is completely invalid, or if it is just outside of the 8×8 matrix, or if it’s a valid X and Y within the range of the matrix. We defined four constants through some trace statements to specify the location of the 8×8 matrix, named TSBottom, TSTop, TSLeft, and TSRight.

We found advice from reading from a four wire resistive touchpad using an Arduino. It required six wires – four digital, to control the output mode of the touchpad from horizontal and vertical, and two analog, to read the values. We called the four digital ports DB, DT, DL, DR, and the two analog ports AR and AT.

To read the Y value, we used the same thing, but you flip all the ports. DB with DL, DT with DR, and AR with AT.

Getting the touchscreen X and Y is done through the use of the margins. If it’s outside the margins on one side, it’s a -1, and on the other side, a -2.

Getting the row and column on the 8×8 is then as simple as saying round(x / (Width / 8)) and round(y / (Height / 8)).

The touchpad was read every 1ms in the MsTimer update loop.

Buttons

The push buttons were fairly simple to handle. A bit of debouncing, and checking them in the main program loop.

That function is called every 1ms, and it checks all five buttons and the play switch. The play switch will stop/start the threaded update loop and let the play function manually handle updating the LEDs and the output ports.

Debouncing is handled by checking if the time between now and the last button press is less than the debouncing constant, which is 10 milliseconds.

Matrix

Getting the 8×8 LED matrix functioning properly involved a lot of math and bitshifting, which is always fun. That was because it wasn’t simply 8×8 in the program, but rather had a height of 24 and a maximum width of 768. Because memory was a large concern because of the audio requirements, it was not acceptable to have a boolean array of 768*24. Instead, we used an array of 16 bit words that was 48*24. Then, we had a byte array with 8 bytes to store the currently displayed matrix from the full matrix.

To get the selected matrix, we had the following function:

This used some bitshifting tricks to get the bits needed from the overall matrix array into the selectedMatrix array. The remaining matrix operations could not use the selectedMatrix though, because they needed to update the underlying matrix structure, so bitshifting needed to be employed again. Here is the function to set an LED based on a row and column (assumed to be between 0 and 7, inclusive).

The update function which was run every 1 millisecond primarily handles updating the screen. Here’s the relevant excerpt:

The built-in _BV function converts an integer into a byte where that bit number is set. For example, 0 becomes 00000000, and 1 becomes 00000010. This allows us to select one row at a time. Since this is called every 1ms, the entire matrix is updated once every 8ms.

Menu

The 16×2 LCD Matrix was surprisingly easy to work with. It’s a self-refreshing device, there was a library available for use, and the interface was only three wires. To display “Welcome to TouchSynth” and have it stay on the screen, we used delayed function calls using function pointers, timed through the update function. Here is the function to display the LCD title at the beginning of the program.

The update function will call the function passed into startTimer after the delay, which is easy because the update function is called every 1ms consistently.

The menu structure is not something I’m proud of; it was well before I was comfortable with data structures in C, and I hacked it together early into the project’s conception and it worked. Near the end, I spent a few days trying to organize it into something much easier to manage, but it had some bugs and we were running out of time, so I reverted.

Essentially, the entire menu is hard coded into different arrays like so.

and so on. The worst part is the code to handle navigating through the menu. Here is the menuIn function.

It has all sorts of code smells, and it’s impossible to be able to tell what it’s doing without looking up all the various arrays. The interesting part of the menu system are the settings. They are localized, so different parts of each composition can have different tempos and volumes. There’s a byte associated with each of the 16 width sections of the grid, and that byte contains all the settings for that block. It makes it rather confusing to read, and it wasn’t a necessary optimization, as it only saved less than a hundred bytes.

The rightmost bit was whether or not that section used local or global settings. The next four bytes were the tempo of the piece. The last four bytes were supposed to be used for volume, but that feature was removed because it was slowing down playback.

Saving and Loading

Saving and loading didn’t involve too much on the device itself. Press the save button, and it uploads all the settings and the full active composition into the EEPROM. Press the load button, and it downloads all of it. The complexity came from wanting to have support for a Java based computer client that could upload and download files from the device through the serial connection. We ended up creating a really simple serial two way communication protocol. Here’s the download function on the TouchSynth’s side.

As for the Java programs, we ended up creating two separate programs. One program handled song viewing and editing. The songs were stored in a binary “.comp” (for composition) format, and this graphical editor allowed us to quickly create songs. That’s how we ended up programming a simplified full version of Beethoven’s Für Elise. The second programmer was a file manager, which managed the composition files, and can upload and download from the device.

To make the Java interfaces, we used the Processing library, which was an up and coming  technology back in 2011. Here’s the Song Viewer and the File Manager side by side.

Side by Side (Random composition displayed)

Side by Side, with a random composition displayed.

Audio

Of course, the most important component of this project was the audio. It took many attempts for us to figure out a way to get fully dynamic audio playback on the device running on only a 16MHz processor. The first attempt was an idealistic one: Store one array for each waveform at the minimum frequency, and scale it down based on the target frequency when required. If possible, cache the result. As soon as we got to testing that, though, it became obvious that this was much too slow, and the more notes were being played simultaneously, the slower the playback became. It had to be able to play anywhere between 0 to 24 notes with absolutely no difference in playback speed.

The obvious solution is to store every waveform in every possible frequency, but we only had 8KB of ram! We thought to use the EEPROM, but then that would get in the way of the saving functionality. After doing more research, I realized it was possible to store the notes directly in the program memory, of which we had more than enough of – over 100KB free. So I wrote a simple program to generate amplitude lists for sin, triangle, saw, and square waves at every frequency between A3 and G#5, the range of the device. Then I wrote them to the program memory. It doesn’t look great, but it works perfectly.

The pgmLoc array stores the array positions that each different frequency starts at. They get smaller and smaller because we started at a low frequency, long wavelength, and end up at a high frequency, short wavelength. Each waveform is a full period.

From there, we had to design the play function. In short, it had to do the following:

  1. Play the song at full speed with no interruptions and handle varying tempos.
  2. Display the song as it plays and highlight the currently played column.

Here’s how we accomplished each of these.

1.) The play function starts off by calling noInterrupts(), which disables the MsTimer2 update function from being called. Then, interrupts() is called again at the end of the play function. This ensures that nothing will happen other than the play function itself. The play function plays the currently selected column only, and is called again if the play switch is still on at each instance of checkButtonState().

To get the current note information, it uses the checkGrid() function.

This stores the number of selected notes in the column, and fills the array with the row numbers of those notes. Then, the “length” variable is set based on the tempo.

The length variable specifies how many times an audio value should be computed and output.

With that, the final output loop can begin.

PORTC is the 8 bit output port for the digital to audio converter.  The lastAudioValue is stored for after play is stopped – if the lastAudioValue was, say, 128, it’s better to keep PORTC set to 128 in the update loop, because that will generate no sound. If PORTC was later set to 0, it would create an audible click.

By waiting for the amount of microseconds to pass to be 96, it ensures that the loop runs in the same amount of time no matter what. Through testing we found that even with 24 notes activated it ran faster than 96 microseconds per iteration, so we set 96 microseconds as the floor for the iteration time.

The addNotes function takes all the notes in the current column and adds them into one byte. Here’s how it works.

A 16 bit word is used to store the result of the additions before the division to ensure no data is lost.

Then, it iterates through all the notes, and for each one, it calls the built in pgm_read_byte_near() function, which reads from the PROGMEM space that the waveforms are stored in. The waveform variable is a pointer to the selected type of waveform. As mentioned earlier, pgmLoc[] is an array that stores the location in the waveform array for each note, so pgmLoc[0] will get the starting location of the lowest frequency note, pgmLoc[1] will get the starting location of the second lowest frequency note, and so on. Finally, positions[] is an array to store the current index of each note. Because each frequency has a different length, it’s possible that some waveforms will be played 3 times, and some will be played, say 4 and a half times. This array allows for them to each remember their current position and smoothly go from start to finish over and over.

The position is then incremented, and reset if it goes over the length of that note, and then after the loop the tempBuffer is averaged with the number of notes.

2.) To display the song, PORTA and PORTL, the 8 bit ports connected to the 8×8 LED matrix, needed to be updated appropriately. Since they needed to be refreshed quickly, this needed to happen within the main audio loop.

The row variable starts at -1, so the first row to be displayed is 0.

Count also starts at 0. Let’s break down the PORTA assignment.

If count is 0, then PORTA is equal to selectedMatrix[row].

Otherwise, it’s equal to selectedMatrix[row] & 10000000 for everything except the last seven notes. For those, distance is set to the length of the song (numColumnSets*16-8), and is used because the screen can’t scroll anymore during the last 7 notes, so that highlights the 6th, 5th, 4th, etc. column instead of the 7th one.

By doing the bitwise and, every column other than the currently playing column is set to 0 on PORTA when count does not equal 0. Count is equal to 0 one in 7 times because of that conditional at the end. This means that the first column is refreshed seven times as often as the other columns, and is therefore noticeably brighter. Desired effect acheived!

Conclusion

TouchSynth was a remarkably fun project to work on, and taught me a lot about how not to organize code, how to think in terms of data, and how to optimize in the cases where optimization is definitely needed. It also taught me not to pre-optimize, after I saw how ugly the settings code got for no reason. That could have been solved through some abstraction, but it was an unnecessary effort. I’m very happy with the final product, and I’m pleased with the Arduino’s robustness and reliability. It’s amazing what you can do on a 16MHz processor, isn’t it?

4 Responses to TouchSynth

  1. […] Steve Mostovoy and his partner Tim Boy Jr, worked for over 9 months to  create a TouchSynth and share all the documentation about it. […]

  2. […] Steve Mostovoy and his partner Tim Boy Jr, worked for over 9 months to  create a TouchSynth and share all the documentation about it. […]

  3. […] eventually moved to a vector board and a project box to get the project into its final form,” Mostovoy wrote in a detailed blog post describing the project. ”For the stripboard design, we designed it digitally before we got started soldering. It […]

  4. yerpa58 says:

    Excellent project! I just found it, made me wonder what you are up to after high school. Thanks for posting!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">