What do you call something with four arms, eight sticks, and no heart? A music robot named Shimon!

The two masterminds behind Shimon are Ph.D student Mason Bretan and the director of Georgia Tech’s Center for Music Technology, Gill Weinberg.

In 2004, Weinberg started work on robot musicians. He released his first version of Shimon in 2008. Through constant adjustment on Shimon’s programming, Bretan and Weinberg gradually increased the quality of Shimon’s output. They’ve been following Shimon’s music style over the years.

Shimon’s Brain

Shimon has definitely improved from what he was composing seven years ago. In the earlier years, the music robot’s compositions didn’t sound like music. Weinberg didn’t understand why Shimon didn’t sound human. Over time, the two researchers adjusted the robot’s cognitive abilities to understand music on a higher relationship, rather than note-by-note.

“When we play or listen to music, we don’t think about the next note and only that next note,” says Bretan. “An artist has a bigger idea of what he or she is trying to achieve within the next few measures or later in the piece.”

These days, Shimon improvises with other musicians (humans, so far) and even creates its own musical themes, harmonies, and melodies. In fact, watching him play with his jazz band, you’d think Shimon behaves almost like a human. He listens attentively, watches the band or composer’s body language for cues, and improvises with them—all in real-time.

“It’s beautiful to me. I’m trying to understand how he came up with what he came up with—and I can’t.”

– Weinberg

The two researchers fed around 5,000 full songs and more than 2 million motifs, riffs, and licks into Shimon’s database. Artists Shimon already knows range from pop to classical to jazz, including the Beatles, Beethoven, and Miles Davis.

What’s more, besides giving Shimon a seed (the first four measures) to use as a foundation, no human is necessary for Shimon to compose, perform, or improvise music. Wienberg and Bretan created a self-sufficient robot using Artificial Intelligence and Deep Neural Learning. Bretan says it is the first time researchers used deep learning to have a robot create music. Bretan, who receives his doctorate in music technology this summer, plays the keyboards and guitar in his spare time.

Shimon’s Live Performance

Shimon debuted as a solo composer in the Consumer Electronic Show (CES) keynote. He also played his first live performance in late June at the Aspen Ideas Festival in Colorado.

Shimon’s band from their hour-long jazz set at Ideas Fest played songs made collaboratively between Shimon and human musicians. The band, which included drummer Jason Barnes, also played some of Shimon’s originals.

Barnes is missing the lower part of his right arm, and Weinberg used a different doctoral project to help him play the drums better. Barnes played with an enhanced robotic arm that held two drumsticks. The two drumsticks responded to the Barnes’ bicep muscle signals. In return, Barnes played faster and with more accuracy than a non-robotic-armed human drummer.

Shimon’s Future Upgrades

More recently, Weinberg has been working on adding more physical gestures to Shimon’s movements. Maybe the music robot will begin bobbing his head to the beat, or he might sway to a melody. These are too easy for Weinberg, though. He wants to expand Shimon’s physical range even more.

Wienberg says seeing the Rolling Stones is as much about Mick Jagger’s strutting as it is about the songs. Weinberg wants to see Shimon move with more personality, and he wants Shimon to gesture with more precise emotional reactions. Shimon should have an onstage persona and aura. But, as Weinberg put it, “Will it ever have soul?”

Writing an algorithm for soul is elusive, but the ideas that Shimon brings to jam sessions are already inspiring soulful collaborations.

– Weinberg

“The idea is that a robot brings mechanical abilities and processing power, and humans can contribute emotion and what we call ‘soul’ and together they can start something new,” Weinberg said.

Instead of using programming to replace human musicians, Weinberg envisions Shimon as the beginning of a new era for music and augmented human creativity. “It’s all about the humans. I don’t see a scenario where robots will replace humans,” he said.

Shimon will create more original pieces as time goes on. With each different seed, the music robot will produce something unique each time.

While we wait to find out if Shimon is playing at a venue near you, check out some smart electronics that allow you to compose your own music. We love the Seaboard Block, and you can’t go wrong with a Populele.

Sources: Georgia Tech, Aspen Times, Quartz, and Noise Prn