The Q&AI: Harmonizing Innovation with AI: Redefining Music Creation and Education

0:01 [Music]

0:05 hello Bob Friday here and welcome to

0:07 another episode of Q&A in today's

0:10 episode we are going to be talking about

0:12 AI and music and today we are joined by

0:14 Nog who was previously a juniper

0:16 engineer who is now a co-founder of a

0:18 music startup called Bloom nog welcome i

0:21 thought maybe we start the discussion i

0:23 hear you guys won a Southwest South by

0:25 Southwest award last year maybe a little

0:27 bit about what award what was the award

0:29 all about no thank you for having me on

0:30 the show Bob so for those of you who

0:32 don't know South by Southwest is this

0:35 music arts tech festival that happens

0:38 every year around March in Austin uh and

0:41 um interesting story about South by

0:43 Southwest it was in 2023 when we had a

0:46 prototype of LON and we just decided to

0:48 crash the South by Southwest uh event

0:51 with our product on a box that's where

0:53 we actually got our first customer and

0:55 at was at that point we decided to you

0:57 know start the company incorporate the

0:59 company after we got the first customer

1:00 then in 2024 one of the representatives

1:03 at South reached back to us and asked us

1:05 if we wanted to participate in the

1:06 innovation awards so we did and um it

1:10 was a really interesting experience

1:11 because we were competing with uh 55

1:14 other companies or startups but uh we

1:18 were the only unfunded startup at that

1:19 point of time so it was like the four of

1:21 us carrying like huge equipment setting

1:24 up the stage when other people had

1:26 workers so it was a very surreal

1:28 experience when we won the innovation

1:30 award for human computer interaction uh

1:33 to give you a little of bit of

1:34 background on that um what we won the

1:37 award for was creating this device and

1:41 the interaction that you would have when

1:43 you played music for example we

1:46 showcased experiences where we would

1:48 visualize your in the notes on your

1:50 instrument on a screen for example we

1:53 had this game called gravity balls where

1:55 a completely new user of a guitar or a

1:58 ukulele or a piano can plug their

2:00 instrument into LON and as they explore

2:02 the fretboard or the keys they learn

2:04 what notes are present on their

2:06 instrument okay it was really

2:08 interesting to see small kids uh who

2:10 were being introduced for to music for

2:12 the first time actually make interesting

2:14 connections there was in fact one kid

2:17 who played with it for a little bit and

2:20 he was like uh and I asked him what did

2:22 you learn from the Lon and he was like

2:24 oh it looks like the notes repeat itself

2:26 after the 12th fret and he was like oh

2:28 these are the strings of the guitar and

2:30 that was really interesting for us

2:32 because what we kind of learned from

2:34 there was if you provide someone the

2:36 appropriate tools and visualizations for

2:39 them to explore uh they they naturally

2:42 learn and get interested in that

2:44 particular subject so this was the

2:45 inspiration for Loom this how you did

2:47 the whole adventure because I think you

2:49 know as I said you were you worked for

2:51 me you work for Mist you know your

2:52 location you know engineer and uh maybe

2:55 the other thing before we jump into AI

2:57 for music is you jumped off the cliff

2:59 from being an engineer at a big company

3:03 to a founder of a music startup i think

3:06 my words of wisdom was don't do it you

3:09 know you have a child on the way now so

3:12 maybe for the audience what was going

3:14 through your mind when you decided to

3:15 jump off the cliff into this little new

3:18 adventure well I I think um uh I think

3:21 one thing that you probably are aware

3:23 about me is um I am famously known for

3:26 making like passionate decisions like

3:28 this if you remember even back at back

3:30 uh in the day when I was graduating from

3:32 masters um I joined Mist just because I

3:35 loved the environment i loved being in

3:37 that startup ecosystem i loved the fact

3:40 that there were no structures and rules

3:42 and everyone was like into making

3:44 something and uh around the co time is

3:48 when I got back into music a little bit

3:50 i started experimenting with digital

3:52 music i started experimenting with

3:54 production processors effects plugins

3:56 and started my YouTube channel and even

3:59 published some music uh online that's

4:02 when my co-founder Shray uh he was also

4:05 working for another startup at that time

4:07 and he was also learning music on the

4:08 site so we would get together with jam

4:10 sessions just like how we have done our

4:12 jam sessions and from that point is when

4:14 we realized that there's really an

4:16 experiential problem with playing music

4:19 if you think about the regular person's

4:21 music journey you start with one guitar

4:23 you try to play a couple of songs you

4:25 reach a plateau then you realize okay

4:27 maybe I need a better amp maybe I need

4:29 better pedals you take them you reach

4:31 another plateau then you probably have

4:33 to learn better music and uh maybe

4:36 scales and stuff like that before you

4:38 are at a comfortable level to even

4:39 perform yeah so I I can relate because

4:42 you know I I started learning the guitar

4:43 when I was 60 i would tell you guys this

4:45 is something you want to learn when

4:46 you're a kid not when you're 60 years

4:48 old but I thought it was great that you

4:50 know you guys are going to build me this

4:51 AI instructor that's going to help me

4:53 learn to play guitar now I think we

4:55 talked about you know AI for learning AI

4:58 for content creation yeah you know I'm

5:00 not sure you actually are building

5:01 something for me yet but maybe around AI

5:04 for learning how do you see AI is going

5:06 to help the next generation of musicians

5:08 start the journey yeah so um when you

5:11 think about music learning and how AI

5:13 plays a role in music learning we can u

5:16 break it down into several different

5:18 verticals right like so the if if you

5:20 think about how you learn music with

5:22 your music tutor the first part is the

5:24 music tutor has to in the background

5:27 know what song you want to play and have

5:29 to break down that song for you right

5:31 like he has to tell you like these are

5:32 the chords that you've play in the song

5:34 this is the downstrum this is the

5:35 upstrum so the first part of learning

5:38 how to play a song or a musical piece

5:40 involves understanding what that musical

5:42 piece is so this is the transcription

5:44 problem of music which we have been

5:46 trying to solve for a very long time and

5:48 now there are like uh base models

5:51 available for doing things like chord

5:53 detection beat detection pitch detection

5:56 so an AI for example can help you in the

5:58 following way you can feed it a song and

6:01 it can return to you exactly what chords

6:03 to play at what time exactly what beats

6:06 are present at what time it could

6:07 automatically tell you the key of the

6:09 song and the BPM of the song right well

6:13 I was I would say when I was learning to

6:15 play music it was more like it took me a

6:17 long time to get the muscle memory so

6:19 when you talk about the award and South

6:21 by Southwest and gamification you know I

6:24 thought that was brilliant because I

6:25 like you know Guitar Hero you know do

6:27 you think AI is really going to help

6:29 solve that problem because part of the

6:30 problem learning is just getting that

6:32 muscle memory down and learning the

6:33 basics yeah it's it's a good point you

6:36 brought up the gamification uh because

6:38 the I think the gamification is directly

6:40 related to the muscle memory one of the

6:42 insights that we had was uh I went to

6:45 play uh FIFA uh on an Xbox but I'm used

6:48 to playing it on a PlayStation but I was

6:50 able to quickly pick it up that's when I

6:52 realized that when you are playing a

6:54 game uh you you are naturally gaining

6:57 the dexterity on using the instrument

7:00 that you are using for playing that game

7:02 now my co-founder Shay his background

7:04 comes with game development so he's been

7:06 a director of technology at Unity so

7:08 he's really good at creating these games

7:10 and that's when we started by creating a

7:13 small subway surfer type of game where

7:16 different notes make the player go in

7:18 different directions the motivation

7:20 behind that was if I can just take your

7:23 mind off the playing music for a second

7:25 and make you focus on maybe some other

7:27 goal like playing this game you

7:29 naturally gain the dexterity on your

7:30 fingers while playing the game now how

7:32 can AI help in this place would be AI

7:35 would help could help bring an infinite

7:38 generation of such sequences right for

7:40 you to play so as long as you tell okay

7:42 these are my practice scales i want to

7:44 practice the pentatonic scale i want to

7:46 practice the major scale ai could take

7:48 that scale extract a subset of notes

7:50 from that scale and create the game play

7:53 for you right create like what notes to

7:55 hit yeah you know my neuroscience for

7:57 instance you know when I was learning to

7:58 play you know which I think is true is

8:00 like you learn to practice a little bit

8:02 you go to sleep and you have that sleep

8:04 process actually reinforces that muscle

8:06 memory and so I think that is that you

8:08 know that gamification will help that

8:10 piece of the puzzle you know now the

8:12 other interesting topic maybe for the

8:14 audience any startups you know when you

8:16 look at the startups out there for the

8:18 learning process yeah any startups you'd

8:20 recommend the audience to take a look at

8:22 if they're just starting their music

8:23 journey yeah so uh back to my

8:25 transcription there are a couple of

8:27 startups coming today that are putting

8:29 up like transcription models out there

8:32 uh two uh notable ones that I have used

8:35 back in the past are uh songster and

8:38 moises so uh songster allows you to

8:42 upload your music like upload a YouTube

8:44 uh link for example and it can generate

8:46 the sheet music right out of the audio

8:49 and that is uh a machine learning model

8:51 they use that convert um audio to MIDI

8:55 and Moises uh has a whole suite of

8:57 different models that they use so you

8:59 could uh upload your audio file in there

9:02 and you could find the BPM of your song

9:04 you could find uh the chords of your

9:07 song and you could even do things like

9:09 speed up your song slow down your song

9:10 without affecting the frequency so there

9:12 are a lot of startups that have solved

9:14 the problem of um the transcription so

9:18 understanding what music that what music

9:21 you are giving to them but the next part

9:24 of learning which is the actual

9:25 real-time part where either your music

9:28 tutor sends you a video or you're

9:29 sitting with the music tutor and

9:31 following him along currently there are

9:34 not really any startups that are working

9:36 in the visualization field of the

9:38 transcription right now all

9:40 transcriptions are just visualized in a

9:41 very basic sheet music sense and that is

9:44 where we are uh focusing some of our

9:46 efforts on yeah because I mean if I

9:47 talked to most of my friends who are

9:48 like trying to learn music seem like

9:50 it's a combination of YouTube friends

9:52 getting pointed thing i mean when you

9:54 look at the you know kind of the cost of

9:56 a music instructor is usually $100 an

9:58 hour or something you know do you see us

10:00 getting to a point where we have kind of

10:02 a combination of AI instructor plus real

10:04 instructors helping this journey yes uh

10:07 and I I see it in two ways right one is

10:11 I see the music instructor be using AI

10:14 in order to ease his workflows right so

10:17 using AI to string better songs for his

10:19 students together using AI to break down

10:21 songs better for his students that is

10:22 one but two we at LON we believe that

10:26 any art form should be approached with

10:29 uh the sequence of play learn and create

10:32 while today most of our art forms we

10:34 approach it with learn play and create

10:36 so the idea is to give you this um uh

10:40 this playground of tools that you can

10:42 use to understand or at least dabble in

10:45 music and I feel a a big chunk of the

10:49 initial person's journey can be replaced

10:51 with these AI tools and experiences till

10:53 they at a point where they really need a

10:54 human to take them to the next level

10:56 yeah yeah now for me personally you know

10:58 it looks like the journey starts and

10:59 I've been on the I think stuck in this

11:01 fiveyear part of like practicing it seem

11:03 like you know you go through from that

11:05 to open mic to band and then we finally

11:07 get to the I want to create content for

11:09 the YouTube or something yeah and I

11:11 think that's the other thing you're

11:12 focused on now is you know how you know

11:14 once you get to this other level yeah

11:16 you know how is AI going to help the

11:18 professional you know when you're at the

11:19 I'm trying to create content you know to

11:22 publish you know it seems like AI has a

11:24 role here too also yeah i I and we think

11:27 of this role as like the bridging the

11:29 gap type of role if you think about at

11:31 least my personal use case was when I uh

11:35 started like playing a little bit bit of

11:38 music and producing it I realized that

11:40 in order to just produce a video and

11:42 like from you know even through your

11:44 podcast journey you know it's not just

11:45 about the content you have to do the

11:47 post-prouction you need to know how to

11:48 capture the audio and the video sync it

11:51 there are so many extra things to learn

11:53 and I feel this is where AI can help the

11:55 content creators so for example in one

11:57 of the last compositions I did I just

12:00 made the composition raw and I sent it

12:01 to an AI that mixed all my tracks

12:04 together so it it adjusted the volume of

12:06 my bass my guitars and everything to

12:08 make it sound good now how's that AI

12:10 because you know in networking I always

12:12 tell people you know AI is really just

12:14 the next step in automation you know you

12:17 know if I look at music you know we've

12:18 had synthesizers and pedals and all

12:21 types of tools for adjusting the sound

12:24 of our music you know where does the AI

12:26 really come in why is AI any different

12:28 than the the pedals I have at home

12:31 that's a really good question Bob and I

12:33 think before we touch upon that question

12:35 it comes back down first to the

12:37 architecture uh you talked about

12:39 synthesizers so in the past they used to

12:42 have a really big machine called a

12:44 Eurorack where they used to plug cables

12:46 together in order to create sounds now

12:49 we are talking about sounds being able

12:51 to uh we are able to create sounds using

12:53 a CPU and a GPU you're right in the fact

12:55 that we probably don't uh we pro we

12:58 probably on a subtle line of where is

13:00 the difference between producing sound

13:02 with a human producing sound with the AI

13:04 but an AI can do things much faster and

13:06 can experiment with way more parameters

13:09 much quicker that than what a human can

13:11 do so the real question to ask is is the

13:15 AI on par with the human uh in these use

13:18 cases so when it comes to creating sound

13:21 and mixing mastering the question is is

13:23 an AI on par with a sound engineer when

13:26 it comes to creating music the question

13:27 is is AI on par with the musician yeah

13:30 well that may lead into another topic

13:32 you know in acting you know a year ago a

13:34 year or two ago right it was all about

13:36 you know is AI going to replace acting

13:38 or actors and everything you know when

13:40 you look at music as an art form you

13:42 know is AI ever going to be on par with

13:44 humans and actually starting to replace

13:46 humans musicians are we getting to that

13:48 point uh that's a that's a tough

13:50 question Bob and I don't know if I'm I

13:53 can give an accurate idea of this but

13:55 when in 2022 when these models started

13:58 coming up uh I did not think AI music

14:02 was really good but if you look at the

14:05 recent generations from you know

14:07 companies like Sunno UDO or even a

14:09 couple of open- source models that are

14:11 out there now like notable one is UA Y u

14:14 uh the music is surprisingly become

14:17 better it's still not on par with a

14:19 human uh and there are also some

14:20 technical challenges like most of these

14:22 AIs produce only mono audio it cannot

14:24 produce studio quality audio but if we

14:27 have come this far this soon I don't

14:29 know how to extrapolate that into the

14:31 future yeah and I think we were talking

14:33 a little before the show here about

14:34 lawsuits and yeah you know we know music

14:37 is copyrighted for 17 20 years yeah is

14:39 that copyright protection actually

14:41 protecting that music from being used to

14:43 train the models or or is that an issue

14:45 right now that is a big issue right now

14:47 and let me go back to one of your famous

14:49 things right like good AI is like making

14:51 good wine you need good grapes to make

14:53 good wine you need good data to make

14:55 good AI so obviously like it's obvious

14:57 that the good data that you want to

15:00 train a music model with is music that

15:02 is available out there but um if you

15:04 have been following since last year

15:06 there's been like a lawsuit that Sunno

15:08 and Yo are under because they did admit

15:11 that they have used uh you know

15:13 unlicensed music for training their

15:15 models now they claim this is

15:17 inspiration or fair use but that's what

15:19 the lawsuit going on is about like is it

15:21 really fair use if your model is using

15:24 the work of our artist to train itself

15:26 and what do you think

15:29 I thought it was inspiration till one of

15:32 these models spit out like something

15:35 verbatim then if if it spits out

15:37 something verbatim that really makes you

15:39 think uh at a philosophical level okay

15:42 what did this model learn did it learn

15:44 it or did it take inspiration

15:46 You think you think the copyright

15:47 protection would protect that music from

15:49 being used for publishing or even for

15:51 training you think the copyright would

15:53 be protecting that i think the game is

15:56 more about the lawsuits after you break

15:57 the copyright

15:59 yeah now now the other thing I'm known

16:01 for saying is like you know when I

16:02 started Mist I built an access point

16:04 because I wanted to make sure I could

16:06 get the data I needed to answer the

16:08 question around why you're having a bad

16:09 poor user experience yeah you know is

16:11 this true in music where does the data

16:12 come from do I need specialized hardware

16:15 or how do I get the data for all this

16:16 great AI that I'm doing you're doing

16:18 yeah so uh if you think about mist the

16:22 access point existed because another

16:25 generic access point could not do what

16:27 you needed in order to drive the

16:28 experiences the same way if you look at

16:31 a lot of music equipment today most of

16:32 them either run with analog circuits or

16:35 FPGAs by converting them to a CPU what

16:39 we are able to do is we are able to

16:42 train models on each of these different

16:44 pedals so we take an actual analog pedal

16:48 find out its impulse response using an

16:50 impulse response matching algorithm and

16:52 then that can be converted into a

16:54 software module that goes in our product

16:56 so in essence what the specialized

16:59 hardware or more so the change in

17:02 architecture that we have brought in

17:04 what it helps you do is it helps you

17:06 create a software module for almost all

17:08 the physical products that exist out

17:10 there in music and put it into this one

17:12 product well now I have to say all my

17:15 engineering friends most of them go off

17:16 and do other engineering startups i

17:18 think you're the only one who jumped off

17:20 the cliff to do a music startup i want

17:21 to thank you for joining us today i want

17:23 to thank the audience for joining us

17:25 today and for other episodes of Q&A

17:27 please check it out on Spotify thanks a

17:29 lot Bob