Juniper Marvis Minis, LEM, GenAl/Customer Troubleshooting, and Marvis Actions 2.0

0:00 marvelous Minis this is Sadira's favorite baby new baby in the block uh

0:06 if you listen to Sedir there's only two problems life should be simple right

0:11 either Wi-Fi can't connect i don't know if anyone's familiar with that problem this morning or Wi-Fi sucks right and as

0:20 long as we have users on a network we have plenty of data to understand their postcon pre-connection and

0:26 postconnection problems but what happens when you don't have users on the network

0:31 that is where Marvis minis comes into play marvis minis is the first synthetic

0:37 user on the market right now that allows us to ensure that all critical network

0:43 services and critical business applications are actually up and running

0:49 before the business opens in the mornings so this is what we announced

0:54 last year at MFD this year what we're announcing is

0:59 Marvis Minis client to cloud application visibility this is where Marvis Minis

1:07 now has complete visibility from the client application

1:12 to the application in the public and private cloud so we have synthetic users now going

1:21 from everywhere on every miss device and in the internet pops across the internet

1:28 this will allow us now to get to the root cause of a client the application

1:33 problem and isolate it down to the router or the service provider in between those two aspects and what

1:41 you're going to see coming up here in Sedir's diml is that Marvis minis sole

1:46 that is going to be the industry's first way of actually taking data and getting

1:51 to the root cause of that client cloud user experience

1:56 time for razledazzle all right good morning everybody thank you so much uh for hosting us we are

2:03 excited to be here uh you know if you haven't uh seen the press we actually do our big press announcements uh like SUJ

2:11 said uh on mobility field day there's actually a press release that went out today around this Marvis client to cloud

2:18 story we're very very excited because you know as we look at this thing Marvis

2:23 Minis has become sort of the new source of truth in networking what does that mean what that means is you know Marvis

2:29 minis doesn't take a weekend off they don't have a bank holiday if they're in Europe you know they uh they you know

2:34 doesn't actually work on a a laptop with CPU pegged or is in a a bad Wi-Fi area

2:40 marvis minis has become deterministic so with that determinism what we've been

2:46 able to do is actually at an entire organization level I want to say hey I

2:51 want to understand how network services are doing at the full organization right because very deterministic data is

2:58 feeding into this to say okay which sites are having DHCP issues which you know which locations are having uh DHCP

3:05 DNS radius issues you know being able to understand at a global level for an

3:11 organization then we say okay this is great but what about you know

3:16 applications now for the first time this is the Marvis nin application service

3:22 level what this is doing is at an end toend level being able to say which

3:27 sites are having failures and for an individual site where is an application degrading so think about you know a

3:35 million minis around the uh around the world feeding into my application experience and basically you know for us

3:43 you know most of the other systems in the world focus on in the industry focus on observability for us it is all about

3:50 actionable insight we want to bring you to an actionable data point to say hey

3:56 for this user from for for on this application at this particular site it is either the wireless the van the LAN

4:05 it may be some somewhere deep into the internet circuits uh you know maybe it is closer into the public private clouds

4:11 you know so now with minis deployed globally right uh we've taken an

4:17 architectural approach to this historically every other vendor wants you to deploy overlay sensors agents on

4:24 laptops Raspberry Pies you know uh you know costs a lot of money built into the

4:30 subscription of current Marvis subscription for our global customers marvis minis client to cloud means that

4:38 architecturally no sensors no agents no Raspberry Pies no deploying on your you

4:43 know an agent on your router minis is running across the global internet across all the pops and so what does

4:50 that mean what that means is now there are minis running in CDN pops around the

4:56 world and so we are establishing this last mile baseline for every one of our

5:02 customers and so now that we know this a million minis around the world are contributing to the answer on if one

5:09 individual application is slowed down at one site at several sites multiple

5:15 applications at a site multiple applications across the entire organization we're able to actually

5:21 pinpoint to be able to get to you know is it an application issue is it a a

5:27 point in the circuit where in the circuit uh you know is the application degraded this entire picture we've now

5:34 appended to Marvis last year what we launched was on the LAN uh wireless side

5:40 uh with an agent sitting on the on the SD van circuit as well right so this is

5:45 Yeah Peter good um you've talked about across multiple sites within an organization but are you doing anything

5:51 across organizations that there's a cloud-based application that's gone down at cloud level that's used by multiple

5:57 organizations are you looking at that yeah so so actually the the reason we keep saying a million minis around the

6:04 world is with the millions of you know Juniper APs deployed out there all

6:09 contributing to the health of a van circuit right and so if if everything's

6:15 pointing towards hey you know my SAP application or my Salesforce application

6:20 is degraded yeah that's a global thing that we're learning but we want to

6:25 localize the insight I'll give you an example we had a customer open a ticket saying "Hey I have 10,000 APs down

6:32 right?" And you know we step back and say "10,000 APs don't commit suicide at the same time you know what really is

6:38 going on?" Looked back at other customers and they've had outages as well and turns out you know um a a

6:45 regional pop in Western Canada was down so now with this this insight Peter what

6:50 uh what Marvis actions is going to show is not 10,000 APs are down hey you know

6:56 uh um you know for your own network or someone else's network it's going to say this regional ISP in Western Canada is

7:03 actually down and and that's a global learning right so that that's the whole point of this thing is the global

7:10 learning yeah you said you're getting the data on the POPs is that because you've got agents or business

7:16 partnership within the POP or is that coming from the sensors or the organizations that are connected to that POP um uh within the POPs as well yeah

7:24 and and we are we're managing that globally customers don't have to uh lift a finger for that and if a customer did

7:30 want to deploy one of their own agents at a pop that you did not yet service correct how would they do that so uh

7:36 that is part of the future roadmap it's not there in uh on our first release but we do want to enable customers to be

7:42 able to deploy Marvis workers uh in their own data centers in in a VPC all

7:48 all that kind of stuff yeah we're that's not part of you know this this initial uh realm of the launch

7:55 so you're talking about taking pop data Marvis minis there and using every other

8:02 customer that has minis enabled to be like dist distributed sensors and you're

8:08 then using your own learning based on all that information to tell everybody where yeah so so again if it is within

8:15 the network wireless LAN van obviously that's your own data if it is on the

8:21 wide internet and we are realizing hey a western Canada top is down now for every

8:26 organization in their own Marvis actions it's going to say "Hey it's this ISP that's contributing right how they got

8:32 the learning is through their own sensors or through other people's sensors." Yeah okay super very good uh

8:39 with that Bob uh you know we'll uh we'll actually maybe a couple of quick stories marvis Mini because it has become truly

8:46 the source of truth in networking uh has become very deterministic in in in

8:51 delivering some amazing customer outcomes uh this is an example of a a a you know uh a a company where we were

8:58 detecting application reachability issues uh they were saying well nobody really opened a ticket yet uh again

9:04 every Marvis Minis failure has a dynamic packet capture waiting for you and that basically actually it's it's imagine

9:11 having the video footage of every crime committed in the world and this is not even a real crime it's a simulated crime

9:17 committed in the world we have the video footage waiting for it so they went back and you know a good old MTU problem uh

9:24 actually turned out to be what was causing this and and so again uh without users having to experience failure we

9:30 can get that get them ahead of this similarly we find a lot of scenarios where hey uh all my SD van is deployed

9:36 all the hubs should be active and up and running active active scenarios and sometimes one of the nodes is actually

9:42 not a real customer uh in a large retailer we found uh so we're finding all kinds of interesting scenarios with

9:49 Marvis minis with that Bob over to you so Bob or Sadir can I ask um so it

9:57 sounds like Marvis minis is enabled on the access point yes and so then my question is you said if if it's enabled

10:02 in the organization what what does that entail and then what kind of telemetry is it sending up and which organizations

10:07 might choose to do or not do that good question J so basically um at an organization level they can uh turn it

10:15 off Marvis if they would like to right uh so step one to but to enable it means

10:21 that they will come in they there is no configuration they will autolearn what

10:27 SSIDs there are meaning what VLANs those SSIDs drop into what the DHCP DNS radius

10:33 services are we autolearn all of that and then we start to validate and and from a workload perspective some

10:39 customers say oh my god is am is am network going to get overwhelmed with all this synthetic testing at all times

10:44 we're watching that our workload load is less than 5% of the overall workload

10:50 that's number one and so um but from a configuration perspective enabling it perspective is really one checkbox to

10:57 say "Yep I want this," or "Nope I don't want this." The only other configuration that we allow at this point is "Hey I

11:04 may have a private application that you may not have learned or known." And so you know go to this IP address do an

11:10 HTTP curl get me if I'm getting a reachability response uh so we can actually define in this new context of

11:18 deep into the network you know application visibility uh you know we have a certain set of applications we're

11:24 going to let them do you know complete visibility into there we're doing continuous monitoring literally uh you

11:31 know every minute uh you know loss latency jitter across the entire van path okay so that makes sense to me for

11:37 everything inside the organization yeah but you also kind of mentioned this the kind of distributed prop pop that everybody participating in this client

11:45 to cloud yes experience reporting so I'm assuming it's also reporting up in for

11:50 that that's correct if if but if they turn it off for their organization there is no last last mile latency none of

11:57 that for them because they've turned it off okay if they're enabled then we're learning that last mile latency across

12:02 the global CDN pops okay yeah it's a large experience model so

12:09 this is where we're taking Zoom Teams data we're joining it with network data

12:16 and we're basically training models now that can actually predict that Zoom teams user experience and then we're

12:23 using some fancy math called Shappley that actually looks at the contribution margin of every network

12:30 feature to that user experience minute so this is like second generation SLES

12:36 first generation is around trying to get network features how they correlated and predicted a bad user minute this is

12:42 actually looking at how each network feature contributes to that bad user minutes so when we look at latency we

12:48 can now tell which features contributing the most latency to the problem now this year what you're seeing us announcing

12:55 we're basically bringing new network features to the model we're bringing new

13:00 Zoom Teams features to the model and then we're bringing a new genai summarization to the chappley results

13:07 but most more importantly what we're announcing here today is a generalized

13:13 large experience model that will now let us to predict the video collaboration

13:18 user experience on any network even if they don't have zoom teams users this is

13:24 what I call the needle in the haystack right most if the signal is big enough

13:30 they don't need fancy math to see the problem the average IT person can see it that's why they value the SLES but when

13:36 you have what I call needle in the haststack this is like where we had a customer that was having a problem but

13:43 the model highlighted it was really a radar channel that was causing a problem that we weren't able to see on the radio

13:48 narrowband radar we had another issue where basically they were having problems at a

13:54 site in India they couldn't find out why certain people were having a problem it turns out it was a misconfigured VPN

14:00 gateway people coming that were getting routed to Australia we had a professor at a university was having problems at

14:07 lunch this model was able to track down it was basically a bunch of students walking past the office during lunchtime

14:13 was causing problem so large experience model is what I would tell you is your needle in the haststack problem said

14:18 dear you want to come up we'll give them a demo oh no this is the first thing I'll

14:26 talk to yeah thank you Bob uh so basically what we're doing is the large experience model so far uh was was

14:34 deciphering okay uh what is contributing to my team's zoom problem now the Marvis

14:41 minis client to cloud data is going to add to the features that we show under that model to say oh yeah your teams or

14:49 zoom is degraded not just because it is Wi-Fi or you know or the wired or the

14:54 van but maybe it is you know this third hop into the uh into the data center

14:59 that's causing that sub features now Marvis minis is feeding into our large experience model and together the idea

15:07 is that we're landing them into Marvis action right into what do you need to do

15:12 for us the crusade has always been for Bob AI is artificial intelligence for me

15:17 AI is actionable insight that's it it's not that much more complicated than that is how do we deliver actionable insight

15:24 to when you need to take action what do you need to do so what we've done in terms of the demo uh in terms of the

15:31 product delivery here so um last year we've talked about teams and zoom integration and so what that is now you

15:38 know fully available for our customers Marvis customers worldwide and what that entails is you know if I connect to my

15:45 teams in Zoom using uh um an API token so now cloud to cloud we're integrating

15:51 with teams and zoom data and getting that data for every individual user

15:56 every minute we have an a judgment on that minute is that a good team's minute

16:01 or a bad team's minute so every user every team's minute if there's a 100 people on a team's call that's 100 user

16:09 minutes for every physical minute and then once you have that kind of data our true north is we got to get this to you

16:16 know 100% good right and so here's an example this is Juniper's global network

16:21 when this model was started you know we had 30% failure 20% failure lots of

16:27 actually really bad you know Teams and Zoom user minutes this is our headquarters in Sunnyale and and you can

16:33 go in and say okay 16 APs are having you know delivering some of these bad teams calls out of the 629 APS now how do I

16:41 understand this how do I make this better and that's where the sharapply model came in and said ah okay we can

16:47 actually you know pinpoint to this was VPN this was CPU on your laptop is cing

16:53 a causing a bad team's call Wi-Fi switching this is the end result basically where we were able to come to

16:59 almost 0% % on um the Juniper uh teams experience today at a global level now

17:06 what we're talking about is without connecting teams in Zoom you know god forbid you're still using WebEx right or

17:12 uh you know you know Google Meet or something without connecting to those um

17:17 you know collaboration systems can we pass judgment on your experience so what

17:23 this you know generalized model is saying with no data coming from uh any of the collaboration applications can we

17:30 tell you if you're having a bad team's if you could be having a bad team's experience and if that is the case what

17:38 could the the contributors be and this is the sharply model that's deciphering that and so here's an example of an

17:44 organization where we don't have any teams and zoom data coming from that organization and we are saying you know

17:50 what likely you're having almost um you know 175 overall bad minutes a lot of it

17:56 is contributed by the van so this is the purpose of the generalized model right think of how large language models were

18:04 built is just learning language learning the internet learning the pages everything that's available for us what

18:09 we have done is now literally billions of teams minutes that we've studied over

18:15 the last uh you know 12 months or so we now have been able to legitimately build

18:21 a generalized you know model for collaboration judgment and and that's what we've been able to break through

18:27 reactive that's I really like the idea of uh finding like a a general model i think people are expecting that nowadays

18:33 with AI correct my next question would be uh could you extend the model for

18:39 other applications the use case I'm thinking about is you know university and they have like a online exam and

18:45 they want to know if in that room the network will support the capacity for x amount of devices it would be great if

18:51 we have like a model one day that we can just ask you know do you think you know the network will support that in that

18:58 room and then it will tell us yes or no you probably need to add APs or something like I think it would be a

19:03 great kind of um um enhancement to a model like this i I'll give an answer

19:10 Bob and and you step in as well so Fran the the the most stringent app on our

19:16 networks today is the real-time collaboration apps right this is the hardest app on our network which is

19:22 teams and zoom with basically real time you know voice and video and so the we

19:28 went after this saying this is the canary in the coal mine if if this one is saying all green you know yeah you're

19:34 able to do the the assumption is a salesforce or a service now will work

19:40 okay or even an education app or an EMR app in healthcare should work okay given

19:46 this is actually declaring victory so inert that's why this is sort of the the

19:51 the harbinger for okay what is actually the application experience on the network that's that's how we focused on

19:57 this bob your thoughts yeah I would say I mean the other thing we're looking at right now is that Marvis min's large

20:03 experience model Marvis mini is going to bring much better WAN feature set into

20:09 that particular model you know the other thing you talk about other applications

20:14 I think this is where Marvis minis is going to come in because Marvis minis is going to give us a bunch of connectivity

20:20 information across all these other applications you know that's where we're looking to see if we can't apply

20:26 Shappley to th those type of models so I think that's how you extend what we're doing to other applications and how

20:33 minis is going to bring better data into the existing model and how can you apart from installing an agent on the laptop

20:39 and the devices itself um how can you get data a client data you know into the

20:47 cloud i know it's a challenge but I'm thinking it would be great to have like a Wi-Fi frame that allows the clients to

20:52 tell the AP this is how my application is doing yeah I think analogy theme I mean there is a Marvis SDK you know so

20:59 for customers actually want to bring the client side into the model um I would say probably most of our customers who

21:05 don't want to put something else on the application we're looking to like zoom teams actually gives us visibility into

21:12 the client so usually that application that end toend application has a visibility on this side here like zoom

21:19 and teams they know the CPU is they know when someone's downloading something in the background right so I would say 5 to

21:26 10% of all our zoom team issues are due to something here and so that you know the application itself is starting to

21:31 bring us in client information short answer is actually it's a great idea and we should take that to standards because

21:37 right now without an agent I can't tell if the CPU is pegged on a on a device right That's a fact and so that's why we

21:44 had to go down the Marvis agent either we we're getting the data from Zoom in Teams then we don't put any agents if we

21:50 don't have that then the Marvis client is is an option to put out there to feed into this model yeah we could start

21:57 simple with just having the clients telling their RSSI that would be a good start correct yes yeah well h how much I

22:05 guess the question is really with like Zoom and Teams and maybe other collaboration platforms is how much data

22:12 do you collect from those platforms that helps feed into the Shappley model i

22:18 would say we're on the order of hundreds of gigabytes right now you know so we're getting up close to terabytes of

22:24 training data okay yeah and so how would that impact the metrics you provide here

22:30 using the generalized large experience model when you don't have the

22:35 collaboration um or the API integration there yeah so so Navaj who's actually building the

22:42 models in the back there i would say we are very close to making sure that the generalized model is almost on par with

22:50 the Zoom teams model mhm and so that's the generalization effort we're going through is make sure the generalized model can accurately predict i think

22:58 this is answering your question right how close is the generalized model to the Zoom teams model i'll let Navage

23:04 come get the answer but it's getting very close in terms of the results what we do is we test the network with the

23:09 data pull the data out and make sure we get the same sharapply results okay yeah right and those are getting very close

23:15 90% i mean um Wes I don't know if your mic is on but um you know just one of

23:20 our customers has literally millions of teams minutes in a day right I think in one of the large universities correct so

23:27 so Roel basically with one customer over the last year per day we were getting

23:34 over a million teams minutes so the model actually has had enough learning to and that it that you're right it is

23:42 all how much we were able to learn and the data But but I think we've been blessed lots

23:47 of customers obviously as you can imagine you know an organization of 30,000 employees the number of team

23:53 calls that happen and if we measure every user every minute every team's call that's a lot of data and that's

23:58 what we've been relying on I mean so the model here is more about you need a lot of data to train the model it's back to

24:05 the right data yeah you know that's where minis is going to bring in much better WAN data so we'll get much better

24:11 visibility if there's a WAN problem going You guys mentioned the Marvis client um

24:17 are you integrating data that you're receiving from Marvis client today for the AI models and the generalized models

24:22 or is that a roadmap item because we do have some clients that would prefer to deploy an agent out to certain laptops i

24:29 would say that's a roadmap item we have not integrated the Marvis SDK into the generalized model yet kumar is saying

24:34 we're we're doing it today is that correct thank you maybe don't tell me anything anymore

24:40 i think u Yeah so from the Marvis client if a if a laptops have a Marvis client or any Android device or iPhones we do

24:46 get the CPU and the battery status and the RSSI being reported from the client we do uh kind of include that into the

24:53 large experience model so cuz uh um uh Allan has a lot of existing Juniper Mist

25:00 customers and so he's asking on their behalf is this something that they can use now or you know in a in a in a

25:07 couple of months and quarters type of thing this is uh in couple of months so right now this is in our staging we are

25:12 evalating like like Bob said we are doing out of sight uh model evalations where we kind of take the data out

25:18 ensure that the without label data we're close enough 90 95% we are evolving this

25:24 and then within 2 months you will be able to enable it across your customers those who can't give either teams or

25:29 zoom label data or they are using a different collaboration tool we'll be able to enable it g in couple of months

25:35 if you haven't figured out Alan Bob and I don't do any work here we just take credit for other people's work he's the lead product manager for Marvis so thank

25:41 you Kamar and we do have a question um so I'm trying to monitor Blue Sky i think others are monitoring Twitter and

25:46 other LinkedIn things um from Brian who Thanks Brian um wants to know about the

25:52 minis and can you tune it for checking um what it's check what it's checking if

25:57 you do something like if it realizes you failed over to like a backup link like cellular or something that's going to be

26:02 less bandwidth maybe higher latency that's a good question

26:08 yeah so u minis as they are always on um so Marvis is something like in the back

26:14 end in the cloud it's it's constantly observing right it can pick up config changes and it it if it knows that there

26:19 has been a policy change on any of your SDAN policy it does understand that now it has to check the traffic path through

26:26 both node zero and your backup node as well and it does that and one of our customers did catch an issue as they

26:31 were doing the such a change window where their policy hadn't taken care of traffic being passed through the node

26:37 one and min right away kind of indicated that hey this application is not reachable through your node one anymore

26:42 but but let me rephrase uh JJ's question here with Brian's question is if the network automatically fails over to

26:50 cellular because there was a outage on my primary link will Marvis now alert oh

26:55 my god something happened or will it learn the new baseline is that a question or or let me read it verbatim

27:00 reduce the checking it does if it detects you're on a metered WAN circuit like a fail over so I think he's kind of

27:07 getting the how much money are you going to cost me while I'm over that is that is true so the way uh we kind of ensure

27:14 that that doesn't happen is for example minis does a speed test it's smart enough where if it's if it's a LT link

27:21 it doesn't do that speed test on that LT link actually so it does understand things where which which upink primary

27:26 up link or secondary upink is being used as such i would say the access points are very lightly loaded they look like a

27:32 a 40 type of call 60k bit per second call in the telemetry okay so you're

27:38 you're not going to be racking up measurements good question i want to ask you guys about you know some of the teams monitoring because I understand

27:44 with all of these communication platforms and I don't manage these but I know with customers you know some of them are always client to cloud but a

27:51 lot of them have proxies on site that's right are you guys differentiating that user experience metric so actually uh

27:57 the one of the most important outcomes of this Marvis minis client to cloud is

28:02 now we actually can tell you um uh because we understand the entire path

28:08 right not just the last mile last hop is you use for baselining but we are actually you know tracing the entire

28:15 path and saying you know and we have a baseline for last time what was the latency at each hop what is it now and

28:23 hey was this call now going through zscaler or some VPN you know and if that's contributing to it we actually

28:29 will highlight that right okay so it's path aware enough that you know if it's going through a proxy and the expectations very much path aware yes on

28:35 day one yeah very good keep so next genai conversational interface as I

28:43 mentioned I have been a fan of natural language interfaces since I started miss I'm a Star Trek fan talking to my

28:49 computer is one of the goals of life um I would say we have had a great convers

28:55 conversational interface to 2018 it did a very good job of understanding queries

29:00 but it really didn't do a great job of answering you know with chat GPT in 2022

29:06 and 2023 Marvis now has a voice right and so this is where we're starting to

29:12 integrate this Genai technology the genai back into Marvis last year was all

29:17 about public doc making it easier to get those answers out of our Juniper public docs this year it's all about customer

29:25 data you know you can think of this as a replacement for Looker Tableau when you

29:31 want to know what's going on inside of your network Marvis is going to be able to help you understand what your network

29:37 is doing you know it'll be able to tell you what clients are on the network how many clients been on the network how

29:43 many iOS clients how many Windows clients basic questions that most IT people like to know without having to be

29:49 an SQL expert going forward so that's what we're doing sadir

29:55 this is one more this is um definitely a lunch bet I lost to Bob because when Bob

30:01 in 2018 said we're going to have a conversational assistant I said who the hell is going to talk to their network

30:07 this is networking nerds nobody wants a uh you know a you know chatbot and then

30:12 chat GPT comes and uh you know definitely I had to pay up on that and so now this is the new interface for

30:20 Marvis's conversational assistant this is uh um uh something we're excited about um a brand new interface uh

30:27 essentially very chaty chaty PTesque and gives us a framework to go into agentic

30:34 AI and MCP and uh you know all the new technologies but basically now the

30:40 Marvis conversational assistant will grow up into this kind of uh you know interface and and essentially be able to

30:46 answer questions for previously you know uh you know uh un uh uh you know

30:53 previously you know people had to build customized dashboards for now you can

30:58 ask unstructured questions that weren't you know nicely properly laid out in a database an example being hey how many

31:06 of my APs across my entire organization have 100 megabits right we deployed a a

31:11 network with 27,000 APs uh you know 6 years ago um 272 of the APs came up in

31:19 10100 and that was uh you know the epiphany saying shouldn't the network just tell you that and so that's why

31:24 there's a a Marvis action for it but you can ask a question you know uh you know uh in terms of this new interface to be

31:31 able to get answers now u another thing is summarization of some of the uh some

31:38 of the visualization here not only now Marvis uh the new Marvis conversation assistant will have you

31:45 completely customizable visualizations but summarization of what you're actually seeing when did it happen what

31:51 did it happen again a lot of um uh LLM uh you know work involved in this the

31:58 one thing that I want to completely uh you know uh um you know emphatically

32:03 state is 100% of our customer data remains private is not exposed to any

32:11 LLMs while we're using LLMs um in a lot of this formulation of the dashboards um

32:18 you know our data our customer data remains completely private this is you know the the the tough part it took us a

32:25 long time to actually make sure um that uh you know we we keep this data private keep these uh customer concerns of any

32:32 data exposure to LLM uh off the table and then you know now that we have that

32:37 now that that's in the past basically we're able to say okay I want to be able to create customizable dashboards um you

32:44 know as an example for us you know uh wireless service levels and wired service levels are uh you know sort of

32:51 how we see if the network is healthy and being able to create sort of customizable dashboards that reside in

32:57 the network for us are super critical right now the last frontier here is um

33:04 sorry you had a question yeah my my question is tied into that are you learning the questions

33:11 people are asking good question that's that is that is the piece I will answer

33:16 Keith right now but tie into that how do you teach people to ask good question very you you've got

33:24 solutions and dashboards to show them but they they don't even know it's there how do they know to ask Great question

33:30 uh so let me answer the first one uh first is you know u agentic AI is all

33:37 the rage these days in in across verticals across industries and so our

33:43 first forier into this is going to be you know leveraging and and one of the

33:49 aspects of agentic AI is that there is reasoning and there's memory and and so

33:56 this is a for the first time as we launch into this multi- agent you know

34:01 um framework for us um under the hood for Marvis conversation assistant yes it

34:07 will actually learn it'll have reasoning it'll have memory and and that's sort of the the basis of this right so you know

34:14 agentic AI for us is a fundamentally different um you know we have lots of work underway on that and and you'll see

34:20 more uh you know through the rest of the year and and and and beyond but this is our first foray into agentai you know

34:28 and and Bob talked about you know MCP servers and stuff you'll see more from us uh on that as well but uh that's the

34:34 your first question I think your second question was how do how do we teach people to ask good questions and um

34:42 that's is something we I think we have to think through I can answer so um we have um like intent

34:50 classification agents which wherein whenever you ask any questions it could be like trend list or whether uh it

34:56 could like how is my network doing how is my network doing yesterday versus today or for last seven days right any

35:02 question that you want to whether it's a debug question or whether it's a documentation where you want to understand how is like how do you

35:08 configure like missed access assurance right anything you ask um there is an intent classification agent that goes

35:15 through and then based on that it classifies hey this is a general documentation question versus this is like a trend list or like probably like

35:22 a troubleshoot question right and when it kind of gets into intent further the agent keeps classifying on every input

35:28 like for example in one of the example as we indicated hey minis is already indicating our failures like u there is

35:35 an action already indicating our failures and as we do that like when when we say hey give me more details it

35:41 does understand that the user is still trying to get more details around that intent and goes in further in line that

35:46 right so that way yes um the intent classification is the one that does the magic where we don't have to train

35:51 people as it kind of as you as the questions are asked these are saved and then the model is trained over and over

35:58 again where if somebody gives us a thumbs down like hey this was not my intent that kind of acts to adds to that

36:03 training back to the the next time when it answers the question yeah I I would I would add Keith I mean I don't know if

36:08 you use chat GPT lately the user experience is slightly changing you know chatbt will come up and say I

36:16 think the user is asking something about this now all these technologies now have history in it so I think as people start

36:23 to learn to use this technology I think they're going to see a history of it right that's the kind of how they get

36:28 feedback from this is what I think you asked if you get the wrong answer you can reask the question again and

36:34 actually the LLM will actually take that history into account so I think that's kind of that people are going to learn

36:40 how to use this the reason I brought that up is you had some dashboards that were dynamically built because someone

36:45 asked a question if Sam built one of those I would like to know what Sam's asking right because

36:51 it's already been done and I didn't think about it so there's there's a lot of We could do better on I think I think

36:57 there's an add to your dashboard feature i think there's a little feature up there i don't know if it was in demo

37:02 yeah but but I think Keith's question is how do I know the intelligent questions others are asking i think we could do a

37:08 better job yeah we uh Yeah let's let's move forward because we're running out of time yeah Peter yes um in terms of

37:14 the data that you're using to feed it so obviously you've got your network data and you've got your docs are you also

37:23 using any external data um so I don't know like data from the network alliance

37:29 on client capabilities is any external data being fed in or is it just data held by I I'll take a first stab and

37:37 then um so what we are doing what Marvis has actually become now is we are

37:43 voracious consumers of data from external sources as well such as teams and zoom is external to us right it's

37:50 and teams and zoom is not just telling us about teams and zoom it's actually telling us about the laptops you know

37:56 CPU and memory all this kind of stuff so that's one category of of people that are giving us data we integrate natively

38:01 with Cradle Point and so we get all of that data directly into the dashboard we integrated natively with Salona we're

38:08 getting data there we're integrating natively with um you know some of the uh the uh security application providers

38:15 the top two major security providers they both have their digital experience uh uh applications that data is also

38:23 natively integrated to Mist and now you know Slava is going to present we're integrated with MDM solutions so we

38:30 basically have created this northbound pipeline that is easy for us to bring in

38:35 external data so we are voracious intentionally like as an example one of

38:41 the next things that we were thinking about is should we integrate with application performance monitoring tools right um we obviously already integrate

38:47 with service now should we make that birectional where service now tells us something that there was a config change

38:53 and a firewall that we don't manage and that kind of stuff so there's there's a few more things we could do but we are

39:00 open to actually consuming a lot of this and making meaning I was more thinking of likeformational documentation like

39:07 extendeds and things like that because so for example if I've got a Wi-Fi um 6

39:13 network and I say how many of my clients are currently Wi-Fi 7 capable probably

39:18 from data on the network because it only understands Wi-Fi 6 no but it pro because you classify devices and you

39:24 know there's an iPhone 16 or whatever if you knew that in external information

39:29 you could answer that question I think I would say that's on the road map I mean right now we're ingesting data from the

39:36 network from Zoom you know third party partners you know your question is yes we will start ingesting data from the

39:42 internet that'd be the next step you know if you ask me a question I'll probably go out to the internet to see if there's an answer on the internet

39:48 there's an answer yeah okay very good okay all right wrap it up here u Marvis

39:54 actions this is our baby this is cup of coffee self-driving component of Marvis

39:59 um after I took my self-driving Whimo Uber I am on the mission now to move us

40:05 from assisted self-driving to fully self-driving if you look at Marva's

40:10 actions Marva's actions has always done a great job on assisted actions right we

40:16 do a great job of helping it tease you know here's an issue here's a recommendation here's the evidence for

40:22 the recommendation with this Marvis Actions facelift you're going to see us start driving more towards south driving

40:30 this is where Marvis knows how to fix a problem marvis is going to start asking

40:36 permission to fix that stuck port fix that missing VLAN and then ask would you

40:42 like me to put this on the trusted self-driving list so this is the Marvis action facelift where we are going to be

40:49 moving from that assisted driving experience to more of that Whimo Uber fully self-driving and earning the trust

40:56 of it awesome so uh this has been uh ever since Bob came to WLPC two years ago I

41:04 think we've been on this thing uh crusade and so um so what we are doing now is is this new facelift for Marvis

41:12 actions there is a new self-drivable but you know icon there

41:17 saying hey these actions you know if you give us permission we could fix them we don't even have to just alert you and so

41:24 now these are the list of things port stuck is an interesting example you would think that's uh you know that's

41:31 something we may not be able to actually self-drive it is it is fascinating you

41:36 know basically think of a thermostat a video camera a sensor that has a certain

41:42 periodicity of traffic and we're observing that periodicity and saying hm you know this port didn't go down but

41:49 suddenly the traffic disappeared and with high efficacy we could actually say

41:54 hm this video camera something needs to happen and we need to bounce this port and literally we bounce uh uh you know

42:02 ports uh like this around the world uh I would say north of uh 20,000 Ethernet

42:09 ports we bounce um on a daily basis around our universe and so essentially

42:16 what this eliminates is if somebody go somebody goes in to check the video footage you know 5 days later and they

42:23 don't have the epiphany oh my god this thing did not actually send video for the last 5 days we are remediating

42:29 so self-driving can can come to even sort of very legacy use cases like that

42:34 another example is if we have all the access uh VLANs on a switch but one of

42:41 them isn't P you know trunked up to the up on the uplink port right actually we could go change configuration on

42:48 switching so the trust part of this is obviously you're going to first you know

42:53 we we have to earn the trust of network IT because at this point we're starting to touch config and that's scary for a

43:00 lot of people so but the moment you get that trust one item at a time you know our goal is everything we can we want to

43:08 get Marvis to be self-driving right it is really powerful

43:16 right so um with that what uh you know our

43:23 fundamental focus as SUJ pointed out at the very top of the session is look

43:29 every vendor is going to come here and talk about AI and models and all kinds of stuff our claim to fame is we

43:37 basically are focused on customer outcomes right and so um you know for

43:43 all of the Marvis uh you know things that you've seen today um you know our focus has been around how do we deliver

43:51 a customer experience enabler how do we enable a better customer experience

43:56 that's right uh you know uh take all of the entire Marvis session the summary of

44:01 it for us is happy customers and and that's what you know across verticals imagine what we do today we go in and

44:09 get deployed in a small lawyer office or a dentist office to literally the

44:14 largest networks in the world the the our current largest network from a wireless perspective 400,000 access

44:21 points deployed in production uh after deploying 400,000 AP PS across their

44:28 global estate they've chosen to deploy um switching with us right and you know

44:33 that's why we have a a small switching session at the end of this uh thing 90,000 switches deployed right it is

44:39 delivering amazing customer outcomes is what the summary of this Marvis session is with on the actions so your your your

44:46 goal is to add more and more into into the actions what's the goal when it comes to uh minimizing risk with

44:54 detection of issues and roll back so you've taken an action and oh my goodness things have gone wrong um

45:01 what's the plan to detect that and roll back those changes so Chris I think the um our entire focus from day one has

45:08 been efficacy right because if we don't get efficacy right this is no different

45:13 than you know our grandfather's alerting that we had just 5 years ago and you know oodles of alerts and and you know

45:20 which one do I even start with so our entire focus has been efficacy so we

45:26 when we take a self-driving action it's near 99% efficacy your question is but

45:32 what about that 1% right because happens in real world right and so for that you know we we actually we have to

45:39 institute I don't think today we have an auto roll back of this but uh we we uh

45:45 but predominantly anything that is self-driving we have extreme high

45:50 efficacy right so Bob got a question on that as a network administrator how do

45:57 you know what changes were made because generally you want to go through some kind the change control process in case

46:02 you do need to roll back right very good question so actually uh Marvis actions previously did not have that and so um

46:10 the very last part of the demo uh is we actually ended up essentially with a a

46:16 now a history not on of all the both self-driven or driver assisted actions

46:23 now there's a complete history and visualization for that it didn't have we didn't have that uh till this launch

46:28 yeah throughout for for the customers that have tested this for you how how do they tie this to their change control

46:34 process like uh what I'm thinking through is since you've got AI can it learn when my maintenance windows are

46:42 submit a change ticket uh cuz I I have an organization where the change control

46:47 person is probably going to be very scared of this that's correct that's correct so actually Roel u this port

46:52 stuck as an example the only reason we do it is

46:57 there is such high efficacy that we knew that port is shot we know there is no

47:04 video coming in this is not a change control think of this as a for that microcosm of an application which is

47:11 that video camera it's a network down situation right and so it's not a change control kind of situation it's a hey

47:17 this application is not functioning go remediated for me and that's the only time we actually doing self private yeah

47:24 I I would say more like RM I would think this as radio resource management right and we started that 20 years ago it did

47:31 not trust it right they had had an onoff switch and most you know like I can do better we're in the same point with

47:38 network resource management so I mean so your question if customers don't trust it they're not going to turn it on so I

47:45 think this is the transition to network resource management going forward Now to your question about change I would tell

47:51 you looking to see if anything change statistically before and after a change

47:58 that's much easier math than try to do anomaly detection because I have the event i can look to see did anything

48:03 statistically change in the network and I think you'll see us go back to roll back you know we will be able to check

48:09 if we change if they update something statistically change in the network that would would trigger a roll back i had a

48:16 question to go along the same lines too because a lot of organizations have zero trust solutions network access control

48:23 solutions that are tied into the infrastructure so obviously you know when we do that we we

48:28 have to have one source of truth of where those the change control stuff is happening from I what's going to be the

48:34 answer for that for the organizations that have that type of integration for for for for the most part every action

48:40 that we produce on Marvis actions is coming in as a web hook into a service now or or some other uh their source of

48:48 truth of of of all global actions in the enterprise so that's where we record all

48:53 of this right there is they will never miss one of these and and now there's a trail of you know Yeah and I'm less

49:00 concerned about auditing that it happened but more that a lot of the products that integrate with like local

49:05 network resources especially on the wired side can't have can't have

49:10 something else making changes to the infrastructure when it's trying to do port changes VLAN status changes and I

49:17 know obviously you guys have access assurance will that be able to I think the answer will be something like it's

49:23 like right now we work very closely with service now right and so service now becomes that source of truth you know

49:30 where Marvis will send the request to service now to actually execute something Bob what JJ's asking is um a

49:37 port may be intentionally changing profile because of network access policy change and so how do how do we not have

49:44 a race condition between us trying to self-drive I think my point is we we don't touch report we send the message

49:51 to the source of the change like service now because you know a customer may trust service now to make the workflow

49:58 change right they may want all changes in network go through service now i think that's what you're thinking yeah

50:03 not not quite but I guess my my question is will will the access solution be able to be a will it fit into this if if

50:11 somebody needs to be doing that type of port level control yes yeah and and I would say you know you to your point of

50:18 you know are there multiple sources of truth i mean no you don't you want one right and so depending on where your

50:24 source of truth is you know if it's missed most of the time it is missed especially if the switches are managed by Mist um and that's where you know

50:31 this port stock would apply uh and so if if you choose to turn on the you know

50:36 the self-driving aspect of it then Mist is your source of truth and and with um

50:42 you know some of the uh Knack solutions that you know want full control over network state um they're already talking

50:48 to our APIs get network state and uh they're already talking to the APIs

50:55 to get network state pulling that back in and so there's already a sync that's happening because you know configuration

51:01 is already coming from Mist