Talking Automation: Challenges of Integrating Automation Tools into Your Network

0:00 [Music]

0:14 we talked about build your own

0:16 automation some of the some of the

0:17 challenges with it um you know we talked

0:20 about you know vendor

0:22 automation have you ever run into trying

0:25 to take

0:26 a vendor's automation and integrate that

0:29 with some build your own automation it's

0:31 not an answer to your specific question

0:32 but i think it's somewhat relevant so

0:35 in an environment i worked in

0:38 we had a vendor who had their own

0:40 automation platform

0:42 that we were not leveraging because when

0:44 that platform was stood up we were just

0:46 doing it like we always did you know

0:48 line by line it there are some

0:50 automation platforms that will allow you

0:52 to also be in the cli and do it manually

0:54 so that's what we were doing for a very

0:57 long time fast forward to

0:59 a business need comes up and something

1:01 has to be updated i know it's a security

1:02 thing or whatever like hey we have to

1:04 update this config line

1:06 in all the devices it could be an ntp

1:08 server it could be snmp you know

1:10 management type stuff that's going to be

1:11 a common setting in all of them so this

1:13 poor unsuspecting soul

1:15 went to this vendor's

1:17 automation platform did what he thought

1:19 he was supposed to do

1:20 put into this

1:22 platform the change he wanted to make

1:25 and that platform went out made that

1:26 change to every device

1:28 the problem was

1:30 that

1:31 because we were managing the configs

1:33 manually that platform all it knew about

1:36 the existing config was that they were

1:37 empty so when it looked and it said oh

1:40 well i have to add this one thing so it

1:42 went into the devices it wiped every

1:45 config out of thousands of switches

1:47 across multiple fabrics and only put in

1:49 this little management change that we

1:50 needed which caused a huge outage so

1:54 it's not the answer to your specific

1:56 question

1:57 where you're trying to integrate

1:59 you know uh open source automation to a

2:02 vendor's automation but i can tell you

2:03 that the simple act of

2:06 manually configuring and managing your

2:09 platform which is what we've been doing

2:11 forever

2:12 and then bringing an automation platform

2:14 in and not really understanding

2:18 the interplay of those two and you know

2:21 i mean now i know that you should not

2:23 manually

2:24 manage your configs if you're using an

2:26 automation platform because there will

2:27 be conflicts and a good automation

2:29 platform like an intent based one like

2:30 abstra will

2:32 look at the whatever you want to call

2:34 golden config but this is what should be

2:35 there and if it does a diff and it sees

2:38 things that it determines shouldn't be

2:40 there they go bye-bye

2:42 which is what happened in that example i

2:43 just gave you oh i see what should be

2:45 here because this is what the

2:46 intelligence knows about so i'm going to

2:48 wipe everything out that shouldn't be

2:49 there and you know that was i don't know

2:52 how common that is in the industry

2:54 but

2:54 you know one of the things with

2:55 automation that i've seen happen and is

2:58 a concern for older school guys like

3:00 myself is

3:02 you can do things very fast

3:04 which is great as long as you're not

3:06 breaking something

3:08 but you can break all the things really

3:10 fast with automation so

3:13 um i think it ties into a couple things

3:15 we're talking about challenges with

3:16 automation challenges with net ops

3:18 expectations of leadership is it easy

3:21 you know it's easy to say well we're

3:22 just going to buy into this vendor

3:23 automation solution and use it but you

3:26 have to look at how your environment is

3:28 working before that automation comes in

3:32 what's different about this automation

3:33 solution

3:35 is there a disconnect between those two

3:36 how do you rectify them and then you

3:38 have to get all of your people on board

3:40 this guy who used the automation

3:42 platform thinking he was doing the right

3:44 thing it was never communicated to him

3:46 that that could be a problem and that

3:48 the fabric had been managed manually all

3:50 this time he had no idea because in a

3:52 larger company people aren't talking you

3:54 know it's

3:55 so there's a lot there right if you're

3:57 going to bring in an automation platform

3:59 that's a vendor

4:01 you have to get it working you have to

4:02 get trained up on it you have to

4:03 understand how it relates to your

4:05 existing infrastructure you have to know

4:07 how to use it you have to know how to

4:08 back out if something weird happens and

4:11 i could tell you that outage that i just

4:12 mentioned it took way too long i mean it

4:15 took so long just to figure out what was

4:17 happening oh my god why did all the

4:18 fabric just disappear

4:20 in most of our data centers right and

4:23 what it wiped out was all the management

4:25 stuff so i don't think we could even get

4:26 in to like fix stuff so

4:30 you know that that's i mean automation's

4:32 great

4:33 but

4:34 you want to go fast

4:36 but you also want to ensure that you

4:37 understand

4:39 the platform you know hopefully the

4:41 platform is simple enough for the user

4:44 to not be able to blow up all your

4:45 fabric and all of your data centers

4:46 right and

4:48 the better solutions i've seen

4:50 they spell that all out it's very you

4:52 know and you know multiple migrations

4:54 like if somebody wants to go to a vendor

4:55 automation platform and they're

4:56 concerned well hey here's you know 300

4:59 fortune 500 companies that have done

5:01 this and they're kicking butt

5:03 you know it's hard to be the first guy

5:04 on the block like hey try this thing we

5:06 won't we won't break your stuff i swear

5:08 but you know products like appstr have a

5:12 track record of hey look at all these

5:13 huge global technical companies that

5:15 have gone to this it went well they

5:17 didn't blow up all their stuff and now

5:19 they're more agile

5:20 they're they're up time you know i've

5:21 seen statistics you know the the up

5:25 error uh what is it outages based on

5:27 configuration most outages i've seen i

5:30 mean yeah circuits go down but you plan

5:33 and you design networks to route around

5:35 physical failures and most outages i've

5:37 seen are the result of

5:40 configuration changes

5:43 that were incorrect we thought they were

5:45 correct they looked correct we lab them

5:46 we had meetings with all that you know

5:48 everybody signed off on it it was

5:49 reviewed but whoops

5:51 we put it in and it did something

5:52 strange

5:53 i love the intent based model because

5:55 you don't need to manually write a

5:57 config and hope that

5:59 the intent that you meant to write in

6:01 all those lines of code

6:03 gets interpreted by the platform and the

6:05 infrastructure as the same thing you can

6:07 just tell an intent-based system this is

6:09 what i want and it just abstracts all

6:11 that minutiae away and it goes and does

6:13 it the intelligence of the system like

6:14 oh you want a new whatever sure i'll go

6:18 do that as opposed to me who something

6:20 as simple as adding a vlan it's it's a

6:22 segment in layer two and networking

6:24 you if you you can add it in a way that

6:28 you wipe out all the existing vlans and

6:30 break all the connectivity that's

6:32 already in place you have to add it to

6:34 these interconnects called trunks and

6:36 if you forget to put switchboard add

6:38 trunk i forget what the command is but

6:40 this is the problem about like manual

6:41 stuff is if you forget this one command

6:44 it adds that and deletes everything that

6:47 already existed and every network

6:49 engineer has done that at one point we

6:51 need new connectivity in this domain

6:53 you add the new net connectivity and you

6:55 break everything else that was already

6:56 working right in an intent based model

7:00 you know like apps it's not going to do

7:01 that it's going to know oh nobody ever

7:04 wipes out all the existing connectivity

7:06 to add new

7:07 but you know again for me who might be

7:09 tired at three in the morning on my

7:10 third maintenance window of the week

7:12 maybe i got a baby at home and i'm not

7:14 sleeping good anyway i mean you know

7:16 accidents happen with config changes it

7:17 breaks stuff

7:19 and any kind of automation system that

7:21 can

7:22 reduce outages because of configuration

7:24 errors to me

7:26 i mean that's just solid gold like why

7:28 wouldn't you

7:29 do that right like you don't want to

7:31 break your stuff

7:32 so

7:33 use a platform that helps you not break

7:35 stuff it's again you can tell i'm not

7:37 trying to be like a company shill guy

7:39 here

7:40 but i think of all the pain that i've

7:42 caused in production

7:44 that could have been you know avoided

7:48 by using a really smart automation

7:50 platform

7:51 so right yeah and you kind of hit on a

7:54 on an interesting point there um

7:57 you know

7:58 fast is great and that's the that's

8:00 always the goal is we want to go fast we

8:02 want to go faster but if you take

8:04 automation whether it's automation you

8:06 build yourself whether it's automation

8:08 from a vendor that

8:09 you know is focused on the speed or even

8:12 if you're just you know trying to type

8:13 real fast right so do it in 23 hours

8:16 instead of 24.

8:18 you're bound to make mistakes you lose

8:20 the reliability so

8:22 having an automation tool that you know

8:25 also has that focus on reliability is

8:28 what's really going to allow you to go

8:30 fast um

8:31 you know there's a saying that the navy

8:33 seals have slow is smooth and smooth is

8:35 fast

8:36 right so you you you pay a little bit

8:39 more attention to exactly what you're

8:41 doing and make sure you're not making

8:43 mistakes which makes you smooth and as

8:45 long as you're smooth the whole

8:47 operation is going to go a lot faster

8:49 and and you know you're not going to

8:50 have to go back and redo it um you know

8:53 consequences a little more dire in their

8:55 circumstances

8:56 um but you know it's like you you get a

8:59 new automation tool you build some new

9:01 automation you just focus on this can

9:03 make it go really fast but if you don't

9:04 focus on making sure it's correct or the

9:07 tool doesn't have the backup to say oh

9:09 hold on stop take a break here we gotta

9:12 we got to make sure this is what you

9:13 really want to do that's when you run

9:15 into problems and it actually takes you

9:16 a lot longer than it would have

9:20 [Music]

9:25 you