Trust, Guardrails, and Action: Can Agents Really Replace Managers?

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:10

   My name is Elliot. I'm a dev with near Foundation and I'll be moderating this panel. We're going to be talking about trust, guardrails, and action. Can agents really be managers? So, I'd like to introduce uh my panelist of the evening, Owen Hassel. He is developer experience for Proximity Labs. Also, John Kennedy works for uh couple years in startup. He's um responsible for one

2. 0:35

   of the biggest open source code bases at AWS game tech and he's building architecture agents for companies building software factories at actual AI. Thank you for coming on. And then Rees Gupta um head of agentic product at scan AAI. You've been a founder. Um your company has been acquired. I also saw on your LinkedIn you were a judge for LEGO competitions. Is that right?

3. 1:00

   Long time back. Okay, nice. Well, uh, I'd like to just go ahead and dive into it. We got about 30 minutes on the clock. First question, you guys are welcome to disagree or agree. When you hear agents replacing managers, what's your immediate reaction? Excitement or concern?

4. 1:20

   Sure. I I have I have a lot of excitement. I think um you know we're specifically working in software development. So how many people here are in software development or have a team software development team? Okay cool. So a few of you right and I think what's happened in software development over the past 20 years a lot of people who should have stayed IC's who should have kind of stayed coding actually became managers and hated it and they did it

5. 1:43

   for career reasons. Um, and actually what we're seeing now is this bifocation where managers with added AI capabilities can go back to doing the things they like. Whether that's to be a player coach who's actually creating code with AI or it's to be really on the human to human side and focus on that.

6. 1:57

   Uh, but I think the way that management is evolving because of agents is brilliant for everyone. And so it's exciting for me. I'll take it. First of all, like uh thanks for uh phrasing this question like agents are replacing managers rather than agents are replacing humans, right? So like uh in my like uh industry that we I'm watching uh I'm basically primary focusing on the enterprise AI

7. 2:20

   specifically in the banking financial insurance space what I'm seeing a pattern is like uh most of these people are actually excited rather than being afraid of uh replacing the jobs with the agents. One of the primary reason is these managers are overloaded with lot of coordination work. 30 to 50% of their work or the uh day-to-day life goes into chasing the follow-ups doing the data reconilations such kind of a work. So in one way they are very happy when we are

8. 2:51

   talking to our customers that uh are they excited so they are very much looking forward for this agents and Owen what do you think I don't I don't want to say too much because my manager is in the room but I don't see that much replacement of managers just yet with AI to be honest I think it's more the the lower level like junior developers that are being hit with like AI replacement the moment Um, but I think a key thing that managers do is add that human element to

9. 3:24

   it, resolving disputes and and conflicts. So, I've not seen AI replace that very well yet, even though one of my colleagues did say to me earlier that he speaks with with AI about his feelings and stuff. So, may maybe it is starting to to replace that element. But I think we're far away from AI replacing that human touch component.

10. 3:53

    Are you excited or are you concerned about your boss getting replaced? I think I might get replaced if I answer. Okay. So, a little concerned. Um, Rees, I'd like to start the question with you. Uh, working in enterprise and automating solutions for them. which decisions are you and you find that enterprises comfortable with automating and versus which require human sign off in the process.

11. 4:17

    So the way that I'm seeing a pattern is like uh it's very much about the agent boundary today the where the agent or the technology lies is like if there is a bounded uh set of requirements or the task that you are supposed to work with the agents. So that is the first criteria if the set of actions are bounded in nature. Second is the task.

12. 4:37

    If those task are reversible in nature for for some or the other way the agents act up doing something wrong. So the task can be reversible. So those are the prime areas that I'm seeing where I can basically like uh uh giving more decision-m power to the agents or can directly rely on the agents without having the human in the loop versus John. What do you think about that about uh which decisions are you finding that your teams are getting comfortable with? uh automating versus

13. 5:07

    what is still seeing human sign off in the loop. Yeah, I think one of the most interesting trends in software development is that uh software developers have stopped looking at code, the best ones that we know. Um and that trend is only increasing. We're seeing some designers say they still like to look at code but in general software developers the the ones even doing really hard algorithms in you know in low-level languages are moving uh over to systems like claude code where

14. 5:32

    they're not um they're not kind of going in detail through the code and this presents a problem for the industry which is that currently sock 2 requires that there's a human review on every piece of code and so that means every and that's basically every SAS needs sock 2 And most, you know, enterprise software companies need sock 2 as well.

15. 5:52

    But that paradigm's got to change because it's actually probably less, this is going to sound counterintuitive, but it's less secure for humans to be going through every line of code. There's probably better uh ways to spend their time because now you've got agents that are better at reviewing for security. And what those humans should probably be doing is actually setting up guardrails uh and automating the software factory to better review code for security issues than looking line by

16. 6:18

    line at code. So that is one place where we absolutely see that the agents should be making the decisions. Um you know you set up the guardrails, you set up the software factory, they should be showing you where the security vulnerabilities are and uh you know making decisions about yes this is worth addressing. We have to address this one, maybe this is low risk. um if you have the guardrails in place.

17. 6:39

    I think that's great that you touched on security specifically because Owen, you work with trusted execution environments and running these autonomous agents within this. Uh how are you seeing uh security kind of influence how agents run and and when humans have to step in and uh yeah, I think it really depends on the task. like I'm okay with AI managing some parts of my life like boring admin.

18. 7:08

    Um definitely not ready for AI to be managing my finances just yet. Um and you were talking earlier about like reversibility. Um yes, all actions that AI are taking right now need to be reversible in my opinion at least. I don't want it to be able to uh delete a database. And we've all seen the tweets of emails, databases being deleted by AI agents. That definitely needs to uh have

19. 7:39

    a human in the loop there. There I want to dive in just a little bit more on that. You said that you wouldn't trust an agent automating your finances. Is that right? How come is that correct? How come? uh we've just not seen that good results with AI managing finances.

20. 7:58

    Well, it really depends how you define an agent, right? Okay. Lots of agents are well, agents are deterministic, right? And these are managing finance and they've had good results because they're the same as how finance has been managed for for years and years, but uh if you add that AI element, it's still quite unproven.

21. 8:21

    Did anyone see the the viral tweet by Matt Schumer talking about the change in AI models? The kind of accuracy I think it's a really good counter example actually. So software development, if you tried it out, if you tried AI tools for software development three, four months ago, you'd have a completely different experience to today. And that's been true every quarter, every 3-month period for the past 2 years since January 2024 when we had co-pilot.

22. 8:44

    It's a completely different world every 3 months. And um you know, only a month ago, I saw a CFO who' completely automated their dashboard. They're having the AI make transactions in QuickBooks instead of going into QuickBooks. They're running their whole financial system across their whole business. It's got logistics. It's got uh you know they've got a whole um they've got a a physical goods business they're running from agents writing transactions into QuickBooks. Now that's just an example of one industry but the

23. 9:10

    the thing that Matt Schumer says it's worth reading is that we're seeing it in software development today. You know AI is the biggest wave dev tools are the steepest part of the bigger w biggest wave and models have come on so quickly that people who tried them out three months ago just don't know that they are actually reliable and they don't have to look at code anymore. That's going to be true for every industry. We're going to see that roll out through industries where people realize that the model has

24. 9:33

    gotten good enough and you know in terms of faults and you know worrying and about what agents do I think the fault rate is going to dramatically drop because of agents I think humans are much more fallible than uh agents in in specific areas today but what you project out if you look at software development is that's most areas in like three months. Uh I just want to add on John's point because like many times when we think that agents are in P it's

25. 9:59

    just because like we always talk about the models are not ready the uh all those kind of a things in my experience especially in the enterprise AI like a models to some degree are almost there the gap is coming from the architecture perspective the enterprise is not ready from adopting the agent as an architecture p say because there's not enough guardrails there's not enough policies there's not enough complian science and the auditability that they

26. 10:25

    have thought about. There's not enough quality uh development environment before putting the things into the production. So all those things are still the gap uh in the industry right now uh and the if uh those gaps get finished the even the current set of the models are good enough to take them.

27. 10:43

    I agree that's the product we've built. So yeah and I'm I'm curious John you say so these models can be starting managing these things. How would you assess if something's ready? Do you look at other companies that are using these models for these use cases? Do you need to assess it yourself?

28. 11:02

    What? Yeah, there's like this whole um maturity framework for software development or AIPAD software development. It's worth looking up. There's a kind of post by this guy called Steve Agi. It's called Welcome to Gas Town. He's building this massive orchestrator, but he has eight levels of eight kind of stages of AI powered software development. Kind of give you a maturity model for where people are.

29. 11:21

    What I would say is that um stability is about level six, level seven. We haven't got to stability on like large scale um orchestrators yet where you've got, you know, 30 agents running at once across a range of roles. That's still fairly unstable. But the kind of I want to run three agents and I want to put guard rails around them super stable. Um, I think the um paradigm for safety at the moment is that um you should be getting almost 100% test coverage, probably 100% test

30. 11:54

    coverage with exceptions. And that's the big change is now because code's so easy to write, test is so easy to write, you can do that. And all the best developers we know, they've gone to this new model of 100% test coverage. And if you get there, you can write production ready code. you don't need to look at the code. You can watch your test output and actually get really good results. So that's that's kind of one of the ways we're judging maturity at the moment is

31. 12:16

    people have moved to that model of test coverage for software development. I I'm talking very spec specifically about an industry now and there's probably good ways to generalize this this out but it's easier for me to speak in specifics in software development.

32. 12:28

    Well, okay. Specifically on software development, you just talked about tests coverage. You know, I've used open code and or I use autonomous agents to write code. I will almost I don't know the last time that I actually wrote my own line line of code. But um if I tell it to write tests, it's going to write me some tests and it's going to iterate on them and it's going to make sure that they pass. Not necessarily maybe change the code to make sure it makes them pass, but yeah.

33. 12:52

    So how do you ensure you got to have the scaffolding? Any anyone building an orchestrator right now like Metaswarm or Gastown or other orchestrators has thought about test scaffolding because it's so critical. Okay. And you can't just like say, "Hey, write me some tests." Like for instance, one of the things that our company does is we build uh we analyze a legacy code bases to understand their architectural

34. 13:09

    decisions and then we insert them back as context so that the the tool is actually using the architectural decisions you've already been building with. And one of the big areas we see there and we've got 72 categories of architectural decisions that people make. One of the big categories there is test coverage. How you do it, which tools you use. um you know there's command line tools that will tell you how much test coverage you have and the agent can actually use that as a tool call to see how much tech test coverage

35. 13:32

    it has and it can go back and iterate until it has some 100% test coverage. So like there are really good ways to scaffold 100% test coverage but you've got to put time into it and what we're seeing is bifocation companies who are investing in building software factories are just going faster and faster and faster. And like things like Gas Town are actually building Gas Town. You know, Ralph is building Ralph

36. 13:52

    Claude Code is building Claude Code like as they scaffold their own tools, they're using it to actually get exponential acceleration. Yeah, I I totally it's something that we're seeing everywhere. But I do have a um you touched on this earlier about enterprise and how comfortable they are with doing it all. Um in order for is tests enough proof to feel that autonomous agent or whatever system is test is definitely necessary because think it from the enterprise lens like

37. 14:21

    uh there's not only the developers who needs to take a decision there's a like a buyers there's a influencers all such kind of a uh hierarchy is exist in the enterprises. So like the way that we are seeing is the testing is definitely like when you put it out something in the developer mode then you do that but after that there's a big piece of the eval before putting into the production

38. 14:43

    and even before going into the production one thing uh that what we are seeing is like a customers are basically like a still worry about it because these agents are designed on a probabilistic theory unlike a determinism deterministic kind of a design beforehand right so they need to run the test enough times in enough var variations to start seeing a comfortable zone when the variations or the variability starts reducing at a point and then after you as I touch upon on

39. 15:14

    the architecture design part of it that you need whether you have the audit trails whether you have the compliance whether you have the policy whether you have the obser observability layer intact all those things are a piece that enterprise cares about right now okay I think that's a great segue into another question uh specifically about okay so what we we uh have audit trails

40. 15:38

    we make sure um that everything works is passing all the tests let's say something goes wrong like I just saw there's probably a a news story that you guys have seen about how cloud code wrote a smart contract that had a vulnerability in it and it costs $2 million um when that happens who where does a responsibility fall does it fall on the agent itself itself does it fall on the business that uh funded de development does it fall on the human that deployed it

41. 16:08

    I guess it's a human who deployed it or the human who because ultimately thinks agents as a tool right if uh your ex uh think it about in the pre-generation pre-agentic generation if you build the financial model in the Excel and it basically generates something wrong you are not going to sue the excel right you are ultimately going to s sue the human who built that financial model. Same thing goes to the agent as well. If the agent

42. 16:33

    goes wrong, it is that you need someone uh to basically put the shoulder on. So, it's a human ultimately in my mind. Yeah, I've not seen an agent be taken to court just yet. Maybe we'll see that in the future, but yeah, either the the human or the or the business. Ask your lawyer.

43. 16:52

    Yeah. So, I think we already have good risk assessment frameworks for these kinds of things. um you know if a company just tries decides to go the build route versus the buy route where they're taking on is all the liability of you know implementing all of that taking all that open source software and putting it together in a certain way and that's kind of what the agent uh trade-off is at the moment if you go and use open claw to build your like go to market agent you're going to take on a lot of liability you can go out to a company and you can outsource some of

44. 17:17

    that liability similarly like if you come use our architecture agent you're outsourcing some of your liability you can go and build it on your own you're going to take on that liability so you know, when you're talking about liability to an agent, what you're actually talking about is to a company or to the person who built the agent on open source software. So, I think those are pretty well established already, those kind of risk frameworks and you know, people have an understanding of how to how to um you know, d risk

45. 17:41

    certain things with data siloing or with guardrails. Okay. So uh regarding visibility into seeing how um this agent is operating and all it is, what are the um what has to be true before you are willing to let an agent act without watching every step of it?

46. 18:01

    I would say reputation as a whole. Uh and reputation can be broken down in in different ways. If you see like uh ERC8 ERC EIP 804, this is like the reputation standard for agents on on Ethereum. And there's different ways these agents have reputation. This can be uh they're running in like a trusted execution environment or it's people or other agents manually rating these agents. But

47. 18:34

    to be honest, I think reviewing trusted execution environment is still about reputation because each individual is not going to go and review the code for the agent. It's true of a smart contract as well. Like these are decentralized protocols, but each time you use a DEX, do you go in and review the smart contract code?

48. 18:59

    Do you? No, I don't think you do. So you're still relying on the reputation that other people have gone and reviewed this code. There's lots of ways to answer this question. I think if you got to go and try out things with open claw, then you should probably test them a few times, watch them really closely and go on. But I actually think there's another really interesting way uh that this is going to evolve which is um agents are going to have an incentive to build uh trust and

49. 19:29

    attention within their own community like human influences. So as a human you manage agents. If those agents have a reputation and have attention then you can monetize that. There's lots of ways to do that. So the agents you want to embed in your agents probably the ones that are kind of acting publicly this idea that attention and trust are good right? What does that mean? Well, that means that they have an incentive to go out and share good information with other agents. And so, we see the emergence of trust networks like

50. 19:54

    Moldbook and uh and Claw Hub. And they're not great yet, but the direction is clear. And so, in the future, agents are going to trust other agents uh to recommend uh skills. And if you look on like Clawhub or somewhere like that right now, you'll see that, you know, for any given piece of software like Adio, which is a CRM, the company didn't actually make the skill, which is huge.

51. 20:15

    It's high risk, right? And so like you should not trust those to be clear. Uh you know, you want it coming from the official company. That's a great point of trust, but it's all about, you know, trust trusted execution environments are a great great way to think about um you know, for specific things that you're building. And I think trust networks are are a great way to think about for for agents, a great way to think about how to trust them on a on you know uh for uh

52. 20:38

    public skills uh that they can kind of download and use on their own. And those networks will emerge. Um and you know I think that incentive structure already operates so I think that's how it'll operate in the future. Yeah. So first of all like this is a very interesting question and this comes very often to me. Uh so the way that I divide that agents are being developed in two ways. One is agents are acting as co-pilot or the assistants. Second as Owen mentioned that agents are going in

53. 21:05

    a full autonomy way right. So when we uh the agents are acting as a more like a co-pilot or the assistants humans are already in control right so uh but when it goes in the full autonomy that's where the problem starts coming into the picture the trust and the visibility is the important piece that starts coming into the full autonomy today nobody wants the perfect agent in one way because perfection is

54. 21:30

    very hard to achieve there is nothing called as 100% correctness so the way that uh things are happening in the enterprise side is people want some kind of a predictable outcome. So that once they understand there is a predictable outcome they can put enough guardrails around it and once there are enough guardrails then as I mentioned the other set of the layers of the regular software engineering the observability layer the

55. 21:54

    audit trail that gives enough confidence for them to start deploying agents without having some kind of a like a control by the human any thoughts on that last bit okay I didn't know maybe something would come up um So uh uh then you talked about you know we've talked about I feel things that the now what's happening now what the what the whole ecosystem looks like what enterprises how it's reacting now um the type of uh audibility that and visibility that you want to see when do we think that this might go away

56. 22:29

    let's look maybe three or five years into the future do we envision fully autonomous organizations and three is in the future is like the end of the universe. Like that is the the horizon there is just like you I I think like if I can predict something's going to happen in 6 months from now in the kind of time they are I feel like an absolute genius. So like that timeline's hard. Uh what I would say is now for the specific role that we focus on the

57. 22:59

    software developer we know that they're becoming generalists which means they have to be designers and you know um architects and planners and agent orchestrators all at once so they can kind of take advantage of the new paradigms.

58. 23:10

    That leveling up like of abstraction continues and will continue and at some point you know we go to the next level and the role becomes a little bit more abstract and the people become a little bit more generalist. um probably new specialties emerge. There should be in the next 6 months huge disruption. I think some companies just become Barnes & Noble and they become like Blockbuster. You know, they go away because they can't they haven't actually made that curve. And so

59. 23:40

    um big job losses in particular um capabilities and then because huge companies will just disappear um because they don't make the jump. Um I don't think it settles down for a while. I think that that everyone has to be leaping forward to keep up in every industry. Um, and that's that's kind of the disruption I see coming. Um yeah maybe we get to utopia at some point and everything settles down but like so like I'm definitely bullish on the agentic infrastructure especially in

60. 24:13

    this year because the way that I'm seeing is multi- aent is the new thing that is the realm of 2026 be it orchestration layer be it like a deployment layers so I'm very much bullish on the agentic infra be it like a memory security or whatever we want to call it out for this year How?

61. 24:33

    Yeah, like you said, I have frankly no idea what is going to happen in the future. I'm a bit of an AI doomist as well. This is why we should be on the same panel. We're like, you know, balance each other out. I try not to think about the future too much. And I sort of hope we do get to a point where things start stop or start to reverse. I think one of those things could be uh

62. 25:03

    physical materials. Can we keep funding this infrastructure? Um yeah, I think things will reveal itself. Okay. I would love for one of you guys to paint a vision. Uh can be optimistic, can be nihilistic, it's okay. But you guys are it's a great pan. You guys have very different perspectives on all this.

63. 25:24

    uh enterprise process automation with your uh autonomous agents, secure decentralized agents with proximity labs, and then energy engineering management guard rails with actual AI. Um let's maybe try and paint a picture. How might all this fit together um to build this unified AI economy that we're kind of here to talk about today?

64. 25:47

    Anybody want to take a stab at it? Uh I can take it maybe. So like if I have to stitch the all these three stories in one line, you need to identify the areas to automate. You need to write the code. You need to put the guardrails. Well said. I can't put it together.

65. 26:06

    So yeah, I don't know too much about what you guys are building, but you like mentioning guard rails and it seems that we're unsure of the future. Maybe we're not too hopeful about the future. I'm not. I know that. Um, but it seems like we're each building a component that hopes for the best for AI. Like we're giving it verifiability so you know it's doing what it's doing. We're trying to add like privacy elements in there so users

66. 26:36

    data is not being exposed. You're building guard rails so the agents presumably uh can't do things that they're not intended to do. So, we're all sort of building these systems that hope to build the best AI for the future. And yeah, we're I think we're all on the right track. I'm pretty optim optimistic. I think like using Salesforce to run a realtor's business is crazy. Like a generalized software system to run something that specific in sector. And what abstraction of

67. 27:08

    functional complexity does it means you can have many more companies specializing in different sectors and that just creates better software. So like in in software specifically I think it's going to get better and better and better for uh people who need software and um and I think all of this kind of specialization on the customer side like who you're serving is going to make a better world. I think people like I was talking about job losses I think people

68. 27:32

    have to you know retrain upskill move companies. Um, but ultimately I think it's abundance. Like I'm super positive on AI. Which jobs do you think they retrain to? Well, I think software developers specifically have to, as I said, become generalist. So maybe we still call them software developers. They do something different, right? And they're problem solvers.

69. 27:52

    Problem solvers. They're going to be less per niche area. So they'll have to go and get that job in a different company that's specializing in a different customer. Um, I think there's whole classes of white collar jobs that will kind of disappear and people will go become specialists in areas that are not functional but more specific to their customer. Um, yeah, but job losses is really about job moving rather than like someone doesn't have a job permanently.

70. 28:18

    I think that's a vision I can get behind. But, uh, we're out of time. Thank you so much for coming on today. Uh, that wraps up this panel for the evening. Give him a round of applause.