NEAR AI Office Hours #13 with Atlas Computing

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:00

   okay hello hello welcome to the new episode of AI office hours uh we have Atlas Computing here today I'm excited to dive in about what you guys are doing as well as discuss more in general about uh kind of AI safety in different uh aspects uh I'll I'll let you guys introduce yourself first maybe Danielle and then ban and then we can dive in sure so um I have a background software

2. 0:30

   engineering uh for the five years before joining Atlas I was working at a dark Air Force contractor on how do we rapidly adapt and and integrate uh complex High you know High Assurance or or otherwise you know safety critical systems uh and the role I was playing was on tool development of the stitches systems integration tool chain which um was actually one of the most successful Dev programs of the past decade hey everyone uh thanks for having usia

3. 1:03

   um and near um I'm bdan I'm playing a product role at Atlas started recently so I concern myself with product development product management design and a little bit of marketing awesome yeah marketing is the most important as uh we're stoked to have with us I mean that's the thing is like we can build whatever you want but if nobody knows about this it doesn't matter so that's that's right exactly as a developer you know reform developer at this point that's

4. 1:43

   uh generalization but yeah I guess we can maybe kick it off with kind with kind of what are you guys thinking about From perspective of like what what are the biggest risks um and then let's just let's just time bound it in like next two years versus you know not in longer longer term but what are the biggest risks where uh do we see kind of potential failure modes and is the current approaches people taking are

5. 2:20

   sufficient uh Slash like addressing those risks right uh and I'm talking about kind of General a safety that um and dropic and few others are doing let's let's kick off with that topic and then we can Spa off in other directions all right so um sure it's never enough nothing is ever enough right that's the short answer um but to frame it let's tell the audience who Atlas Computing is so uh we're nonprofit um trying to run like a

6. 2:54

   startup um it has a very high research profile so we bring a lot of researchers in high performance industry people to walk work on this like hard targeted AI problems uh one of them is safe Ai and even more specific guaranteed safe AI um the challenges one of the biggest challenges is that uh nobody knows exactly what are the issues and what are the kind of issues we're going to

7. 3:25

   encounter you know uh next year and two years from now um the only good news here is that the field is making progress all these fields connected fields are making progress so hopefully we get better as we go but there's no right answer in terms of how the world is going to look like two years from now um what's working in our favor is that uh a lot of smart people are paying attention to this and they bring their do domain knowledge to

8. 3:59

   to Safe Ai and other aspect of AI safe AI is not the only one right so that's what we're doing we're trying to bring all these people together to to work on this very targeted problems I'll add in a little bit one of the things for instance one of the motivators for me coming into the work that Alice is doing is if you look at our critical infrastructure whether that's in our Energy Systems in uh

9. 4:34

   critical privacy in uh you know military defense systems um the approaches to establishing really high confidence when you you can't be wrong are really quite mature and there are many different angles that are much more than just testing your system but if you look at you know what we know how to do and what we've built out and deployed in AI safety uh it's a lot more limited

10. 5:05

    it's a lot more narrowly to different uh test based approaches whether we call it red teaming or evals at the end of the day you're treating the system like a black box for potentially a really wide application domain um and if we extrapolate forward to places where uh some of these modern AI capabilities is maybe more and more in the loop of places with critical safety concerns oh man I hope we don't think that that's good

11. 5:42

    enough but I mean to kind of somewhat of a contra point right uh when we evaluate people for this critical decision making critical roles right we are also using tests and evals um and we're not even you know like not trying to be that you know uh adversarial I would say for the most of the most of the times um I think the only difference is with people we kind of usually do this test throughout probably like 20 years of you know their career before they get to role which you

12. 6:23

    know like the criticality of their decisions is growing over time which obviously is not happening here I'd be curious I for for you when you think about whether to trust a human being on you know on your team or or someone who's providing services to you maybe it's a human organization and it has to be very reliable and robust getting you know in some circumstance we're getting it wrong with

13. 6:53

    be catastrophic for for you personally or your community um how do you approach trusting would be I mean you usually you have like set of processes that a like critical decisions are not in the hands of any single person right I mean that's why we're building decentralized blockchains which run you know not it's not a single server that's run by a single person it's you know a number of validators run by independent parties you know they all have different acceptance criteria and

14. 7:28

    processes how they you know pre pre- War pre- roll out but then before that I mean let's say just like release process for a protocol just to give a very like critical example right if something goes wrong you know multi-billion dollar protocal stops and so you want to have like a whole back process right with scaryy runs with you know pool request reviews with kind of lots and lots of different steps stages as well as you want to know that person people who working on it are you know at a certain

15. 7:58

    bar of proficiency and uh skill and understanding of the system and I think the understanding of the system is probably the hardest part from perspective of uh the kind of safety but again I think that my my point is more that generally individuals are as susceptible to you know adversarial Behavior as you know computers are right if if it's not not like PR injection then it's you know

16. 8:31

    at $5 range that you know physically trying to curse you to do something so uh I think the I think the to me indeed is it's always and and this is the kind of I've used this few times I think most of the safety issues are not the AI issues they're the systems issues that we don't have for post AI world right if we assume AI here and we have all those kind of AI systems and again they have some properties right which will be hard to observe but again humans are hard to like know what are the properties you

17. 9:09

    know any individual person have like you never know when somebody will you know have like uh you know mental for example or burnout or whatever and so you need to build systems around to modulate that and and deal with that um in general so that's kind of why to me and again there's some things that are on the other side kind of from a attack side AI is becoming just a tool for multiplying something that like individually be already hard right so

18. 9:43

    the good example I like is AI is really good at finding bucks right and like it's probably on an average side of any individual developer but like developers are not really good like not really good at paying attention to read somebody else's code and so here you can scale that specific instance of look at this function find all the possible options how it can go wrong and so if you have that that at scale now running across

19. 10:13

    every function of some code base they just going to like statistically given the know that is you know hundreds of bugs per thousand lines of code it'll just find all the possible VAR abilities now you can just go and attack that code and like yes a human could have done all of that it just like you can do it at like massive scale now for reasonably cheap right it's going to cost you potentially like thousands of dollars for per like major vulnerability in some

20. 10:40

    systemat systematically important code so I think that that's to me is like a very much realtime security issue that we need to be addressing and I know you guys are looking at form modification as a system around that that actually ensures that the code again there's nothing about AI this is a human problem as well and so like how do we ensure that the humans or eyes doesn't matter are

21. 11:04

    actually producing correct code and the systems behave in exactly the same behavior we expect right so formal verification is not necessarily A a method to around humans is to you know verify more what humans do I think guaranteed Safi is bringing both humans and um formal methods J this uh formal verification being one of the one of the methods and I think people in in blockchain and crypto are um more familiar with formal verification of

22. 11:39

    lately because it has been pushed uh with uh pretty good success uh in the last five years I would say yeah I mean definitely the the risks in crypto are just usually higher per line code right the smart contracts you know like under thousand lines of code for the most part or like couple thousand but you know if you mess up online you lose potentially hundreds of millions of dollars and so right that's

23. 12:13

    where kind of the the area the for developing uh this is I mean people are willing to spend millions of dollars to develop better tooling fortification for that right yeah there is that nice opportunity in the blockchain space where you know if you spend 2% budget against the the size of your contract uh on safety well it's very easy to to you know make the argument that's a reasonable investment um I'm I'm much

24. 12:46

    newer than bogdon to to working in formal verification uh so I'll I'll say a little bit of what I'm seeing and and then he can add in um I think a lot of people who are in the formal verification space and and try to sell it think oh the reason to formally verify a system is to stop a bad thing from happening to make sure okay we don't have some critical uh you know vulnerability that uh whether a malicious actor is is

25. 13:19

    involved or just a a fat finger we don't take down all the transportation systems that use Microsoft throughout the entire world um but it's very hard to sell people on oh I you know do this difficult thing and you'll you'll rule out something that wasn't likely to happen anyway from your perspective or or we can't show you we can show you bugs that we found but um interestingly uh instead of selling

26. 13:51

    the point about security often people are more successful selling trust uh so a place outside of of crypto and blockchain where formal verification has really made Headway in the industry is that Amazon's AWS AWS uh has an incredible value in their brand of having their customers trust that they will you know be stable robust private secure uh you know many things and so selling trust to the customer instead of you know security internally has helped AWS make a lot of progress uh really

27. 14:31

    getting value out of formal verification but closing that full uh feedback loop up to up to senior staff and then another exciting thing we're seeing is we now get some of the very highest performance systems by using formal verification people used to think oh if I'm going to formally verify code well it's very hard to reason about code I'm probably to use a narrow subset of say C or whatever whatever language I'm working in uh and that's going to hit my performance instead we now see that if

28. 15:09

    you want to remove like Final Extra assembly instructions from a routine that's running billions of times an hour which big companies like AWS absolutely would love to shave incremental instructions off uh sometimes the approaches are so unintuitive and and hard to reason about as humans that if you hadn't formally verified that it really still does your you know shot 56 hash correctly or or

29. 15:41

    whatever you know narrow functional contract you have you would never get to that level of performance yeah there are there are a lot of examples like that and um where formal verification helps uh formal verification is not the only method um formal specification formal design also a matter a lot uh you know how you need to know how to to handle that um yes one of the the risks and kind of

30. 16:17

    barriers is the the cost and the time U and the expertise that it takes to uh perform formal verification um there's a question of of where formal verification is necessary and where formal verification is necessary now um AI systems have a lot of components and um it's not very clear which ones need to be verified like right away um so establishing a order of priority for that and kind of attacking

31. 16:50

    the problem as a as a you know kind of Complete product problem is also important um to Daniel's example about AWS there's another um um I think really good example of incremental help um with the comert uh the C compiler comert um C is not you know easy to formally verify uh as a language um but um Comer proved itself a more reliable than GCC clang that been around for a long time and sure you know um the

32. 17:34

    code extracted is not verified the EML compiler is not verified EML you know runtime environment is not verified but that doesn't mean you shouldn't verify anything right so even small incremental verifications along the way or like of components in the system still bring an advantage at least of safety so I the it's very hard to negotiate this um formal verification is the silver bullet and don't do it at all right like people expect everything

33. 18:15

    from formication and if it doesn't give them everything like let's not do it you have to do it but at least on components and try it and and I think what AI brings now and machine learning and um large models bring to this equation is a lot of help of course that help can be taken like for granted and automatically um that needs to be verified the whole solution needs to be verified but I think

34. 18:48

    it's um you know against the bigger problem you big you bring bigger help in a way right so they balance each other right like you bring a bigger gun to a a gunfight yeah I think I mean you mentioned right the specifications the defining the conditions of what you're trying to prove that's that's a big part of the problem in the first place because like whatever you prove you know if you have unspecified kind of uh

35. 19:20

    conditions like for example that the memory doesn't get overrun like if you don't specify that then you know your pro proof will be correct even though like the code is still doing something unexpected from General sense and I think that's uh that's kind of generally the interesting challenge of you know even specifying all of the things that system should do or should not do right is is extremely hard but at the same

36. 19:51

    time what llms are getting good at is kind of this Common Sense type reasoning right where yeah they're not maybe very good at like structur reasoning this is why we have the you know more kind of symbolic machines right for proving Etc but LMS can actually convert like hey I want a program that like sorts an array to you know sort an array and don't run out of bounds right and stuff like this and so they actually convert into set of conditions which like are

37. 20:25

    common sense and you would not think of specifying but you know M kind of knows based on statistical kind of data that those are the conditions and I think kind of going throughout the steps adding that semantic understanding and semantic kind of Leverage is indeed starting to help to leverage these tools which before you know like forication for smaller smart contract side it's

38. 20:51

    like a three three men month job for like 100 lines of code or something it's like a pretty ridiculous uh you know complexity and so getting getting to a level where and it's because like actually writing specification first is taking like first human months right and then proving it and so I think that those are fundamentally the pieces as we decompos it but I mean why I like this kind of because you can map that to a lot of other processes right because I mean

39. 21:24

    code is just like describing a process and kind of some transformation and so if we're talking about medical if we're talking about you know you know shipping stuff to space or whatever like all of those places they have kind of similar like hey defining sometimes it's really hard to Define all of the criteria like medical you know the the drug should not you know have like weird side effects

40. 21:49

    you know attacking part of your body that it wasn't targeting but like defining that is you know pretty complicated because we pretty much want to say it should not attack anything right Define everything else in your body so I think there's a lot of tooling that will need to exist that will just help with this definition process and I know you guys have published right the tool chain for AI assistant code for specifications since this is verification so you're definitely working on Sal

41. 22:19

    Direction yeah to kind of plus one and lift out some of what you're saying for where can AI actually accelerate or otherwise improve proove the workflow of doing formal verification to to create high Assurance um exactly as as you're saying well llms today can pull from a vast pool of knowledge of kind of what humans have have built up over time can then leverage that back to have conversations and and uh discussions around what should be required what are

42. 22:54

    considerations that you don't want to forget about um you know if uh if people are doing code generation today they just get the code and it either works or it doesn't and maybe they're going to review all the details of it but often not uh it'd be really good to have a bit of an intervention of a conversation that says okay you're you're working with medical data or uh or financial

43. 23:22

    information okay uh maybe you should consider authentication let's think about some r rules you would want about privacy um so that's a way of leveraging context um we also know okay alms are good with language and translating uh between representations of the same information and so just as we're seeing with like GitHub co-pilot or these other code synthesis and code explanation summarizing tools uh one of the challenges with formal verification is

44. 23:52

    okay you're writing these specifications uh that need to be machine understandable so they look like code you want to move between the highle human intent and and uh again those discussions of well what should this be and push that into formal specifications that uh machines can have reason about effectively that's another thing that I think there's lots and lots of promise for modern generative AI to be helping

45. 24:21

    out and and finally and ier you named this one as well but uh there's a lot of brunt work when it comes to them uh doing formal verification or or even other kinds of formal reasoning that might give uh you know a lower confidence bound um and a lot of work is very repetitive and yet only some Specialists have that that builtup set of skills right now to write you know just the

46. 24:47

    right Loop in variant for you know don't overflow this integer boundary in in this s routine or whatever it is and like those kinds of Highly repetitive fundamentally coding tasks it's proof coding now look a lot like what we're seeing uh L be very successful with in software yeah I think there's an interesting point just for people who are not familiar um because a lot of those tools are coming from uh pretty much mathematics where this I I use for

47. 25:26

    mathematical approves and and the interesting property that proofs have is compatibility because if you have a proof of some property and then you use this property in your other in proving something else right it kind of naturally either works or doesn't like that proof itself is kind of plugged in into the kind of big appr proof you're making and so I think that's a very kind of important piece that's needed here as you know AI generating more and

48. 26:04

    more code and obviously this is one of the things I'm working on that we want to reuse you know libraries we want to reuse some other code all of this will one way or another be generated by I or built by people but you have no guarantees around its kind of quality and and safety and so having this proofs is fundamentally allowing now to compose and then if your precondition if your prec condition says like hey it cannot overrun memory and you're using something that doesn't have

49. 26:36

    those precondition it actually will and like the proof is not there it will actually fail and so you have this like really nice properties of composability that you don't have right now in the code because for example tests don't usually compose in the same way where you like testing for kind of some other like Edge case of some other module in the library in your tests so I think that's a really like powerful property

50. 27:05

    that at least like I wasn't wasn't immediately obvious to me until I learned that kind of how the mathematicians now work where it moved away from you know sending each other PDFs with proofs but actually just sending lean proof and you can plug it in into yours and so now you can work with anybody like you don't really care like who that proof came from from like who that lean file came from like you can plug it in into yours and then if it all compiles then you know things are

51. 27:35

    working so and I think rust is the only like programming language that's like close enough to that state where you know things compil most probably they actually work there still maybe like logical bugs but um from from a coding perspective that was also an interesting experience um yeah I like what you said about tests not composing uh one of the almost mantras from the previous organization I worked with was constraints compose examples don't if

52. 28:10

    you have a list of requirements you know if you got the requirements correct and now someone else has their requirements your requirements are still necessary and these these do compose really nicely um certainly at the at the informal language level there can be challenges as we you know talking about formal verification well now there could be syntactic challenges with aligning just

53. 28:34

    the definitions what are what are you semantically saying but conceptually at least we're in a better starting point and and to make things more complicated uh you know guaranteed safei is trying to bring uh users people in the loop uh allowing them to write the rules um write the specifications and the specifications to be verified take into consideration this um user opinion and um I know you guys uh

54. 29:07

    promoted this um user own data and user own AI um you said previously that uh Alex is is working on a a product yasna um that we don't know much about yet what I don't know much about so maybe we can talk about this overlap of the human Center approach of how the end user is is included here yeah I mean that's a good question and this I mean part of it is spans from that right

55. 29:41

    now you know open Ai and Tropic right these organizations are very much like hey we need to make AI safe and their definition of safe is safe for them because they don't want to get sued by some that that's really the main like and I you know entropic is maybe a little bit more trying to figure out whates that mean but like generally like the safety for a company is safety for the company and so so there's no actual like and yeah I mean they like because

56. 30:16

    of this the filtering they're doing they're like declining to give you legal advice Etc it's all spans from that like their business that needs you know that make trying to make money and trying not to get into trouble for providing that service and so I think that that's why like I I think it's important to understand that when people think of safety kind a from coming from companies um and so kind of the opposite side of

57. 30:41

    this similarly how it's opposite for you know companies trying to make money from you user on the eyes about how do you make money or how do you benefit not just in monitor but also like in education in you know opportunities and ability to to you know learn explore the world and kind of intrinsically motivated as well right and so that that's where like we want to really focus on and kind of provide really an alternative to what uh kind of

58. 31:13

    uh I call it for-profit AI because it's it's less about open or closed Source it's more about the motivation and kind of incentives that they're driving it and now similarly on safety side it's the same idea is like now it's not about safety of the company that produces the model but it's about like what is the safety for individual right for the safety for Organization for company for

59. 31:38

    community and I think there it's actually a lot less about that the model will you know output some garbage or some incorrect legal advice or whatever it's more now about actions it's like you don't want model to take actions which are not benefiting you or you know potentially creating concerns and this is again where some form of creating set of constraints that you want models to live in um and like I mean the example to use in web three right imagine you have a trading bot or whatever that you

60. 32:10

    instructed to do things and you know it potentially like creates makes a mistake or whatever gets hacked by prompt injection like it doesn't matter but it should be you know guaranteed to not be able to execute something that's like not beneficial right so so for that you need like what are those constraints what are the ways to kind of Define and again there you can actually escalate to a user and say like hey you know this is happening these are like it's outside of constraints like for example you know you you're swapping

61. 32:44

    something or like it's trying to unbind your positions Etc and it's not uh it's like too much slipage or something right it's like you know do you want to proceed with this so you always have an ability to kind of go back I think was knowledge retrieval and was anything like that I think designing as well in such a way where you have multiple points of view you have a way to see

62. 33:09

    kind of the sources and and like voices that's at it again this is more I mean not it's not about like again safety in the same sense it's more about understanding that there's something that are facts and there's something that are opinions and and really distinguishing those and again this is something that I think uh it's been we we in general like with social media and search engines are kind of starting to lose the grip on and

63. 33:41

    so I think it's important to again design products around that and so and again this is like this only will happen if you start designing around the user right because if you're trying to design around a profit for a company you don't care you just like show whatever they spend most time on they click on what they buy and so you don't have the same motivation to like okay well you want to

64. 34:04

    build an informed public you want people to be you know more educated like a year later we want people to be more educated on average and be have a more informed opinions like nobody has such a goal right even even the government doesn't have such a goal so like how how do we actually set incentives to generate that to drive that toward individuals and I think this is where I mean blockchain to me is really powerful tool to do that because it has so much individual incentives as

65. 34:35

    well right we want more people to be kind of educated uh participating and being active members of this community so we have the motivation to actually create value for them right so it let's make it clear for the audience that you know safei is not the only approach out there and they're more you know there's human centered Ai and trustworthy Ai and explainable Ai and responsible AI that are coming from this angle of representing people and keeping

66. 35:09

    people's interest at heart and I think that makes it very complex on both Technical and product side uh on the technical side yes you can employ formal methods formal verification but if you're dealing with components that inform each other or change each other and the system changes as you go whatever um tests or properties um you wrote might not hold or they might change as during that conversation between systems or parts of the system right so that's that's kind of the technical

67. 35:49

    Challenge and on the product challenges getting out of the just the technical approach right like starting with human goals and also verifying if those are accomplished and it's not just about the company and it's not about the you know the profit maker yeah I'm curious how you guys see um on the on the product side for use cases where it's not Engineers who are actually interacting how will

68. 36:34

    people uh both Define their criteria as well as uh understand kind of if things are matching that um it's hard I I think there's a lot of fear uh first of all about how AI reacts in general so everything you you read or like here in the Press it's it's about this um doomday view that something's going to go wrong and it's going to go terribly wrong and we're going to have a nuclear war in our hands because of

69. 37:14

    AI um maybe uh we don't know and I think the product approach here probably the the safe product approach is is start looking at how each component uh determines an outcome and still keep it manageable and trying to verify um components as they relate with other components in in in the larger system as much as possible I mean there is no like you said it's it's hard enough to do it for a thousand lines of code um it's really really hard to do it for

70. 37:52

    a million lines of code right so um no one knows but again the good news is that these components are in the focus of very smart people so you know what is the best language to write specifications for from the from the human point of view right um and and so many so many details that need to kind of be ironed out and so Atlas is just a you know kind of small part into that configuration of researchers industry policy makers think thanks I

71. 38:34

    mean it's it's the whole society you know U merging here it's not just one company with one technical solution yeah I strongly Echo the sense that there's not you know One Singular solution for every industry and application we you know domain expertise matters a lot in reasoning about what do we want what uh you know what would be okay and and not okay and and how could we achieve that and so um

72. 39:15

    I I I don't think we've seen too much domain specific safety in especially the the larger dialogue around well okay how should we be moving forward um maybe this is because a lot of people's idea of of modern AI is chat GPT which you can kind of talk about anything it's it's like super domain agnostic um but I'm looking forward to that shift towards domain specificity I guess add to that one one more thing actually would

73. 39:53

    be uh there's where it hits consumers directly you know a customer is using some service that's explicitly involving llms artificial intelligence maybe it's a natural language interface and so they know there's AI in this maybe they're in authoring an agent to go off and do something or authorizing an agent to do something and then there will be lots and lots of uh products and capabilities using modern AI capabilities that are not immediately visible to the consumer

74. 40:26

    oh this is AI but it will have many of the same big picture questions were concerned around around you know when can I trust juristic reasoning what will and what generalize am I in the safe domain area or not what do I care about ethically and from a safety perspective and this is where you know of course as a society we have to start having conversations about governance and we have to have ideas and and approaches for how governance and policy and Regulation and so on will integrate into

75. 40:56

    into these system this I think I think uh you know hopefully gets at a lot of the values you've been naming Ilia of it's not one company deciding this is what's best for us and we'll pretend it's best for the world come on every company is changing the world to the way they want it right um yeah I just wanted to mention like I think to to me a lot of the fear for AI I mean outside of it's coming from movie side is really coming from people

76. 41:29

    projecting their own stuff onto right like the nuclear war part is I mean literally like humans are the ones who get us on the brink of nuclear war right they like at the force time right now in in the last like 60 years to get there or I guess like 80 years so there reality is like it's it's mostly and and same as like oh it's going to kill a soul which I'm like the there's no there's no value function there's no incentive because like it's

77. 42:01

    so it's so much energy to actually try to kill everyone it's way easier to just wait uh you know like create a social media platform that you know doesn't like removes people from wanting to reproduce and then wait for 80 years right oh wait this is already happening right like our birst rate is reducing dramatically already and like we we kind of getting extinct ourself like without any additional uh like AI systems right it's

78. 42:34

    literally our own incentive structures at designing that that kind of led to that so I think like a lot of it is just our projections and so I think it's really important to understand kind of again even even when we have AGI it's still going to be a tool like it's still going to be somebody initiated and run it and defined it what the goal it is and yeah I mean it can like mutate the goal

79. 43:00

    whatever but like those are still based on some inputs right it's not going to be like uh like out of nowhere and so I think those are import really important points and then the other point and yes I mean somebody can design an AI that like is designed to kill everyone right that that's important to note that's that's a part of the problem where now we have like tools that are like can behave at scale and plan potentially long term but it's the same thing as if somebody right now

80. 43:28

    sits in their evil layer and plant you know destruction of the world right like we need mechanisms to understand detect and kind of prevent that independently of it's like a or a person is planning that and kind of you know some some Evil Genius you know James Bond style planning like a thing so I think that that's a really again like from a systematic approach like how do we how do we design things and and like the the nuclear war is interesting because like how do we design a world where everybody's in

81. 44:01

    check right it's it wasn't AI problem it was like a people problem that we need we need a set of motivations such that everybody knows they can destroy each other and so like okay we're g to not do that um so we kind of I mean not not that uh obviously um negative but ideally we have positive systems that like everybody actually wants to work together to benefit uh and so even if there is an AI it actually just fits into the same system right that like oh

82. 44:32

    actually the most beneficial for everyone is like contribute here and again this like block blockchains a really like examp like a simplistic examples like Bitcoin is a simplistic example of like bunch of self- motivated agents participating because it actually benefits them uh to do that in following protocol yeah it's actually very uh extraordinary how fast asked um and how well you know researchers and academics from US Canada UK came together and how

83. 45:05

    this organizations are spanning out to to discuss and address the problems it's comforting to see this body of like um really really uh experts that been in the field for decades uh coming together and discussing this in a in a open Forum uh it's beyond the academic conferences it's it's becoming more and more public uh governments are taking action by you know maybe in different ways like UK has Arya who's you know more focused more uh

84. 45:41

    maybe government controlled and funded where us is more decentralized and kind of um it allows a combination of government and private entities to to take smaller parts um and so I us doesn't make much I mean us at Le current set doesn't make much sense right there's like limit on compute there's like you know hey it needs to check that it's not doing bad things like right that's very yeah right

85. 46:20

    but that's just a government position right there are the players uh that are having a voice and you know I think I think the cumulative voices around the government are also um playing a a very important part and that resonates with you know what scientists in other countries are saying so uh again I would I would see that as a as a huge positive um to kind of uh balance the the the doomday um approach that

86. 46:50

    circulates uh uh in the media so there was a tweet uh just hours before we started this conversation from Yan Lun the chief scientist AI scientist at meta that talks about the llms and like you know they cannot solve problems that they haven't been trained on they cannot acquire new skills without human help so you know don't don't go there yet don't you know create this like apocalyptic view without knowing what's uh what's involved

87. 47:32

    so yeah but I feel like his his position is also somewhat limited because he's like yes llm cannot do it but but a system that uses llm can right there's actually a paper today as well which is a researcher or scientist paper from Sakana folks and uh you know it's coming up with new Concepts test hypothesis run them writes a paper and a process uh so yes a single LM may not be able to do it but you know Monte Carlo search around it a a set of you know systems

88. 48:05

    around it can so and and that's the beauty of having all these voices kind of in the open forum and you know being fairly transparent to date and kind of address it together so to me that's a that's a plus oh for sure I think I think the fact that we're having this conversation and able to do that versus it sits somewhere in Manhattan you know on a Manhattan Project in Los alos right on a

89. 48:34

    hill and nobody knows about this I think is is hugely beneficial um you know I really liked the conversations here about how do we think about the system around the AI or the human and how do we put safety into that system one of the things that Atlas has uh really gotten to to chat with other people in The Wider Community about is uh when we look at Safety and Security in our most critical systems

90. 49:06

    today almost always the the software is the worst most vulnerable place if there's AI okay that's even worse that's even worse in terms of our confidence and our insurance levels um and then of course the worst software vulnerabilities are in the old stuff and and we have this tremendous glut of Legacy uh software that does important things maybe decades old is not easy to replace both because it's massive and sprawling and complex and because incredible amounts of effort

91. 49:46

    went into establishing confidence in in the behavioral correctness at least in domains it was originally developed in we don't want to throw out all that good work uh but now the context has shifted a little bit or now you know we have 30 years of accumulated uh expertise on how to break in to uh insecure software and so the there's then this question okay how do we shift these these infrastructural systems that underly

92. 50:22

    many of the surface level tasks that that humans or AI are are working on because that's often the the place that our systems are getting hammered badly today yeah I mean I think the the interesting part is that people actually see those systems that been there for 40 years and trust them more because they've been around 40 years and I think there's an interesting kind of effect here

93. 50:53

    that like on one side especially if it's a I mean I think in in in open source it's it's definitely better because like if it's been an open source for some period of time with the closed Source like the fact that it's been sitting there and not being hacked doesn't actually mean anything because you haven't really exposed it like it may have not been exposed you don't actually know because you you you don't

94. 51:17

    have a good way to measure like how much like approaches to finding vulnerability to actually have um but I do think I mean I I actually fundamentally think we'll need to rewrite the whole internet and now that we have ai we can actually probably do that pretty quickly and we will need to rebuild every single piece of Stack like starting from literally tcpip you know Network in stack and up

95. 51:45

    to level from there and you know while we there we add cryptographic you know cryptoeconomic security and all those things as well to make sure that we actually have like enter encryption you know we don't need to have Telos because we can just pay our way through the networking protocols Etc um but all of that will take time to mature the tooling that is able to do that and then it will take some immense amount of uh like product and kind of

96. 52:17

    mental shift to really adopt that instead of having another Frankenstein where now we like attached another thing on top uh and that I'm I'm excited again because I see the kind of blockchain world is where we found ways to really have living protocols right like the example I love to use is like ipv4 right to IPv6 transition IPv6 is a 15y old protocol right we still haven't transition to it I'm pretty sure I'm still using right now on my laptop ip4 uh and this is because there's no like way to come to a global consensus

97. 52:58

    that like hey now we all use ip4 get out of it right like you know you have six months but also like all of the code every system is ready to do that and it's just doesn't transition but in blockchain because we have a global consensus we're actually able to do that we can coordinate um upgrades like this and so I'm I'm pretty excited on like this combination of where blockchain will

98. 53:20

    lead the way of kind of rewriting the stack and and then leveraging that to build more Safe Systems because the stack is built on kind of the safe concept again so so I'll jump in and say I don't believe in global consensus I don't believe it's actually ever possible um I think it's uh you know I'm educated from my background working with and around military systems uh there's always this question of how do you interoperate or get everybody on the

99. 53:54

    same page if you're answer is will you know locally coordinate every pair of people well dear God you just signed up for a very expensive task but if your answer is everybody will agree on how to interact then how do you know what everybody needs and wants is it the same do you have everybody in that conversation um even if somehow you had everybody in that conversation which you didn't you don't have the future in that

100. 54:27

     conversation but I do think there's something compelling in the ability to evolve the protocols that that you are naming um and the more that we think about forward and back compatibility as part of the design and think about interoperability with the things we don't know inside of the design is is uh a helpful place to go um and then finally I I think again shifting more towards orienting in terms of requirements and and being a bit hierarchical and what really matters can we get clear on what matters well now we

101. 55:07

     have you know sub problems we decomposed this we recurse that structure but the more we stay oriented towards those design considerations those are the parts that we have higher levels of consensus on those are the parts that we can reuse much better and if we're hopeful big warning of being optimistic and hopeful here but if we're hopeful we can start to imagine the paths by which uh you know modern generative AI can help us keep Rec concretizing from those high

102. 55:40

     level designs at a pace that that matches the upgrade Pace we actually need to rely on at the implementation level at the solutions that are getting deployed all right my my my closing thoughts if I'm allowed um are put people first um solve a solve a problem for people um and you know take as much as you can chew and um make that solution uh transparent and put it out in the world and we'll see how it all fits together

103. 56:12

     and how we all benefit from it and thanks for inviting us here sure thanks for coming uh let's uh you know let's keep making progress in all these directions there's there's a ton to do but I'm excited and uh I think changing the conversation more toward this kind of like how do we make systems uh behave in in expected ways right I think away from Fear mongering around AI you know running out of the box and you know killing everyone I think is is really important and excited excited to have this conversation with

104. 56:51

     you guys thank you so much cool really great to out here all right um I didn't know if I were supposed to stay or um