Standards, Policy, and Safeguards for AI Systems

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:09

   Happy afternoon everybody. Happy Near Con. Do a quick round of introductions starting with Mike. Hi, I'm Mike Perry. Um I have been at the tour project for almost two decades now. I've worked on all sorts of things. Uh tour browser and tour VPN threat models and I'm a technical product manager and I'm responsible for the product backlog of the tour network basically.

2. 0:34

   I'm SJ Klein. I'm a Wikipedian and co-founder of the public AI network and currently building the AI commons for data and systems and evals for agents and larger systems. I'm David Casey. I run an organization called funding the commons and we run events including conferences, hackathons and residencies to support open source digital public infrastructure and we work closely with the near ecosystem which we love.

3. 1:03

   So I'll get right into it since our time is ticking. Um almost everyone agrees that we need AI governance. Not everyone but almost everyone. Uh but nobody really agrees what that means in practice or most people don't agree. So Mike, you've said the history of privacy regulation makes you wary of policy approaches uh top down approaches that could end up looking like censorship of model weights. Sam, you are pushing for minimum benchmarks and insurance frameworks.

4. 1:39

   Where is the line between necessary guard rails and uh return of the crypto wars? Uh either of you take feel free to start. I mean I guess I could give a little bit of context for uh my feelings on privacy by policy. Uh I was the lead on the tour browser back when the do not track header um was a thing. It was basically a regulatory push of this HTTP header for websites to obey by policy your desire not to be tracked. Had all sorts of problems eventually was basically

5. 2:16

   more or less abandoned um because of its just inability to be enforced and instead we focused on privacy by design. The censorship piece is that, you know, the crypto wars one, two, three are were all kind of this reaction to, oh, crypto is munitions, crypto is uh needs to be blocked, censored, whatever. I feel like there's some energy behind this regulatory capture or this this incentive for particularly the leading labs to argue for uh regulation and

6. 2:47

   censorship of their competitors that may be open weight in China or even in the US so that they can you know maintain their lead and um under the guise of safety or maybe legitimately trying to like censor uh potentially malicious model weights and I feel like that is also very fraud um probably will end in the same way that crypto wars 1.0 know ended with people publishing these things on I don't know t-shirts or optical mechanism you know print on

7. 3:16

   their shirts or what what have you so a little nervous about it but I feel there there's different definitions of governance and what it means to the extent that some people don't want governance I think everyone would like to know that if they're interacting with another agent that the agent is somehow self-governable that it's able to stick to agreements. It's able to make uh commitments that it can follow up on that has enough self-control and enough consistency over

8. 3:46

   time that you are interacting with a persistent endpoint of some amorphous ecosystem. So if we start from there, a minimum that we that we should all want is mechanisms for self-governance. Uh ways to attest who you are and where you're coming from. Ways to uh describe your intents and your desires and then evaluate yourself over time to see if you are continuing to proceed toward along those in that direction. and ways

9. 4:20

   to build a cohort that you trust to give you feedback about whether you're continuing to live up to your initial direction and goals. Uh if we expand the notion of governance to include that kind of self-governance, hopefully everyone agrees that uh governance is useful and you should want ways to describe how much or how how transparent or how independently evaluated some governance framework is. for u like the kinds of minimum benchmarks that I think

10. 4:52

    we our ecosystems need include u ways to know whether a system has any sort of insurance. Uh ways to know whether there are safeguards in place for known failure modes. Whether whether the agents you're interacting with or the context in which you're interacting has defined its failure modes over time has some kind of um visible overview of system health so that you can decide based at any point how whether your context has changed so

11. 5:30

    that you would like to ask for some increased confirmation that you're still engaging in the in the contracts or um procedures that you thought you were. Thank you. And I you mentioned the crypto wars. I think there is this idea that there will be external bodies that are surveilling activity and saying this activity has to stop or can only happen within these constrained environments. there's a different flavor which is whatever you're doing, you'd like to you'd like to be able to describe to your own

12. 6:02

    satisfaction um which of those control systems exist. And then it's not that there is some external force that's trying to limit what's happening. It's that you should know what you think you're getting into and then be able to All right. And uh speaking of control systems and standards, let's get concrete with this audience. Let's say you are a builder um or a user running a private AI agent in a decentralized

13. 6:33

    cloud. Uh let's say you're using Near's infrastructure. What's actually missing from the trust chain today? And what standards would you build first and what is the fall back when they fail? So I feel like again I'm going to go back to my uh the I guess cipher punk history. Um, in 2013, the the Snowden Revelations, there was this famous slide of from the NSA's uh documents. It was kind of like a yellow post-it note. It

14. 7:05

    had uh the diagram of Google's network infrastructure and then a little arrow pointing the center of the uh of a SSL concent concentrator with a quote SSL added and removed here smiley face. And this was obviously the the vantage point or the attack point that the NSA sought to compromise within Google's infrastructure caused a lot of uproar among Google engineers and and even public comments by Google about how Google is making this push to secure

15. 7:36

    their traffic and protect people's data in transit. And here's the NSA um scheming to subvert it at the point where they are decrypting it. It also encouraged um and led to an effort to make that the data center connections encrypted and more difficult to surveil by any arbitrary adversary or data center operator. I feel like the the uh endto end the the private inference story has these same failure modes in at least three places. Um there is the

16. 8:10

    client where your web browser is talking to potentially a private inference portal uh that where the the it says okay I'm talking I have attested that near AI is running the infrastructure uh and the there's a confidential VM running this agent and then that's talking to a confidential inference provider that is has an end toend encryption but your the JavaScript is just telling you that somebody could have compromised or done a cross- site scripting ejection or other infrastructure compromise. You have no

17. 8:42

    guarantee or trust as a user. So a similar problem has been like recognized by the freedom of the press foundation for their secure drop software which is basically a platform for whistleblowers to upload documents to news organizations. And as a result of this problem, they've developed this add-on called Webcat. And this is a Firefox add-on that's available now. It's very early preliminary release that um they developed to basically certify all of the contents of a web app and have

18. 9:14

    verify the hashes of all the the JavaScript and verify that the signatures are correct and are um fresh in a blockchain to give the user an assurance that they're talking to secure drop instance or any other web a web app such as a cryptocurrency wallet or private inference provider that this JavaScript is authentic. hasn't been tampered with is signed by the actual web developer and then because it's an add-on certifying this you have this

19. 9:44

    guarantee that JavaScript is authentic and then the JavaScript there has certified the private inference provider um which then um you know is providing you with with this this encrypted inference there's another weakness there where the LM provider APIs themselves near has developed this N E2 chat encryption AP API to give you that encrypted inference connection for your actual JSON blobs. I that I feel like that needs to see more widely wide deployment. It's not by default used in

20. 10:16

    any of the agent frameworks. For example, the research agent frameworks like to see that you know uh deployed more and and actually used by these confidential research agents. Um and then finally on the network side where the provider if the provider is like renting their own services and you have this like distributed inference provider scenario they may be trying to do their own value extraction by monitoring the

21. 10:41

    traffic that the research agent is generating. What websites is it visiting? What is this what is the user of this agent interested in? Even if it's just the SSL domains that may be enough to extract value. So in those scenarios you could have tour inside that confidential VM um that is published open source and attested by the attestation framework performing that network activity and protecting

22. 11:05

    that traffic from the observation by that data center operator or ad hoc infrastructure operator. I feel like that is like the the the end that gives you the end to end chain of preventing that SSL added and removed here vantage point at at various points of this the weaknesses in this uh in this certification of that that private inference.

23. 11:30

    Yeah, we need attestation. We need chains of attestation. We need chains of agency. When you see an output, you should know all of the agents that were involved. Um and some measure of agent loyalty. What is the agent's end goal? Especially if the agents can be part of a swarm, enough attestation of individual agents and clustering of agents of the same type to get a sense of whether there's some swarming behavior in your system.

24. 11:56

    Once you have attestation, you can have agent cards, right? You can you can associate a bunch of standard metadata that agents attest to themselves that they represent as describing their operation. And then we need third party eval spot checking the data about an agent in their agent cards. Um we need dispute resolution. Is anyone here working on dispute resolution for near? We need dispute

25. 12:25

    resolution between agents. You want some kind of onchain resolution of disputes to the extent that that's easy and feasible. Um and then we need insurance. When people want insurance, there needs to be a mechanism for a given transaction to have associated insurance. It can be no fault insurance just associated with the possibility that things go wrong. Especially as some of these um interactions scale and

26. 12:52

    agentic workflows can spawn other agentic workflows, you need some bounding of how bad a failure mode of your system can be. uh on the non-standard side, on the technical side, you need safety valves, but on the standard side, you need some some model for ensuring the um a potentially unbounded system interaction.

27. 13:17

    All right. So we've gone into safety security when uh I'm sure a number of you folks have launched blockchain protocols in the past and when we launch one we undergo security and safety audits often if it's a DeFi protocol especially we undergo economic audits.

28. 13:37

    So let's say security and safety is taken care of. Um this is kind of a real world thing happening right now. There's a fellow named Sigil who launched something called automatons which are essentially uh self-propagating agents, autonomous agents on decentralized networks like Ethereum um that are designed to extract value and uh capture it for themselves and selfreplicate.

29. 14:07

    Um they're doing so by things like launching meme coins and then pumping them on Twitter. Not surprising. They probably saw some humans do that. Um, please don't do that agents if you're if you're listening to us. Um, it's not a good way to make friends with us. Um, so let's say uh everything is done well on the security side, on the evaluation side, yet uh there are in my mind threat

30. 14:31

    exposures on the economic side that can lead to massive value capture by autonomous agents. And uh Vitalik uh said recently um like blockchains were building these to free humans, not AIS, um to free ourselves. And so um if these decentralized blockchain infrastructures are being used uh for other things that are not necessarily forwarding human liberation, um how do do you guys think

31. 15:06

    that we could use also safeguards at the economic layer? And how would you think about this? Um how do you feel about it? Uh do you have opinions? I suppose yeah again I'm going to go back to kind of the the the tour um and cipher punk roots here. One of the problems that we have have dealt with for a long time on tour is the IP reputation of exit relays and I have been looking for both funding and

32. 15:35

    solutions for uh improving this problem and reducing the rate of humans being banned from while just using tour from various websites. looking at things like anonymous credentials and uh various rate limiting tokens to maybe provide a a protected set of of high IP reputation exits that can be only be used with these tokens. One of the economic mechanisms I've toyed with is maybe like

33. 15:59

    a proof of stake where these anonymous credentials can be problematic as Brave pointed out in one of their uh recent blog posts when an adversary can try and targeted like um accuse uh particular users of misbehavior to try and narrow in on who they are uh by revoking their access. So um one one or or you know dean the flavors of anonymous credentials that can be deanonymized in the case of misbehavior. So those have kind of been ruled out in my mind.

34. 16:31

    But one of the things that strikes me as like a either a proof of stake or a proof of work backed token where the penalty for misbehavior is maybe losing your your stake in a smart contract or having to re repeat a proof of work or maybe even a a useful human uh piece of uh of effort like um uh an aspect of an evaluation that you have have have proven. As we discussed earlier with Sam, um these sorts of things seem like

35. 17:03

    kind of the shape of a solution. I feel like there's still a lot to be done in this area and I'm definitely interested in working with anybody who's kind of who's interested in exploring these approaches of of like various kinds of anonymous credentials and reputation systems to help both these kinds of problems. So, I guess in interest of Roo's Basilisk, you know, want to help free the humans and the well- behaved agents to uh to be able to to transact freely.

36. 17:30

    Yeah, I would start by saying I don't like the us versus them narratives. Like we are living in a universe that it has all kinds of automation. Agents are a new kind of automation uh and a new kind of autonomy in the world. Like we are also autonomous agents. So, we're all in this together. And the question is really how do we keep systems that how do we keep our systems operating with coherence that doesn't fracture into subp parts that are in opposition or end

37. 18:02

    up like spiraling into some dark attractor that then defeats the initial purpose of the system whatever it is. And to that end, a lot of what we've been talking about to date, like historical models of safety and security are often onetoone transactions. How do you make sure a transaction is legitimate or safe? But now we're talking more about systems and ecosystems. Agents are less an individual. Like using the language of agent is a little confusing because

38. 18:33

    really they're forkable processes in a family of processes that all can have sub agents. And the kind of autonomy we're talking about has very easy inexpensive access to uh versions of extending autonomy into into like other sectors of their operation space. So we need ways to um understand the scope of the operation space maybe bound that have uh transparent ways to perceive the control loops that exist. All systems have control loops.

39. 19:13

    Nothing can infinitely extract from other parts of the system. Then it just becomes the system and other things extract from it. And so we should take advantage of the fact that we have all of this localized compute inside an extended system and build ways of tracking what's happening and um parameterizing or influencing features of a a collaborative environment or a transactional environment that maintain the the parameters we want. We'll often talk abstractly about safety or security. Just define what that means.

40. 19:51

    Define what unacceptable failure modes are and identify control loops that allow you to avoid those failure modes or allow you to fall back like have a fail safe if things aren't working out. There are a lot of great examples in biological systems where individual cells in the body all usually contain some self-destruct mechanism that was built in billions of years ago early on in the evolution of cells to make sure that if you get a bunch of signals from outside that things aren't working out.

41. 20:18

    You have a way to say all right we'll let the next generation take care of this. Obviously there's some problem that I can't diagnose entirely within myself. So we should design for those sorts of facilities and then um some of the evals and benchmarks we should have are how like do you have these mechanisms for taking in feedback from the environment about whether you are screwing up the environment and then

42. 20:43

    that would make all all parties in the environment feel a little bit safer that it's not going to oscillate out of control. Yeah. Fascinating. All right, we got uh three minutes, so room for one more question. Uh this one's pretty open. We're here at Nearcon in a room full of decentralized AI builders. Um what would each of you say uh to this community that like what what would you have uh folks focus on that's one thing that

43. 21:15

    that isn't currently being uh developed or invested in or given sufficient attention to? Um again I feel like from my point of view obviously the tour project the key differentiator that that we can provide and that near private inference can provide is this private agent where you know what you're getting the atestation the the the the proof of what it is and the proof of its capabilities. But in order for that to be true we need to solve the SSL added and removed here problem. We need uh something like webcat to authenticate the JavaScript.

44. 21:49

    We need that true uh end encryption to a load balancer and then connection through the load balancer to the actual machine providing the inference and we need all of this to be supported by uh the agent frameworks in use so that there's no gaps for an adversary to either extract value or maliciously attack to observe the traffic um uh of of what's going on.

45. 22:15

    Yeah, that's very needed. I would add near has a marketplace. Uh build mechanisms in the marketplace for third party evaluation for sandboxing as a service for people who are filling in some of these gaps in observability. Um that that is also a marketplace. And then where there is like where oper making your systems work well means you have to build out cataloges of behaviors and patterns to look for or tools to

46. 22:48

    identify them. Make those public. Build public cataloges of behaviors, public cataloges of of classifiers. I was having a chat with some of the near team about how internally there were patterns of behavior in the near ecosystem that people had developed which helped them understand what was going to happen if there was some anomaly. Um but it wasn't public. It was just operationally it was operationally useful. We should be

47. 23:12

    sharing these kinds of things so that together we can improve these glosseries of behavior these glosseries of um what makes for good features of a stable system. We're going to have all kinds of systems. Let's just name the like the public references that all systems can ground their structure to.

48. 23:37

    So when you're trying to evaluate how a new system works, you're entering into a new agreement. You can say you can just double check what's our environment. Has the environment changed? And if so, which of these patterns hold? And then again, maybe there's a marketplace like one of the audits could be um context audits. you're about to engage in what you think is this kind of transaction, but we'll double check the context to make sure that you aren't actually embedded in some larger system that

49. 24:02

    changes the boundary conditions of how it works. All right. Well, we uh took the journey from safety, security, economics, and uh foresight into the future. So, wanted to thank uh all of you for for being here and taking your time to be with us and for Nearcon for putting this amazing event on. Yeah. Thank you.