NEAR AI Office Hours #14 - FAR AI

Captions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.

1. 0:04

   blockchain flows near the one we're close to won't stop us now together we'll redefine the game with near we'll F the flame empowered Minds rise and so use your own a I we explore hold the future hand in hand in this Tech driven Wonder hello hello welcome to another office hours uh we have a very special guests from the far and they trying to be more near and so uh there's going to be a lot of punch on this I'm

2. 1:03

   sorry from the start but yeah I mean great to have you guys here I'll let you guys introduce yourself uh and we can dive in in a bunch of topics we have yeah thanks I have to see the pun hadn't really occurred to me um but you know perfect perfect pairing so I'm Adam I'm the founder and CEO of firei we were set up a couple of years ago research nonprofit focused on AIC safety both doing inhouse research and also education generally helping support the

3. 1:35

   field and B you want to yeah absolutely yeah I'm Ben uh great to um virtually meet everybody on this stream I suppose uh I'm a director here at farai uh focusing on supporting the broader research Community around AI safety particularly in the Bay Area but also around the world so great to be connected and we have obviously me and Alex here joining from near side uh to yeah let's dive in into this um well I

4. 2:05

   guess you know the the general topic to ask is how do you define AI safety and I guess to dive in a little bit more why you set up far maybe why is it far it should be near should be well no we wanted to you know leave that leave that for you guys um you know I think I think it's a great question like what what is AI safety I tend to conceptualize of it is it's fairly Broad that AI is likely to be one of the most transformative Technologies we have this century and so there's just a whole host

5. 2:36

   of things that need to happen to make sure that's really beneficial um for society and I'm relatively speaking and say an optimist I think there's a huge amount of benefits from AI but technology doesn't necessarily just go well by default it goes well because we make a concerted effort to mitigate the risks take advantage of the benefits and so I think that there's both some really key technical problems in AI safety

6. 2:59

   making sure that AI systems actually do what designers intend um so that's sort of alignment problem make sure that they do so reliably so that's robustness they don't fail in weird edge cases they're not exploitable by adversaries um which current AI systems you trivially are with with jailbreaks and then there's also a broader soot technical problem of okay you know aligned to whose values um so I think that there's you know this is

7. 3:23

   something where probably quite near to to your hearts right do we want to make sure that the power is decentralized is everyone has access to AI do we want to do some kind of preference aggregation so we can make sure that the AI systems are being built reflect a large number of different people's preferences and I think there's lots of different potential answers to this question people have barely explored it but right now uh you've got a handful of AI system

8. 3:46

   is being developed by a really small number of players um and that's that's you know problematic just from an usual concentration of power um aspect of things so I would say AI safety is all of those things um although you in order to actually make progress and of course you want to scope it down and focus on one of the sub problems on AI safety I'm not going to say I'm going to solve all of those absolutely not um and Ben do you want have anything

9. 4:10

   you want to add to the definition of AI safety you can also you know you can share a little bit about what far set up yeah I mean I think that's spoton I uh often conceptualize it as having like two main components like the first is something like is can we have the AI do what we want it to do um this often like encompasses traditional ideas of safety uh and then something that fall feels

10. 4:34

    more like alignment or ethics like does the AI reflect the values whatever that means of like the broader society and like have we built it in a way that is doing the kind of preference aggregation that's doing the type of like um uh broadly not concentrated power the kinds of goals that like I yeah exactly as Adam said I expect near cares about um like what is the design process for this

11. 4:58

    AI so like the first maybe focusing more on a lot of the technical aspects but the second looking at like broader society and its involvement in the AI design and development process um and yeah again echoing what Adam said like this is a broad problem especially when we include that second bucket when we think about Society r at large and far is a growing organization but we're not going to grow so big that we can tackle all of that uh it's one of the reasons

12. 5:23

    why we're really excited to do more field building more Outreach to be able to build strong research collaborations so that we have many shots on goal so to speak so um uh one quick aside is that um for for quite a while since since way before chat GPT I I was I was an AI Doomer and it's not very um it's it's not a a popular thing to be these days it's all about effective acceleration doomers are being uh you

13. 5:56

    know Fred upon but I'm still a very big Doomer I think I think we uh the risk of a pretty high and they kind of outweigh the in many in many ways the potential benefits and when I think about thei safety uh for me the bigger question has always been can we at all and if we can then what should we do uh defend against uh adversarial players who are not uh who intentionally not building saf right so like it's one thing to do like you know

14. 6:30

    what entropic does effectively hey let's build the most safe AI system ourselves right like you know let's let's CL be no un abusable Etc and let's do a lot of research to interpret what she's doing sorry he's doing and the and whatnot right and it's another thing to say hey there's this uh you know Russian hackers in a basement with the 10,000 h100s building AOG which is about to you know to launch North Korea produced missiles into what ever right so I I

15. 7:01

    wonder if this is something you you look into at all or if you have like any thoughts about how we can is there anything we should slash can slash must do to prevent Rogue players who intentionally build on safei or this is completely out of scope for you no I I think this is this is very much um in scope although it's not going to be something that we can solve single-handedly I think it's a in large part A governance problem and we can you know advise people on the governance side you know we're not a policy

16. 7:34

    organization but I think there's a lot of technical research that you can do that make a lot of these governance methods more tractable and also just have a less s of stock tradeoff because what I don't want to do is create an authoritarian state to stop the Rogue AI right like that's a problem in its in its own right as well but you know but good news is is right now ai models do require substantial compete resources to

17. 7:57

    train so you know 10,000 h 100s in a basement I mean it' have to be a pretty big basement with some really good air conditioning so you know actually um it might be feasible to at least monitor that and tell if some you know country is building a big data center so I do think that that kind of compute governance approach where you know anyone can do whatever they want their laptop or a rack of servers but you got a hundred million um set of compete

18. 8:24

    maybe you need to register and maybe you need to have someone go in and just you know check what you are running on that and then you build a bunch of tooling into goodd chips that would allow you to have some basic Telemetry so that isn't necessarily like you know what code you're runting which is saying how much interconnect bandwidth is there between these chips so you if you say I've got a billion dollars of gpus but it's just doing inference or fine tuning on an open source model you could attach to

19. 8:49

    that in a verifiable way without anyone needing to go in and look at any of your you know code or infrastructure um and then be able to distinguish between that versus a huge new training run which might require more scrutiny so I think there are some technical interventions you can do there which aren't AI interventions these are like cryptography security computer architecture but like very tractable we already have stuff like secure enclaves or confidential Computing in some of the

20. 9:14

    latest gpus like h100s but in theory could support a lot of these features it just need to be some software buil on top of it and then also making it actually robust to adversaries that have Hardware access and are trying to tamper with this but that's in theory um you know tractable or at least you could detect if that tampering has has occurred with some kind of inperson audit so I'm sort of an optimist in

21. 9:37

    principle here but I think it's going to require quite a lot of work to actually build that out especially if we're talking about an international regime now other thing I'm worried about in terms of this is not actually necessarily Rogue actors or Rogue States training in AI system because that's really expensive why do you that when you can just steal it right um like I think this sort of model weight theft is

22. 9:59

    underrated um and I I was quite sort of entertained but but zuck's most recent letter for for llama free um was basically saying well we don't think there's any downsides to open sourcing models because if anyone wants to steal Open the Eyes model they just do it anyway um which I thought was sort of you know a bit of a strange argument it's kind of like saying we're not going to even bother trying to protect your

23. 10:22

    emails because people are just going to steal it anywhere we'll just put them all publicly right it's like you know security is hard but it is a solvable problem um but but I think you know there's there's some truth to this point but right now I think if a nation state wanted to steal an AI model probably could um and so there's a lot of work that needs to be done to to show that up as well but it is a solvable problem

24. 10:44

    like you know big tech companies like Google have faced off against nation state adversaries in the past and being able to to stay secure so it just requires treating that with the same degree of sensitivity we would treat customer data I'll take the opposite uh kind of on all of this because I think you know I'm definitely anti- dumer ISM uh although I I do think we do need to do a lot of things to make

25. 11:16

    sure uh we have systems in place but my point here is any any of this like you know a bunch of hackers aing a rogue model like that's not the problem the problem is how when they actually going to use it and they can use you know you can use right now already chat GPT to hack into software like you like the even with current all the protections it still actually is pretty good at finding Rel abilities in the code um and obviously you know you can use open source models

26. 11:50

    so like that to me is not a problem at all that I feel is we probably like it it kind of distracts us from a real problem which is like our systems are not designed and by systems I don't mean AI systems I mean our society systems not designed for the world where AIS are available they haven't been designed even for social media right like we see a pretty big cracks in society due to kind of social media creating this you

27. 12:20

    know very per perturbed view right and you know people complaining about Google like results being biased Etc right like all of that is just examples of kind of our algorithms that run you know everybody like everybody's social feed and you know search results are not designed for you know more and more content being pumped out and like as AI is proliferating like that is just gonna

28. 12:46

    um accelerate but you know the misinformation by rers in the basement uh existed before like you didn't need AI you could just hire a thousand people writing a bunch of stuff right um so my Point here is like to me it's not about like trying to protect you know even h100s from running in some of this places Etc and I'm actually a lot more with Zach that yeah like if people really want to get the weights like they'll get the weights it's not um you

29. 13:16

    know hiring a private Army to ascend onto open air building please nobody do that um but uh but instead it's it's it's really about about how do we build systems like you know the government systems in us are all currently working still working a lot on like paperwork and you know like physical signatures there are still checks with physical signatures right like if we have cryptography for you know 50 years plus and we still have physical signature checks that people

30. 13:50

    you know validating if this is a real thing so to authenticate people yeah the number that doesn't rotate that you tell everyone you know from your bank to your employer um as as an as a private your private key right like all of that is like like there's a systematic problems in I call Society operating system and I feel when people are talking about you know AI safety they should be talking more about this

31. 14:21

    versus trying to kind of contain thei research which again leads to I feel very limiting like policy as you said right of like hey you know if you have 10,000 GPU h100s now you need to report record on that and it's like what if I have you know 10,000 fbga that I designed specifically for Transformers do I report or not is that like fits into the or like if I'm designing new chip now I need to register is like at which stage when I tape out when I design it when I it came to my mind and

32. 14:54

    I have this idea of you know beautiful chip that will run Transformers at like THX speed or somebody comes up with a new model and like completely all of the parameters don't make any sense and all those uh you know floating number points maybe don't make sense because somebody runs it on BI on biochemistry uh for example substrate so to me that like it feels like we're zooming down on onto a

33. 15:17

    specific set of Technology choices that right now showed like an interesting result instead of like addressing the fundamental problem that our society kind of runs on I mean honestly like pre- internet a lot of it is pre- internet Tech some of it is internet Tech and we haven't really transitioned to what I I call it post AGI you know Society operating system like when we have AIS and AIS are everywhere and

34. 15:42

    everybody can use them for good things and bad things we need to have a system that can handle that yeah I mean I feel like I end up just agreeing with with quite a lot of that so so maybe I'm somewhere in between and that I think I see a lot of benefits from Ai and about something like you know 10% catastrophic risk this Century from AI which is enough to make me feel really serious about it and want

35. 16:07

    to try try and work to make that number get lower but also you know 90% of cases I'm like this is going to be this is going to be awesome and we shouldn't so strangle this awesome technology but you know the reason my I'm as low as 10% is because I think people are going to take these issues seriously and both try and Sh up the rest of society and try and develop AI responsibly I think if we just had people raise their head with

36. 16:31

    just to build cool technology with no thinking thought of a bigger picture then then my you know my catastrophic risk number would be a lot higher um but yeah I think sort of our plan a where possible should be to sort of make Society more resilient including with AI so right now computer security is kind of a mess that you can get zero days for iOS Android for about a million dollars

37. 16:53

    and the reason we don't see people just exploiting this all the time is more of economic so that that you know you're burning a zero day for a million dollars it's going to get patched how do you actually take make money from this right but if you were you know more of a nation state adversary and you just wanted to go after a particular Target you can do this right and that does happen so it's very Patchwork and I think AI could really destabilize that

38. 17:18

    and shift for offense defense balance but you could also sh it up because yeah you know you can use gbd4 to detect vulnerabilities and do automatic attacks that was a interesting paper by D Kang's lab but you can also do that in your PLL request right you've got an automatic reviewer to find vulnerabilities and just improve improve a security we might even be able to start actually generating formally verified code

39. 17:42

    because right now it's like that's possible people have done so 10,000 plus line micronel that are formally verified it's just really slow and expensive you make programming so much cheaper with AI we might actually be able to start doing this for more significant amounts of software so I think there's you know how how do you start using this tooling to make Society more resilient is a really key question and my concern is just that we're not going to be able to move fast

40. 18:04

    enough on that and I think bcks is a good example like I was recently trying to get um get rent from one of my housemates and like the fastest way for him to transfer it was to write a check in front of me and I scan it with my mobile phone because like there's no like Mobile payment system and it it's just crazy right I mean and yeah like we have we have blockchain but like yeah you need to use BL dat right um so

41. 18:27

    Society moves slowly um and I think we should be trying to make it move as quickly as possible but if we have an option of saying let's slow down certain proliferations of dangerous technology until we can put in place ways of making Society more resilient that would be great um but but it you know it's it's hard but I wouldn't want a situation where we're gonna like permanently cut off part of a technology tree and I

42. 18:51

    don't think it's feasible for exactly reasons you're saying like yeah people are going to come up with some better AI architecture come with fpga but far from gpus as soon as you put in place regulation people try and find ways around it but I think it can be a way of buying time for other kinds of activities and you know right now My worry is that we're just accelerating faster and faster um at technology we've got very good at rapidly making progress on that we've not got better at like

43. 19:17

    rapidly adapting our governance institutions um and I I I don't see about improving unfortunately in the short ter but I'd love to be proven wrong and Ben you've probably got plenty of thoughts well I really agree with all this and I I feel like when I was hearing both like Alex and Ilia you speak it feels I don't know like maybe there's more agreement actually between y'all than there is like real disagreement like it's I always find like the split to be a bit of a false dichotomy like I suspect that a lot of

44. 19:47

    AI Safety Solutions are actually like full stack or scalable Solutions the kind of things that should work on like addressing some amount of near-term risks probably are also going to be important for the broader Doomer concerns and and I say that as somebody who has a fair bit higher probability on catastrophic risk from AI this Century than does Adam but I'm still like very excited about all the investments in

45. 20:11

    near-term mitigations against uh AI misuse um I I saw actually I guess a few weeks ago or I don't remember when yall had um Atlas Computing on your office hour as well I think they're a great example of a group that's trying to apply uh these advances in AI to for verification and addressing the kind of uh lack of of robustness in our cyber physical infrastructure and I think there's a couple initiatives like this that seemed great regardless of where how much weight you put on um uh longer

46. 20:42

    term concerns uh from AI um I think just the other thing i' I'd like throw in there is I'm really yeah I like want to kind of plus one the uh speed issue for cultural and societal responses to AI being one of the key issues like yeah if if we if governance is operating at something like uh you know two to five year timeline for passing policy or mitigations to new developments in AI I

47. 21:13

    think that's just so much slower than what we'll actually need that experiments in other forms of governments like ones that like are going to be quicker uh or like kind of empowering more actors to be able to do their own forms of like like like for instance whether it's company self-regulating whether it's experiments in like um uh like quicker response Cycles to AI developments that seems

48. 21:36

    incredibly important because otherwise we are going to enter into this like situation where the UDA Loop basically for capabilities is g to be spinning way faster than like society's response and that I think leads to both like near-term misuse problems and to catastrophic risk I I never know the format for these things so I wanted to actually throw out a question to y'all if that's all right I'm kind of curious where like I was looking into the the near Ai and the

49. 22:07

    kind of like open source AI research agent I think is how i' term it that y'all are like particularly interested in working on I'm like kind of C giving your respective views on like where AI is going why like what's the what's the um what's the rationale what's the like the drive for building that so so there's a um I I I think one one important consideration here is that we we do

50. 22:38

    believe uh in uh like effectively on the if you like plot the whole uh spectrum of how accessible should the machine learning models be uh From perspective of having uh the safest uh sort of deployment of AI you can be on the on the on the end of the spectrum saying hey it should be a prerogative the very small number of entities that are you know heavily regulated and that's something that I think open AI is pushes very heavily for uh or you can be on the

51. 23:09

    on the end of the spectrum saying hey everybody should have access to the largest models to the weights uh and uh we are heavily on the on the for on the letter right so effectively there is no trust uh in the small set of entities I think historically small set of entities were always sort of heavily incentivized to optimize for themselves not for uh for the majority and as of today uh we sort

52. 23:40

    of in that regime in the sense that there are very large models which are somewhat competitive with the proprietary model right so like Lama 400 5B is a very competent model maybe like you know it is not as good on the benchmarks as GPT 4 and some of them but it is definitely usable in a very large set of cases for you would want to have a model right so you have uh like like if you're trying to build you know like

53. 24:03

    a long-term uh product and you don't want to depend on open AI API or if you want to have a product where you know that you know uh no like nobody can pull the rug underneath you or like ban you for violating some policies Etc right you can you can use that and the problem is that we do not have a uh it is not a future proof solution in the sense that as of today like you know a while ago there was at least two

54. 24:32

    different uh companies consistently publishing such models right there was mistel and now suddenly mistel for some of their newer models is changing the license right and there was stability and stability did not age well with the open source efforts right and like like we're slowly getting into the mode where the the whole open source uh sort of model story is fully dependent on Facebook continuously publishing state-of-the-art models right and you don't want it to be the case right so you want to be in a state where

55. 25:05

    if Facebook chooses not to do that any anymore uh there is someone who is uh ready to pick up the torch uh and the way I usually outline it is that you have certain number of requirements the requirement number one is that that entity needs to be sufficiently um funded to be able to you know to train the state of art model it needs to have future proof like effectively it needs to have incentives aligned in such a way that it will continues to do do so right so so so far you know like hypothetical M

56. 25:38

    instability they did not have incentives aligned long term right and so and thirdly it needs to have expertise and from that perspective near is somewhat well positioned to be there but but it doesn't make any sense for us to be training large language models today because you know there there there do exist very large language model today that are competitive and it would be a massive weight of resources so so we so

57. 26:03

    our focus is in in a somewhat different areas today right we invest much more heavily into like a agentic infrastructure private Vector stores Etc yeah maybe to add kind of from this perspective of you know the spectrum that there's few entities that are controlling all of the intellig toolings right and you know potentially they becoming more and more government like because they have more and more

58. 26:33

    capabilities and as well they they're pretty much becoming Monopoly on some new type of power which is you know how people perceive reality through this interactions with AI as you you know as we see this happening already was Google and Facebook and so we want to have an alternative which is I mean that's why call user owned it's not just about open source but actually it's about having a very

59. 27:04

    different approach to kind of your value function right the global value function is where individuals and communities like this models are focused on their success versus on kind of profit or you know control of a specific company and then and then requires a different approach right it requires also kind of as as Alex mentioned like how do we design a system that's less about one

60. 27:31

    company doing something or one organization but actually how to like coordinate a whole ecosystem of different partners so you know different agent different researchers to really work together to kind of create co-create that and from a safety perspective at least you know my hypothesis here is that if everybody has access to you know state-ofthe-art or near state-of-the-art AI tooling than like any individual group right

61. 28:03

    trying to use it kind of use it for malicious purposes right you have a lot more people you know kind of on the on the side that's trying to protect this and can have access to you know very similar amount of uh of intelligence tooling and compute to be able to to stop that versus if there's only like one or few companies that are able to do this and one of them becomes malicious for whatever ever reason be that you know internal reason or indeed you know a physical attack of a

62. 28:35

    malicious State like State and then they use this tooling that only available and like nobody else is able to use something of that order to protect kind of system that seems like a massive fail mode for the whole for the whole ecosystem and for the whole world and so kind of Distributing power have always been um you know a way to to really democratize something and allow everybody to kind of keep checks and balances and and as well as like that's potentially a way to your point about government moving

63. 29:09

    slowly if you have kind of AI tooling and you have some of this identity systems that you know blockchain provides you potentially can have governance as well sped up because now you know product wise you can actually embed governance directly into some of thetic Frameworks right there's an agent on your behalf actually interacting with other agents to decide and S the decisions yeah I'm interested in

64. 29:35

    experiments there and I and I'll just say like um I really agree with something I think you were pointing out which is like the way in which governance isn't just about like voting in the upcoming election it's like also very much how companies and ecosystems of people end up governing the decisions that influence the products that are going to influence everyone like I really like what anthropic has experimented with I think in partnering with the collective intelligence project on their like constitutional AI appro

65. 30:05

    like trying to aggregate more preferences to inform the underlying policy that the AI executes like I'd love to see more experiments like this happening in the world and seems like it's the kind of thing you're pointing out with near yeah just add I really like emphasis of user owned AI because I think there's been a bit of a dichotomy some between open weight and proprietary model but unlike most open- Source

66. 30:29

    software where you can go in you can Tinker you can Fork it you can change it if someone just gives me a you know 405 billion parameter you know set of Matrix weights it's like what do I do with this I mean I can find chune it but there's been a lot of recent research suggesting that actually even if you do do something like RL from Human feedback or DPO the pre-training process really restricts a set of preferences that are

67. 30:55

    expressable after that fine tuning so if doesn't have some representation of a value that's important to you you're going to really struggle to find tun it back in later so um it's not like this is it's fully user owned I mean it's more so of in a proprietary model but if meta wanted to restrict things or just unintentionally based on how they Curative a data set they still have a lot of influence here so I think being

68. 31:18

    able to at least understand those those limitations but also you know having more infrastructure where academic or open source collaborations can can actually build those models I I'm you know actually a pretty big proponent of at least right now um openweight AI models we use them all the time in our research I think there has to become a point where you say well you can't wash your hands of responsibility of what

69. 31:43

    people do with your models if you make them available whether it be behind an API or making it giving everyone access to the weights but you know you also by the same token should be getting some benefit from the the positive uses of model so if yeah 100 people use them for something great to defend against Bad actors and 1% of people abuse it if a net you know effects on society has been

70. 32:06

    positive we should still allow it I just want to kind of at least be open to there being potentially some restrictions we do want to put in place on on training and sort of irreversible proliferation of models and I think that you know the good news is there are things you can do to mitigate some of these risks while still having open weight models so I'm quite excited by have a model on learning literature for example where you're basically saying okay we can get rid of a particular set

71. 32:32

    of dangerous capabilities or knowledge that a model has well it's still being able to do everything else um you know B basically the same as original model so we could identify a particular subset of capabilities that maybe not most people don't need to have direct access to and then put more restrictions just on those that could be one way forward um but I certainly don't think that just having a few AI companies have exclusive access

72. 32:57

    to these models is a good path forward I mean if we think of something like um nuclear power where I think most people agree we shouldn't just say okay everyone can build a nuclear power plant in their backyard the reason people aren't up in arms about the you know the the power of these utility companies exert is they are a utility company they sell to everyone equally it's a commodity you can't say we're not going to provide you with power because we

73. 33:20

    don't like your political views or something like that but right now there's no similar protection for AI right like you know these companies can just write their terms of service they can choose what views these um AI models Express there maybe there's a need for something like net neutrality for AI if we're going to have this kind of kind of Monopoly but I definitely don't know what the right um combination here is I think open weight models have a part to

74. 33:45

    play in it but I do expect both for sort of the the governance aspect but also just economic fundamentals of proprietary models are probably going to be at least a generation ahead of open weight models for the foreseeable future you don't believe in like a a model competitive with gp5 but open source dropping anytime soon um I I mean things move pretty quickly like I wouldn't bet against it two or three years from now I just think the proprietory models are going to have a one or two year lead and then you know

75. 34:19

    other interesting thing right is I mean lmer 3.1 45b we started to play around with it and it's pretty interesting from a research perspective but it's it's hard to work with right like you know you can't just put it on um you know even a fairly beefy server you've got to start shoting it across multiple gpus multiple nodes so yeah I mean most people are going to be using these models behind an API it might be an API served by a third party you might be

76. 34:44

    able to have free fine tuning access for real enthusiasts can work with it but I think for models getting bigger and bigger it's still going to restrict the number of people who use it like this isn't something you download for a IP and run it on your laptop right um so from that perspective I think we're seeing the line blurred between this kind of utility view of AI models um and um the sort of more proprietary um set

77. 35:08

    you know model and really it's just a question of how much competition is there in those models being served how are structures of terms of use do you have an ultimate recourse to to you know build inhouse infrastructure I think there is a meaningful difference but for most people I just expect them to be interacting with these models behind API whether open from many perspectives Lama 400 5B behind an API is strictly Superior to proprietary

78. 35:34

    model because like the biggest risks right if you build a company and you build it around gp4 and then for any reason you get locked out right like you know you happen to be in a geography that becomes sanction or you um violate some policy right that then uh you're in a very bad state right because if as you switch to different model I mean models are not dropping Replacements of each

79. 35:59

    other right but if you use an API like you use fireworks right and for some reason fireworks limits your access that's chill right you just go to together. a and it's exactly the same model right maybe with a different pricing or speed it doesn't matter but you know your product has consistent experience right and in the worst case uh like a fact it allows you to heavily delay the time you

80. 36:23

    will have you will have to figure out how to self it but you always have that option as well right so if all of them look you out right you happen to be like I mean we have we have already projects right like hyperbolic which are offering that kind of across different provid like decentralized providers right who themselves don't know where you're coming from and you're paying

81. 36:47

    crypto right so you kind of creating a Marketplace of providers that is you know similarly how it's done with crypto don't really care where it's coming from and just provide this service so to your point around like utility the ultimate utility is when all the providers are indeed just kind of commodity providers behind a shared API and you know they can go offline they can come in online and they also don't care where the requests are coming from and that's that is actually

82. 37:19

    what you know blockchain is really good at to facilitating yeah and and so but that only can work with obviously open weight models to to facilitate yes so I think you can commoditize the inference um the the challenge is that in many cases these models do actually need to be updated I mean they've got a fixed training window so depending on your application you may need to be continually fine tuning them

83. 37:49

    now yeah you can do that on an open weight model yourself but it could be prohibitively expensive if you're trying to add a lot of additional information and like train them all the latest news so I think you know to your point earlier like yeah right now the open source ecosystem is pretty dependent on on Mata and they have a lot of influence there so I think if if you know really wanted to make sure that there was s of

84. 38:13

    you know business continuity here no single provider has lock in then you'd really want sort of a a set of competing um openweight model providers and I I don't know if I prefer to be you know if I put myself in the shoes of someone building a up on top of an AI model I don't know if I'd prefer it to be in a situation where I've got free proprietary providers whose models are not perfectly drop and replaceable but they're like pretty similarly competitive and I know if one of them

85. 38:37

    locks me out I can go on to the next or be in a situation where I've got a hundred you know competing inference providers but I know that my model is going to go out of date in a year and I don't know if there's going to be another model dropping anytime soon um so I think there's just sort of different risk profile depending on the the time Horizon you're looking for but I I definitely think there needs to be some degree of competition here and the less competition there is and the more

86. 39:02

    important it is there's some kind of Monopoly Protections in place like like that neutraly yeah that's an interesting point so effectively you're saying you might prefer to have a risk of being locked out of proprietary API uh but have the benefit of uh uh of having the state of the art performance going forward yeah I mean not if there was only a single proprietary provider right

87. 39:36

    but if you say well I can do use Claud I can use gp4 I can use um you know Gemini then okay if you're doing if your business is really at sort of the edge of these terms of service then maybe you know you are better off with open weight models but I don't see many people worrying like oh what if I get locked out of Amazon web services and then also Google Cloud platform like it just doesn't really happen I mean these are companies that want to take your money um I think you know people do worry about like what happens if I get

88. 40:07

    locked out of my personal Gmail account because in that case you're not for you know you're not for customer your product you're not paying for money um but I don't think these companies are going to have a very good business model of they arbitrarily shut in people's accounts um but but yeah I wouldn't want to be buing a company where I'm wholly dependent on a single company but if it's like you know a three companies that's already lot better if it's 20 companies it's actually pretty good um

89. 40:30

    so I I do think that we could you know potentially get into that ecosystem without too much trouble I then we're also seeing things like anthropics partnering with with AWS um and I think you know a lot of other Cloud providers interested in this to um actually provide inference access to their models that might also be another way around this where you say okay there are us to specialized R&D companies but they're

90. 40:53

    not necessarily the ones that are that are doing um doing the model inference yeah I just want to mention that there's definitely examples of you know different clouds shutting down people's accounts for no yeah like apparent reason but yeah yeah I mean I think this is indeed kind of a current state where people do prefer to use one of the cpis although I think actually more and more people are switching to uh to open source actually because

91. 41:26

    it's just cheap um like you you can customize a model in such a way to get like a cheaper good same result or or better result than for cheaper because you can like modify it in a way that you that you want which you cannot do you cannot just go like into you know 40 and like downgrade it to whatever 2. imagine is particularly important for folks in the near ecosystem who are doing agent experimentation that the like tinkerers the hobbyists all the developers were

92. 42:00

    like working on like new agents I imagine that being able to do that kind of fine tuning is particularly important it would be my I mean that that's a that's a big part and especially for any research that's outside of those three big Labs but I'm also talking about like a larger scale uh consumer or developer productivity apps which have a really big load the you know let's say you have a code completion you know like carser or one of those plugins you actually

93. 42:29

    don't want to use GPD 40 all the time it's extremely expensive and you actually don't need that uh what you need is like customized you know coding model that's you know probably like 7B or 12b size that's way faster and way cheaper and that that's kind of being I think a big use case as well just kind of go going down to specialized models and I think that that's where a lot of the open source re research is moving is like how do we

94. 43:01

    specialize models better I mean agent is part of that specialization like how do you how do you build set of software around the a around the model that you know potentially uses tools Etc uh that that specializes in specific set of skills versus being the generic I can do anything systems yeah I do feel pretty excited by that sort of specialized AI development and I think there's there's two

95. 43:25

    development Pathways you can take to trade make an AI system better you can actually say what problem am I trying to solve let me train it on that problem let me provide adequate scaffolding for it like you let me actually really we say we train AI systems let's actually train AI system on this task and other is to just say let's stack more layers right let's throw more gpus at it and right now a lot of the the AI companies

96. 43:48

    are primarily focused on the second development pathway that wasn't true five years ago but it really has been true in in recent years and you can sort of see why because a very easy way of turning Venture Capital dollars into a more capable product whereas it requires a lot of domain specific knowledge to start specializing your model to be good at coding good as a health care assistant but um I think at some point

97. 44:11

    it's going to become both you know economically more viable to build these special AED models because we're seeing AI compute um investment has you know vastly grown faster than Mo's law right like you know it's not that the hardware has gotten that much better as that people are now spending billions of dollars on on these training runs and at some you know you don't you can't go that much further before it starts

98. 44:35

    becoming an appreciable fraction of world GDP um and at that point it's going to become hard to scale much further so it's going to be sort of I think an economic imperative but I think it's also actually quite good from a safety standpoint and a you know user ownership standpoint if you do have these specialized models you got this model that does everything then those sort for risk surface area is huge right

99. 44:57

    it could do all sorts of misbehave in all sorts of ways how are we going to test that you've got this like you know one company or handful of companies lot power over it where you say okay this model does this one task like I maybe I don't even care about the political views of coding assistant developers right like it doesn't come up very often and if I do want my summarization of what's going on in the news then I hear a lot about about the political leanings

100. 45:23

     but the good news is okay there's a left r one there's a a libertarian one as a Marxist one I can get these different perspectives so that kind of ecosystem is a lot healthier and you know to the governance question we kind of know how to govern um you know ecosystems of specialized actors because that's just like companies that's that's individuals that have being trained what we don't know how to do is govern sort of

101. 45:47

     millions of individuals that are much faster than us which is be what like a general purpose AI system could could look like um but sort of we're we're just like plants to it because it can act so much faster even if it's not any more capable than us and that would pose a lot more challenges but I think we can get many of the benefits of AI without going down that technology pathway at least in the short term strongly agree

102. 46:10

     and also I just really love the idea which hadn't thought of that maybe GitHub co-pilot has like radical anarco communist views but I just never get to see it because it's coding or it's uh mean just just put a comment in and ask it you know yeah yeah yeah slash slash please complete this political phrase it's all in variable names yeah yeah you start noticing it in the variable names as it start sneaking it in there like subconsciously your your world views you know change a little bit

103. 46:39

     at a time but over the course of a year I also go ahead go yeah no I was also interested in uh you mentioned briefly in passing that you you have like your own internal efforts but then you support a wider ecosystem of project I'm curious like what things are you supporting like in in which in which areas uh and if there are like examples of like uh cool projects you think that uh that are doing some um cool research on the safety side

104. 47:21

     that that you would want to uh yeah yeah absolutely so I mean just to zoom out a little bit the overall Vision as far as to incubate and accelerate new research directions in in AI safety because we don't think that the current bets that let's say the handful of major AI companies are pursuing a sufficient to guarantee safety even putting aside these sort of you know these these social and governance issues and interestingly I think most people walking at those

105. 47:51

     companies kind of agree at least if you ask them in private they're like yeah I mean we're just doing the bare minimum we don't know how to actually make these systems reliably aligned and then uh you know you see Academia pursuing a wider range of directions but it's really hard for most academic groups to really make a serious dent in these problems because it's so compute and Engineering intensive um so we're trying to you know

106. 48:15

     work in this Middle Ground of having a diverse portfolio of different bets operating on medium scale research lets you go from you know an idea to an actual viable prototype that can then either be us as a technology demonstration uh to get AI companies to scale this up and build it or go to governments and say look this is possible so you need to be like asking for better don't let people say this

107. 48:37

     isn't this isn't possible we did it on a you know a sh string budget relative to these companies and then also just have a better understanding of these issues so that we can um you know inform the research going on and the policy going on so to you know question of what are concrete projects that I'm excited by well we've been doing a lot of um events recently bringing together machine

108. 49:00

     learning researchers one um what I want to highlight is the international dialogues on AI safety because I think one of the the big issues we see with governance is like okay great maybe we have an excellent governance regime in the US um we don't but you know we maybe we could get there right you know maybe we get the AI companies um pretty on boards we've got you know a great government agency we have some actual legislation there's no legislation now

109. 49:24

     there's just an executive right I could kind of see is happening but what about the rest of the world what about you know hack is in 10,000 h100s in that huge basement data center um and so I think we do need International cooperation but you know the good news here is that I think there's an emerging scientific consensus and although it's going to be hard to get people to see eye to eye in terms of how to use AI systems at least we can have a consensus

110. 49:50

     of like no one wants to build a system that's going to blow themselves up right um and so we we've seen operation in the past on things like nuclear safety that yeah we'd rather some other nations not have nuclear weapons but if they're going to have it would really rather that they not be insecure so that terrorists can use it um and so the US actually has a standing policy if a

111. 50:13

     nuclear armed Nation can ask the US for assistance in securing their nuclear weapons and the US will share its technology um in that area um so I think that we could meet hopefully get to a similar area where there's some even if political tensions rise we can get some areas of Safety Research carved out and protected and so we've been very excited to at these International dialogues have

112. 50:36

     some very senior um Chinese European Canadian UK US scientists and policy makers um you know all really put some pretty strongly worded statements out um calling for Global action um on on these issues such as red lines in um in AI development like being able to conduct automatic cyber attacks or um be able to assist in chemical biological nuclear weapon development that if cross would require action by by domestic Regulators

113. 51:10

     so you know to be clear you know these statements but don't have any legislative Force right but this kind of track to diplomacy lets us pay for way for um track one diplomacy like actual diplomats to start having these conversations realizing that there is at least some scientific um and um institutional support in their respective Nations for these kinds of topics so that's one just one project I'd really want to

114. 51:41

     highlight and then Ben I mean you also do a lot of work on um you know incubating the fields and maybe want to share some of what's going on at F Labs yeah I'll um I I was also thinking of the international Dialogue on AI safety which I think just one of the coolest examples of the like uh high impact events in field building that far has done in partnership with other groups uh

115. 52:03

     we also run um uh a a co-working space and research Hub here in the Bay Area where we bring in a number of groups to work from uh I think that this is really good for the kind of idea exchange the mutual support that often seems extremely important in research so we've got a number of groups working out of here um it's hard to pick your favorite it would be like you know picking your

116. 52:25

     favorite child so I'm not gonna name too many specific names but I do um maybe maybe I'll I'll say uh you know one group that's been part of like the far ecosystem and far Labs since uh since we started the far Labs um group about like a year and a half ago now Matts um Matts program.org I think is their address uh is does really great work bringing in new researchers into the field of AI safety so they run cohorts for stocks

117. 52:57

     for you know promising Master students that kind of type of people who are interested in like doing independent research or maybe joining a research lab in the future um they've run these kind of like semiannual cohorts uh and I don't know I think that that's incredibly promising for doing the kind of like General field building that's important and uh really plays an important part in our ecosystem because from there we can also start thinking

118. 53:20

     about more of the projects and organizations that might grow out of this and support uh uh new bets um a few of them that also work from Far Labs like tus is a great research group that's pursuing this like uh I think relatively novel Direction this new paradigm for uh interpretability style research um a few others like that come to mind like Simplex which is doing this for computational mechanics so yeah I'm

119. 53:43

     just really excited by what I see as the growth in like um researchers coming into the field of AI safety with their own paradigms and their own unique takes on how to solve some of the like fundamental problems in a safety because I suspect that like yeah a lot of those aren't going to work but some of them might and that kind of like uh new paradigms is exactly what the field

120. 54:06

     needs uh so yeah named a few there I feel like I could keep going on that but I I just say this kind of feels like maybe like popping up for a moment I was thinking about how like one of the things far is excited about is probably the same thing you're excited about like why you're doing like live streams like this in office hours like I was struck at some point by the way crypto and web 3 has these like really formal methods for governance but at the same time you

121. 54:31

     like join discords or slacks to like have the important discussions and share context I feel like this kind of like split is also like a big part of like AI Safety Research where like yeah you have a lot of papers and you have a lot of like formal experiments happening but much of the context sharing that's crucial to people like actually knowing what's going on and coming to good ideas and or in well happens through in-person conversations or on yeah slacks and

122. 55:02

     discords as well so I don't know I think far's trying to invest in the full stack of AI safety which includes making good Discord conversations happen I say Discord we actually use slack I just feel like that was actually you know I don't know if that was the right choice but uh we go back and forth on it yeah I mean both both of them are not great for Technical and kind of a long form conversation so yeah yeah what's that

123. 55:26

     XKCD we need now we need a new solution and now there's 14 competing standards for online conversations Google doc a lot of Google docs yeah that's actually probably the main social network is like being added like various Google docs for commenting I feel like yeah we I mean the Google Google wave was the beautiful solution for all this this on Google too wave and reader that they shut down and that would have we would have been in perfect Utopia exactly so maybe just to include

124. 55:56

     on on kind of a positive side so we have this blockchain community right listening to us there's a lot of indeed governance tooling and experiments that are uh that exist and so I'm actually curious if you guys have ideas on what you know interesting experiments that blockchain Community can facilitate in governance right to help with you know what you guys are working on and maybe

125. 56:21

     collaborate with some of the efforts uh that you have in the safety ecosystem I've got an idea which I think is going to be maybe a bad idea but it'll also give that way I'll set up Adam's idea to sound better um but all right so I was thinking about this because I saw that near was partnering with gauntlet for your like stake proposal um and I haven't followed Gauntlet too much

126. 56:45

     recently but like a year or two ago or like three years ago I was like pretty interested in them as an example of agent-based modeling in like an applied like useful way um that's no knock against the ABN Community I just feel like most of their stuff is done in like ecology research there aren't that many examples of being like used um as much in like real world like high Stak situations like governing near and I'm

127. 57:08

     like yeah I'm really excited about directions of experiments that look like this where I think that's like a prototype of what AI powered Tools For Thought will end up looking like like I how much money I I actually just don't know this but like a tremendous amount of like uh money is now going to be governed by a proposal that was shaped by like AI tools for thought and I'm like okay that seems like if it's happening in the crypto space it's a

128. 57:34

     good test bed for the kinds of like social mechanisms that maybe can get adopted by Society at large to make it more robust and if like crypto is this almost like experimental area for like the kinds of things that could happen in the rest of the world um that seems like a great role to play and like really important and highly leveraged so um I'm excited about more experiments of like

129. 58:00

     using AI to help AI for wisdom AI for epistemics like using that help make better decisions about how to govern communities especially ones that are going to have all these adversarial attacks that have all these people who are trying to like mess with it and not do the good thing and see if like it's robust to that and then maybe Society can adopt it yeah I think it's it's great Ben so I I'm you know you're too

130. 58:25

     humble with with your idea but yeah I mean definitely plus one that no government or even major corporation is going to adopt an idea just because there's nice white paper and open source implementation of it they want to see real world uses of it with actual self-governing communities that have gone well I mean you can kind of walk up a value chain from being like yeah you know some small Dao to okay now it's a

131. 58:49

     you know it's a small company now it's a major company now it's a small town okay now it's a you know maybe a city or a nation and we don't see much innovation in these kinds of voting systems or governance mechanisms I mean most of a worldall is still using first P or post voting even though that's sort of a one voting mechanism but everyone agrees is the worst one you know right like there controvers is what the right one but

132. 59:12

     it's definitely not this one but I think you know part of it is that we've not seen that many examples of other things working well so it's this gradual process um but in terms of more like research flavored um contributions that people could make I mean one thing I want to highlight is this idea of Cooperative AI so I really like the notion that you mentioned earlier user owned AI you have your AI system be your

133. 59:36

     principal um or so you have a principal it's the agent it's acting on your behalf it's negotiating of AI systems I think there's this question like well how do you make that negotiation go well because you could end up these AI system just like Waging War on each other like here's an adversarial attack right I mean AI system is like very insecure right now um and then you end up in an equilibria you know less good and the AI system could have cooperated on behalf

134. 1:00:02

     of their principles so it's a problem we' seen with you know human uh institutions it's a problem would ideally not recreate with with AI um and I think there's a a lot of to say okay can can you make AI systems to better cooperate with other AI systems can you make provable commitments here as well because that something you can actually say here's my source code so you can tell how I'm going to respond in this situation um so yeah that's I'm excited about yeah makes sense I I just want to

135. 1:00:31

     mention yeah on on the work with gauntlet one of the experiments we're going to be running a little bit later on is uh having AI delegate and so can AI delegate make indeed interest interesting or useful decisions and pretty much use the economy as a way to like see if if people willing to delegate pretty much stake on this AI delegate as as their representative so think of it as AI

136. 1:01:03

     representative vote for if if they're making good decisions and what one of the interesting experiment there is like do you want to have lots of different Representatives like specialized who are like together coming to a consensus or do you want one that is you know becoming like AI president and that's I feel I don't I don't have an answer and it would be an interesting experiment to

137. 1:01:27

     see how how it will end up but yeah we're out of time so thanks a lot for uh coming on and and kind of walking us through what you guys are working on uh there is a huge kind of uh need for continue figuring out how we going to approach the AI across the board and building as you as you mentioned kind of the space out attracting researchers and um also educating people so thanks a lot for the work you're doing and uh I'm sure we'll we'll continue collaborating

138. 1:01:59

     in different ways uh and thanks everyone for watching hope uh this was educational and see you next time yeah