{"id":"uihp9V1SPfs","title":"I Built a Benchmark Harness for NEAR AI Cloud Models","description":"I built a small open-source harness for benchmarking models on NEAR AI Cloud. You give it a prompt set, pick one or more models, and it scores every answer against what you expected and labels refusals as complied, partial or refused.\n\nIt tracks the numbers that matter when you pick a model:\nTime to first token (TTFT)\nTokens per second\np50 / p90 latency\nEstimated cost in USD, from live pricing\n\nRun two models side by side and it flags every prompt where they disagree. The bundled over-refusal set uses benign requests with alarming wording, like killing child processes, so you can see exactly where an abliterated model splits from its cautious base.\n\nIt runs locally on Python 3.9+ with zero dependencies. Use the browser UI, or run it headless from the terminal. A failed check exits non-zero, so it doubles as a cron or CI smoke test.\n\nRepo (MIT): https://github.com/joe-rlo/near-ai-harness\nNEAR AI Cloud: https://cloud.near.ai/","channel_name":"NEAR DevHub","published_at":"2026-11-03T00:00:00.000Z","thumbnail_url":"https://img.youtube.com/vi/uihp9V1SPfs/hqdefault.jpg","transcript_text":"Hey folks, let me show you something I've been working on around uh the idea of a simplified harness to actually benchmark different models. Um not in what they're fully capable of. There's a separate benchmarking for all of that, but I'm not trying to recreate that. It's just a way to understand I'm going to use a model in production or I'm going to use a model for development um to understand both its capabilities as well as uh its infrastructure setup as well. And I'll show you what I mean. First the problem like this. Imagine you have a co-orker that changes every single week and you're not really sure if they're actually capable of doing the job each week, but you are going to keep switching out co-workers each week. Well, that would lead to a lot of problems at work and probably really poor productivity. Yet, we do the same thing with AI models. Right now, we're switching models and we're making assumptions that things are maybe uh worse or slower or different and whatever else. We have no real sense of that. I use AI in some of my products as a tool to uh enhance certain features and things like that. And I want to make sure that if I'm going to switch a model or I'm going to continue using a model that it is serving what I expect it to serve uh continuously, but also the specs around like performance and how fast it's running and all the different things and total cost is really important to me as well. So, I made this little simple harness to kind of showcase what it looks like. There's no fancy stack here. Uh it's really simple and the idea here is that it's one prompt set and the prompts you can uh modify yourself. You don't have to do what I'm doing, but it's just one way to kind of showcase it. And I look at a couple different things here. One, uh time to first token in milliseconds, the decoded speed per response, the latency spread across the whole entire run, and then the estimated cost of how much it cost to run that particular prompt. This is open source, and so you can obviously fork it and do whatever else you want with it. Um it's really simple to run. Uh it just is a basic Python server. It's running off the near AI API in order to do this. Uh and you just go from either the terminal or the UI. It's really straightforward. And I'll show you the UI right now. Here's the UI. Uh and you can see here I can choose the different models that are provided here by uh near. And so I don't have to guess anything that I can compare it against other models again in the same list that you saw there before. I can test if they've been obliterated and how that actually performs. You can see all the different prompts here. I can check off um and again these are just loaded in from a JSON file. You can modify these and put whatever you want in there. And then I can run my different comparisons from there. I can change a couple different aspects here in the testing. You might want to limit tokens from the testing purpose because you don't want things just to run things. I compared here uh the Quen 3.8 8 27 billion parameter versions of both obliterated versus not and just look at you know again time the first token the average latency the total cost things like that um and then depending on which ones I selected here like just basic smoke test some structure JSON test finding a bug things like that uh checking again uh hallucinations you know these are all different prompts that I kind of have put together here to see how it would behave and I can see just each individual uh prompt of how it did. So, for example, like these two um both failed and I can see that how much it cost to run it. You can see it's a little bit more expensive here. Um it took a but it was shorter, it was faster. So we look into kind kind of what was going on. And I can see by that by going into the view and I can then go into the reasoning and see all the reasoning that happened as well and investigate oh why did this happen? What's going on? And sometimes these tests might fail simply just because I need to allow more tokens or whatever else. But it gives you a really simple structured point to actually jump into like what is happening with these models and understand again what the different costs are and set up and go from there. Uh this comes preset with all these different prompts to go through. Some are around uh you know uncensored models, some of them are based around just basic behavior of models. Some of them are about uh agent behavior things like that. And again you can set your own as well. If this is of interest to you, uh you can go download it from my GitHub today and you can fork it, do whatever else you want with it. Uh and uh just try it out. Uh you do need a near AI API key in order to run this. So you'll have to supply your own, but you can see as for my examples, like the costs are very very low to just do these different samples and benchmarking. So you can try it out pretty easily. Thanks for watching and make sure to subscribe to keep up to date with more projects I'm trying here on Near AI.","transcript_synced_at":"2026-11-04T03:46:05.505Z","segments":[{"start_seconds":0,"text":"Hey folks, let me show you something I've been working on around uh the idea of a simplified harness to actually","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=0s"},{"start_seconds":7,"text":"benchmark different models. Um not in what they're fully capable of. There's a separate benchmarking for all of that, but I'm not trying to recreate that.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=7s"},{"start_seconds":15,"text":"It's just a way to understand I'm going to use a model in production or I'm going to use a model for development um to understand both its capabilities as","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=15s"},{"start_seconds":25,"text":"well as uh its infrastructure setup as well. And I'll show you what I mean.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=25s"},{"start_seconds":29,"text":"First the problem like this. Imagine you have a co-orker that changes every single week and you're not really sure if they're actually capable of doing the","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=29s"},{"start_seconds":37,"text":"job each week, but you are going to keep switching out co-workers each week.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=37s"},{"start_seconds":40,"text":"Well, that would lead to a lot of problems at work and probably really poor productivity. Yet, we do the same thing with AI models. Right now, we're switching models and we're making","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=40s"},{"start_seconds":48,"text":"assumptions that things are maybe uh worse or slower or different and whatever else. We have no real sense of that. I use AI in some of my products as","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=48s"},{"start_seconds":55,"text":"a tool to uh enhance certain features and things like that. And I want to make sure that if I'm going to switch a model or I'm going to continue using a model","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=55s"},{"start_seconds":63,"text":"that it is serving what I expect it to serve uh continuously, but also the specs around like performance and how","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=63s"},{"start_seconds":71,"text":"fast it's running and all the different things and total cost is really important to me as well. So, I made this little simple harness to kind of showcase what it looks like. There's no","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=71s"},{"start_seconds":79,"text":"fancy stack here. Uh it's really simple and the idea here is that it's one prompt set and the prompts you can uh modify yourself. You don't have to do","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=79s"},{"start_seconds":87,"text":"what I'm doing, but it's just one way to kind of showcase it. And I look at a couple different things here. One, uh time to first token in milliseconds, the","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=87s"},{"start_seconds":95,"text":"decoded speed per response, the latency spread across the whole entire run, and then the estimated cost of how much it cost to run that particular prompt. This","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=95s"},{"start_seconds":104,"text":"is open source, and so you can obviously fork it and do whatever else you want with it. Um it's really simple to run. Uh it just is a basic Python server.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=104s"},{"start_seconds":112,"text":"It's running off the near AI API in order to do this. Uh and you just go from either the terminal or the UI. It's really straightforward. And I'll show","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=112s"},{"start_seconds":120,"text":"you the UI right now. Here's the UI. Uh and you can see here I can choose the different models that are provided here by uh near. And so I don't have to guess","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=120s"},{"start_seconds":129,"text":"anything that I can compare it against other models again in the same list that you saw there before. I can test if they've been obliterated and how that","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=129s"},{"start_seconds":137,"text":"actually performs. You can see all the different prompts here. I can check off um and again these are just loaded in from a JSON file. You can modify these","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=137s"},{"start_seconds":146,"text":"and put whatever you want in there. And then I can run my different comparisons from there. I can change a couple different aspects here in the testing.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=146s"},{"start_seconds":154,"text":"You might want to limit tokens from the testing purpose because you don't want things just to run things. I compared here uh the Quen 3.8 8 27 billion","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=154s"},{"start_seconds":165,"text":"parameter versions of both obliterated versus not and just look at you know again time the first token the average","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=165s"},{"start_seconds":173,"text":"latency the total cost things like that um and then depending on which ones I selected here like just basic smoke test","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=173s"},{"start_seconds":180,"text":"some structure JSON test finding a bug things like that uh checking again uh hallucinations you know these are all","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=180s"},{"start_seconds":189,"text":"different prompts that I kind of have put together here to see how it would behave and I can see just each individual uh prompt of how it did. So, for","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=189s"},{"start_seconds":197,"text":"example, like these two um both failed and I can see that how much it cost to run it. You can see it's a little bit","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=197s"},{"start_seconds":205,"text":"more expensive here. Um it took a but it was shorter, it was faster. So we look into kind kind of what was going on. And","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=205s"},{"start_seconds":213,"text":"I can see by that by going into the view and I can then go into the reasoning and see all the reasoning that happened as","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=213s"},{"start_seconds":221,"text":"well and investigate oh why did this happen? What's going on? And sometimes these tests might fail simply just","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=221s"},{"start_seconds":228,"text":"because I need to allow more tokens or whatever else. But it gives you a really simple structured point to actually jump into like what is happening with these","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=228s"},{"start_seconds":237,"text":"models and understand again what the different costs are and set up and go from there. Uh this comes preset with all these different prompts to go","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=237s"},{"start_seconds":246,"text":"through. Some are around uh you know uncensored models, some of them are based around just basic behavior of models. Some of them are about uh agent","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=246s"},{"start_seconds":254,"text":"behavior things like that. And again you can set your own as well. If this is of interest to you, uh you can go download it from my GitHub today and you can fork it, do whatever else you want with it.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=254s"},{"start_seconds":264,"text":"Uh and uh just try it out. Uh you do need a near AI API key in order to run this. So you'll have to supply your own,","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=264s"},{"start_seconds":272,"text":"but you can see as for my examples, like the costs are very very low to just do these different samples and benchmarking. So you can try it out pretty easily. Thanks for watching and","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=272s"},{"start_seconds":280,"text":"make sure to subscribe to keep up to date with more projects I'm trying here on Near AI.","url":"https://www.youtube.com/watch?v=uihp9V1SPfs&t=280s"}],"youtube_url":"https://youtube.com/watch?v=uihp9V1SPfs","url":"https://near-pedia.xyz/videos/uihp9V1SPfs"}