{"id":"Vw_YfujRFuE","title":"NEAR AI Paperclub March 10th","description":"NEAR is the Blockchain for AI\nWebsite: https://near.org/\nNEAR AI: https://near.ai/\nIntents: https://near.org/intents\nChain Abstraction: https://www.proximity.dev/\nSharding: https://nearone.org/\n\nFor the devs\nGitHub: https://github.com/near \nDev Docs: https://docs.near.org/\nDevhub: https://devhub.near.page/ @NEARDevHub\n\nConnect with socials\nTwitter:   / nearprotocol  \nBlog: https://near.org/blog/\nMedium:   / nearprotocol  \n#NEAR #nearprotocol #NEARAI","channel_name":"NEAR Protocol","published_at":"2025-03-12T00:00:00.000Z","thumbnail_url":"https://img.youtube.com/vi/Vw_YfujRFuE/hqdefault.jpg","transcript_text":"yeah let's dive into the paper um if you were here a month ago for R1 uh this this presentation will be quite different because R1 had a lot of moth and a lot of uh novel ideas and S1 is quite different S1 has effectively two very simple ideas behind it but those simple ideas Pro result in very good results so there's going to be way less of U um hardcore math and more of a interesting adhoc engineering so two ideas behind the paper are and they largely I guess they they both contribute towards the the end goal but they're large largely independent one is a particular data set which only consists of 1,000 examples which is extremely few that when you train on it um makes the model significantly smarter and these 1,000 examples are distilled from a larger data set to 59,000 examples which is more of the size of a what people usually use for fine tuning and those 1,000 examples give you almost the whole benefit of of the entire data set right so if you fine tune on that th000 and on all 59,000 you get the model over very similar capabilities and the second idea is called budget forcing and it's a very nice very simple trick which may uh which you apply at at test time at inference time which makes model um smarter but also it makes model work like depending on the question at hand you can you can force her to to work cheap more cheaper or more or spend more compute uh to get to to the result faster or you know to get to a better result okay let's talk about the data set so the data set the goal uh that they had was they they started with the quen quen 32b and they wanted to make quity to be smart they wanted it to uh to do reasoning uh sort of like what one does uh provide the provide the result and they wanted it to be as smart as possible and uh their first idea was that uh the data set on which they're going to train needs to be sufficiently diverse if the data set is homogeneous then the model will over optimize for a particular um for a particular uh area of expertise of that homogeneous data set and so they outline all the all the sources that went into the into the data set and you will see that some of them most of them are mathematics but even within the mathematics they quite diverse they have they spent combinatorics algebra geometry Etc and some of them have like Olympia bench uh is a data set that goes into other stem disciplines not mathematics specifically then there's English law brain teasers so it's a very sufficiently uh you will notice that the majority of examples is still math right the first two combined is uh 41 42,000 examples out of 59 so it's still predominantly math but it has it has sufficiently many examples from other areas and every entity so so the uh the the data that they pull together is a problem and most of the problems have Solutions not all of them by by solution I mean the the final answer but I don't think any of those data sets has uh have a have the actual reasoning Trace annotated by human right uh but in their data set they do want to have a reasoning Trace they want to have a problem statement reasoning something like what R1 or o1 would do and the solution which is again answer the way they do that uh in the original so the paper was written I guess and then when they were ready to publish it R1 came out uh but because they did most of the work already on the original they didn't change and so majority of the paper is written based on Germany flash thinking which is a Google's model that allows you to get those reasoning traces but then at the end they mention hey guys when when the paper was ready to publish R1 came out so we tried some of the experiments using R1 instead of G flash thinking uh and everything that uses R1 was significantly better we will get to the results right but what they do here is that they give they get the problem statement give it to to Germany flash thinking flash thinking she generates a reasoning trace an answer okay and then they want uh when they distill those 59,000 right so they they they they put this goal in front of themselves right they have this 59,000 examples they want to get 1,000 and their goal is to uh to get that 1,000 to be as good as possible in terms of the final model performance and they identifi three um qualities that those 1,000 examples should should have one of them is quality uh and uh when it comes to Quality it consists of two steps the first step is they just filter out a lot of garbage right so while they were doing Gman I thinking there were some API errors some of the output was not Parable and what not it only gets rid of a small number of examples uh then they do difficulty and then after difficulty there will be a second step of quality when it comes to difficulty what they do is they get the problem send the problem and the number of models that they use on the PIP will be growing very fast right they send it to quen 7B and quen 32b and if neither of them solves it uh they consider that example as a good example right and the reason they use two models and not one model is that with one model you can have it could be a very easy problem but the model just I don't know like like you know models models mess up right so with two models they reduce the chance of such simple problems of slipping through and the way they um derive whether the model solved the problem uh is they get the solution of the model and they get the answer that they have right and the model also produces the answer I guess in many cases they could just compare it but they don't do that instead they get the solution of the model and the answer and they feed it to clae CLA 3.5 and they say hey clo look at this solution this is the answer we know that is correct this is the solution and an answer and you know if the answer is actually maybe I have it here no I don't and if the answer you know is specific just compare them and if it's not specific look through the solution tell me if it's good right so cl cl is the final arbitrator of whether um the solution is correct or not and so they filter out out of 51,000 examples that were remaining after removing garbage they filter it down to 25,000 right so which on itself is interesting I guess it's a it's an interesting quality of the data set of the initial data set right that qu 32b which is a relatively capable model was not able to solve half of the of the problems okay and so once that is done the third part is diversity uh and but first first speaking of quality so they want to get 1,000 examples and the first thing they do is they say if if the gery solution was correct as as judged by CL uh and it's one of the uh very high quality data sets uh ime or GP QA then they just put it put the example into the data set right uh and if it's U also if it's correct and it's month any month doesn't have to be from those two high quality data sets but the thinking length is very long right so Gemini and Gemini is not instructed in any way for how long to think she she decides right so if she chose to think for a very long time more than 6,000 tokens they're like hey that must be a good example as well because presumably longer thinking correlates with the with the harder problems right and that results in Having Eight 100 sorry 384 problems out of a thousand they want to have right so they need to to get the remaining 600 uh whatever 16 U and that's where the third quality comes in which is called diversity they want the 1,000 examples to be as heterogeneous as possible the way they do that is again they use CLA to um to assign labels or like tags to problems you know combinatorics biology geometry algebra so they have those domains and there's a picture in the paper I don't have it on the slides which shows like a tag cloud of how many problems they have of which in each domain and it's pretty much evenly distributed right and so what they do is they say you know until we have 1,000 examples we sample randomly one of those domains and from that domain we get one of the examples and the way they get an example is they they build a power uh Power lad distribution uh over the lengths of the thinking and and they favor examples which are which have longer thinking and as a matter of fact if you look at the exact for exact formula they fa favor them drastically uh if I'm reading it correctly right if you look at lines 2021 2022 effectively the longest sequence will have 50% chance of being chosen the second longest will have 25% Etc right because the the uh the the weight is 2 to the power of the rank right so the the uh the longest one will have weight of .5 the second longest will have weight of 025 Etc right so the longest ones will get chosen disproportionally frequently cool and and they do it until they get to a th example so the the goal here is to get uh you know every you see when when they choose the domain uh it is not weighted right it is not weighted by the number of examples in the domain right so so they want the final data set remaining 600 uh 16 examples to be you know representing every domain equally and among those get the most complex ones okay oh yeah so so grading is how they were okay I do have the prompt this is the prompt they were giving to CL uh it's in the paper uh to tell if the answer is correct right and it was used it was used in two places it was used here when they were trying you know to get the first 300 examples which are of extremely high quality and it was used to decide to begin with if they want an example right so if uh if quen would already give you an example that clae sorry give you a solution that CL deems correct they don't even want to have that example in the data set right and interest something interesting that I guess is worth highlighting highlighting here is that while the first 300 examples all had a correct solution uh the lower part of the algorithm does not check for correctness right so they will sample incorrect Solutions uh and more over uh as it says here the actual model the actual data set that was used for the majority of the paper only had 53% of the problems solved correctly in the data set right 47% were incorrect which I think on tell is very interesting right because you you you like one of the goals is to is to create 1,000 high quality examples and yet like they go at length you know to to device those U algorithms and yet they end up having 53% of only 53% of correct Solutions right and and presumably it is not it is not something they do in ablation testing they do a lot of ablation but they do not specifically they do not test what would what would happen if they only got the correct Solutions which is would be very interesting to see what happens right and so SK s1k 1.1 uh is the one that they mention in appendix a that is the one that uses deep seek R1 instead of Gemini flash thinking that one has 63% correct Solutions cool so that so that's the data set right so once the data set is ready they fine-tune quen 32b on the data set and then they test it with a another with a second idea in the paper called budget forcing the idea of the budget forcing is that they want they want to for a particular problem they have a problem they want to solve it they want to to have a particular budget in terms of tokens and they have both a lower bound and an upper Bound in that budget right and they want the model to be thinking within that band right and uh uh like like in many cases you don't want it to think shorter because you want you know you're like hey I have more resources think longer uh but but do come up with a you know with a higher quality answer right and they don't want to to to think much longer because you know you you have limited resources that you're willing to spend and so when they want the the to limit how uh the the upper bound of the model thinking that's relatively straightforward you just you know if you reach the upper bound you just insert and thinking token and say hey final answer is and the model presumably will stop thinking and provide produce the answer and that's exactly what it does but um continuing to think is a more interesting idea right so the the simple idea would be to just say you know if it tries to create end of thinking token you just sample any other Tok token right because you know like like at the end of the day you choose which token to sample right you just say you know until until I reach the lower bound I will not even consider I will mask out and of thinking token right so it will have to continue thinking indefinitely and so they they do test against it that's one of the ablation tests and they found that it is not the best way to do that the best way to do that is whenever you see end of thinking to sample weight instead right so it it thinks about it thinks reasons reasons reasons then it's ready to give you the answer it says end of thinking and you suppress end of thinking and instead sample waight and maybe triple dots I don't remember right so waight and then the model has to continue from that state after weight right and so it will you know it will challenge her assumptions and presumably uh think further and so so this is the the results on on the data set on a data set called Aime 24 it's a thir problems I think that was a qualifying round for uh Us olympiads in mathematics right so it's a pretty complex 30 problems and you will see that uh the the x-axis is how many tokens it can think for right so if you um if you you know if it's 1,000 so generally uh when uh we will see it on future tables generally if you don't if you do nothing on average model things for six you know you know between like five five and 6,000 tokens this is a reasonable thinking Trace right so if you wanted to think for 1,000 tokens that means you will almost always have to uh cut it short right so like it will M mid sentence you will inser insert end of thinking and send it to uh to get the answer and similarly you will notice on 2,000 and 4,000 uh it's also effectively more often than not you will be forcing it to stop while for you know uh points to the right uh this is where it would actually try to stop earlier and they introduce this weight uh interference and you know not not entirely surprisingly here right the longer it thinks the batter it solves right so at 1,24 thinking tokens it barely solves I guess zero is not zero here right that would be like what 16% uh right but uh you know it reaches almost 60% uh at it as it goes up and I think very interesting uh comparison they make here is yeah very very interesting comparison they make here is that uh there's another way to spend more tokens during test time right so so like let's say let's say that an average thinking trajector is 5,000 tokens but you at test time have sufficient resources to you know to spend 20,000 tokens tokens what you can do is you can unroll four thinking trajectories uh and each of them come up with an answer and then you can it's called majority voting and then you choose the answer that is uh that is the most frequent answer among those right because there's there are many ways to mess it up but there's only one way to solve it correctly right so if you solve 1,000 times uh and you know 100 of those the answer was five and then the remaining 900 it was all over the place then you know the answer is probably five right it's called majority voting uh and you will see here that um with the same budget uh sequential scaling so you know making longer a single uh longer thinking trajectory scales much better uh than doing majority voting and motivation behind it is that with a single trajectory uh every consecutive token sees all the mistakes you the model has made but with parallel scaling they have to redo all the mistakes again okay uh sign not very they use tempure water foring temperature Z for and I think that in generalic up lower temperature you know it Capt so it's just right so so but using the saure yeah to make sure everything yeah so so the note was that uh the parallel scaling here use temperature one and the sequential uses temperature zero I guess you cannot quite use temperature zero for parallel right because you will get the same the same trajectory but I guess you can use lower temperature that's a or I guess for fairness what you can do is you can use temperature one for sequential that would make for I see okay okay the results and there are two tables for the results in the paper uh one is in the in the actual paper which I don't have here on the slides and then another one this is from appendix a this one has s1.1 the one that was trained using deeps and this particular table is missing one of the rows that the P that the table in the paper has specifically this table misses doesn't have a a quen 32b which is a very interesting Benchmark because qu 32b is the base model for SK right and so uh so okay a couple observations here one is that uh the the resulting model is not even close to O3 which is presumably something we we would expect uh and yeah in the original paper they also have o1 And1 numbers are also noticeably better than U than uh S1 then if you compare it to R1 R1 is also better so this is a misspelling uh this is a typo that is actually 97 .3 right but if you compare to dis still but R1 is the one that is the result of the actual reinforcement learning right it's not a fine tuned model it's significantly more compute right so it's interesting to compare to the distilled models right because the distilled model is distilled on so this is a very interesting comparison right the these three rows uh is a three models and I guess the most interesting one will be would be qu 32b because it's the same base model which is fine tuned on 800,000 examples right 8 800 times more uh from R1 right and you will notice that S1 is outperforming it on all three not not on ime but it's outperforming it on Ma 500 and GP QA right so while only being trained on 1,000 examples right so it's a it's interesting comparison and the the base numbers for Quin 2.5b uh is u on ime it would be 26% right so so so like the Improvement relative to the base model in both cases in both uh uh distilling from R1 and the training from uh training on S1 is significant right effec it solves twice as many problems as the base model cool and and here you will notice they also they also show the result so so first of all you will notice that uh the model trend on the on um deep seek trajectories is uh noticeably better uh SK 1.1 sorry s1.1 than the one trained on Gemini also not surprising and it shows the result of you know introducing those weight you know forcing it to think longer uh you see it gets like an extra percent on math math 500 extra 2% on the GP QA cool 14 B Oh you mean for Amy right yes yeah so so S1 s1.1 is 32b right yeah well a more interesting question here is why the result of the distilled 32b is lower than distilled 14b right so so I guess there's a couple of things here one is that generally The Benchmark is not very accurate right there is some Randomness involved uh so 800,000 yeah yeah so so yeah so so so big difference between this row and this row is that R1 is in fact distilled model was trained on 800 times more examples fine tuned right it's 800k no no the distilled one it's a distill right right so it it has seen way more examples right it still doesn't explain why between the two of them 3 to be performs worse uh but U yeah it's uh could be some intricacy of a bigger model right like for example I don't think it applies here right but you know when you start training when you pre-train a model you know for the first many many iterations the smaller model will perform much better than the larger one right it's it's much later into the training that the large model uh catches up with the small one and uh I don't know if the same applies to fine tuning but I wouldn't be surprised if that's something that potentially has an effect here okay so so they did some ablations not not as many as one would want uh you you might always I don't know I I always want more oblations because there is a you know if the paper has 10 ideas in the perfect world you will have 2 to the power of 10 ablations not quite what we get but we get quite a few uh so first one is they say hey is our sophisticated algorithm of choosing 1,000 examples is even is it even necessary could we just choose you know 1,000 examples using only subset of those ideas right and so they they test four different ideas here here first idea is they just sample randomly uh and you will you will notice that randomly is not doing particularly well so on IM it solves uh 36% which is a 10% improvement over the over the base model uh on math yeah so on math the base model has 80% right so they get sorry 80 84% so this is 6% Improvement and this uh on GP QA the base model has 48% so that's a 3% Improvement right so Random doesn't give you as much it still improves the model but doesn't give you it's nowhere close to what s1k gets right uh diverse says we will we will choose examples from different domains you know the second part of the algorithm but we will do nothing in terms of ensuring the quality um all right in particular after choosing the domain presumably they just choose uh each example uniformly as opposed to you know that waiting where they choose the the one with the longest thinking trajectory uh Works uh better on these two data sets significantly worse on IM on ime actually it does exactly as poorly as the base model which is fascinating I guess longest for the longest they just got 1,000 longest trajectories no um not looking at all at the domains or quality again it's interesting right because on on each data set performs differently right on G on gpq does uh very well uh but on math and imy it's doing much worse than s1k and the last one is just train on all 59,000 examples right do not do not remove the ones that were solved by uh quen do not remove low quality ones do not do any keep yeah you know keep all of them uh and I I guess the last two of a particular interest right because training on all 59k is actually better than training on 1,000 right but but obviously it's 60 times slower so it's uh it's interesting to see how much and and this is this confidence interval right so it's not just 53 uh the the numbers in the square bracket say hey the with within some probability I don't remember like 90 something perc it's between minus 13 and 20% relative to s1k right so so yeah effectively those 1,000 examples oop sorry are getting you all the way there to what you would get with with 59,000 which is pretty cool okay uh then in terms of you know cutting it short or introducing this weight token they compare against three different approaches in one approach they just tell the model you know think for that many tokens um that just doesn't work right The Prompt is on the left right it says hey what is the uh answer to this think for 248 tokens right and if you think at how many tokens it actually thinks there's no correlation uh the upper upper part of the table right there's no correlation between how many tokens it's instructed to think for and how many it actually does right it's uh no matter what it thinks for 7 8,000 tokens so it completely ignores it has no you know no inherent ability to understand how many tokens it has emitted so far which I guess makes a lot of sense right because nothing in its data set was ever presumably mentioning this concept uh and you will notice that despite thinking for different number of tokens it is actually increasing in terms of uh quality which is which is very fascinating right you will see that you know even though it was thinking in both cases for seven you know almost 8,000 tokens the quality here is only 26% which is the exactly the quality of the base model right and it increases within with the number of tokens requested and on the lower part of the table is the uh they they still like I this no example interesting question so the question is would it qualify as a testm scaling if it's just a supervised fine tuning I I I think when they say test time scaling they specifically refer to this you know saying weight the data set is uh is just a cherry on the cake in this case in the paper yeah yeah but this is not test time as you properly mentioned as you properly said any any sort of supervised fine tuning is a train train time scaling when they say test time scaling they specifically refer to the fact that uh they they can SC they can scale the amount of resources you spend the test Time by saying waight H the weights are fixed yes but but but the amount of thinking it does scales by by saying weight instead of end of thinking so that's what the title of the paper refers to not the the what oh the chart um there is I don't know if I have it here no I don't have the main the main chart here but yeah it's uh yeah like the very first chart in the paper shows you that as a maybe it was at the beginning oh yeah this one yeah so so so this is one of them yeah yeah so so this is entirely test time right so how many thinking tokens that's entirely test time the model is fixed at this point right and the more thinking tokens you spent the higher is the quality like what like it or no no this is only during testing when when I'm pretty sure when they were doing Gemini from Gemini they get the whole trajectory right they never truncated and never do weight and whatnot as a matter of fact gini API doesn't even give you this opportunity you cannot say oh you know what do not sample end of thinking here with our one they could have done it thaten yeah doesn't actually matter absolutely I I think the entire point of this paper is that the whole paper was written before R1 right so so so I think the the motivation why they even went ahead to build this data set why that why that even is part of the paper is that when they were building when they were writing the paper there was no open source thinking model right so in order for you to have a model to do this experiment with test time scaling you had to create one to begin with and the only thinking model available I guess was Gemini that allows you to use it for training you know like o1 existed but opena explicitly prohibits using their outputs for fine tuning right so yeah so so I guess the whole the first half of the paper is there precisely because R1 did not exist with R1 uh but it is it is a very interesting you know finding that you can use that 1,000 examples it's still value could have been presumably a paper in itself yeah but this you know S1 the simple simple scale test time scaling refers I think specifically to trating or introducing weight tokens okay coming back to this uh ablation on the lower part of the table they also instructed to think for number of tokens but then they truncate at that number of Tok right and I'm pretty sure 3.3 is is is another typo uh I don't think it gets that that bad it's probably 33.3 but yeah it's uh yeah okay the second one is the second one is interesting because they say think for that many steps right and they empirically uh they empirically I guess they they run some experiments and notice that if you ask her to think for some number of steps has step on average is 100 tokens right and so they came up with the idea of trying to think it so if you ask it to think for 64 steps you're looking for like a ballpark of 6,400 tokens which could be exactly what you want right and so they say they tell the model to think for some number of steps and then I think uh the part that says that many number of steps left is also engineered into the into the model right so so this is interesting right because this time around the model obviously does have a concept of the number of steps right because the number of steps is explicitly written that is left right so you would expect that it cannot think for more steps than than it is instructed but you will see in practice it does right so if it says think for 16 steps right and then it says 16 steps left 15 Steps left zero steps left it goes to minus one steps left and it will continue for on average 123 steps so that's fascinating that the model after reaching zero steps left does not and you know forcefully stop thinking right and and that and that reproduces for all number of of steps all the way to uh 256 which is also interesting right if it says think for 256 tokens I would also expect the model to always go all the way down but it does not right it stops at some number of thinking tokens uh and the results are comparable right it's again 33% give or take on every on every execution and the again the same the same lower half they they they truncated it at zero steps forcefully uh the results are comparable and interesting thing here is you will notice that if the model is instructed to think for more steps the steps get shorter right so if you say hey do 16 steps the steps on average are um 60 tokens long right um well I guess 60 is an outlier but you see 70 69 66 56 it's gradually decreasing with the number of steps so you know the the model is uh tricking itself into thinking for approximately the same number of of um of tokens yeah and the last ablation they did is they just told her think for a little bit or think for a lot and that worked relatively well uh in the sense that you will notice that when when it's instructed so the first prompt is think for a little bit the second one says think for a lot you will see that the number of tokens is in fact much lower uh in the not much but it is noticeably lower in the first case than in the second and thinking for longer does increase the quality uh on the on the first two data sets although I think something interesting is the third row here which I might I might not understand it correctly the paper doesn't say anything about it it only exists in the table right but the way I understand the last row is that uh they run three experiments in the first they have this after the problem statements in the second they have this and and in the third they just have nothing and you will notice that having nothing does much better uh than saying either think for a little bit or think for a lot which I think is a very interesting observation but paper completely you know completely ignores it they they never mention the fact that not instructing it to think for a particular duration is better okay the last one they do is the rejection sampling and effectively this is this is purely this is only um any application in the in the ablation test you would never do rejection sampling like that in practice I guess but what they do is that they would sample a thinking trajectory and if it does not have the required number of tokens they will completely discard uh the the sample and Sample it again right and so they will sample until they get the desired number of tokens right does it make sense and so interestingly if you sample until you get the more tokens you get you wait for the worse is the quality right so if you sample until you get 2,000 tokens the quality is pretty good it's like 40% uh on uh on ime which is a decently good result but if you if you sample until you get more than 5,000 the quality is suddenly 22% uh the way they they actually have a whole example in in one of the appendixes appendix E2 they have an example where they look at one of the problems where this phenomenon occurs and they observe that uh the short thinking trajectory is usually the result of the model immediately jumping to the correct idea you know and it develops it and it gives you the answer the answer has high high percentage of being correct while a longer trajectory means that the model initially came up with the wrong idea and then it had to backtrack many times and so this a much higher likelihood in this case to never arrive at the correct answer right and so um that presumably explains this weird uh you know downward facing graph uh within with the with with a bigger budget of tokens okay the very last ablation which I think is interesting right so because when they when they when they reach the thinking limit right they just introduce they just forcefully sample end of thinking right and proceed to the answer so a natural uh you know the natural uh reverse of that you know if you want to think longer you just sample anything which is not end of thinking token right so you know you just mask it out uh until you get to the desired budget of tokens right and so so that's a very interesting Baseline to compare against and they do here right they so the 2x without string is effectively um is effectively sampling just masking out and of and end of thinking until you get to the number of tokens you want and then they introduce different no extrapolation means they just you know end of thinking means end of thinking just sample it and stop and then they have different versions of what they can sample right and you will see that on U um that note introducing weight on all the data sets results in a strictly worse performance right so so if you just force the model to think longer uh it is more likely to digress than to improve right while sampling weight increases on two out of the three data sets and you know Math 500 is the only one on which it does not improve but it's already 93% right meaning that you know it only has very hard problems left unsolved right so it's interesting right because like I could imagine if someone came up with an idea uh of forcefully thinking you know like if you try this first experiment and you you get worst results it's interesting that people actually went ahead and and even tried something else uh let alone actually got a very positive result okay cool this is the last slide I have uh that's the paper um exciting exciting results okay cool let's continue conversation back at the pizza stamp","transcript_synced_at":"2026-11-03T05:01:15.703Z","segments":[{"start_seconds":0,"text":"yeah let's dive into the paper um if you were here a month ago","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=0s"},{"start_seconds":8,"text":"for R1 uh this this presentation will be quite different because R1 had a lot of moth and a lot of uh novel","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=8s"},{"start_seconds":18,"text":"ideas and S1 is quite different S1 has effectively two very simple ideas","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=18s"},{"start_seconds":25,"text":"behind it but those simple ideas Pro result in very good results so there's going to be way less of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=25s"},{"start_seconds":32,"text":"U um hardcore math and more of a interesting adhoc engineering so two ideas behind","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=32s"},{"start_seconds":41,"text":"the paper are and they largely I guess they they both contribute towards the the end goal but they're large largely","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=41s"},{"start_seconds":49,"text":"independent one is a particular data set which only consists of 1,000 examples which is extremely few that when you","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=49s"},{"start_seconds":56,"text":"train on it um makes the model significantly smarter and these 1,000 examples are distilled from a larger data set to 59,000","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=56s"},{"start_seconds":65,"text":"examples which is more of the size of a what people usually use for fine tuning and those 1,000 examples give you almost","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=65s"},{"start_seconds":72,"text":"the whole benefit of of the entire data set right so if you fine tune on that th000 and on all 59,000 you get the model over very","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=72s"},{"start_seconds":80,"text":"similar capabilities and the second idea is called budget forcing and it's a very nice very simple trick which may uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=80s"},{"start_seconds":88,"text":"which you apply at at test time at inference time which makes model um smarter but also it makes model","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=88s"},{"start_seconds":98,"text":"work like depending on the question at hand you can you can force her to to work cheap more cheaper or more or spend","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=98s"},{"start_seconds":105,"text":"more compute uh to get to to the result faster or you know to get to a better result okay let's talk about the data","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=105s"},{"start_seconds":111,"text":"set so the data set the goal uh that they had was they they started with the quen quen 32b and they wanted to make","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=111s"},{"start_seconds":120,"text":"quity to be smart they wanted it to uh to do reasoning uh sort of like what one does uh provide the provide the result","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=120s"},{"start_seconds":128,"text":"and they wanted it to be as smart as possible and uh their first idea was that uh the data set on which they're going to train needs to be sufficiently","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=128s"},{"start_seconds":136,"text":"diverse if the data set is homogeneous then the model will over optimize for a particular um for a particular uh area","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=136s"},{"start_seconds":144,"text":"of expertise of that homogeneous data set and so they outline all the all the sources that went into the into the data set and you will see that","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=144s"},{"start_seconds":151,"text":"some of them most of them are mathematics but even within the mathematics they quite diverse they have they spent combinatorics algebra geometry Etc and","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=151s"},{"start_seconds":161,"text":"some of them have like Olympia bench uh is a data set that goes into other stem","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=161s"},{"start_seconds":169,"text":"disciplines not mathematics specifically then there's English law brain teasers so it's a very sufficiently uh you will notice that the majority of examples is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=169s"},{"start_seconds":176,"text":"still math right the first two combined is uh 41 42,000 examples out of 59 so it's still predominantly math but it has","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=176s"},{"start_seconds":186,"text":"it has sufficiently many examples from other","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=186s"},{"start_seconds":192,"text":"areas and every entity so so the uh the the data that they pull together is a problem and most of the problems have","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=192s"},{"start_seconds":200,"text":"Solutions not all of them by by solution I mean the the final answer but I don't think any of those data sets has uh have","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=200s"},{"start_seconds":208,"text":"a have the actual reasoning Trace annotated by human right uh but in their data set they do want to have a reasoning Trace they want to have a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=208s"},{"start_seconds":215,"text":"problem statement reasoning something like what R1 or o1 would do and the solution which is again","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=215s"},{"start_seconds":223,"text":"answer the way they do that uh in the original so the paper was written I guess and then when they were ready to","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=223s"},{"start_seconds":231,"text":"publish it R1 came out uh but because they did most of the work already on the original they didn't change and so","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=231s"},{"start_seconds":238,"text":"majority of the paper is written based on Germany flash thinking which is a Google's model that allows you to get those reasoning traces but then at the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=238s"},{"start_seconds":247,"text":"end they mention hey guys when when the paper was ready to publish R1 came out so we tried some of the experiments using R1 instead of G flash thinking uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=247s"},{"start_seconds":255,"text":"and everything that uses R1 was significantly better we will get to the results right but what they do here is that they give they get the problem","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=255s"},{"start_seconds":264,"text":"statement give it to to Germany flash thinking flash thinking she generates a reasoning trace an","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=264s"},{"start_seconds":272,"text":"answer okay and then they want uh when they distill those 59,000 right so they they they they put this goal in front of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=272s"},{"start_seconds":279,"text":"themselves right they have this 59,000 examples they want to get 1,000 and their goal is to uh to get that 1,000 to be as good as possible in terms of the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=279s"},{"start_seconds":288,"text":"final model performance and they identifi three um qualities that those 1,000 examples should should have one of them","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=288s"},{"start_seconds":296,"text":"is quality uh and uh when it comes to Quality it consists of two steps the first step is they just filter out a lot","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=296s"},{"start_seconds":305,"text":"of garbage right so while they were doing Gman I thinking there were some API errors some of the output was not Parable and what not it only gets rid of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=305s"},{"start_seconds":314,"text":"a small number of examples uh then they do difficulty and then after difficulty there will be a second step of quality when it comes to","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=314s"},{"start_seconds":323,"text":"difficulty what they do is they get the problem send the problem and the number of models that they use on the PIP will","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=323s"},{"start_seconds":329,"text":"be growing very fast right they send it to quen 7B and quen 32b and if neither","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=329s"},{"start_seconds":336,"text":"of them solves it uh they consider that example as a good example right and the reason they use two models and not one model is that with one model you can","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=336s"},{"start_seconds":344,"text":"have it could be a very easy problem but the model just I don't know like like you know models models mess up right so with two models they reduce the chance","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=344s"},{"start_seconds":352,"text":"of such simple problems of slipping through and the way they um derive whether the model solved","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=352s"},{"start_seconds":360,"text":"the problem uh is they get the solution of the model and they get the answer that they have right and the model also","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=360s"},{"start_seconds":368,"text":"produces the answer I guess in many cases they could just compare it but they don't do that instead they get the solution of the model and the answer and they feed it to clae CLA 3.5 and they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=368s"},{"start_seconds":377,"text":"say hey clo look at this solution this is the answer we know that is correct this is the solution and an answer and you know if the answer is actually maybe","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=377s"},{"start_seconds":385,"text":"I have it here no I don't and if the answer you know is specific just compare them and if it's not specific look through the solution tell me if it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=385s"},{"start_seconds":392,"text":"good right so cl cl is the final arbitrator of whether um the solution is correct or not and so they filter out","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=392s"},{"start_seconds":402,"text":"out of 51,000 examples that were remaining after removing garbage they filter it down to 25,000 right so which on itself is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=402s"},{"start_seconds":411,"text":"interesting I guess it's a it's an interesting quality of the data set of the initial data set right that qu 32b which is a relatively capable model was","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=411s"},{"start_seconds":418,"text":"not able to solve half of the of the problems okay and so once that is done the third part is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=418s"},{"start_seconds":424,"text":"diversity uh and but first first speaking of quality so they want to get 1,000 examples and the first thing they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=424s"},{"start_seconds":431,"text":"do is they say if if the gery solution was correct as","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=431s"},{"start_seconds":438,"text":"as judged by CL uh and it's one of the uh very high quality data sets uh ime or","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=438s"},{"start_seconds":445,"text":"GP QA then they just put it put the example into the data set right uh and if it's U also if it's correct and it's month","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=445s"},{"start_seconds":455,"text":"any month doesn't have to be from those two high quality data sets but the thinking length is very long right so Gemini and Gemini is not instructed in","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=455s"},{"start_seconds":463,"text":"any way for how long to think she she decides right so if she chose to think for a very long time more than 6,000 tokens they're like hey that must be a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=463s"},{"start_seconds":470,"text":"good example as well because presumably longer thinking correlates with the with the harder problems right and","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=470s"},{"start_seconds":478,"text":"that results in Having Eight 100 sorry 384 problems out of a thousand they want to have right so they need to to get the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=478s"},{"start_seconds":486,"text":"remaining 600 uh whatever 16 U and that's where the third quality comes in which is called diversity they want the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=486s"},{"start_seconds":494,"text":"1,000 examples to be as heterogeneous as possible the way they do that is again they use CLA to um to assign labels or","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=494s"},{"start_seconds":502,"text":"like tags to problems you know combinatorics biology geometry algebra so they have those domains and there's a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=502s"},{"start_seconds":511,"text":"picture in the paper I don't have it on the slides which shows like a tag cloud of how many problems they have of which","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=511s"},{"start_seconds":518,"text":"in each domain and it's pretty much evenly distributed right and so what they do is they say you know until we have 1,000 examples we sample randomly","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=518s"},{"start_seconds":527,"text":"one of those domains and from that domain we get one of the examples and the way they get an example is they they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=527s"},{"start_seconds":535,"text":"build a power uh Power lad distribution uh over the lengths of the thinking and and they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=535s"},{"start_seconds":542,"text":"favor examples which are which have longer thinking and as a matter of fact if you look at the exact for exact formula they fa favor them drastically","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=542s"},{"start_seconds":551,"text":"uh if I'm reading it correctly right if you look at lines 2021 2022 effectively the longest sequence will have 50%","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=551s"},{"start_seconds":559,"text":"chance of being chosen the second longest will have 25% Etc right because the the uh the the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=559s"},{"start_seconds":567,"text":"weight is 2 to the power of the rank right so the the uh the longest one will have weight of .5 the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=567s"},{"start_seconds":576,"text":"second longest will have weight of 025 Etc right so the longest ones will get chosen disproportionally frequently cool and and they do it until","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=576s"},{"start_seconds":584,"text":"they get to a th example so the the goal here is to get uh you know every you see when when they choose","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=584s"},{"start_seconds":592,"text":"the domain uh it is not weighted right it is not weighted by the number of examples in the domain right so so they want the final data set remaining 600 uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=592s"},{"start_seconds":601,"text":"16 examples to be you know representing every domain equally and among those get the most complex ones","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=601s"},{"start_seconds":610,"text":"okay oh yeah so so grading is how they were okay I do have the prompt this is the prompt they were giving to CL uh it's in the paper uh to tell if the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=610s"},{"start_seconds":618,"text":"answer is correct right and it was used it was used in two places it was used here when they were trying you know to get the first 300 examples which are of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=618s"},{"start_seconds":626,"text":"extremely high quality and it was used to decide to begin with if they want an example right so if uh if quen would","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=626s"},{"start_seconds":635,"text":"already give you an example that clae sorry give you a solution that CL deems correct they don't even want to have that example in the data","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=635s"},{"start_seconds":643,"text":"set right and interest something interesting that I guess is worth highlighting highlighting here is that while the first 300 examples all had a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=643s"},{"start_seconds":650,"text":"correct solution uh the lower part of the algorithm does not check for correctness right so they will sample","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=650s"},{"start_seconds":657,"text":"incorrect Solutions uh and more over uh as it says here the actual model the actual data set that was used for the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=657s"},{"start_seconds":665,"text":"majority of the paper only had 53% of the problems solved correctly in the data set right 47% were incorrect which","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=665s"},{"start_seconds":674,"text":"I think on tell is very interesting right because you you you like one of the goals is to is to create 1,000 high quality examples and yet like they go at","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=674s"},{"start_seconds":684,"text":"length you know to to device those U algorithms and yet they end up having 53% of only 53% of correct Solutions","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=684s"},{"start_seconds":693,"text":"right and and presumably it is not it is not something they do in ablation testing they do a lot of ablation but they do not specifically they do not test what would what would happen if","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=693s"},{"start_seconds":701,"text":"they only got the correct Solutions which is would be very interesting to see what happens right and so SK s1k 1.1","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=701s"},{"start_seconds":709,"text":"uh is the one that they mention in appendix a that is the one that uses deep seek R1 instead of Gemini flash","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=709s"},{"start_seconds":718,"text":"thinking that one has 63% correct Solutions cool so that so that's the data set right so once the data set is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=718s"},{"start_seconds":726,"text":"ready they fine-tune quen 32b on the data set and then they test it with a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=726s"},{"start_seconds":735,"text":"another with a second idea in the paper called budget forcing the idea of the budget forcing is that they want they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=735s"},{"start_seconds":743,"text":"want to for a particular problem they have a problem they want to solve it they want to to have a particular budget in terms of tokens and they have both a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=743s"},{"start_seconds":751,"text":"lower bound and an upper Bound in that budget right and they want the model to be thinking within that band right and","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=751s"},{"start_seconds":761,"text":"uh uh like like in many cases you don't want it to think shorter because you want you know you're like hey I have more resources think longer uh but but","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=761s"},{"start_seconds":769,"text":"do come up with a you know with a higher quality answer right and they don't want to to to think much longer because you know you you have limited resources that you're willing to spend and so when they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=769s"},{"start_seconds":778,"text":"want the the to limit how uh the the upper bound of the model thinking that's relatively straightforward you just you know if you reach the upper bound you","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=778s"},{"start_seconds":786,"text":"just insert and thinking token and say hey final answer is and the model presumably will stop thinking and provide produce the answer and that's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=786s"},{"start_seconds":794,"text":"exactly what it does but um continuing to think is a more interesting idea right so the the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=794s"},{"start_seconds":803,"text":"simple idea would be to just say you know if it tries to create end of thinking token you just sample any other Tok token right because you know like","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=803s"},{"start_seconds":811,"text":"like at the end of the day you choose which token to sample right you just say you know until until I reach the lower bound I will not even consider I will","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=811s"},{"start_seconds":819,"text":"mask out and of thinking token right so it will have to continue thinking indefinitely and so they they do test against it that's one of the ablation","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=819s"},{"start_seconds":826,"text":"tests and they found that it is not the best way to do that the best way to do that is whenever you see end of thinking","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=826s"},{"start_seconds":833,"text":"to sample weight instead right so it it thinks about it thinks reasons reasons reasons then it's ready to give you the answer it says end of thinking and you","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=833s"},{"start_seconds":842,"text":"suppress end of thinking and instead sample waight and maybe triple dots I don't remember right so waight and then","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=842s"},{"start_seconds":849,"text":"the model has to continue from that state after weight right and so it will you know it will challenge her","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=849s"},{"start_seconds":857,"text":"assumptions and presumably uh think further and so so this is the the results on on the data","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=857s"},{"start_seconds":865,"text":"set on a data set called Aime 24 it's a thir problems I think that was a qualifying round for uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=865s"},{"start_seconds":874,"text":"Us olympiads in mathematics right so it's a pretty complex 30 problems and you will see that uh the the x-axis is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=874s"},{"start_seconds":883,"text":"how many tokens it can think for right so if you um if you you know if it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=883s"},{"start_seconds":889,"text":"1,000 so generally uh when uh we will see it on future tables generally if you don't if you do nothing on average model","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=889s"},{"start_seconds":897,"text":"things for six you know you know between like five five and 6,000 tokens this is a reasonable thinking Trace right so if you wanted to think for 1,000 tokens","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=897s"},{"start_seconds":905,"text":"that means you will almost always have to uh cut it short right so like it will M mid sentence you will inser insert end","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=905s"},{"start_seconds":912,"text":"of thinking and send it to uh to get the answer and similarly you will notice on 2,000 and 4,000 uh it's also effectively","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=912s"},{"start_seconds":921,"text":"more often than not you will be forcing it to stop while for you know uh points to the right uh this is where","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=921s"},{"start_seconds":930,"text":"it would actually try to stop earlier and they introduce this weight uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=930s"},{"start_seconds":938,"text":"interference and you know not not entirely surprisingly here right the longer it thinks the batter it solves","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=938s"},{"start_seconds":946,"text":"right so at 1,24 thinking tokens it barely solves I guess zero is not zero here right that would be like what","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=946s"},{"start_seconds":954,"text":"16% uh right but uh you know it reaches almost 60% uh at it as it goes up and I think very","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=954s"},{"start_seconds":962,"text":"interesting uh comparison they make here","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=962s"},{"start_seconds":972,"text":"is yeah very very interesting comparison they make here is that uh there's another way to spend more tokens during test time right so so like let's say","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=972s"},{"start_seconds":981,"text":"let's say that an average thinking trajector is 5,000 tokens but you at test time have sufficient resources to you know to spend 20,000 tokens tokens","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=981s"},{"start_seconds":989,"text":"what you can do is you can unroll four thinking trajectories uh and each of them come up with an answer and then you can it's called majority voting and then","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=989s"},{"start_seconds":998,"text":"you choose the answer that is uh that is the most frequent answer among those right because there's there are many ways to mess it up but there's only one way to solve it correctly right so if","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=998s"},{"start_seconds":1006,"text":"you solve 1,000 times uh and you know 100 of those the answer was five and then the remaining 900 it was all over","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1006s"},{"start_seconds":1014,"text":"the place then you know the answer is probably five right it's called majority voting uh and you will see here that um with the same budget uh sequential","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1014s"},{"start_seconds":1023,"text":"scaling so you know making longer a single uh longer thinking trajectory scales much better uh than doing","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1023s"},{"start_seconds":1031,"text":"majority voting and motivation behind it is that with a single trajectory uh every consecutive token sees all the mistakes you the model has made but with","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1031s"},{"start_seconds":1040,"text":"parallel scaling they have to redo all the mistakes again","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1040s"},{"start_seconds":1047,"text":"okay uh sign not very they use tempure water","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1047s"},{"start_seconds":1056,"text":"foring temperature Z for and I think that in","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1056s"},{"start_seconds":1062,"text":"generalic up lower temperature you know","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1062s"},{"start_seconds":1067,"text":"it Capt so it's just right so","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1067s"},{"start_seconds":1074,"text":"so but using the saure yeah to make sure everything yeah so so the note was that uh the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1074s"},{"start_seconds":1082,"text":"parallel scaling here use temperature one and the sequential uses temperature zero I guess you cannot quite use temperature zero for parallel right","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1082s"},{"start_seconds":1089,"text":"because you will get the same the same trajectory but I guess you can use lower temperature that's a or I guess for fairness what you can","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1089s"},{"start_seconds":1097,"text":"do is you can use temperature one for sequential that would make for I see","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1097s"},{"start_seconds":1105,"text":"okay okay the results and there are two tables for the results in the paper uh one is in the in the actual paper which I don't have here on the slides and then","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1105s"},{"start_seconds":1113,"text":"another one this is from appendix a this one has s1.1 the one that was trained using deeps and this particular table is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1113s"},{"start_seconds":1122,"text":"missing one of the rows that the P that the table in the paper has specifically this table misses doesn't have a a quen","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1122s"},{"start_seconds":1130,"text":"32b which is a very interesting Benchmark because qu 32b is the base model for SK right and so uh so","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1130s"},{"start_seconds":1139,"text":"okay a couple observations here one is that uh the the resulting model is not even close to O3 which is presumably","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1139s"},{"start_seconds":1147,"text":"something we we would expect uh and yeah in the original paper they also have o1 And1 numbers are also noticeably better","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1147s"},{"start_seconds":1156,"text":"than U than uh S1 then if you compare it to R1 R1 is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1156s"},{"start_seconds":1164,"text":"also better so this is a misspelling uh this is a typo that is actually 97","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1164s"},{"start_seconds":1172,"text":".3 right but if you compare to dis still but R1 is the one that is the result of the actual reinforcement learning right it's not a fine tuned model it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1172s"},{"start_seconds":1179,"text":"significantly more compute right so it's interesting to compare to the distilled models right because the distilled model is distilled on so this is a very","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1179s"},{"start_seconds":1188,"text":"interesting comparison right the these three rows uh is a three models and I guess the most interesting one will be would be qu 32b because it's the same","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1188s"},{"start_seconds":1196,"text":"base model which is fine tuned on 800,000 examples right 8 800 times more uh from R1 right and you will notice","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1196s"},{"start_seconds":1205,"text":"that S1 is outperforming it on all three not not on ime but it's outperforming it on Ma 500","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1205s"},{"start_seconds":1212,"text":"and GP QA right so while only being trained on 1,000 examples right so it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1212s"},{"start_seconds":1220,"text":"a it's interesting comparison and the the base numbers for Quin 2.5b uh is u","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1220s"},{"start_seconds":1229,"text":"on ime it would be 26% right so so so like the Improvement relative to the base model in both cases in both uh uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1229s"},{"start_seconds":1237,"text":"distilling from R1 and the training from uh training on S1 is significant right","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1237s"},{"start_seconds":1244,"text":"effec it solves twice as many problems as the base model cool and and here you will notice","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1244s"},{"start_seconds":1252,"text":"they also they also show the result so so first of all you will notice that uh the model trend on the on um deep seek trajectories is uh noticeably better uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1252s"},{"start_seconds":1262,"text":"SK 1.1 sorry s1.1 than the one trained on Gemini also not surprising and it shows the result of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1262s"},{"start_seconds":1271,"text":"you know introducing those weight you know forcing it to think longer uh you see it gets like an extra percent on","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1271s"},{"start_seconds":1278,"text":"math math 500 extra 2% on the GP QA cool","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1278s"},{"start_seconds":1299,"text":"14 B Oh you mean for Amy right yes","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1299s"},{"start_seconds":1311,"text":"yeah so so S1 s1.1 is 32b right yeah well a more interesting question","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1311s"},{"start_seconds":1319,"text":"here is why the result of the distilled 32b is lower than distilled 14b right so so I guess there's a couple of things","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1319s"},{"start_seconds":1327,"text":"here one is that generally The Benchmark is not very accurate right there is some Randomness involved uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1327s"},{"start_seconds":1338,"text":"so 800,000 yeah yeah so so yeah so so so big difference between this row and this","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1338s"},{"start_seconds":1346,"text":"row is that R1 is in fact distilled model was trained on 800 times more examples fine tuned right it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1346s"},{"start_seconds":1354,"text":"800k no no the distilled one it's a distill right right so it it has seen way more examples right it still doesn't","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1354s"},{"start_seconds":1362,"text":"explain why between the two of them 3 to be performs worse uh but U","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1362s"},{"start_seconds":1369,"text":"yeah it's uh could be some intricacy of a bigger model right like for example I don't","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1369s"},{"start_seconds":1378,"text":"think it applies here right but you know when you start training when you pre-train a model you know for the first many many iterations the smaller model will perform much better than the larger","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1378s"},{"start_seconds":1386,"text":"one right it's it's much later into the training that the large model uh catches up with the small one and uh I don't know if the same applies to fine tuning","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1386s"},{"start_seconds":1394,"text":"but I wouldn't be surprised if that's something that potentially has an effect","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1394s"},{"start_seconds":1402,"text":"here okay so so they did some ablations not not as many as one would want uh you you might always I don't know I I always","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1402s"},{"start_seconds":1410,"text":"want more oblations because there is a you know if the paper has 10 ideas in the perfect world you will have 2 to the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1410s"},{"start_seconds":1418,"text":"power of 10 ablations not quite what we get but we get quite a few uh so first one is they say hey is our sophisticated algorithm","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1418s"},{"start_seconds":1427,"text":"of choosing 1,000 examples is even is it even necessary could we just choose you know 1,000 examples using only subset of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1427s"},{"start_seconds":1435,"text":"those ideas right and so they they test four different ideas here here first idea is they just sample randomly uh and you will you will notice that randomly","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1435s"},{"start_seconds":1443,"text":"is not doing particularly well so on IM it solves uh 36% which is a 10% improvement over the over the base","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1443s"},{"start_seconds":1452,"text":"model uh on math yeah so on math the base model has 80% right so they get sorry 80 84% so","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1452s"},{"start_seconds":1461,"text":"this is 6% Improvement and this uh on GP QA the base model has 48% so that's a 3% Improvement right so Random doesn't give","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1461s"},{"start_seconds":1470,"text":"you as much it still improves the model but doesn't give you it's nowhere close to what s1k gets right uh diverse says","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1470s"},{"start_seconds":1479,"text":"we will we will choose examples from different domains you know the second part of the algorithm but we will do nothing in terms of ensuring the quality","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1479s"},{"start_seconds":1487,"text":"um all right in particular after choosing the domain presumably they just choose uh each example uniformly as opposed to you know that waiting where","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1487s"},{"start_seconds":1496,"text":"they choose the the one with the longest thinking trajectory uh Works uh better on these","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1496s"},{"start_seconds":1503,"text":"two data sets significantly worse on IM on ime actually it does exactly as poorly as the base model which is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1503s"},{"start_seconds":1510,"text":"fascinating I guess longest for the longest they just got 1,000 longest trajectories no um not","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1510s"},{"start_seconds":1519,"text":"looking at all at the domains or quality again it's interesting right","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1519s"},{"start_seconds":1526,"text":"because on on each data set performs differently right on G on gpq does uh very well uh but on math and imy it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1526s"},{"start_seconds":1536,"text":"doing much worse than s1k and the last one is just train on all 59,000 examples right do not do not remove the ones that","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1536s"},{"start_seconds":1542,"text":"were solved by uh quen do not remove low quality ones do not do","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1542s"},{"start_seconds":1549,"text":"any keep yeah you know keep all of them uh and I I guess the last two of a particular interest right because","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1549s"},{"start_seconds":1557,"text":"training on all 59k is actually better than training on 1,000 right but but obviously it's 60 times slower so it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1557s"},{"start_seconds":1566,"text":"uh it's interesting to see how much and and this is this confidence interval right so it's not just 53 uh the the numbers in the square bracket say hey","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1566s"},{"start_seconds":1575,"text":"the with within some probability I don't remember like 90 something perc it's between minus 13 and 20% relative to s1k","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1575s"},{"start_seconds":1583,"text":"right so so yeah effectively those 1,000 examples oop sorry","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1583s"},{"start_seconds":1591,"text":"are getting you all the way there to what you would get with with 59,000 which is pretty cool okay uh then in","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1591s"},{"start_seconds":1600,"text":"terms of you know cutting it short or introducing this weight token they compare against three different approaches in one approach they just tell the model you know think for that","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1600s"},{"start_seconds":1609,"text":"many tokens um that just doesn't work right The Prompt is on the left right it says hey what is the uh answer to this think","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1609s"},{"start_seconds":1618,"text":"for 248 tokens right and if you think at how many tokens it actually thinks there's no correlation uh the upper upper part of the table right there's no","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1618s"},{"start_seconds":1626,"text":"correlation between how many tokens it's instructed to think for and how many it actually does right it's uh no matter","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1626s"},{"start_seconds":1633,"text":"what it thinks for 7 8,000 tokens so it completely ignores it has no you know no inherent ability to understand how many","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1633s"},{"start_seconds":1641,"text":"tokens it has emitted so far which I guess makes a lot of sense right because nothing in its data set was ever presumably mentioning this","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1641s"},{"start_seconds":1650,"text":"concept uh and you will notice that despite thinking for different number of tokens it is actually increasing in terms of uh quality which is which is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1650s"},{"start_seconds":1659,"text":"very fascinating right you will see that you know even though it was thinking in both","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1659s"},{"start_seconds":1666,"text":"cases for seven you know almost 8,000 tokens the quality here is only 26% which is the exactly the quality of the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1666s"},{"start_seconds":1674,"text":"base model right and it increases within with the number of tokens requested and on the lower part of the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1674s"},{"start_seconds":1681,"text":"table is the uh they they still","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1681s"},{"start_seconds":1694,"text":"like I this no example interesting question so the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1694s"},{"start_seconds":1703,"text":"question is would it qualify as a testm scaling if it's just a supervised fine tuning I I I think when they say test time scaling they specifically refer to","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1703s"},{"start_seconds":1710,"text":"this you know saying weight the data set is uh is just a cherry on the cake in this case in the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1710s"},{"start_seconds":1721,"text":"paper yeah yeah but this is not test time as you properly mentioned as you properly said any any sort of supervised","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1721s"},{"start_seconds":1728,"text":"fine tuning is a train train time scaling when they say test time scaling they specifically refer to the fact that","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1728s"},{"start_seconds":1735,"text":"uh they they can SC they can scale the amount of resources you spend the test Time by saying","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1735s"},{"start_seconds":1742,"text":"waight H the weights are fixed yes but but but the amount of thinking it does scales by","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1742s"},{"start_seconds":1751,"text":"by saying weight instead of end of thinking so that's what the title of the paper refers to not","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1751s"},{"start_seconds":1758,"text":"the the what oh the chart","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1758s"},{"start_seconds":1765,"text":"um there is I don't know if I have it here no I don't have the main the main chart here but yeah it's uh yeah like the very first chart in the paper shows","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1765s"},{"start_seconds":1773,"text":"you that as a maybe it was at the beginning oh yeah this","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1773s"},{"start_seconds":1780,"text":"one yeah so so so this is one of them yeah yeah so so this is entirely test","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1780s"},{"start_seconds":1788,"text":"time right so how many thinking tokens that's entirely test time the model is fixed at this point right and the more thinking tokens you spent the higher is","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1788s"},{"start_seconds":1796,"text":"the quality","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1796s"},{"start_seconds":1808,"text":"like what like it or no no this is only during","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1808s"},{"start_seconds":1817,"text":"testing when when I'm pretty sure when they were doing Gemini from Gemini they get the whole trajectory right they never truncated and never do weight and","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1817s"},{"start_seconds":1826,"text":"whatnot as a matter of fact gini API doesn't even give you this opportunity you cannot say oh you know what do not","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1826s"},{"start_seconds":1833,"text":"sample end of thinking here with our one they could have done","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1833s"},{"start_seconds":1854,"text":"it thaten yeah doesn't actually","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1854s"},{"start_seconds":1870,"text":"matter absolutely I I think the entire point of this paper is that the whole paper was written before R1 right so so so I think the the motivation why they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1870s"},{"start_seconds":1879,"text":"even went ahead to build this data set why that why that even is part of the paper is that when they were building when they were writing the paper there","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1879s"},{"start_seconds":1887,"text":"was no open source thinking model right so in order for you to have a model to do this experiment with test time","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1887s"},{"start_seconds":1894,"text":"scaling you had to create one to begin with and the only thinking model available I guess was Gemini that allows you to use it for training you know like","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1894s"},{"start_seconds":1902,"text":"o1 existed but opena explicitly prohibits using their outputs for fine tuning right so yeah so so I guess the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1902s"},{"start_seconds":1910,"text":"whole the first half of the paper is there precisely because R1 did not exist with R1 uh but it is it is a very","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1910s"},{"start_seconds":1918,"text":"interesting you know finding that you can use that 1,000 examples it's still value could have been presumably a paper in itself yeah but this you know S1 the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1918s"},{"start_seconds":1927,"text":"simple simple scale test time scaling refers I think specifically to trating or introducing weight","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1927s"},{"start_seconds":1935,"text":"tokens okay coming back to this uh ablation on the lower part of the table they also","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1935s"},{"start_seconds":1944,"text":"instructed to think for number of tokens but then they truncate at that number of Tok right and I'm pretty sure 3.3 is is is another","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1944s"},{"start_seconds":1953,"text":"typo uh I don't think it gets that that bad it's probably 33.3 but yeah it's","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1953s"},{"start_seconds":1961,"text":"uh yeah okay the second one is the second one is interesting because they say","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1961s"},{"start_seconds":1968,"text":"think for that many steps right and they empirically uh they empirically I guess they they run some experiments and notice that if you ask her to think for","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1968s"},{"start_seconds":1977,"text":"some number of steps has step on average is 100 tokens right and so they came up with the idea of trying to think it so","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1977s"},{"start_seconds":1984,"text":"if you ask it to think for 64 steps you're looking for like a ballpark of 6,400 tokens which could be exactly","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1984s"},{"start_seconds":1992,"text":"what you want right and so they say they tell the model to think for some number of steps and then I think uh the part that says that many number of steps left","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1992s"},{"start_seconds":1999,"text":"is also engineered into the into the model right so so this is interesting right because","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=1999s"},{"start_seconds":2007,"text":"this time around the model obviously does have a concept of the number of steps right because the number of steps is explicitly written that is left right","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2007s"},{"start_seconds":2015,"text":"so you would expect that it cannot think for more steps than than it is instructed but you will see in practice it does right so if it says think for 16","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2015s"},{"start_seconds":2023,"text":"steps right and then it says 16 steps left 15 Steps left zero steps left it goes to minus one steps left and it will","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2023s"},{"start_seconds":2030,"text":"continue for on average 123 steps so that's fascinating that the model after reaching zero steps left does","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2030s"},{"start_seconds":2038,"text":"not and you know forcefully stop thinking right and and that and that reproduces for all number of of steps","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2038s"},{"start_seconds":2044,"text":"all the way to uh 256 which is also interesting right if it says think for 256 tokens I would also expect the model","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2044s"},{"start_seconds":2051,"text":"to always go all the way down but it does not right it stops at some number of thinking","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2051s"},{"start_seconds":2059,"text":"tokens uh and the results are comparable right it's again 33% give or take on every on every","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2059s"},{"start_seconds":2066,"text":"execution and the again the same the same lower half they they they truncated it at zero steps forcefully uh the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2066s"},{"start_seconds":2074,"text":"results are comparable and interesting thing here is you will notice that if the model is instructed to think for more steps the steps get shorter right","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2074s"},{"start_seconds":2083,"text":"so if you say hey do 16 steps the steps on average are um 60 tokens long right um well I guess 60 is an","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2083s"},{"start_seconds":2092,"text":"outlier but you see 70 69 66 56 it's gradually decreasing with the number of","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2092s"},{"start_seconds":2098,"text":"steps so you know the the model is uh tricking itself into thinking for approximately the same number","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2098s"},{"start_seconds":2105,"text":"of of um of tokens yeah and the last ablation they did is they just told her think for a","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2105s"},{"start_seconds":2114,"text":"little bit or think for a lot and that worked relatively well uh in the sense that you will notice that when when it's instructed so the first prompt is think","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2114s"},{"start_seconds":2122,"text":"for a little bit the second one says think for a lot you will see that the number of tokens is in fact much lower uh in the not much but it is noticeably","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2122s"},{"start_seconds":2130,"text":"lower in the first case than in the second and thinking for longer does increase the quality uh on the on the first two data","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2130s"},{"start_seconds":2139,"text":"sets although I think something interesting is the third row here which I might I might not understand it correctly the paper doesn't say anything","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2139s"},{"start_seconds":2147,"text":"about it it only exists in the table right but the way I understand the last row is that uh they run three experiments in the first they have this","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2147s"},{"start_seconds":2155,"text":"after the problem statements in the second they have this and and in the third they just have nothing and you will notice that having nothing does","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2155s"},{"start_seconds":2163,"text":"much better uh than saying either think for a little bit or think for a lot which I think is a very interesting observation but paper","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2163s"},{"start_seconds":2170,"text":"completely you know completely ignores it they they never mention the fact that not instructing it to think for a particular","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2170s"},{"start_seconds":2179,"text":"duration is better okay the last one they do is the rejection sampling and effectively this","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2179s"},{"start_seconds":2188,"text":"is this is purely this is only um any application in the in the ablation test you would never do rejection sampling","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2188s"},{"start_seconds":2197,"text":"like that in practice I guess but what they do is that they would sample a thinking trajectory and if it does not have the required number of tokens they","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2197s"},{"start_seconds":2204,"text":"will completely discard uh the the sample and Sample it again right and so they will sample until they get the desired number of tokens right does it","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2204s"},{"start_seconds":2213,"text":"make sense and so interestingly if you sample until you get the more tokens you get you wait for the worse is the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2213s"},{"start_seconds":2222,"text":"quality right so if you sample until you get 2,000 tokens the quality is pretty good it's like 40% uh on uh on ime which is a decently","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2222s"},{"start_seconds":2231,"text":"good result but if you if you sample until you get more than 5,000 the quality is suddenly","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2231s"},{"start_seconds":2238,"text":"22% uh the way they they actually have a whole example in in one of the appendixes appendix E2 they have an","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2238s"},{"start_seconds":2245,"text":"example where they look at one of the problems where this phenomenon occurs and they observe that uh the short thinking","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2245s"},{"start_seconds":2253,"text":"trajectory is usually the result of the model immediately jumping to the correct idea you know and it develops it and it","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2253s"},{"start_seconds":2261,"text":"gives you the answer the answer has high high percentage of being correct while a longer trajectory means that the model initially came up with the wrong idea","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2261s"},{"start_seconds":2268,"text":"and then it had to backtrack many times and so this a much higher likelihood in this case to never arrive at the correct","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2268s"},{"start_seconds":2275,"text":"answer right and so um that presumably explains this weird uh you know downward facing graph uh","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2275s"},{"start_seconds":2284,"text":"within with the with with a bigger budget of tokens okay the very last ablation which","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2284s"},{"start_seconds":2292,"text":"I think is interesting right so because when they when they when they reach the thinking limit right they just introduce","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2292s"},{"start_seconds":2301,"text":"they just forcefully sample end of thinking right and proceed to the answer so a natural uh you know the","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2301s"},{"start_seconds":2310,"text":"natural uh reverse of that you know if you want to think longer you just sample anything which is not end of thinking token right so you know you just mask it","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2310s"},{"start_seconds":2318,"text":"out uh until you get to the desired budget of tokens right and so so that's a very","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2318s"},{"start_seconds":2326,"text":"interesting Baseline to compare against and they do here right they so the 2x without string is effectively um is effectively","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2326s"},{"start_seconds":2334,"text":"sampling just masking out and of and end of thinking until you get to the number of tokens you want and then they introduce","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2334s"},{"start_seconds":2342,"text":"different no extrapolation means they just you know end of thinking means end of thinking just sample it and","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2342s"},{"start_seconds":2349,"text":"stop and then they have different versions of what they can sample right and you will see that on","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2349s"},{"start_seconds":2357,"text":"U um that note introducing weight on all the data sets results in a strictly","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2357s"},{"start_seconds":2365,"text":"worse performance right so so if you just force the model to think longer uh it is more likely to digress","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2365s"},{"start_seconds":2373,"text":"than to improve right while sampling weight increases on two out of the three","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2373s"},{"start_seconds":2381,"text":"data sets and you know Math 500 is the only one on which it does not improve but it's already 93% right meaning that","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2381s"},{"start_seconds":2388,"text":"you know it only has very hard problems left unsolved right so it's interesting right","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2388s"},{"start_seconds":2395,"text":"because like I could imagine if someone came up with an idea uh of forcefully thinking you know like if you try this first experiment and you you get worst","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2395s"},{"start_seconds":2403,"text":"results it's interesting that people actually went ahead and and even tried something else uh let alone actually got","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2403s"},{"start_seconds":2411,"text":"a very positive result okay cool this is the last slide I have uh that's the paper um exciting exciting results okay","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2411s"},{"start_seconds":2419,"text":"cool let's continue conversation back at the pizza stamp","url":"https://www.youtube.com/watch?v=Vw_YfujRFuE&t=2419s"}],"youtube_url":"https://youtube.com/watch?v=Vw_YfujRFuE","url":"https://near-pedia.xyz/videos/Vw_YfujRFuE"}