Token Processing: Overcoming Quantization Challenges
Nothing is loaded from YouTube until you press play, and then without tracking cookies.
About this video
The complexities of quantization in machine learning model inference. strategies to enhance token processing efficiency #MachineLearning #TokenProcessing #QuantizationChallenges #DataHandling #ModelInference #AI #TechInsights #MachineLearningTips #DataScience #OptimizationStrategies #NEARAI
Transcript
100 wordsCaptions as YouTube publishes them, usually machine-generated, so product and people names can be misheard.
- 0:00
let's see let's see why is challenging and what can be done about it so let's zoom in into single Math mode right so let's say we look into this one right so we have Vector X here this is the model and usually we don't look at a single Vector we look at a bunch of them uh so there's your whole context window of data right so if you doing inference you you fit the prompt let's say the
- 0:33
prompt is already 100 tokens so this is your 100 tokens your prompt uh we call it uh let