Skip to main content

ayeung.dev

Tag: Deep Learning

🤖 Mastering BERT Tokenization and Encoding

To use a pre-trained BERT model, we need to convert the input data into an appropriate format so that each sentence can be sent to the pre-trained model to obtain the corresponding embedding. This article introduces how this can be done using modules and functions available in Hugging Face’s transformers package (https://huggingface.co/transformers/index.html).