From the course: Generative AI: Working with Large Language Models

Unlock the full course today

Join today to access over 23,100 courses taught by industry experts.

OPT and BLOOM

OPT and BLOOM

From the course: Generative AI: Working with Large Language Models

Start my 1-month free trial Buy for my team

OPT and BLOOM

“

- [Instructor] You've probably noticed that up to this point all of the language models are from big tech firms. Now although OpenAI made GPT-3 available via an API, no access was given to the actual weights of the model making it difficult for smaller research organizations and institutions to study these models. The Meta, or Facebook, AI team then released OPT, or Open Pre-trained Transformers. This was a couple of decoder-only pre-trained transformers ranging from 125 million to 66 billion parameters, which they shared with everyone. Interested research teams could also apply for access to the 175 billion parameter model. Now, this effectively gave researchers access to a large language model that was very similar to GPT-3. The Facebook team also detailed the infrastructure challenges they faced, along with providing code for experimenting with the models. This model was primarily trained on English text. The research teams…

Contents