In 2020, the AI company OpenAI presented an incredibly large neural network called “Generative Pre-trained Transformer 3” (GPT-3). The network uses 175 billion parameters and during training it read terabytes of texts from all over the Internet.
Since then, GPT-3 has amazed experts with its stylish and error-free texts in several languages, including programming languages. Now the access to GPT-3 is released for the general public.
Transformers are neural networks that can read sequences as input and output new sequences. As you compute the output, you can focus your attention on specific parts of the input and learn to tailor an internal vector representation of the meaning of the input to the context. They are similar to the older long-short-term memory networks (LSTM), but can be calculated in parallel and scaled better.
The most obvious application of such an AI is language: Sentences are sequences of words that the Transformers read. The output is a new sentence word for word. Such a language model is trained to continue writing a text that has been started by always calculating the best next word.
Risk of misinformation
So far, OpenAI has rarely granted external programmers access to the language model. The company justified its reluctance to say that such a good language model can easily be misused for misinformation campaigns, spam or fraud. But now OpenAI has built in enough security measures internally that the risk of misuse of the AI is acceptably low in the company’s opinion.
Registration for the API is now public and released without a waiting list. Programmers who want to use GPT-3 in their applications must agree to guidelines that prohibit, for example, fraud and political interference and limit the permitted areas of application. For many dictatorships, access is completely blocked. A content filter classifies texts as “safe”, “sensitive” or “unsafe”.
API access is paid for per 1000 words or punctuation marks (tokens) with prices between 0.08 and 6 US cents, depending on which variant of the AI is used.
Securing a language model against unethical use is so difficult that misuse is still to be expected. At the beginning of this year, researchers have already shown that GPT-3 has learned prejudices about Muslims from its training data, which it also reproduces in its own texts.
Replicating biases from the dataset is a problem that almost all language AIs struggle with. Since GPT-3 writes more eloquently than many small neural networks, its texts are very difficult to recognize as the work of a machine. A source check for texts on the Internet is therefore becoming more and more important, also because GPT-3 will not be left alone.
The start-up AlephAlpha is already working on a similarly large language model, as are Microsoft and Nvidia, who have given their network the intimidating name “Megatron-Turing Natural Language Generation Model” (MT-NLG) – Megatron is a villain from the Transformers- Franchise.
In c’t 26/2021 we show how you can switch from Windows 10 to 11 with just a few clicks of the mouse – and how you can overcome higher hurdles, for example if your PC does not meet the hardware requirements. In addition, we have compiled hand-picked gift tips from the c’t editors on six double pages and tested clever suction bots with object recognition. You will find issue 26/2021 from December 3rd in Heise shop and at the well-stocked newspaper kiosk.