This project originally started in 2021. I had just started programming and somewhere online I read about GPT2 and that you could finetune this yourself. At the time, I had no idea what finetuning was or how it worked, but it seemed cool. At the same time my twitter was full of dumb (mostly false, probably) fact-tweets. I noticed that the facts were often of a similar structure so I started writing a script to gather as many of these facts as I could, and I wanted to see what would happen if I finetuned GPT2 on these tweets. The first version made with GPT2 made many language mistakes, however I still found it extremely cool to see what this 'AI' could do. Then, in 2024 I remembered I still had the dataset lying around somewhere, and I wanted to see the differences between GPT2 and the then newly released Llama3.1. After only a couple hours of running Apple's LoRA finetuning script, it turned out to work much, much better than before. Below are some randomly picked facts created by the LoRA. You can refresh the page to load more facts.
One interesting thing I noticed is that sometimes, the model will use its prior knowledge to create actual true facts. For example, facts about certain countries or celebrities that, when Googled turn out to be true. When searching the dataset for the same names or countries the fact does not show up, meaning that the model does not simply regurgitate previous learned facts. Sometimes however, the model just makes up facts that sound kinda true but, when Googled are actually false. The older GPT2 version of the facts model had this issue almost for all facts. Some other times, the model just creates a quote like something written in a diary.