They started by giving the bots prompts to solve grade-school math problems, some of which began with encouraging phrases like “you are an expert mathematician” and ended with things like “this will be fun!” The bots, however, did not consistently perform better with the positive reinforcement. So, they turned to AI to improve their methods.
They used an automated process to tweak the phrasing of the prompts based on whether or not the chatbots’ accuracy in solving the problems improved. This process was overall more effective in making the bots better at math, but the ones that were the best were the most surprising. When the chatbots were asked to start their answers with “Captain’s Log, Stardate [insert date here]:,” a phrase even casual Star Trek fans will recognize, the bots’ accuracy consistently improved.
“Surprisingly, it appears that the model’s proficiency in mathematical reasoning can be enhanced by the expression of an affinity for Star Trek,” the team wrote in their findings.
