Launched by the San Francisco-based artificial intelligence research firm OpenAI last November, ChatGPT can execute various tasks including answering follow-up questions, challenging incorrect premises, admitting to mistakes and rejecting inappropriate requests, according to a company blog. The tool uses reinforcement learning from human feedback and is fashioned after OpenAI’s InstructGPT, a tool that follows instructions and provides detailed responses.
According to an email from NYC DoE spokeswoman Jenna Lyle, the department blocked the tool in December after schools requested that it do so. Some educators have expressed concern about the potential for AI to enable cheating by writing essays for students, and Lyle said the negatives of ChatGPT outweigh the positives.
“Due to concerns about negative impacts on student learning, and concerns regarding the safety and accuracy of content, access to ChatGPT is restricted on New York City Public Schools’ networks and devices,” Lyle wrote in a prepared statement. “While the tool may be able to provide quick and easy answers to questions, it does not build critical-thinking and problem-solving skills, which are essential for academic and lifelong success.”
OpenAI said in its blog that the tool is not without limitations. It said the technology “sometimes writes plausible-sounding but incorrect or nonsensical answers.” The fix is somewhat challenging, the company wrote, as the chatbot is “sensitive to tweaks to the input phrasing or attempting the same prompt multiple times.” The company said the chatbot tends to be verbose, using certain phrases more often than not, and sometimes asks clarifying questions if the user is being ambiguous but typically guesses what the user is asking. OpenAI also said on its website that ChatGPT sometimes exhibits biased behavior, and the development team is still working to prevent it from giving unsafe recommendations to users.
Lyle said that her department still strives to give students access to emerging technologies for the purposes of AI and technology-related education, and despite city schools banning ChatGPT, students and staff can request temporary access to it.
When Government Technology had an opportunity to “interview” ChatGPT recently, the chatbot responded to a question about how state and local leaders should regulate AI by saying, in part, that they should establish standards, guidelines and safeguards.
“State and local leaders should approach the regulation of AI systems, such as myself, in a thoughtful and measured way,” it wrote. “It is important to recognize the potential benefits that AI can bring in terms of improving efficiency, accuracy and decision-making, as well as the potential risks and challenges that it may pose. As such, leaders should consider implementing regulations that balance the need to promote the use of AI with the need to protect the public and ensure that AI systems are used responsibly.”
OpenAI, which has raised $1 billion over five funding rounds since 2016, according to Crunchbase, responded to requests for comment with an emailed statement from spokesperson Alex Beck calling for transparency and more research around the use of AI-generated text.
"We don’t want ChatGPT to be used for misleading purposes in schools or anywhere else, so we’re already developing mitigations to help anyone identify text generated by that system," the statement read. "We look forward to working with educators on useful solutions, and other ways to help teachers and students benefit from artificial intelligence."
Editor's note: This story has been updated to include an emailed statement from an OpenAI spokesperson.
