During a workshop at the Consortium for School Networking’s annual conference in Seattle this month, Fry — who worked with an outside firm to develop four custom chatbots for WASA members last year — shared the lessons she’s learned when it comes to deciding whether to build your own chatbot or pay someone else to do it for you.
IN-HOUSE VERSUS OUTSIDE HELP
For example, if the chatbot will draw from a fairly static content source, such as an employee handbook or a document of school policies and procedures, she said it will likely require less time and skill. Chatbots that use more dynamic content sources, such as the student information system where attendance and grades are logged, tend to be more complex.
School IT leaders who don’t have the in-house time and expertise but do have the funding, Fry said, can hire an outside firm to do the work. She encouraged school leaders with tight budgets to look to students and the community at large for help and said WASA chose a local company called Swift Labs to develop its own chatbots last year.
WASA'S CHATBOTS
The result of those listening sessions was the development of four chatbots: one trained on state laws that pertain to education, another that creates student behavior plans, and two more that generate feedback drafts for teacher and principal evaluations. To use the chatbots, K-12 administrators must pay a monthly fee.
For the student behavior bot, Fry said users answer 11 questions about the issue at hand and can upload documents, then receive a detailed student support plan. The teacher and principal evaluation chatbots are trained on specific performance rubrics, she said, and use uploaded observation notes to create draft feedback reports.
“It gives a whole bunch of information for the evaluator to look at and to think through, then offers three to five growth-focused questions that the evaluator can use to have a conference with that educator," Fry said. "And it does it in, like, 30 seconds."
MAINTENANCE REQUIRED
Even if you have help building your bots, Fry said maintaining them requires someone with deep knowledge of their content to consistently check for errors and updates — and that’s not typically a task that can be outsourced.
“How many of you have someone who you can dedicate just for checking back on these bots that you’ve developed? If you don’t and you are developing these for public use, you need to assign somebody to do that work,” Fry said. “It doesn’t necessarily have to take a long time, but you have to continually check back and make sure that it’s getting accurate information and that you’re updating it off whatever you happen to have as source documents.”
Fry said WASA’s legal chatbot requires lots of updates due to ongoing changes in the state Legislature, and that free AI tools have come far enough to provide users with most of the same information. In retrospect, she said she’s not sure WASA would have created that bot “had we known how far AI that is available to all of us would evolve in its ability to go out and get that information.”
A MOVING TARGET
As AI continues to change at a rapid pace, Fry said education leaders who choose to make custom chatbots, whether in-house or with help from paid consultants, will need to modify those tools often to keep pace with the moving target of what this technology can do and what users expect from it.
“We’re constantly adjusting them, and it does take quite some time to keep tinkering with them and making them more and more user friendly. And the bar is being raised day after day about what people expect AI to do for them,” Fry said. “We have to think about, are we going to continue to enhance these tools, or will there be a time when they’re no longer viable and the other options have caught up and would be more effective for our members to use?”
