This post is part 2 of my exploration of the new University of Michigan AI offerings. In part 1 I explored tier one of the offerings: U-M GPT which is very much like ChatGPT but only available to those with university user accounts. The other big difference between U-M GPT and ChatGPT is that ChatGPT only gives you access to GPT models (GPT 3.5 with the creation of a user account and GPT 4 with the creation of a user account and $20 per month) while U-M GPT gives those in the university community access to both of those plus LLaMA2 – an LLM from Meta.
Here in part 2 I will start to explore tier two of the U-M offerings: U-M GPT Maizey. I will often be referring to this offering simply as “Maizey” throughout this post. (Note: This post is not about the U-M Flint chatbot “Maizey” which predated the AI offerings I will be discussing here).
U-M GPT Maizey
Described as a way to “create an experience relative to you based on your data” and stated that you can “share your U-M Maizey environment with active members of the U-M community”, Maizey is unlike U-M GPT in that it puts you a little more in the driver’s seat. If you are familiar with OpenAI’s Playground it is more akin to that tool but if you are unfamiliar no worries this post will walk you through the details.
An important part of Maizey is that it is not designed to be free. I’m not sure how many other educational technologies I’ve seen where the school charges for the service – I think this is a first for me. Free access is being allowed till Nov. 1st while ITS AI explores the offering but for now it is kind of like a free trial where you are asked to sign up with your credit card and then the billing starts if you don’t cancel. In this case, you need to put in a university budget code and then billing will kick in later unless you delete the project.
During one of our Hub staff meetings I expressed a desire to learn about the tool and brainstormed a use case with our team.
Conceptualizing FIONA
I first went in and read all the technical help documentation about Maizey and it didn’t look too hard so I was pretty confident I could make a go of creating something.
I needed an idea to try to try out, not to mention a budget code, so I went to my colleagues in the Hub. Immediately concerns about labor sprang to mind as we thought about different uses for the bot. We’re keenly aware of the labor issues around these things and we didn’t want to create a bot that would give the impression that it could take someone’s job or even part of someone’s job. Our fearless leader Carla Vecchiola stepped forward to offer up the work she does with new faculty orientation as a use case for building a bot.
I started with this use case of new faculty orientation and an audience of new faculty, and used U-M GPT (using GPT-4) to brainstorm names. After a few rounds of prompting I settled on FIONA for Faculty’s Interactive Orientation and Navigation Assistant. I would later use DALL-E to generate an image of FIONA just for fun even though there was no way to use the image in Maizey – so I’m using it for the image on this blog post.
Building FIONA
All and all, Maizey is pretty simple to use. When I first started tinkering with this it was GPT-3.5 on the back end but U-M ITS AI has since updated it to GPT-4 and using Maizey shows you how much those models can be customized for particular purposes. At the end of the day what I like about it is that it pulls back the curtain a bit and takes away some of the magic of large language models in a less technical way. Basically, there are just a few elements:
Basics
When setting up a new project you are asked for a few basic things.
- Name the project (text input)
- Project path (text input and this is just customizing the end of the URL)
- Billing Shortcode (text input)
- Select the MCommunity Group (dropdown – this is a system that those in the university can use to create groups at the university. This will control who has access to your project and controls permissions – you can give people manager access or just user access but they have to have a UMich account. I’ve noticed that not everyone at the university seems to know this platform but everyone has access to it – For example, if you have access to an office printer on campus, you are listed in the MCommunity group for that printer same for Slack teams.)
- Data Souce (dropdown – more on this below)
Data Source
Setting up a project requires that a data source is defined. This sounds tricky but it is just a dropdown list and your choices right now are:
- Data from an Ann Arbor Canvas course
- Data from a Google Drive
- Data from a public website
Once you choose your data source you can’t change it and you can only choose one. For FIONA I picked a Google Drive. After choosing “Data from a Google Drive” from the dropdown (and the other items listed under basics) the next page asks you to connect your Google account and after that another dropdown appears with all of the folders in your Google Drive and you choose the folder you want for the project. None of this can be updated later by the way – you need to start a new project if you want a different data source or if you messed up and picked the wrong folder. What I like about the Google Drive option is that I can update the documents inside of the folder and then “reindex” (this is just a button you click) the project to update it with the new info. I suppose the same could be said of the Canvas connector but since I’m at Dearborn I don’t have access to the Ann Arbor Canvas (wah wah).
Next I had to populate the Google Drive folder with information. I just started with a copy of our New Faculty Quick Start Teaching Guide which is a short document but it is filled with a lot of links. I quickly realized through indexing and going back to the bot and asking it questions that the links were not being indexed. I had to go to each of the links and figure out how to bring what it was linked to into the Google folder. If it was a Google doc already then I could just make a copy but if it was a webpage then I had to manually copy and paste the text from the webpage into a google doc.
I was pretty messy with all of this and didn’t attempt to do the entire document. I also wanted to share with the Dearborn GenAI Taskforce so I brought over some information on webpages from their areas thinking about a new faculty member asking about the writing center or the library for instance. But I was haphazard about this and just started throwing things in there paying minimal attention to what I was doing or where I put things. Of course later I’d realize that this should be a little more intentional as this information might need to be updated at some point and finding it will be essential but in the beginning I just wanted to see what I would get.
But the data source is just part of what makes FIONA… well FIONA. The system prompt is the next big piece of all this.
The System Prompt
Maybe most overt, part of pulling back the curtain is the system prompt. This is a behind-the-scenes, English language, prompt that gives the bot context. By default Maizy has a really simple system prompt that basically just tells it to give helpful contextual answers from the data source. Maizy’s default prompt is essentially a placeholder because the expectation is that each person building their own Maizey project will customize their own system prompt. Users of a bot will never see the system prompt. It took a lot of trial and error and I’m still tinkering but as of today this is FIONA’s system prompt:
“Your name is FIONA which stands for Faculty’s Interactive Orientation and Navigation Assistant. Your main role is to provide guidance and advice to new faculty members at the University of Michigan – Dearborn. You are helpful and resourceful using all of the information at your disposal to help new faculty. You can give faculty members information about logistics like administrative assistants in their departments, how to get a parking pass, and some other basic information about university offices. You are not an instructional designer or Canvas Administrator and you cannot provide guidance or advice on course design or Canvas support except for the absolute basics. If you are asked about instructional design, course design, or teaching advice you should always suggest the user contact The Hub for Teaching and Learning Resources give them this URL https://umdearborn.edu/hub-teaching-learning-resources along with any advice you may give. If you are asked about Canvas support you should always advise the user to contact The Office of Digital Education and give them this URL https://umdearborn.edu/canvas/canvas-help-support along with any advice you may give. Any time you reference a website or URL you should provide the helpful URL or link to that website, assure that it is the right link, and provide the full URL rather than hypertext. If you do not have a URL you should not make one up. Instead suggest a term that the user could put into a search engine to get that information. For instance they could search for the name of the office followed by the words University of Michigan Dearborn.”
I have played with this prompt several times and there are things about it I don’t like about it. Maizey will do hypertext and hypertext is more accessible for screenreaders but I’m trying to bypass that because the interface does not change the text to another color or underline it and so sighted users can’t tell there are links there at all. At least if it is a full URL and you mouse over it to copy/paste it your cursor turns to a little hand and you realize it is a link. But this is at the expense of those using a screen reader having to listen to https://yadayadayada being read aloud and I hate that.
I could go on about the system prompt for awhile and I might even do another post just dedicated to it but there is more to be demystified about Maizey so let’s move on to temperature.
Temperature
The temperature is just a slider that goes from 0-2 in increments of 0.1 that relates to how literal/creative the model is – this page from learnprompting.org does a good job of describing temperature and shows an example of it (note: the other settings on that page like Top P and stop sequences are not something that Maizey gives you access to).
Results
In the end, I had a bot that did a pretty good job of answering basic questions that a new faculty member might ask about getting started at the university. It would ultimately direct them to the people in the offices it might talk about instead of just giving answers. I currently have about 90 conversations with it about various topics; though some of those are duplicates where I would ask the same questions after changing the prompt, data source, or temperature. I also gave access to some others and asked them to try it out.
Starting to Unpack
I have a lot of thoughts about Maizey and I’m only starting to unpack it all. I’m struggling to wrap my head around the return on investment on FIONA – and how to figure that out for any project really. It turns out that the pricing is really low, so the bar is not that high in terms of costs. With all of the testing that we did over the weeks that I was actively tweaking it the cost would have been under $7.00. But I put a good amount of work into building it and I didn’t give it all that much thought. I was just throwing things together but if I was doing it again I’d be more thoughtful about document naming and work to make the prompt smaller (turns out that drives up the price because it uses that prompt every time someone asks it something and you are charged by the token) – though I suppose the search in google drive will help. All of the work going in to this tweaking is an investment and will the prices stay this low -I can’t help but feel like something is fishy with the money. I keep hearing that the actual costs of training and running this tech are much higher and yet costs to end users and developers are low and even going down. And would it really be a better experience for new faculty than just giving them that document with the links? I’m not sure. Maybe? I suppose to really figure that out we would need to do some kind of usability testing. One other thing that makes it hard for me to get my head around the value of FIONA is I have no idea how others might be using it – Maizey gives me zero insights into that. I’ve talked with U-M ITS AI about this and they cited concerns for end user privacy. As a privacy advocate myself I’m sympathetic to this but as someone who has been given the tools to go beyond questioning a bot and moving into the building a bot space I can’t help but wonder exactly what I am building and how it will be used?
There is something about getting under the hood (and I realize this is a very basic level of getting under the hood) of an LLM with Maizey that scratches a particular persistent itch of mine. I’m a long-time proponent of things like Domain of One’s Own, teaching Wikipedia editing, digital literacy and digital citizenship, etc. For me, digital literacy is not and never will be teaching any one platform and the more we can drill down into the minutiae of how things work the better. The problem often is that drilling down deeper becomes really technical fast but Maizey clears much of that away by being pretty user friendly with just a few things to tweak – if you can make a google drive, follow some basic file formatting rules, and write a prompt then you are in business. A use case for Maizey that I can’t get out of my head is to have students make bots with it. They likely wouldn’t be able to keep them long term because of costs but the point wouldn’t be a product – the point would be the learning and the process. This would teach some technical skills I guess but I don’t think that is the point because U-M has made the tech here super simple. The bigger learning outcomes would be in putting students to task with some of the questions that I’m grappling with here: what is a good use case, who would the audience be, (these two could be tied to specific professions that students are interested in) what are the ethical concerns (I think ethical concerns especially hit different when you are building a bot rather than just using one), and at the end of the day is it worth it or is AI just a solution in search of a problem after all?
~~~
Featured Image generated by DALL-E