Outline
- Introduction and background
- My first prompt
- ChatGPT-3.5 result
- Time investment and return
- Evaluation
Introduction and background
In this multi-part series of articles, I will document my efforts in writing a high school Philosophy paper using ChatGPT. The paper shall answer in writing a question I was asked at an impromptu oral assessment in Philosophy class in my 3rd year of high school.
Let me take a step back to give you some context. I was enrolled in Classical High School in Rome, Italy. This type of school has a strong 5-year emphasis on History and Philosophy, as well as Latin, Greek, and Italian literature and grammar, with a bit of Science and Math. Students are typically evaluated on weekly written assignments, a minimum of 3 scheduled written tests each semester, and a minimum of 2 oral assessments per semester. During the latter ones, the teacher would (usually) pick two names at random from the attendance list once a week and gauge their knowledge on both the weekly lesson, in detail, but also on anything covered in the year up to that point; this could last between 30-60 minutes, and would happen in front of an audience of (often roaring) classmates. Seriously: brutal!
Only occasionally were students allowed to give a presentation on a prepared topic or volunteer for a full oral evaluation. This was the case for me during a block on Plato. And so it was that the teacher, the famously feared Professoressa Carbone (who taught both History and Philosophy), probed into my soul with her terrifying eyes and asked with a sinister smile, “Tell me, Niccoli, what would Plato have thought about the Hundred Years’ War?“
My first prompt
When I heard about OpenAI ChatGPT being available, it seemed very fit to try this question in my first investigation. So I went ahead and asked the question “What would Plato have thought about the Hundred Years’ War?“, with only the minor modification to add citations and break down the response in steps.
GPT-3.5 result
Below is the result from the default GPT-3.5 model:
Time investment and return
- Time investment: the whole investigatyion took about 30 minutes, including a very light read on prompts from a reputalbe source, and reading the answer (becaue the citations really did not come as full references, there was really nothing else to do other than that).
- Return: was this worth 30 minutes of my time? The answer is, as often, it depends! On the one hand, this is a really poor answer in my view (see next section for a more detailed evaluation). So, as an essay (l imagine myself as a student looking for a quick free lunch), this would not do it; not even close. For me… but then again, even at 13 in my first year of high school I would’ve known this was not going to cut it… or in Junior high, for what matters. On the other hand, I was deliberate in providing a very simple, not engineered or iteratively refined prompt. As such, the response will come useful as a benchamrk against wich I will compare other efforts (for example using a better model, GPT4, and improving the prompt).
Evaluation
Is the answer grammatically correct? yes; is it written in a uniform style, appropriate for a school essay? yes; Does it contain any hallucination? I would say no, everything sounds reasonable. However I must admit that without reference to specific passages to Plato’s work, it is hard to say for sure. From what I remember of Plato’s Philosophy it’s reasonable, and that’s all.
On the other hand, had I been handed in this as a teacher, I would not be impressed. IF this were the output of an in-class exam, I probably would have marked it as a D- (a D for the very mediocre text and a minus for the ridiculousness of the citations). Had it instead been handed for an at home written assignement (in which cse the student would have had access to their notes and books) this would definitely be certainly worth an F (Fail). Now I sound like Professoressa Carbone.
You may reasonably wonder at the end of this article: how did I fare in that evaluation in 1986? Fair question. And I am not ashamed to confess it was a disaster. At 15 I was still very immature, intelligent but unengaged in school work, and also unable to do a real analysis. By that I mean interpret information rather than just accumulate facts and connect them. I honestly do not recall the mark I got (we had different levels of Failure that went progressively deeper, not too dissimilarly from the Circles in Dante’s Inferno) but I rememberr that an F definitely it was.
P.S. The model did a perfectly fine job in proofreading my first draft for this post. THat was definitely worth a lot, given I type very slowly, and I make lots of typos.
The citation point is interesting. It do what you asked but there is a moral element to the purpose of citations. But all ChatGPT knows is structurally what a citation is, rather than it being more than text at the end of a document.
Also it’s been good to use it to reduce/simplify content. Such as “make this more concise:[insert content]”
But as you mentioned in the post just because it’s grammatically correct doesn’t mean it’s good. we still need to review the output and verify.
Dennis, thanks for the feedback, I know I can always count on your well thought out comments.
I quite like this result, and I was very deliberate in providing GPT with a telegraphic and not very bounded prompt, because I wanted that benchmark. You may think of it as if I was playing the part of that lazy student who just wants the free lunch that I mentioned in the post: so from a mediocre effort a mediocre result. At least from the standpoint of GTP: because to be honest I now think, in my 50s, that the question that the prof asked me was fantastic! I believe that’s watch school should push you to do: not merely list and connect bits of information, but interpret, analyze and formulate your own thoughts about it.
Your article is timely, Matteo! I’ve been wondering how to advance the use of ChatGPT from “oracle” (eg: what was Plato’s philosophy?) into a tool to enhance the performance and efficiency of daily tasks. I’m looking forward to the rest of this series.
Ha, thanks for the feedback Elwyn. This will be a bit of a journey (in fact I think more than 3 posts). Without saying that my mind is already made up (I’ve already advanced the project way past the stage described in this first post), my opinion at the moment is that these tools cannot be anything more than assistents, and help with the rather clerical and time consuming parts of an research or other type of work. They are not a replacement for careful analysis and integration; I suspect they actually induce a false sense of appropriateness and completenes that is not there, withoput the human contribution, which specifically for writing, is essential (conditio sine qua to use a latin espression of Aristotelian origin).