So I’ve started an experiment. You may have noticed these appearing:
We are looking at using AI to summarise the first 40 posts on topics.
This data is being sent to Open AI. We will only be doing this on public, crawlable Categories which are consumed by Google and Microsoft in any case. Data is not being added to the remote AI model, but it will be used in their macro analytics around usage of the API.
The summaries themselves are only shown to people who are logged in, so crawlers will not see them.
There’s a downvoting button if you don’t like a summary. Sufficient downvotes (currently > 2) will prompt the system to retrieve a new summary in the hope it will be a better one.
Any concerns or feedback please Reply here.
I’m aware of the current impact that people are unable to post New Topics or Replies. We are working on that now.
OK that should be resolved. If not we will take another look!
I don’t like the summary. I have reduced the medication to zero, this is not reflected in the summary, it gives the wrong impression.
1 Like
Yep, I noticed that, let’s downvote it!
1 Like
OK any new reply or additional downvotes to that Topic will cause it to re-retrieve.
If this is consistently a problem I will remove this feature. We’ll also add a tooltip to explain the potential pitfalls of the summary.
I am not sure whether the summary is useful at all. I gained lots of benefit by reading through each individual post within a topic. If the topic summary does not sound interesting to me, I would not read it, and consequently miss out on all the information in the individual posts.
2 Likes
I hear you. Appreciate the feedback. I’m going to leave it up a couple of days or so then run a community poll on keeping it.
I’ve added an important Disclaimer to cover your point:
“Disclaimer: this is highly experimental! This summary is generated by a Large Language Model run by openai.com. It is very sophisticated and can create convincing, relevant, syntactically correct text but is prone to inaccuracies which can sometimes be misleading. Use it to get a flavour, but read the full Topic for the definite information”
OK so they just released “GPT-4” which I’m told is much superior, so I’ve left the summarising system up for now until we can evaluate that.
I love the intent of the AI and feel it can be useful as a quick overview of the first 40 posts of the topic, as you state it does. Obviously, many topics have massive amounts of posts and information from many different people over the course of years, which can make the topic very time consuming to read from start to finish. It has to be tough for AI (or even a single person) to summarize those 1st 40 posts, especially because many posts contain cross-topics. With time, I feel AI will get even better as newer versions roll out. I hope it’s here to stay!
1 Like
it doesn’t work for longer threads because GPT3 doesn’t have long enough context length
1 Like
It is often incorrect. For example, one time it stated that “Katherina has Meniere’s disease”. I definitely don’t have Meniere’s disease. I am not reading it anymore.
Yeah Open AI is taking its sweet time providing GPT4 access. We will move to that asap.
Yes. Oooops. Hence caveat and the downvoting system. As soon as we get three downvotes a new summary is created, hopefully a better one. It is also regenerated after three additional replies.
Hopefully GPT4 will be an improvement.
We might also get it to read user bios to avoid this kind of mistake.
@Katharina I will try to improve that summary, can you remind me which Topic it is on?
I think even with GPT4, there will not be enough context for longer threads. Once the number of words has exceeded the maximum number of tokens that can be fed into the model, the model will essentially “forget” what happened at the beginning of the thread and just summarize the end. This is why on many of the threads, it seems like its only really summarizing the final post – it ran out of context
1 Like
That’s not how it works currently. It simply stops updating the summary after the 40th Post. It never attempts to summarise after that.
But your point that context is important carries. It definitely is!
1 Like
oh okay interesting. didn’t know that
1 Like
I am unable to find it. I think it might have been in the post “Next steps after 50% reduction in symptoms”. But it is not there anymore.
We’ve just deployed an experimental upgrade to this system that should be able to summarise Topics taking into account now up to the first 160 Posts (that’s an increase of x4!!) 
2 Likes
