feat: Improve Memory System #5209

Closed
opened 2025-11-11 16:14:40 -06:00 by GiteaMirror · 0 comments
Owner

Originally created by @Davixk on GitHub (May 18, 2025).

Check Existing Issues

  • I have searched the existing issues and discussions.

Problem Description

Currently, the Memories system will select and feed one single Memory inside the model context, and the prompting only says User Context:.

This results most of the times either in:

  • The selected Memory being irrelevant
  • The LLM misunderstanding how to use it

Desired Solution you'd like

This can be improved by:

  1. Retrieving top K most relevant Memories for any given Chat message
  2. Feeding them all inside the context
  3. Improving the system message attached to them

I believe the Memories should be retrieved and injected into each message on the server side, rather than the client side.
This would allow the server to use the internal retrieval system and fetch a number of accurate matches for any given Chat message.


The system message should include:

  • When Memories are relevant
  • How they should be interpreted

For example, ChatGPT uses a combination of the bio tool description, which is something like this:

The bio tool allows you to persist information across conversations. Address your message to=bio and write whatever information you want to remember. The information will appear in the model set context below in future conversations. DO NOT USE THE BIO TOOL TO SAVE SENSITIVE INFORMATION. Sensitive information includes the user’s race, ethnicity, religion, sexual orientation, political ideologies and party affiliations, sex life, criminal history, medical diagnoses and prescriptions, and trade union membership. DO NOT SAVE SHORT TERM INFORMATION. Short term information includes information about short term things the user is interested in, projects the user is working on, desires or wishes, etc.

which then allows it to use much simpler wording later when addressing the User Memories.

Except since currently, no tool is natively available for models to call to store Memories, then this information about Memories should be instead included in the section title.

Alternatives Considered

No response

Additional Context

No response

Originally created by @Davixk on GitHub (May 18, 2025). ### Check Existing Issues - [x] I have searched the existing issues and discussions. ### Problem Description Currently, the Memories system will select and feed **one single** Memory inside the model context, and the prompting only says `User Context:`. This results most of the times either in: - The selected Memory being irrelevant - The LLM misunderstanding how to use it ### Desired Solution you'd like This can be improved by: 1. Retrieving top K most relevant Memories for any given Chat message 2. Feeding them all inside the context 3. Improving the system message attached to them I believe the Memories should be retrieved and injected into each message on the server side, rather than the client side. This would allow the server to use the internal retrieval system and fetch a number of accurate matches for any given Chat message. --- The system message should include: - When Memories are relevant - How they should be interpreted For example, ChatGPT uses a combination of the `bio` tool description, which is something like this: > The bio tool allows you to persist information across conversations. Address your message to=bio and write whatever information you want to remember. The information will appear in the model set context below in future conversations. DO NOT USE THE BIO TOOL TO SAVE SENSITIVE INFORMATION. Sensitive information includes the user’s race, ethnicity, religion, sexual orientation, political ideologies and party affiliations, sex life, criminal history, medical diagnoses and prescriptions, and trade union membership. DO NOT SAVE SHORT TERM INFORMATION. Short term information includes information about short term things the user is interested in, projects the user is working on, desires or wishes, etc. which then allows it to use much simpler wording later when addressing the User Memories. Except since currently, **no tool** is natively available for models to call to store Memories, then this information about Memories should be instead included in the section title. ### Alternatives Considered _No response_ ### Additional Context _No response_
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#5209