Ignoring fine tuning for a moment, an LLM that has the tools available to remember and recall bits of information as needed is already possible. No need to dump all of that into active memory (context). You just recall relevant memories (Semantic Search) and add only those.