Is there a way to see and possibly edit the Persistent file?

Rich

Old Fart
Modder
Respected User
Donor
Game Developer
Jun 25, 2017
2,461
6,926
Personally I quit. It's near midnight here, I had a fucking hard day and lost enough time with this, especially since I don't need it.
OK, I couldn't resist. Here's some code that will dump a persistent file. It's raw Python 3.x (I don't have Python 2.x installed), so you'd have to have Python 3 installed on your computer, but, using that, you could run it as:
Code:
python dump_persistent.py PATH_TO_A_PERSISTENT_FILE > output.txt
It doesn't depend on anything in Ren'py - it's completely self-contained.

Code:
import io
import pickle
import pprint
import sys
import zlib


class RevertableDict(dict):
    pass


class RevertableList(list):
    pass


class RevertableSet(set):
    def __setstate__(self, state):
        if isinstance(state, tuple):
            self.update(state[0].keys())
        else:
            self.update(state)


class MyUnpickler(pickle.Unpickler):
    def find_class(self, module: str, name: str) -> type:
        if name == "RevertableDict":
            return RevertableDict
        if name == "RevertableList":
            return RevertableList
        if name == "RevertableSet":
            return RevertableSet
        if name == "set":
            return set
        return type(name, (object,), {})


class DumpPersistent:
    def __init__(self, file_path: str) -> None:
        self._file_path = file_path

    def run(self) -> None:
        with open(self._file_path, "rb") as file:
            file_bytes = file.read()

        decompressed_bytes = zlib.decompress(file_bytes)
        file_like_object = io.BytesIO(decompressed_bytes)
        unpickler = MyUnpickler(file_like_object)
        result = unpickler.load()
        pprint.pprint(result.__dict__)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python dump_persistent.py path_to_file")
        sys.exit(1)

    file_path = sys.argv[1]

    dumper = DumpPersistent(file_path)
    dumper.run()
Basically, it uses the standard Python Unpickler class, but with find_class overridden to handle the fact that the pickle will be looking for Ren'py-specific classes.

Now, having said, that, I don't know what this will do if you have managed to save a class of your own into the persistent file - might work, might blow chunks. But it was an interesting exercise.

What I've been able to determine is the following:

The Persistent object will have an attribute for each persistent variable you've set. No surprise whatsoever there.

Beyond that, it has the following attributes, some of which I've decoded, others of which I haven't:
  • _achievement_progress - a dict. Use unknown
  • _achievements - a set. Use unknown
  • _changed - a dict that seems to record the last time any of the persistent items were changed. Key is the attribute name, value appears to be a timestamp as a floating point number.
  • _character_volume - a dict. Use unknown.
  • _chosen - a dict. Appears to record every menu entry in the game, along with whether or not you've taken that choice.
  • _console_history - a list of tuples. All the lines that have been output to the console
  • _console_line_history - a list of lists. Appears to have all the commands you've typed into the console
  • _console_short - boolean. Meaning unknown
  • _console_traced_short - boolean. Meaning unknown
  • _console_unicode_escaping - boolean. Meaning unknown
  • _director_bottom - boolean. Meaning unknown
  • _file_folder - int. Meaning unknown
  • _file_page - int. Meaning unknown
  • _file_page_name - dict. Meaning unknown.
  • _gl2 - bolean. Meaning unknown
  • _gui_preference - dict. Meaning unknown.
  • _gui_preference_default - dict. Meaning unknown.
  • _iap_purchases - dict. Meaning unknown.
  • _preference_default - dict. Appears to store default values for preferences
  • _preferences - Preferences object, storing the user's preferences.
  • _seen_audio - dict. Key is every audio or movie file, value is whether or not the user has seen it.
  • _seen_ever - dict. Key is every label (ones you provide, or the ones Ren'py generates for every line) and whether or not the user has seen it.
  • _seen_images - dict. Key is a tuple for every image, and value is whether or not the user has seen it.
  • _seen_translates - set. Probably related to the translation system - not sure exactly what the contents are
  • _set_preferences - boolean. Use unknown.
  • _style_preferences - dict. Probably related to the style preference system.
  • _update_last_checked - dict. Use unknown
  • _update_version - dict. Use unknown
  • _virtual_size - tuple that appears to contain the (width,height) of the game.
  • _voice_mute - RevertableSet. Use unknown

I'm sure I could figure out what the rest of those attributes contains if I wanted to spend time poking through the Ren'py code, but now, like anne O'nymous, it's late and I feel I've done my duty for the day.

anne O'nymous, have fun with this - I know you won't be able to resist...
 

anne O'nymous

I'm not grumpy, I'm just coded that way.
Modder
Respected User
Donor
Jun 10, 2017
10,111
14,790
anne O'nymous, have fun with this - I know you won't be able to resist...
Oh, I clearly will. I don't know yet when (still have to finish the refactoring and port for my variable viewer), but I already put your code in my "todo" stack.



Also :
  • _achievement_progress - a dict. Use unknown
  • _achievements - a set. Use unknown
As the name say, it's linked to the achievement feature. As far as I remember my quick look at it, the dict permit you to keep track of the advance in one achievement (when you've to do this X time by example), while the set store the unlocked achievements.


  • _chosen - a dict. Appears to record every menu entry in the game, along with whether or not you've taken that choice.
When you skip, auto-advance, or advance again after a rollback (from memory, therefore it's possible that it don't apply to the three), there's a possibility to pass the menu without being presented a choice ; Ren'py will pick for you the last choice you made.
I guess that it's where the said choices are stored.

[side note: I worth being looked deeper since it can be a way to patch an update when you forgot to use a variable to keep track of a choice.]


  • _console_short - boolean. Meaning unknown
Somewhere after the 6.99.12.4, PyTom added the possibility to limit the size of the variable content that is display in the console ; enabled by default. By example, in place of a full list you'll have something like [ 1, 2, 3, 4 ... 998, 999, 1000 ]. You switch from one to the other with short and long in the console.
The name make me guess that it's where the current state is stored.


  • _console_traced_short - boolean. Meaning unknown
The same as above, but for the traced expression (watch VARIABLE from the console). You change this with watch short and watch long from the console.
Side effect, you can't trace variables named either "short" or "long" ; well, you can, but directly editing the list of the said traced variables.


  • _console_unicode_escaping - boolean. Meaning unknown
Have the console to escape the Unicode characters (\x00\xFE\xA6, or should it show them as effective characters. Appeared with the 7.4.0, therefore the start of the effective port to Python 3.x ; where all string are now Unicode even in .py files. This can be changed with the escape/unescape console commands.
Honestly I'm not sure what he tried to do with that. I mean, historically all string from a rpy file are Unicode, therefore the OS is able to display it. I guess it's with the intent to offer a deeper debugging method, by example for when you came to manipulate the string as BYTE (the old ASCII), what can lead to a real mess with the Unicode value since they are stored on more than one BYTE.


  • _director_bottom - boolean. Meaning unknown
Is the window for the "interactive director" feature (kind of live code tracing, and sprite (name only) building feature) displayed in top or bottom of the screen.


  • _file_folder - int. Meaning unknown
  • _file_page - int. Meaning unknown
  • _file_page_name - dict. Meaning unknown.
It's related to the save/load page.
I don't know what the first do. Seem to mean that you can change the folder where the save files are, but since there's two (one on the system structure, and one directly in the game) I don't know to what it relate ; perhaps used only for Android/iPhone ports.
_file_page is the save/load page actually selected, while I guess that _file_page_name is related to a cool feature almost never used by players (me included). When you're on the save/load page, click on the "page X" text on top of the page and, magic, you can edit the text and give an explicit name to this page.


  • _gl2 - bolean. Meaning unknown
[Guess] Is Ren'py using its second generation of renderers.


  • _gui_preference - dict. Meaning unknown.
  • _gui_preference_default - dict. Meaning unknown.
[Guess] Related to the gui. store.
It was implemented to replace the themes, therefore probably offer the possibility to have more than one value by entry. _gui_preference should probably host all the values (entry = { "theme1": value, "theme2": value} ?), while _gui_preference_default should probably contain the default value for each entry.


  • _iap_purchases - dict. Meaning unknown.
[Guess] Related to the "In Application Purchase" feature. Because yes, Ren'py have a feature to offer micro-transaction to the players. Due to the nature of Ren'py and how easy it is to change any value and/or any part of the code, core included, I never looked at this ; it's so easy to fake what you bought.


  • _seen_translates - set. Probably related to the translation system - not sure exactly what the contents are
[Guess] In the AST dialog lines are stored in two step ; a "say" entry hosting a "translate" entry. Therefore, it's possibly the same than the other "_seen" sets, but for the dialog lines ; what permit the skip feature to works.


  • _update_last_checked - dict. Use unknown
  • _update_version - dict. Use unknown
[Guess] The SDK can auto-update itself, I guess that the same is possible for games. Therefore the first should host the date (timestamp ?) when the feature have checked for an update for the last time, while the second would host the actual version.


For the others, either I don't know, or you said it.
 

Rich

Old Fart
Modder
Respected User
Donor
Game Developer
Jun 25, 2017
2,461
6,926
Oh, I clearly will. I don't know yet when (still have to finish the refactoring and port for my variable viewer), but I already put your code in my "todo" stack.



Also :
...

For the others, either I don't know, or you said it.
Thanks for the info on those. A lot of them I could have guessed based on the names, but figured I'd only state what I could see in the couple of files I tested. And some of the console features I didn't know about.

As always, you're the "Ren'py-in-depth" guru!
 

anne O'nymous

I'm not grumpy, I'm just coded that way.
Modder
Respected User
Donor
Jun 10, 2017
10,111
14,790
Now, having said, that, I don't know what this will do if you have managed to save a class of your own into the persistent file - might work, might blow chunks. But it was an interesting exercise.
I can answer you for this point: it do fucking nothing at all !

And I really mean, "nothing". It don't throw an error, nor show that there's something here.
I know it, I used it (with a really small variation) to look inside a saved file (the "log" one), because I encountered a corruption issue while trying my variable viewer. And well, I have to look inside the file with a text editor to find what variable was saved and what is exactly its class. For your dump (that is still interesting) the said variables wasn't shown.

So, now I have to reworks all the way I deal with the internal variables of all my tools and mods ; yes, I know, it's the base and I should have looked at this first ;) Pfff, long week ahead it seem.
 
  • I just jizzed my pants
Reactions: 79flavors

anne O'nymous

I'm not grumpy, I'm just coded that way.
Modder
Respected User
Donor
Jun 10, 2017
10,111
14,790
anne O'nymous, have fun with this - I know you won't be able to resist...
It needed time, mostly because I didn't really had an effective need for this before, but I encountered an annoying problem and needed to have a better look at the content of a save file, so... Here's my take:

Python:
init python:

    import zlib
    import io
    import cPickle
    import pickle

    # From console: lookSave( FULL_PATH_TO_THE_FILE )
    def lookSave( f ):
        with open(f, "rb") as file:
            file_bytes = file.read()

        if f.find( "persistent" ) != -1:
            decompressed_bytes = zlib.decompress(file_bytes)
            file_like_object = io.BytesIO(decompressed_bytes)
        else:
            file_like_object = io.BytesIO(file_bytes)

        result = cPickle.load( file_like_object ) if renpy.config.use_cpickle else pickle.load( file_like_object )

        FH = open( renpy.os.path.join( config.basedir, 'SAVE.txt' ), "a" )
        if hasattr( result, "__dict__" ): # persistent file
            for k in result.__dict__:
                FH.write( "{} -> {}\n".format( k, result.__dict__[k]  ) )
        else: # "log" save file
            # result is a tuple: ( store, rollback object )
            for k in sorted( result[0].keys() ):
                FH.write( "{} -> {}\n".format( k, t[k]  ) )
        FH.close()
Directly done inside Ren'Py*, so used directly with the right game there will be no issues due to unknown types.
It's a raw take that still need the "log" file to be extracted from the save archive ; I'm too annoyed by the problem I have to care about dealing with the whole zlib part ;)


[*]
We really need to stop forgetting that Ren'Py can be used as script interpreter and that we don't need external scripts to deal with this kind of situations.
 
  • Red Heart
Reactions: gojira667

sebbu

Newbie
May 2, 2022
49
6
I'ld like some help concerning _seen_*.

the _seen_translates (set variable) contains a nice label (it's defined in the translation file), so it's easy to edit, but the _seen_ever contains (dict variable) a tuple as key, containing the file (the translation one), a timestamp (it's not the one of the script file or the translation file), and a small number (slightly smaller than the number of lines in the current label), with True as value.

I'm not sure how to get the tuple values apart from checking in the console the content of _seen_* the first time before seeing the label and after seeing the label for the first time.

EDIT : the end-goal is to remove a specific line from seen labels, and possibly prevent it being added again, to prevent "game over", especially when skipping, so that i can backwheel.
 
Last edited:

anne O'nymous

I'm not grumpy, I'm just coded that way.
Modder
Respected User
Donor
Jun 10, 2017
10,111
14,790
the _seen_ever contains (dict variable) a tuple as key, containing the file (the translation one), a timestamp (it's not the one of the script file or the translation file), and a small number (slightly smaller than the number of lines in the current label), with True as value.
It's keys are ( File name, Dialog magic number, line number ) ; among other possibilities.

Python:
init python:

    def clean_seen_ever( filename, line ):
        for k in persistent._seen_ever:
            if k[0] == filename and k[2] == line:
                del persistent._seen_ever[k]
                break