Frontend Database and other random experiments

[ririmudev]

So if you're curious what this is about... I've been mulling the idea of a frontend database, as something to use in a game design. Since full-stack stuff is what I'm familiar with, I was thinking it could be cool to leverage a database. Either one database for a whole game, or one database per save file.

Basically, one could imagine the game as a web app using some javascript framework. You could go to the site, or just download the relevant assets. But it wouldn't be making API calls to a backend (because who wants their ero-game making calls to some server... not me). All good: can do this client-side.

If having one db across multiple saves, then you'd just need to religiously include a saveId column. So an actor table could be like:
{ "saveId": "00000001", "id": 1, "personaId": 1, "firstName": "Foo", "lastName": "Bar", "horns": null, "ears": "HUMAN", "stats": { "int": -1 } }

Appropriate use of libraries could then read and write to the "db", which is basically a different kind of save file. Besides this being a setup I'm familiar with, I was thinking there's an advantage that you can have large save files, with many actors, stats, & action history, but only a small amount of data needs to be in memory at any point in time, and it's quick to fetch whatever you need.

That opens some interesting doors. It would be easy to save data from all ero-sessions, and later an NPC can say, "It's funny how you never want a BJ. Always straight to the point, lol." Or mid-session, they might recognize a pattern and comment on it (a dev can code versions of that without machine-learning). Edit: this is of course currently possible for most games, but would scale horribly due to large save files, unless it's limited to a few characters.

Any thoughts?

BTW, I assume this will be a short-lived topic, so I made the title with "and other random experiments". I have a few other things I've been pondering, and might throw them onto this thread once the current item seems exhausted.

 

[Winterfire]

Edit: this is of course currently possible for most games, but would scale horribly due to large save files

???

Maybe I am too dumb to understand this thread, but I am absolutely sure that even if you are careless, a game is almost impossible to reach a point where the save size will become an issue.
Most of the size (the great majority) is due to the rollback anyway.

-edit-
And depending on the game/engine, thumbnail is also stored within the save, which contributes to the size, but it is meaningless when it comes to loading the data.

 

[ririmudev]

???

Maybe I am too dumb to understand this thread, but I am absolutely sure that even if you are careless, a game is almost impossible to reach a point where the save size will become an issue.
Most of the size (the great majority) is due to the rollback anyway.

-edit-
And depending on the game/engine, thumbnail is also stored within the save, which contributes to the size, but it is meaningless when it comes to loading the data.

Any dumbness is probably on my side. That aside, I think the big engines (renpy, rpgmaker) do a decent job with the optimization, and a lot of games are limited to < 30 or so actors, and a somewhat limited way of interacting with them. I'm thinking more the games where characters are generated, and you can have 100+ characters. Not too many of these, e.g. FreeCities (pregmod), Lilith's Throne, etc. But those ones do get performance issues - for other reasons to be sure, but there is a large amount of character data, and loading them up could be a chore if there isn't an option to load only what's needed.

Rollback is a good point - that goes with what I'm talking about action history. It's nice to store all that history, so you can build functionality drawing from it. Showing a log is one function, but NPCs making snide remarks based on your history with them is another. Or even remarks based on their interactions with other NPCs.

E.g., every turn they could have an interaction with someone, for example their neighbor, and when you ask them "What's up" the next day, they can say they're still mad about their neighbor's dumb joke. So a bunch of these can happen across a lot of random characters, and you can ask them what's up, and determine the best way to raise points with them that day. I dunno, something like that.

 

[Winterfire]

Any dumbness is probably on my side. That aside, I think the big engines (renpy, rpgmaker) do a decent job with the optimization, and a lot of games are limited to < 30 or so actors, and a somewhat limited way of interacting with them. I'm thinking more the games where characters are generated, and you can have 100+ characters. Not too many of these, e.g. FreeCities (pregmod), Lilith's Throne, etc. But those ones do get performance issues - for other reasons to be sure, but there is a large amount of character data, and loading them up could be a chore if there isn't an option to load only what's needed.

From my experience with my current project, I have a minimap where I store each visited tile as a vector3 (+bool), with hundreds of entries (500+, over a thousand if a player would explore every single map fully), and it is not even efficient as I could have used a vector2 instead. That's possibly one of the heaviest things I am storing.
Then I have characters (to be fair only 6), quests, unlocked abilities, achievements (persistent only), tokens, inventory and of course other variables you'd expect a game to have to keep track of choices made, and current progress in the story (about 50 of those, mostly bools).

All of this, without rollback, is about 100kb. Almost 1mb with rollback.
While I have never tested with a hdd, with a cheap ssd loading 1mb is almost instantaneous, back when I kept 128 steps, it would be close to 4mb, which took a few seconds instead.
This is my experience with Unity (Naninovel Framework), but at least Ren'Py keeps track of the rollback waaaaaay better, so you can exclude the issues with too many steps if you have Ren'Py in mind, not sure about RPGM though.
This also while keeping in mind that I could optimize the saving further (by switching to vector2 for starters).

It is safe to assume that the great majority of games will load and save that much data instantaneously, especially if you optimize it, but for those few games that despite everything need to save 5mb or more of data (See Skyrim or other big games for instance), you can just throw a loading screen while loading, or a spinning icon or something to let them know that the game is currently saving and they shouldn't quit just yet.

When it comes to performance, the issue is almost never saving. Unless they keep writing data on the disk, but that'd be the dev's programming at fault. The only thing I can imagine that would impact performance would be a small freeze when autosaving, again if the data is big (5mb+).
For all the rest, save data is not the issue if those games have performance issues.

 

[JakaiD]

Save files do not have to get that big that you need to worry about it if done correctly. Ex, do not use Int for values when Bytes can store it.

The DB idea is cool for use in your data/assets/resources but I guess that would also need to tie in with whatever engine you use. Probably better to just stick to what an engine already has available when working with something like Ren'Py or even RPG Maker.

Engines like Unity, Godot, or Unreal is a different story since with these you would have to decide how to handle the resources and save files. But even in these cases I would stick to things like ScriptableObject (Unity) for data. For save files I use

You must be registered to see the links

a lot these days since it is really fast and can create small files (even includes compression support if needed). The game I am working on now generates random characters (25 in the stage at a time) and the largest save file is 4.81Kb - not compressed but using appropriate values types and msgpacks's binary serialization in action.

 

[ririmudev]

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

Thanks; I'm not worried about save/load performance - if it takes some seconds, then a loading screen like you mentioned is fine. I'm thinking more about in-game performance...

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

...continuing my response, big save files is basically my objective. The experiment is with having history between any character with any other character, and so characters will have their own stuff going on in a way that isn't predetermined (but not simply RNG either), and the gameplay would partially be about figuring out what's up and responding accordingly.

But it's maybe too ambitious; closest analogy would be The Sims (or LifePlay if the NPCs had more... life). I'd need to have functions that are like: when spawning an NPC, generate some backstory by creating events that happened in the past. Then at the end of each turn, create some events that happened between them and a couple other NPCs. E.g. getting in an argument with a neighbor.

So far, none of that adds to gameplay. There needs to be more functions that give life to that event history. For example: "be in bad mood" if NPC had x arguments over y days (unless personality enjoys arguments). Lots of permutations would be needed to take advantage of historical events and make things lively, although by exposing relevant parts to modding, maybe it could be made pretty fun over time.

 

[danarchist_lewd]

I don't think the issue here is actually with the amount of data being stored, but actually finding a way to create content that is reacting to all the data you're creating in a way that is compelling. The way I've seen many games do this (and that I'm trying to implement) is basically creating a director that takes in all the data and uses it to create narrative that is actually compelling. Games I've seen do this are Left 4 Dead, Hades, and The Mortuary Assistant (amongst others).

Unless you're making a sandbox game like The Sims or Dwarf Fortress.

 

[anne O'nymous]

That aside, I think the big engines (renpy, rpgmaker) do a decent job with the optimization, and a lot of games are limited to < 30 or so actors, and a somewhat limited way of interacting with them.

It's not a question of optimization, nor a question of number of actors, but a question of effective size of the data. And you clearly highly over estimate their size.

As reference, I have a small homemade software to manage the games I follow, have tried and all. Right not it have 2106 entries, that have 44 fields that goes from a simple boolean to long text describing the story. The "save file" is 2.84 MB, and it's a serialized and prettied file, therefore full plain text that include the fields names and is full of white spaces. The whole software (in Perl/Tk totally not optimized) use less than 50 MB in the RAM, and you can easily assume that more than 48 of them are for the software itself.

Python as used by Ren'Py prior to the branch 7.x, able to manage 2 GB of RAM, in the branch 7.x, it's 4GB, and it think that in the branch 8.x it's as much RAM as is available. Said otherwise, Ren'Py have always be able to manage more than 1000 times the size of my 2106 entries serialized database.
If you want more proof, my variables viewer is able to process more than 20 000 variables at once. Not when it display it, there's a way lower limit on this, but live stored on the game memory and all directly addressable. Each one of them being a copy of the variables used both by Ren'Py and the game.
Therefore, to keep your game with auto generated characters, even if you had 400 of them, and each one had 100 fields describing them, it wouldn't be a problem for Ren'Py. And like Python is a script language, and one of the slowest, it also mean that it wouldn't be a problem for whatever game engine you want to use.


So, sorry, but using a frontend database is just ridiculous.
Not only it fix an issue that don't exist (RAM limitation), but it also make the variables not directly available, what slowdown the game. Because yes, all variables need to be directly available all the time, you never know when you'll have to address one of them.

 

[Tompte]

Pardon my frankness, but this sounds overly complicated.

Because web apps are orders of magnitudes slower and less efficient compared to native apps, I doubt you'll find many good ideas or practices from that field for use in game development (unless, of course, it has to be a browser game). You'd be surprised how quickly you can load gigabytes of data into memory when not having to go through a browser, or json, and knowing a little bit about how memory works.

Besides, databases were made for quick random access from large data stores. The data you'd be storing would neither be large nor random. Game data tends to be highly structured (i.e. all of your pieces and their properties are well defined in advance). Also, you would probably want to read all of it, not just individual bits and pieces.

If you need performance in high numbers, you should probably look into some kind of client-side programming language and stay away from the web stuff, imo. 100 isn't exactly a big number on my side of the fence.

As a real-life example, in my game, during startup, I load pretty much all of the game's content (several thousand lines of text, images and data structures in XML), and index up to many hundreds of user created character files (~50mb each with image data) in about 300 milliseconds, yet I still think it could be improved. The majority of that time is spent converting image data into textures, not so much from reading files.
Saving and loading the game is instantaneous. (~50kb compressed data, containing the entire game state).

tl;dr:
You shouldn't need a database for trivial things like basic file I/O when you're all client-side. You may just be used to things being slow.

Sorry, I spent way too long writing this. Feel free to ignore XD

 

[ririmudev]

Jeez, I get it already. Back to the laboratory

 

[Winterfire]

Jeez, I get it already. Back to the laboratory

What are you trying to do, exactly?

You mentioned in-game performance, which has nothing to do with saving and loading data (aside from small freezes due to autosaving if the data is too big, such as 5mb+).
If you want to have hundred of characters on screen (say 2d ones, from the games you have mentioned) with a single call to the GPU, you could just use an atlas rather than having 100 separate calls (which is still a good amount for a total of calls).

Otherwise if it is something heavy like pathfinding, you can use multiple threads and/or optimize the code by avoiding repetition (If the obstacle is static, you only need to find it once).

 

[ririmudev]

What are you trying to do, exactly?

You mentioned in-game performance, which has nothing to do with saving and loading data (aside from small freezes due to autosaving if the data is too big, such as 5mb+).
If you want to have hundred of characters on screen (say 2d ones, from the games you have mentioned) with a single call to the GPU, you could just use an atlas rather than having 100 separate calls (which is still a good amount for a total of calls).

Otherwise if it is something heavy like pathfinding, you can use multiple threads and/or optimize the code by avoiding repetition (If the obstacle is static, you only need to find it once).

A bit belated, but I should have replied; thx for the response.

As for what I was trying to do, it was premature optimization, though to be fair, I was thinking I would be supporting people with... "not-powerful" devices (like the ones that ask for android version and receive a reaction). I definitely don't need optimization for the machines I use. I'm not planning to spend a lot on visual resources (at most a few SVG sprites), but the history stuff I mentioned shouldn't be underestimated; the Umsatz is: saving a bunch of stuff that usually isn't saved. I still think the frontend DB idea is cool and may do it anyway, but it made sense to back-burner it.


Anyway, reason for thread-necro is the next set of experiments: I'm thinking of a game-loop combining day play and week play.
1. Sunday: day play. Do actions, each action passes time, until no more time left in day. Fairly common approach.
2. Week planner: at end of Sunday, plan week (Mon-Sat). E.g. Mon-Fri, 4 slots working, 2 slots skill-building. Sat: 4 slots skill-building, 1 slot hanging out with a known NPC, and 1 slot relaxing at the tavern.
3. Execute week: skips to next Sunday; events during week may trigger (e.g., if progress to promotion accumulates due to working regularly, a promotion event would trigger)

Week-play is somewhat inspired by Free Cities, which plays week-to-week pretty well. Also, for certain things that might take 9 months, you don't want to click "Next day" 270 times.

Tutorial is built-in: for first Sunday, there will hardly be any actions available; maybe it starts with some NPC giving exposition, and then you choose a job. If hour-of-day matters, then some jobs are 9am-5pm, whereas others might be night-shift, and you'd week-plan accordingly (I haven't decided if hour-of-day matters; might be complex for a first iteration, hence why I simply said "slots" above).

Edit: I mentioned day-play and week-play off the top of my head. If they already mean something, such as fapping to a calendar, let me know.

 

[osanaiko]

The curse of the software developer: temptation to use your existing Hammer (web dev experience) to try and do a project (VN or RPG Hgame) which requires a screwdriver.

Savegame handling is a solved technical problem if you use a framework like Renpy or RPGM. Creative energy should be directed toward making a interesting erotic story with fun gameplay and attractive images.

goodluck!

 

[ririmudev]

Meh, efficiency is for the 9-to-5 (or 10-to-3, including nap time). After hours, it's Hammer time.

 

[ririmudev]

Well... a tech demo is deployed. I guess it's only been a month... thought this was a necro.
1. Oof... getting it deployed was a few hours. Half of it was reviving my 2 yr old server in AWS.

2. Anyway, link is as usual:

You must be registered to see the links

. Needs https; I'll get around to it eventually. I don't know the first thing about that joiplay site, but maybe that could be an easy hosting option.

3. Newish browser is needed, since the target is ES2022... I could lower it, but then it won't be a trim 550kb.
Text only, probably for at least 1 yr.

4. There's a View Capabilities button in there that gives a run-down of the systems, but in short:

1. Relation system: includes knowledge repo across actors. Fine-grained: possible to know appearance, but not name...
2. Item system: can trace a lot. You could receive item from P2, who got it from P3, who stole it from P4.
3. Translate system: if digging in the assets, you'll notice an english.json and german.json. Will eventually need heroes to make it not-terrible. No need right now; it's still early.
4. Shell - just a silly library that looks like a linux shell. Will ultimately get replaced.
5. Bunch more; this is probably less than half.

5. Plan: soon will work on the secret lewd system. After that will be polishing the genetics system; don't want to be shooting blanks after all. Then adding the events system. Then one of three main directions, which shall be kept quiet for now. Or not; as always, expect nothing from me. Maybe my next tech demo will be in 3 years.

6. Screenshots below:

Spoiler: Couple more images

You don't have permission to view the spoiler content. Log in or register now.

 

[osanaiko]

Mad respect for getting this far dude!

 

[ririmudev]

View attachment 2624992

Mad respect for getting this far dude!

Crazy S.O.B... I could get used to that