Context of the problem:
- I have made chess GUI (Java)
- The GUI is capable of loading chess puzzles/problems to solve
- Of said puzzles, i have gotten my hands on a database which is just shy of a million entries
How does one most efficiently go about getting a random puzzle from this database?
Obviously keeping the database in memory is not an option, despite it already being compressed. Stripped all data that isn’t needed from it and in the end converted it into byte-arrays. All to no avail. The full database always ends up taking up somewhere between 100 and 200 MB of memory. A tenth of that would be acceptable (for my purposes).
Even worse: When processing the entire database (in attempts to keep it all in memory), the process file->memory took upwards of 700 MB memory.
Let’s say i have the following:
- The database as either a txt or csv file
- The amount of lines in said file (which is equal to the amount of puzzles)
Am i with that, in some way, capable of grabbing either a random or specific puzzle from the file (albeit async, that doesn’t matter) without having to load the entire database into memory?
Thanks for your time!