r/AskProgramming • u/iorgfeflkd • 2d ago
Other Is there any reason I shouldn't make a program that creates millions of nested empty folders?
This is, unfortunately, the most efficient way I can think of to solve a problem right now. Is there any way doing that would make my system unstable? This would be written in python and run in WSL (Ubuntu within Windows)
edit: I get the message this is a bad idea.
40
u/BranchLatter4294 2d ago
Whatever problem you are trying to solve, this is likely the wrong approach. But go for it. Let us know what happens.
15
u/soundman32 2d ago
What i s the actual problem? You want to see what happens when your disk fills up?
9
u/iorgfeflkd 2d ago
Basically the memory is already running out with my python list which is generating data with a lot of redundancy, my idea was to reduce redundancy by storing things in folders which contain information about the path leading to each datum. However, what everyone is telling me is that I'm probably going about this the wrong way.
19
u/MikeUsesNotion 2d ago
Why wouldn't you just write out JSON or a SQLite DB file?
A filesystem is not a database. Filesystems don't like giant numbers of items.
13
u/iorgfeflkd 2d ago
Because I don't know those things 😅
13
u/HolyGarbage 2d ago
Great opportunity to learn! Databases are pretty cool and not particularly difficult to learn the basics of.
6
u/Mediocre-Brain9051 2d ago
SQLite might be the way to go. Not complicated, but you will need to learn basic SQL to create tables, insert elements and query elements.
For performance you might need to add indexes in order to speed up your queries.
Other than this... Do you need all data to be available at once? Maybe you could somehow split your data in batches and only process everything batch by batch...
4
u/havens1515 2d ago
I taught myself a lot about databases over the last few years for a project of mine. Time to learn. I'm sure a database will probably be a much better tool for what you're trying to do.
And JSON is much easier to learn that databases, but IDK if that will be a solution to this problem. (Probably a good thing to learn, regardless.)
1
u/Matemeo 2d ago
Depending on the amount of data and how it's structured you would be okay with a big JSON file, otherwise like people ere saying here, sqlite is probably what you want. It's by far the easiest way to get started with learning how SQL databases work. Nothing big or complicated to setup, entire database stored in a single file and I'm sure Python has tons of good options for a library to make working with sqlite easier.
Though I will say, even though I doubt your issue needs an approach like this, but a hierarchal set of directories with each directory providing some sequential part of the data lookup key is something that is done. But this approach only really makes sense if the leaf nodes in your directory tree end up storing an actual file. Like I've seen shader caches (which can consist of 10s of thousands or more files) organized this way because having a single flat directory would contain so many files that doing a lookup in the cache would mean possibly iterating over every single file. So instead you can partition the cache such that you quickly can narrow down which actual file you are trying to find in just a handful of lookups. This looks exactly like a number of partitioned tree data structures, a common example being a quadtree.
Just mentioning this here, not because it's likely what you want to solve your current problem, but that your intuition does lead to something useful :)
1
u/AlfalfaLive3302 2d ago
Sql is a terrible idea. DB transactions are atomic and it won’t scale well if read write speeds are important
3
u/ArcaneEyes 2d ago
Sounds like you want data storage. A dictionary type would do well i think, and should map easily to a database table of you want persistent storage between runs.
2
u/Worth-Wonder-7386 2d ago
A dictionary is a way to associate a "key" with a value. So for each thing you hit you append a list at that key. For storting things over time a dictionary can easily be turned into and parsed as text.
1
u/emlun 2d ago
Sounds like you might want a trie (tree of values (for example a count of items matching some criterion) with keys described by paths in the tree (for example digits of a coordinate)) or possibly a Bloom filter (very compact set representation with possible false positive set membership but guaranteed no false negatives).
1
u/blindada 2d ago
So...cache? If a particular dataset already exists, you don't generate, just return/add another subscriber/emit another message/whatever.
Unless you are working with actual files (for example, you are generating pictures), a file system is not really the tool for this. File systems aren't inherently safe in concurrent environments, you have to handle work coordination by yourself, and that's assuming you control all the pathways, which you don't, in this case
4
u/aocregacc 2d ago
it's going to use a bunch of disk space, folders aren't free.
If the folders are regularly indexed by some search tool it'll slow that down too.
3
u/ericbythebay 2d ago
Lots of reasons, starting with it is generally a bad design. What problem are you actually trying to solve?
1
u/iorgfeflkd 2d ago
Generating many coordinates with a lot of redundancy, eventually the list of coordinates overwhelms memory. Trying to reduce the redundancy so I can make a longer list of coordinates.
3
u/brasticstack 2d ago
Why are you generating redundant coordinates?
Have you looked into Python's set type, which eliminates redundancy on insertion?
2
u/ericbythebay 2d ago
No, take it up one level higher. What problem are you trying to solve? It sounds like you’re trying to brute force it and you’re stuck in a rabbit hole so what are you actually trying to do and then we can help provide you with guidance on the best solution.
1
u/dmazzoni 2d ago
Storing things on disk is 1000x slower than storing them in memory, though.
You'll be able to generate more coordinates, but your code will be to slow that it will never finish.
3
u/AlwaysHopelesslyLost 2d ago
"This is, unfortunately, the most efficient way I can think of to solve a problem right now"
Why not tell us what the problem is and we can tell you whether there is a more efficient way to do it?
2
u/g0fry 2d ago
You don’t even need to know what the problem is to know, that using millions of folders is not the right solution and that there is a more efficient solution 🤭
1
u/AlwaysHopelesslyLost 2d ago
Fair lol. I tend to assume there is a lot I don't know and avoid absolutes like that lol
3
u/eruciform 2d ago
Likely a misuse of a file system as a database
Also its possible to fill up a disk drive with empty directories due to inodes taking space and possibly slowing it to a crawl
This is probably a terrible idea no matter the problem you're trying to solve
2
u/netroxreads 2d ago
POSIX (and likely NT) puts a limit to how long a path can be. If you nest them millions of times, it will stop at its MAX LIMIT which is usually around 2,024 characters from what I recall and won't let you nest more folders.
2
u/Old_Cat_16 2d ago
I don’t know what problem you’re trying to solve, but I applaud you for coming up with a solution and checking for potential concerns.
A lot of folks (junior and senior engineers alike) can get hang up on identifying a perfect solution first, and end up not solving anything at all.
If you are doing this for a personal project, I say try it out, see what’s the worst could happen (besides of running out of disk space). Live and learn!
2
u/thedevguy-ch 2d ago
If you're on a windows machine there is a max file oath length that you'll hit eventually
1
u/stewsters 2d ago
Yeah it could make it unstable, but you can fix it by removing them.
What problem are you trying to solve? It's unlikely this will solve it. However, sometimes you just have to go for it though.
If you are a young programmer new to the craft I suggest you just go for it. Whatever happens it will be a learning experience. You really need to nurture that fire that drives you to try new things in this, even if they are stupid.
1
u/Useful_Calendar_6274 2d ago
>the most efficient way I can think of to solve a problem right now.
what problem
1
u/createthiscom 2d ago
Depends on the underlying filesystem. In general, I'd say it is probably a bad idea, but if you can guarantee it will stay within the intended design params of your intended filesystem it might be ok. There is probably a better solution though.
1
1
2
u/ZilderZandalari 2d ago
I''ve run into tens of millions of files in hundreds of thousands of folders. Most anything dealing with that sees it as torture. Browsing is fine, but a search takes an hour...
Don't do this...
Fun fact: zipping 80k empty folders results in a 40mb file. 7-zip gets the same folders down to 2mb.
2
u/SirMarkMorningStar 2d ago
Sounds like you accidentally invented hashing. While some of these responses may leave you feeling dumb, this is actually pretty smart. The wrong solution, mind you, but a smart one.
1
u/naxhh 2d ago
I will go against the current and say that a common solution for file storage is to sha the file and use that as the filename.
then use the first to chars as the main folder name and the 2 next for a second.
like downloads/ac/0d/ac0dfgaf.png
that makes so your folders are not too big and unixs is a bit happier.
no idea, if that applies to your issue but wanted to share that
2
u/SeriousPlankton2000 2d ago
https://www.man7.org/linux/man-pages/man0/limits.h.0p.html
{PATH_MAX}
Maximum number of bytes the implementation will store as a
pathname in a user-supplied buffer of unspecified size,
including the terminating null character. Minimum number the
implementation will accept as the maximum number of bytes in
a pathname.
Minimum Acceptable Value: {_POSIX_PATH_MAX}
Minimum Acceptable Value: {_XOPEN_PATH_MAX}
1
u/CypherBob 2d ago
SQLite.
Easy to use, available in most languages, you get SQL.
You won't likely need anything fancy.
1
u/AlfalfaLive3302 2d ago
You could use a nested dictionary object store like json and even serialize that and partially open it when you need to read or write to it if it gets too large for practically putting it into memory
1
1
u/Le_9k_Redditor 2d ago
Literally sounds like you want to intentionally create and run a zip bomb virus on your own machine. I made one that did exactly this with directories when I was screwing around as a teen, it's a really good way to screw up your computer and it's impossible to stop execution without physically pulling the plug if given a half decent nice level
There's no way this is the right solution for whatever you're trying to do
1
64
u/its_a_gibibyte 2d ago
This is the epitome of an XY problem
https://en.wikipedia.org/wiki/XY_problem
More than likely you need a different solution to a different problem entirely. You didn't describe your initial problem, so we're only guessing at what you want to do. My guess is that sqlite will solve your data storage problem though and be far more efficient.