|
|
Post by AnthroHeart on May 22, 2021 5:45:29 GMT
This subforum is for the exciting new Nesting File Creation Utility for use with the Intention Repeater. It creates a nested file structure, which significantly boosts the power of the Repeater. The latest is v2.0. It is available here: github.com/tsweet77/repeater-max/tree/main/Nesting/v2.0There is a NEST-FULLPOWER.ZIP file there. That's what you put in the Repeater. You have your intentions in the INTENTIONS.TXT file, outside of the ZIP. It has 1,000 main NEST files in it, with 10,000 repetitions per file. All this in a 468kB file. I used a maximum compression, which is still very fast to do. You'll see "-mx=9" in the .cpp code, which indicates the level of compression. reden , can you use this utility to create bigger sets possibly? I did it on my Raspberry Pi 4, and there are more powerful computers out there. This gives (# of files) ^ (# of iterations per file) total combinations. If a Googol is 10 ^ 100, this is 1000 ^ 10000. So it's way more. |
|
|
|
Post by AnthroHeart on May 22, 2021 5:51:37 GMT
Note, the Intention Repeater, as of Version 5.0 also can generate
Nesting Files through the --createnestingfiles.
Though it can't do .zip on the fly. That's what this utility is for.
You can choose whether or not (y/n) if you want a .zip.
|
|
|
|
Post by AnthroHeart on May 22, 2021 5:55:50 GMT
NEST-FULLPOWER.ZIP uncompressed = 131.3 MB
NEST-FULLPOWER.ZIP compressed = 454.4 KB
It compressed 295.9X.
Remember, this is created on the fly, so after a NEST-.TXT file is added to the .ZIP,
it is removed. So it never hits 100+ MB on your computer.
I did 1,000 files, because as it gets there it starts to really slow down on my system.
|
|
|
|
Post by reden on May 22, 2021 16:27:24 GMT
This subforum is for the exciting new Nesting File Creation Utility for use with the Intention Repeater. It creates a nested file structure, which significantly boosts the power of the Repeater. The latest is v2.0. It is available here: github.com/tsweet77/repeater-max/tree/main/Nesting/v2.0There is a NEST-FULLPOWER.ZIP file there. That's what you put in the Repeater. You have your intentions in the INTENTIONS.TXT file, outside of the ZIP. It has 1,000 main NEST files in it, with 10,000 repetitions per file. All this in a 468kB file. I used a maximum compression, which is still very fast to do. You'll see "-mx=9" in the .cpp code, which indicates the level of compression. reden , can you use this utility to create bigger sets possibly? I did it on my Raspberry Pi 4, and there are more powerful computers out there. This gives (# of files) ^ (# of iterations per file) total combinations. If a Googol is 10 ^ 100, this is 1000 ^ 10000. So it's way more. I have uploaded my 100k files, 10k repeats in each file to google drive: drive.google.com/file/d/171ZDie5Q3yWY7EskMSaiRdzJ59YTg5xr/view?usp=sharing That's the ultra compressed version. Normally it would weigh 48-50 MB, but the ultra compression makes it only weigh 1.7-2 MB. See if it still works. Here's the normal zip: drive.google.com/file/d/12EAfn-dNFYOL93zvvJOkS3agBuMRJeyF/view?usp=sharing . I did it manually using "zip". I added the files in batches of 5000 and 4999 each. |
|
|
|
Post by AnthroHeart on May 22, 2021 16:34:16 GMT
|
|
|
|
Post by AnthroHeart on May 22, 2021 16:35:48 GMT
This subforum is for the exciting new Nesting File Creation Utility for use with the Intention Repeater. It creates a nested file structure, which significantly boosts the power of the Repeater. The latest is v2.0. It is available here: github.com/tsweet77/repeater-max/tree/main/Nesting/v2.0There is a NEST-FULLPOWER.ZIP file there. That's what you put in the Repeater. You have your intentions in the INTENTIONS.TXT file, outside of the ZIP. It has 1,000 main NEST files in it, with 10,000 repetitions per file. All this in a 468kB file. I used a maximum compression, which is still very fast to do. You'll see "-mx=9" in the .cpp code, which indicates the level of compression. reden , can you use this utility to create bigger sets possibly? I did it on my Raspberry Pi 4, and there are more powerful computers out there. This gives (# of files) ^ (# of iterations per file) total combinations. If a Googol is 10 ^ 100, this is 1000 ^ 10000. So it's way more. I have uploaded my 100k files, 10k repeats in each file to google drive: drive.google.com/file/d/171ZDie5Q3yWY7EskMSaiRdzJ59YTg5xr/view?usp=sharing That's the ultra compressed version. Normally it would weigh 48-50 MB, but the ultra compression makes it only weigh 1.7-2 MB. See if it still works. Here's the normal zip: drive.google.com/file/d/12EAfn-dNFYOL93zvvJOkS3agBuMRJeyF/view?usp=sharing . I did it manually using "zip". I added the files in batches of 5000 and 4999 each. You could play around with my code in the new Nesting Files Repo I posted in another thread. It uses the multithreaded option in 7zip (2-cores). Version 2.1 is MUCH faster than before. |
|
|
|
Post by reden on May 22, 2021 16:46:08 GMT
I have uploaded my 100k files, 10k repeats in each file to google drive: drive.google.com/file/d/171ZDie5Q3yWY7EskMSaiRdzJ59YTg5xr/view?usp=sharing That's the ultra compressed version. Normally it would weigh 48-50 MB, but the ultra compression makes it only weigh 1.7-2 MB. See if it still works. Here's the normal zip: drive.google.com/file/d/12EAfn-dNFYOL93zvvJOkS3agBuMRJeyF/view?usp=sharing . I did it manually using "zip". I added the files in batches of 5000 and 4999 each. You could play around with my code in the new Nesting Files Repo I posted in another thread. It uses the multithreaded option in 7zip (2-cores). Version 2.1 is MUCH faster than before. I told it to create 1M files with 100k repeats each. Whole steps up. This is so easy and safe. There's no risk of the hard drive becoming filled up. |
|
|
|
Post by AnthroHeart on May 22, 2021 16:50:23 GMT
Funny thing, with Text, compression level 9 (max) produces same exact filesize of ZIP
as compression level 1.
So I believe I just left it off.
Does it seem like it slows down running longer?
|
|
|
|
Post by AnthroHeart on May 22, 2021 16:53:36 GMT
|
|
|
|
Post by reden on May 22, 2021 16:54:46 GMT
I added -mmt2 to the compression options. This makes it use two threads, and so it goes double fast.
|
|
|
|
Post by AnthroHeart on May 22, 2021 16:56:07 GMT
I added -mmt2 to the compression options. This makes it use two threads, and so it goes double fast. Cool. If you find more optimizations, let me know and I can add them to the next version. Also see if .zip vs .7z or other 7z formats are better for TXT. |
|
|
|
Post by reden on May 22, 2021 16:56:07 GMT
Funny thing, with Text, compression level 9 (max) produces same exact filesize of ZIP as compression level 1. So I believe I just left it off. Does it seem like it slows down running longer? You left it at 9. |
|
|
|
Post by reden on May 22, 2021 17:37:03 GMT
The buffer!
You have to make it so the program adds two, or more, files to the zip at once.
|
|
|
|
Post by AnthroHeart on May 24, 2021 9:18:16 GMT
I have released v4.0 of the Nesting File Creation Utility. It should be faster than ever.
You can find it in my gitHub, mentioned above.
There is only one bug that I know of. Though it is minor.
If you choose N to create zip archive, it still deletes all the files and creates a ZIP anyway
with just one P0 file. So choose Y to create an archive.
If you want the nesting files themselves, you can use the previous version until I get this bug worked out.
I changed, for example, NEST-0.TXT -> P0
P is for Power.
So 100 files would be P0 - P100.
P0 is the reference to INTENTIONS.TXT.
If you want to run the Repeater with Power Level 40, for instance, you would just put in the Repeater: P40
Or to run at full power, put in the Repeater: NESTFILES.ZIP
This is the fastest conversion tool yet. They should compress even faster.
When I can figure out how to remove spaces between the P0 and such in the files, that would be great.
If you look in the code, you'll see a CHUNK_SIZE variable. That is how many Power Files to write to the ZIP at a time.
I compile using what's in the code as well: clang++-9 -O3 -Wall -static ./Repeater_Nesting_Utility_4.0.cpp -o ./Repeater_Nesting_Utility_4.0
|
|
|
|
Post by reden on May 24, 2021 13:30:06 GMT
I have released v4.0 of the Nesting File Creation Utility. It should be faster than ever. You can find it in my gitHub, mentioned above. There is only one bug that I know of. Though it is minor. If you choose N to create zip archive, it still deletes all the files and creates a ZIP anyway with just one P0 file. So choose Y to create an archive. If you want the nesting files themselves, you can use the previous version until I get this bug worked out. I changed, for example, NEST-0.TXT -> P0 P is for Power. So 100 files would be P0 - P100. P0 is the reference to INTENTIONS.TXT. If you want to run the Repeater with Power Level 40, for instance, you would just put in the Repeater: P40 Or to run at full power, put in the Repeater: NESTFILES.ZIP This is the fastest conversion tool yet. They should compress even faster. When I can figure out how to remove spaces between the P0 and such in the files, that would be great. If you look in the code, you'll see a CHUNK_SIZE variable. That is how many Power Files to write to the ZIP at a time. I compile using what's in the code as well: clang++-9 -O3 -Wall -static ./Repeater_Nesting_Utility_4.0.cpp -o ./Repeater_Nesting_Utility_4.0 "If you look in the code, you'll see a CHUNK_SIZE variable. That is how many Power Files to write to the ZIP at a time." Will you adapt the "How many files to write at a time" question to this new CHUNK_SIZE variable? |
|
