Sepulchre, to you specifically:
You might not believe this, but, I came up with YET another engine, MUCH faster than before, but not THAT much different... worlds faster though, hugely!
(& it MAINLY depended on IF I used a sorted array or not. The engine I was using before/earlier (variation & improvement on that one I put the source out to here a few pages back even) is equally good on sorted, vs. unsorted arrays, whereas this even NEWER one is only good on sorted ones)...
Yes - you probably recall this from academia in working w/ stringlists (just took me a bit to 're-realize' it again is all), or sorts, in the course "datastructures"...
See, when the array IS sorted? I quite easily get a 1000x fold speed increase, IF NOT MORE, quite literally by using a SMALL variation in my last motor/engine/algorithm + sorted arrays!
E.G.'s (after some time studying it all, via ProcessExplorer & other tools), & FROM THE ORDER OF OPERATIONS YOU MAKE USING IT:
-------------------------------------------------------
IMPORTING HOSTS FROM OTHERS & "FORMAT/SCRUB/CONVERSION" PHASE (under NEWEST GUI engine) FROM 127.0.0.1 OR 0.0.0.0, TO PLAIN 0:
-------------------------------------------------------
250,000 RECORDS 1.5 MINUTES...
-------------------------------------------------------
LOADING (under NEWEST GUI engine) FOR OTHER PROCESSING:
-------------------------------------------------------
250,000 RECORDS 28 SECONDS
-------------------------------------------------------
SEEK-REPLACE/ADD/REJECT ENTRIES (under NEWEST GUI engine):
-------------------------------------------------------
50 SECONDS TO ADD NEW ENTRY or CHECK FOR PRE-EXISTING ONE (vs. one you are trying to add)
-------------------------------------------------------
NORMALIZATION (under NEWEST GUI engine - "HUGE IMPROVEMENT" & in its slowest part too):
-------------------------------------------------------
250,000 records NORMALIZED/NO REPEATED LINES IN THE FILE, in 9 minutes time... with a group of HIGHLY repeated items.
250,000 records NORMALIZED/NO REPEATED LINES IN THE FILE, in 30 minutes time (as opposed to 10 days time from the last engine I put up here, which worked but, took WAY too long)... with a group of NONE (& one with nearly none) repeated items (testing total vector time this way).
RAM consumption varied with HOSTS that big & during processing from 67-157mb of RAM max range across diff. HOSTS.TXT file sizes - & as low as 540k... 118 hWnd's (window handle contexts), + 2 Thread multithreaded designed single monolithic .exe file, running on an AMD Athlon64 X2 4800+ CPU @ 2.6ghz stock/oem speed, no overclocks, 512mb PC-400 DDR Ram onboard.
-----
* Brainstorming, & observing: IT PAYS OFF - THE SCRUB, LOAD, & SEEK-REPLACE/ADD/REJECT ENTRIES AREA? They're the same, maybe a BIT faster...
HOWEVER - The 'artist formerly known as "the slowest part"' (in NORMALIZATION (process of removing repeats) STAGE)? Ha - a HUGE improvement resulted, huge orders of magnitude in less time taken by far!
(Now a "little something" to note: I have what is probably very near to the largest HOSTS file out there for public consumption, that is fully normalized... mine is only 533,576 entries (yes, double this dataset, but I would just double its time then to compensate in a LINEAR fashion (most likely its somewhat shorter actually))
THUS... max, a 15 minute - 1/2 hr. wait over a normal-to-largish HOSTS file weighing in @ about 10mb/250,000 entries, & most folks won't have 1/2 of THAT mind you? That small wait, for a perfect no repeats hosts file!
(& you end up w/ a HOSTS file w/ the least RAM consumption YET SAME BLOCKING FUNCTIONS for safety + speed, & diskspeed efficient while loading internal format possible inside the file itself, fully alphabetized & normalized too)...
APK
P.S.=>
Heh - world's faster now, & BEST OF ALL, in its formerly slowest part (bigtime, would take days to do the same job this one does in 20 minutes time max)... happy boy here now!
(Soooo, @ this point - I truly am pretty fairly convinced @ this point that I can do no better now, short of going pure DOS/Console mode/tty terminal type of apps, or using inline ASSEMBLER dumps inlined into my code @ this point... as fast as she gets I think, by this point)... apk