Reply
Old Jun 26, 2008, 01:04am Default Re: Today !
Junior Member

Posts: 29
Karma: Kanolo1 is on a distinguished road
Today I buy a lot of furniture for my new store.
Oh, it take a lot of money.
__________________
Diet l Business l Automotive l Technology l Online Store
Kanolo1 is offline
Reply With Quote
View Public Profile Visit Kanolo1's homepage! Send a private message to Kanolo1
 
Old Jun 26, 2008, 02:32pm Default Re: Today ! "THE ENGINE IS 'NEARING COMPLETION', @ last"... apk
APK
APK's Avatar
Elite Member

Posts: 354
Name: The Duke of URL
Karma: APK will become famous soon enough
Today, I "knock the 2 of 3 slated tasks out", from my 2nd list below (since it's raining outside, & that means NO cutting the grass, OR, washing my car)!

Here is how it appears, thusfar:



-----

This program will {COPYRIGHT APK - Alexander Peter Kowalski 06/01/2008 onwards}:
  • 1.) Build you a CUSTOM HOSTS FILE that is in * THE * MOST EFFICIENT FORM THERE IS (0<singlespace>URL<cr+lf>) line-by-line format, so the file loads as FAST as possible (since it is tinier on disk, e.g.-> going from 127.0.0.1 to 0.0.0.0 alone? Well, I went from 19mb to 14mb - & THEN, in going from 0.0.0.0 to 0?? I went from 14mb to 11mb size on disk), AND also consumes the LEAST RAM in your local DNS cache...with NO repeat entries & alphabetically ordered.
  • 2.) It builds the CUSTOM HOSTS FILE that secures you vs. the threats on bad sites &/or adbanners out there too
  • 3.) Also, in addition to speeding up your websurfing (by NOT loading adbanners (again, especially malware infested ones) & such) - it will also optionally speed-up your fav. websites, by your hardcoding your fav sites into it, as far as IP to URL conversions - doing this, locally, is FAR faster than calling out to DNS servers (which can be poisoned, or "man-in-the-middle" attacked, too)).
  • 4.) It will graphically allow you to search thru a HOSTS FILES' listings (mine are done in alphabetically-arranged order, so if you choose to do them in notepad.exe even, it is simpler) in their "sites to block" list, for adding in new ones (or, it will reject them as already existing, preventing repeat entries, aiding further in "normalization" (lack of repeats)).
  • 5.) It will also, in BOTH the original list to process (on left) & in the "FINAL LIST" (on right) for save to disk, allow you to selectively remove elements in its lists, @ YOUR discretion, prior to saves to disk (easy to do, via rightclick on list popup menus)
-----

I took out the progressbar, it was a LOT of graphic & cpu overhead, vs. the simple text output of it noted above (what record it is comparing to the rest of the list dataset array, to remove repetitions for normalization)

PLUS, ProgressBar's properties (vs. listbox count, for total max array elements) no longer "meshes" w/ the code loop engine I redid (diff. than shown here earlier on last page now in fact, even BETTER!))...

THUS, THESE 2 TASKS * SHOULD * GET DONE, TODAY:

-----
  • 1.) The ability to PING a list of sites in a .txt file, for folks that like to use the HOSTS file to speedup access to their fav. websites (which I for one, do, since it takes less time to get an IP resolution from your local DNS cache OR the HOSTS file, than it does taking time to call out to a DNS Server, many orders of magnitude faster in fact (I have this code already having written MANY apps like it) into another tab - &, to "merge it" with the blocking section format scrub output file from #1 above also!): ADDITIONALLY, this very thing, can secure you vs. DNS poisonings &/or "man-in-the-middle" attacks too!
  • 2.) Storing my PREBUILT 11mb HOSTS file in another tab & memo field for save to disk overwriting ones folks have stored, as an option, for save to disk (either I will do it THIS way, OR, store it as a programmatic resource to extract, which I do in my APK Doctor Who ScreenSaver (w/ its .avi file it plays)), as a "CUSTOM HOSTS FILE" folks can use as a GREAT starter one for themselves, if need be (it's VERY complete/comprehensive).
-----

LEAVING ONLY THIS 1 TASK, REMAINING:

-----
  • 3.) Allow the option (in the "file format scrub" portion, topmost button) to generate HOSTS files that use either 127.0.0.1 (slowest/largest option, BUT most compatible), 0.0.0.0 (next largest/slowest option, less compatible across diff. OS', but, is faster loadtime & smaller RAM occupancy in the local DNS cache your OS + tcp/ip stack keeps than 127.0.0.1) & lastly a NEW more efficient format I am using ( 0<singlespace>URLtoBLOCK<cr+lf> ) that actually WORKS for bad adbanners/sites, as it quite literally shaved my HOSTS file down from 14mb size, to 11mb, already, yet yields FULL blocking function!
-----

(And, this last remaining task? It's far from "impossible to do" (the base code is MOSTLY the same, repetitive, just diff. things to replace) - It's more just TIME-CONSUMING (in testing the data outputs is all - bit 'touchy'))...

-----

* Nicest part is, so far?

The dataset gets SMALLER each run, & thus, like a "TESLA TURBINE"? It gets MORE EFFICIENT, each iteration of the loop, the more data you "shove thru it"... This happens, because since each time it runs, it grows smaller (just "good physics", in its engine/algorithm, really)!

E.G.-> This test dataset run? Was operating @ 2005 records per minute (@ the start, mind you), & FAR faster once it passed the "midpoint" of the dataset/array in the lists above!

(Which I use (listbox controls) for "graphic debugging" (so I can literally SEE what is going on) as 'dynamic arrays', vs. non-gui arrays))

That is - for now: See, I know if I did this in charmode/DOS tty term mode, OR, using "true non-visible dynamic arrays"?? I would have literally 10x this speed (did already w/ sepulchre in this thread, showing him that optimization method)... but, nobody today wants console mode apps though!

(+, the more repeats there is? The FASTER it operates there, too - because as it vectors thru the array, it pulls HUGE swathes of repeats, & the more of them there is?? Faster she goes, because the dataset gets smaller by removing said repeats (during normalization - the process of removing repeats))!



(Heh, it's "ALL IN KEEPING WITH MY SIGNATURE" below, lol - this thing's like a dog (man's best friend) albeit online! To quote Reese from the film "THE TERMINATOR" - "Dogs... we use them to spot terminators..." & sometimes, imo @ least?? Dogs are BETTER PEOPLE, than people...)

APK

P.S.=> So far, since it is largely about string comparison, via math (which Delphi EXCELS @, more than 2-3x as fast as MSVB + MSVC++ even, in BOTH strings &/or math processing work)?

I am also using the VERY BEST & FASTEST language combination I know of for this type of work (Delphi 7.x optimized by hand AND compiler code, CPU priority assignments (optional), Win32 API code, & inline Assembler code)

-----

In fact, the last version of this program I did (circa 2002 or so)? Took me over 150++ lines to do what I am NOW doing, in under 20 lines of "active ingredient" code... pretty cool!

(That only tells ME, personally, that I have 'grown' (gotten better @ this stuff), since then in the past nearly 7 intervening years!)

Neat... THUS, programmatically/algorithmically? She's turning up JUST how I was hoping it would (i.e.-> pretty fairly "elegant" & efficient)... apk
Attached Images
File Type: jpg APKHostsFileRewriter4.jpg (93.7 KB, 33 views)
__________________
"I'm Reese: Sgt. TechComVN38416, assigned to protect you - You've been TARGETTED FOR TERMINATION!"

Last edited by APK; Jun 28, 2008 at 10:54pm.
APK is offline
Reply With Quote
View Public Profile Visit APK's homepage! Send a private message to APK
 
Old Jun 27, 2008, 06:56pm Default Re: Today !
DaveMo~'s Avatar
Witty Title

Posts: 1,497
Name: Dave
Karma: DaveMo~ is a jewel in the roughDaveMo~ is a jewel in the roughDaveMo~ is a jewel in the roughDaveMo~ is a jewel in the rough
My today was much simplier....work on here, sleep a lot, fool around.

Dave
__________________
Trying this out: My Dollar Store Let me know what you think!


DaveMo~ is offline
Reply With Quote
View Public Profile Visit DaveMo~'s homepage! Send a private message to DaveMo~
 
Old Jun 28, 2008, 07:39am Default Re: Today !
Babe

Posts: 2,477
Name: Cindy
Karma: Cat-tranz is a jewel in the roughCat-tranz is a jewel in the roughCat-tranz is a jewel in the rough
Oooooo
Quote:
fool around
__________________
New members - before you post please read the TNT Rules
Cat-tranz is offline
Reply With Quote
View Public Profile Send a private message to Cat-tranz
 
Old Jun 28, 2008, 10:10am Default Re: Today ! 1 last thing "2 due"... & "the engine's done"
APK
APK's Avatar
Elite Member

Posts: 354
Name: The Duke of URL
Karma: APK will become famous soon enough
ALL THAT is left for the program above to be done:

-----

3.) Allow the option (in the "file format scrub" portion, topmost button) to generate HOSTS files that use either:
  • 127.0.0.1 (slowest/largest option, BUT most compatible)
    - gave me a 19mb sized HOSTS file
  • 0.0.0.0 (next largest/slowest option, less compatible across diff. OS', but, is faster loadtime & smaller RAM occupancy in the local DNS cache your OS + tcp/ip stack keeps than 127.0.0.1)
    - gave me a 14mb sized HOSTS file
  • OR, a NEW more efficient format I am using ( 0<singlespace>URLtoBLOCK<cr+lf> ) that actually WORKS for bad adbanners/sites, as it quite literally shaved my HOSTS file down from 14mb size, to 11mb, already, yet yields FULL blocking function!
-----



* JUST ABOUT DONE - very close (& I have forgotten how much work even freewares can be @ times!)

APK

P.S.=> This has taken me a lot longer than I had expected even though I have (for example) the PING code already from many apps like that I have done before that use such code (ICMP.DLL functions for ping)...

Cutting & pasting it from those said other programs has a downside:

I have to account for the fact I built in a lot of things SPECIFIC to the app it was in initially, & that means cutting those things out for THIS app (a pain!), but, it's coming along perfectly, thusfar, & DONE RIGHT, in each of its sections/functions... apk
__________________
"I'm Reese: Sgt. TechComVN38416, assigned to protect you - You've been TARGETTED FOR TERMINATION!"

Last edited by APK; Jun 28, 2008 at 11:33am.
APK is offline
Reply With Quote
View Public Profile Visit APK's homepage! Send a private message to APK
 
Old Jun 28, 2008, 01:50pm Default Re: Today !
Christopher's Avatar
Iced Cap

Posts: 652
Name: Christopher
Karma: Christopher is a jewel in the roughChristopher is a jewel in the roughChristopher is a jewel in the roughChristopher is a jewel in the rough
Nice work, APK. How long have you been developing Windows apps?

I'm working on a web app at the moment, designed for web companies to handle clients, projects, products, downloads, payment gateways etc. It will be my company's first actual product
__________________
:: Devlog - New developer blog with useful PHP information
:: WeTalk.tv - TV Forums
Christopher is offline
Reply With Quote
View Public Profile Visit Christopher's homepage! Send a private message to Christopher
 
Old Jun 28, 2008, 05:48pm Default Re: Today ! The "GriNdEr" goes for testing...
APK
APK's Avatar
Elite Member

Posts: 354
Name: The Duke of URL
Karma: APK will become famous soon enough
DONE!



It ALL just works... looks only a BIT diff. than screenshot before (as it now has a dropdown list of your fav sites to ping to help speed them up is all, added now & yes, the Progress Bar is back (probably trayiconized & the "whole 9 yards", later, & @ the "pretty'ifying stage"))

(DONE - Except for the option above, which other than for "flexibilities' sake" is foolish imo, really - as there is NO HOSTS FILE entries format more efficient than the 0<space>URL<cr+lf> format I currently use)

HOWEVER:

For the sake of flexibility, the option to build 127.0.0.1 & 0.0.0.0 types will be built in thru/over the next couple days, tops...



* NOW, it's going off to my pal Jack via email, for "end-user" level testing (useability mostly, because he understands the "internal mechanics" of a HOSTS file, as is, from my turning he onto them earlier/late last year, to present)

APK

P.S.=> I renamed it to "APK HOSTS File Grinder 4.0++", because that IS, what it does... ANYTHING "HOSTS FILE" imaginable, really, & just "grinds it out" as it does:
-----------------------------------------------------------------------------------------------------------------------------------------------
  • EASY Integrating the HOSTS files of others, even if in other internal line-by-line formats, into the MOST EFFICIENT format that is, first, & then...
  • Allowing you to speed up access to your fav sites, via 1st pinging them, & adding them to the normalized non-repeat line items list on the right above
  • Add/remove sites from a hosts file, but by first checking for their pre-existence inside the HOSTS file on ADDS, & rejecting if there already (& adding if NOT present)
  • Lastly, it will FULLY NORMALIZE (accurately 110%) a HOSTS file (removal of duplicates)...leaving you with one in the MOST efficient format line-wise there is (consumes less memory & faster loadtime from disk)
-----------------------------------------------------------------------------------------------------------------------------------------------
{COPYRIGHT APK - Alexander Peter Kowalski 06/01/2008 onwards}

Been fun building it too, good review & neat "documenting its development" + even discussing it here @ times w/ Sepulchre (fellow coder) too!

(AND, of many Delphi programming topics)

...& personally speaking? I really needed a tool like this, because I use a custom HOSTS file for speedups online (visibly humanly perceptible ones, not just ones a benchmark gives you in nanoseconds) & security online too...

This tool beats the combination of Access & SQL "SELECT DISTINCT" queries + File Imports/Exports issues, + pinging in console/dos tty mode, & trying to do search/replace using notepad.exe even...

(By a mile, during normalizations especially, AND importing the HOSTS files data of others too, putting it into a single LARGE comprehensive HOSTS file, in the BEST format that there is for loadtime AND memory consumption (least on both counts))

I sit back, watch a flick (do the laundry, mow the yard, wash the car - you know it, it's doing the work for me, in the background - like doing laundry, multitasking)... apk
Attached Images
File Type: jpg APKHostsFileGrinder40.jpg (102.1 KB, 41 views)
__________________
"I'm Reese: Sgt. TechComVN38416, assigned to protect you - You've been TARGETTED FOR TERMINATION!"

Last edited by APK; Jun 30, 2008 at 07:14pm. Reason: Finalized it, photograph of diff. look (vs. one in post earlier above on same page) is visible/notable... apk
APK is offline
Reply With Quote
View Public Profile Visit APK's homepage! Send a private message to APK
 
Old Jun 28, 2008, 06:38pm Default Re: Today !
APK
APK's Avatar
Elite Member

Posts: 354
Name: The Duke of URL
Karma: APK will become famous soon enough
Quote:
Originally Posted by Christopher View Post
Nice work, APK.
Thanks - it's a start! AND? The feeling of a "job well-done" imo, thusfar @ least??

Is going GREAT w/ Gwen Stefani's/NoDoubt's "'KEEP ON DANCIN'", which I am listening to, now as I write this in reply to you (lol, in celebration - even getting some ideas on doing more w/ it too...)

----------

I actually have an idea (bigger idea) for this, @ this stage... believe-it-or-not:

... & that is that this very program is the base engine that will come w/ the package, & more, for more security online & YET WITH BETTER SPEED TO GO WITH IT, which HOSTS files can & DO give you, both of!

(Updating it (the program itself, OR, regular HOSTS files updates) either ftp &/or an HTTP GET, done via a trayicon resident system (this program, actually, why not? It's pretty light, & F A S T too))

For:
  • Checking for updated HOSTS files, updated bi-weekly @ least usually, to keep you safe(r) online...
  • Then, downloading them (IF present server side, indicating update is ready for D/L etc.)
  • Plus, lastly, installing the newly updated HOSTS file downloaded, or the program itself (vs. spyware/virus/spam/trojans/rootkits (et al), for you <- sight unseen/transparently (it's a fairly fast operation on a single file))!


(... @ that point, imo, the "big hassles" then mostly would be fielding questions via email & taking + making requests etc. et al @ that point))

----------

... I am going to call it "The Achilles Shield" Copyright APK - Alexander Peter Kowalski 06/01/2008 to present:



... & like in the photo above?

"It's GOT YOUR BACK!'...

Something 'catchy' is needed (or, catchier)... I dunno? How about a slogan like "You can't get burned if you don't go into the kitchen!" <-The very basic principal it works on in fact - sayeth 'The Armored Saint'... lol, I dunno!

(You've gotta start somewhere, & w/out goals + dreams, what's the point, right? I've got the foundation @ this point I think... rest is up to me!)

E.G./I.E -> In myself making it as USER-FRIENDLY & simple to manage by end-users, as is possible, w/ as much help as possible in the program, itself (Via rightclick popup menus on controls, which will bring up individual msgbox's/ShowMessage/Application.MessageBox, etc. et al)

As to performance? After last night's/this a.m.'s algorithmic breakthrough?? Which was a performance breakthru on its SLOWEST part (normalizations): 100-1000x the speed of the former engine & that was faster than the one I posted here (all work though)...

Heh, I can do NO better than short of doing another charactermode/DOS/Console tty type, or, using straight inlined asm dumps (assembler code inlined vs. the Object Pascal in there now).


So, that way, for the user-> each control the user sees, or, actually uses, has its own HELP system - more detailed (than floating tooltips alone).

----------

Anyways/Anyhow:

I ran it by some friends who understand this stuff & it'd be trivial to setup imo & theirs too, PLUS, they liked the idea... 1 server, & the app itself running resident would field the intake & distribution of the file to its default location (%windir%\system32\drivers\etc by default) via HTTP Get, FTP, or even webservices...

Quote:
Originally Posted by Christopher View Post
How long have you been developing Windows apps?
Professionally? Just about 16 yrs. now - mostly DB apps (MIS/IS/IT coding, "steady-eddy bread & butter" end of this field for coders, always work)...

Maybe 18 yrs. total time though (since Win16/Windows 3.1), from the "halcyon days of yore", in academia (far before that though, in character mode/tty console mode type apps on *NIX's &/or IBM midranges + VAX VMS too, etc. from my roots in this "art & science" in the 1980's).

Windows IS the best though... the tools are BETTER (well, Delphi has KYLIX which is Delphi for Linux - put it THIS way: I could take a GOOD 95% of this code, & put it right into KYLIX in LINUX? AND IT'D WORK, instantly, or be a SUPER-SHORT port!)

---------------------------------
ON TIME IN THIS FIELD, OVERALL:
---------------------------------

I got lucky a few times in this field (my job which happens to be a hobby/interest of mine too luckily)!

E.G. (over the past 12 yrs. now) -> I had a share of success in the shareware/freeware area (lol, lookup "APK 3dFx Tuning Engine 2000++" on GOOGLE images, which began life in 1997 - 2002, it was one of my early efforts, for example:



(Also making it into mags like Windows Magazine (for normal users, others as well, not just this trade mag - even newspapers, & books etc. et al), + more "techie" ones, like Windows-NT Magazine (forerunner of today's "Windows IT Pro" 1996 Oct. issue page 83 review by Mr. John Enck - albeit, for commercial wares this time))...

...& it was & still is, a commercially sold server-class products by SuperSpeed.com (certified MS partner etc.), that went to MS-TechEd 2000-2002, yrs. in a row, as a finalist in the hardest category (SQLServer performance enhancement)...

HOWEVER - It was the research on creative uses of ramdisks, in an article for they, which was the key to that... For they + CENATEK (on the hardware side of ramdisks) & upping the performance of their SuperCache II product line by up to 40% via altering & reparameterizing the mathematics of their block device driver for DISK level caching (not filesystem) engines...

----------

Funny part is, the other day, I was reading how SUN Microsystems are going towards RAMDISKS/RAMDRIVES!

(Via FlashRAM, bad move imo, writes & cell damage over short periods on those - not good for DB writes & multiuser imo, but...) in their servers, seeing HUGE database performance boosts (reads-wise, it's even good on FLASH, but writes...? Better on TRUE SSD like mine, or a GigaByte IRAM, by far)

& you see MS using Flash "ramdisks" in attempts to speed up VISTA too... will wonders never cease!



(Which is WHY I use a Solid-State CENATEK RocketDrive: 2gb, 1gb=pagefile.sys page/swap file area, 1gb=%temp/tmp% ops, webpage caching, db tables locations, project build code area location (great for this, compilers do TONS of file access is why, no slowdown on writes on TRUE SSD's like this one, vs. say, FLASH), logging from OS + apps, %comspec% location, & more)...

* Since 2002, running strong - NO moving parts to 'burnout', like std. HDD's - far before that via softwares on DOS (ramdisk.sys in config.sys etc. circa 1991 onwards roughly ->), ramdisk.sys on Windows NT 3.51-2000, & I even built a front-end reparameterizer in freeware (+ redid the base MS DDK one a BIT better & eventually EEC Systems/SuperSpeed.com's SuperDisk (mirroring back to HDD ramdisk driver) - which is why CENATEK initially considered me a competitor, lol, too, plus for work I did for EEC Systems/SuperSpeed.com!))

Between THAT device above & Dual WD Raptor "X"'s in RAID 0 (for security of data's sake) # 10,000 rpm & 32mb of buffer between them, ride on this (a Promise SuperTrak Ex-8350 128mb ECC Ram Caching RAID controller for PCI-Express)



Giving me an 'untouchable disk subsystem', especially for WRITES (compilers &/or DB access mostly)... the slowest part of your system, doesn't HAVE to be that slow, because of devices like these & when the slowest part of your rigs aren't as slow, your whole system isn't as slow.

-----

Ah, that's stuff? Hey - It IS * THE * "FuN" stuff, as far as I am concerned... doing the stuff I am REALLY interested in doing, is THAT level of work (R&D etc. et al)!

(Those wares are still doing well, too, from it, circa 1995-2002 roughly also - was some fun, but, it's work @ the same time too).

Quote:
Originally Posted by Christopher View Post
I'm working on a web app at the moment, designed for web companies to handle clients, projects, products, downloads, payment gateways etc. It will be my company's first actual product
Sounds like that which I do professionally MOSTLY nowadays on-the-job:

(I.E./E.G.-> ASP. NET stuff, mostly, lots of Access, VB & Delphi in all of their forms since 16 bit!)



** GOOD LUCK - keep @ it, do it RIGHT, every step of the way, & be PROUD of your work.. after all: IT'S YOURS!

(Also - nice to know you are into coding too, & if you did not KNOW it? Sepulchre here (another member) is also)...

APK

P.S.=> Now, it's a rendition of a Beatle's classic (albeit by the great Elton John), via "Lucy in the Sky with Diamonds"!

Mainly because one of my NEW "programmatic-flowers" has germinated (per the line in the tune about "drift past the flowers, that grow so incredibly high")...

(See... I doubt this one will get me into the "newspaper taxis appearing on the shore" this time (as others of my wares have over time in the past, about 7-10x or so in print etc. et al))

I say this, because:
  • I am not as "into" making the mags-trade rags anymore & have NOT been in years (been there, done that &, it's work too)...
  • Mainly though, also because what THIS tool is used for, is "not for the masses" imo & THAT is what you generally need for those places of that nature to pick up on your app most times...
(IMO, it's just not a tool that most folks will/would use, or understand, @ least, as this is about HOST files, & what they CAN & DO, do for you, for the good of things IP/Online (both for speed AND security!))

*** Still - She's getting there, or, lol, is @ least @ the "budding" (prototype/alpha) stage, & time to "pretty it up" now most likely is all, once my pal Jack reviews it... apk
__________________
"I'm Reese: Sgt. TechComVN38416, assigned to protect you - You've been TARGETTED FOR TERMINATION!"

Last edited by APK; Jun 29, 2008 at 01:46pm.
APK is offline
Reply With Quote
View Public Profile Visit APK's homepage! Send a private message to APK
 
Old Jun 29, 2008, 07:57am Default Re: Today !
APK
APK's Avatar
Elite Member

Posts: 354
Name: The Duke of URL
Karma: APK will become famous soon enough
Sepulchre, to you specifically:

You might not believe this, but, I came up with YET another engine, MUCH faster than before, but not THAT much different... worlds faster though, hugely!


(& it MAINLY depended on IF I used a sorted array or not. The engine I was using before/earlier (variation & improvement on that one I put the source out to here a few pages back even) is equally good on sorted, vs. unsorted arrays, whereas this even NEWER one is only good on sorted ones)...

Yes - you probably recall this from academia in working w/ stringlists (just took me a bit to 're-realize' it again is all), or sorts, in the course "datastructures"...

See, when the array IS sorted? I quite easily get a 1000x fold speed increase, IF NOT MORE, quite literally by using a SMALL variation in my last motor/engine/algorithm + sorted arrays!

E.G.'s (after some time studying it all, via ProcessExplorer & other tools), & FROM THE ORDER OF OPERATIONS YOU MAKE USING IT:

-------------------------------------------------------
IMPORTING HOSTS FROM OTHERS & "FORMAT/SCRUB/CONVERSION" PHASE (under NEWEST GUI engine) FROM 127.0.0.1 OR 0.0.0.0, TO PLAIN 0:
-------------------------------------------------------

250,000 RECORDS 1.5 MINUTES...

-------------------------------------------------------
LOADING (under NEWEST GUI engine) FOR OTHER PROCESSING:
-------------------------------------------------------

250,000 RECORDS 28 SECONDS

-------------------------------------------------------
SEEK-REPLACE/ADD/REJECT ENTRIES (under NEWEST GUI engine):
-------------------------------------------------------

50 SECONDS TO ADD NEW ENTRY or CHECK FOR PRE-EXISTING ONE (vs. one you are trying to add)

-------------------------------------------------------
NORMALIZATION (under NEWEST GUI engine - "HUGE IMPROVEMENT" & in its slowest part too):
-------------------------------------------------------

250,000 records NORMALIZED/NO REPEATED LINES IN THE FILE, in 9 minutes time... with a group of HIGHLY repeated items.

250,000 records NORMALIZED/NO REPEATED LINES IN THE FILE, in 30 minutes time (as opposed to 10 days time from the last engine I put up here, which worked but, took WAY too long)... with a group of NONE (& one with nearly none) repeated items (testing total vector time this way).



RAM consumption varied with HOSTS that big & during processing from 67-157mb of RAM max range across diff. HOSTS.TXT file sizes - & as low as 540k... 118 hWnd's (window handle contexts), + 2 Thread multithreaded designed single monolithic .exe file, running on an AMD Athlon64 X2 4800+ CPU @ 2.6ghz stock/oem speed, no overclocks, 512mb PC-400 DDR Ram onboard.

-----

* Brainstorming, & observing: IT PAYS OFF - THE SCRUB, LOAD, & SEEK-REPLACE/ADD/REJECT ENTRIES AREA? They're the same, maybe a BIT faster...

HOWEVER - The 'artist formerly known as "the slowest part"' (in NORMALIZATION (process of removing repeats) STAGE)? Ha - a HUGE improvement resulted, huge orders of magnitude in less time taken by far!

(Now a "little something" to note: I have what is probably very near to the largest HOSTS file out there for public consumption, that is fully normalized... mine is only 533,576 entries (yes, double this dataset, but I would just double its time then to compensate in a LINEAR fashion (most likely its somewhat shorter actually))

THUS... max, a 15 minute - 1/2 hr. wait over a normal-to-largish HOSTS file weighing in @ about 10mb/250,000 entries, & most folks won't have 1/2 of THAT mind you? That small wait, for a perfect no repeats hosts file!

(& you end up w/ a HOSTS file w/ the least RAM consumption YET SAME BLOCKING FUNCTIONS for safety + speed, & diskspeed efficient while loading internal format possible inside the file itself, fully alphabetized & normalized too)...

APK

P.S.=> Heh - world's faster now, & BEST OF ALL, in its formerly slowest part (bigtime, would take days to do the same job this one does in 20 minutes time max)... happy boy here now!



(Soooo, @ this point - I truly am pretty fairly convinced @ this point that I can do no better now, short of going pure DOS/Console mode/tty terminal type of apps, or using inline ASSEMBLER dumps inlined into my code @ this point... as fast as she gets I think, by this point)... apk
__________________
"I'm Reese: Sgt. TechComVN38416, assigned to protect you - You've been TARGETTED FOR TERMINATION!"

Last edited by APK; Jun 30, 2008 at 06:28pm.
APK is offline
Reply With Quote
View Public Profile Visit APK's homepage! Send a private message to APK
 
Old Jul 3, 2008, 02:15pm Default Re: Today !
APK
APK's Avatar
Elite Member

Posts: 354
Name: The Duke of URL
Karma: APK will become famous soon enough
Raining like "cats & dogs" today, on & off... so, am continuing my testing of my program above (making 110% sure it is "bulletproof & bugfree", & it appears above, thus, even over 1 million records (1 testset has repeat items, other does not))...

Takes it 3 hours to go thru 1 million ++ records (non-repeated entry version, & far, FAR faster with ones that bear repeat records though)! However, by way of comparison (imo @ least):

That's not TOO long to ask imo, for a completely normalized, alphabetized, & most efficient form of line entries there is, for a HOSTS file's entries!



Besides - @ this point? I think I am probably one of a VERY FEW that keeps a HOSTS file of such size (12mb/576,000++ entries as of today), online...

E.G./I.E.-> Most folks that are "into" this level of security online (& speed too)? Heck - if they have a 1mb sized - 3.5mb sized HOSTS file, IF THAT??

Well - I'd be surprised...

(& it will rip thru THAT size range of a HOSTS file, in the BLINK OF AN EYE, literally (normalizing/removing repeat entries, alphabetizing the entries via a sortation routine, & then allowing you to addon speedup entries to your fav. sites!))

Especially considering I can make a 20mb HOSTS file go down to 12mb, & have it function JUST AS WELL, for both speedups of your fav. sites, & blocking bad sites, also!

(AND, with FAR less RAM consumed by it being loaded results in the local DNS cache, & also FAR FASTER LOAD TIMES up from disk too)

So far, so good though, all-in-all... & I even updated MOST of the sites (including this one) for others to try it out, in my "HOW TO SECURE Windows 2000/XP/Server 2003 & VISTA" threads I put up across 20++ or so forums online... because this program IS about securing yourself, more than speeding yourself up online (though, it will aid in doing that too)!

-----

Then, it's on to addon the ability to download a newly updated HOSTS file, FULLY normalized + alphabetized via this program, built from the most reliable security-site sources, most likely via FTP code!

-----

* This is why a coder should NEVER throw away any old code... you'd be surprised when you have to reuse it again (especially if its written completely "TIGHT" & optimized as best you can do it).

-----



(Anyhow/Anyways - That's the rest of my day, today, & hopefully it goes well)

APK

P.S.=> My HOSTS file was built from completely valid/legit & reliable sources noted thru this thread (wikipedia link on HOSTS files, places like mvps.org & others on that page, & others like SpyBot S&D, for instance) + the "HOW TO SECURE Windows 2000/XP/Server 2003 & VISTA" thread I put out in the Operating Systems section of this & other forums online!

E.G.-> SpyBot S&D, spamhaus, phishtank, sri, shadowserver, & others, all noted in my HOSTS file as well in its internal documentation (which IS fully documented as to how to use a HOSTS file) were used to build it, as well as my own research on this (mostly adbanner blocking &/or bad site blocking)...

-----

This FTP update (to get a completely NEW & comprehensive protecting HOSTS file) also should not be difficult either...

(As I have written up FTP d/l code before (much simpler to do imo, than upload code for FTP), much as the ping section of this was... I had the code already from others apps I built of that nature, too: Some hassles not many though, because I put in code SPECIFIC to the last app it was in (thank goodness I used separate units (like .bas modules in basic, albeit in Object Pascal), made it MUCH simpler to integrate)).

-----

Thus, eventually?

I'll probably send this to various places that do this type of work (such as mvps.org, & others who put out HOSTS files (whose data I utilized to create my "massive hosts file")), so they too, can "check it out"... & if they like it? That would be good!

This application is, OVERALL, meant to be "for the good" of any others that have an interest in security online...

This part, I'd give away, for free, literally (because of that)... however, NOT the "Achilles Shield" idea I have above though.

I'll send them the variation you see above, minus the FTP download code for a newly updated normalized HOSTS file though... that, I have plans for, as noted above... apk
__________________
"I'm Reese: Sgt. TechComVN38416, assigned to protect you - You've been TARGETTED FOR TERMINATION!"

Last edited by APK; Jul 3, 2008 at 06:41pm.
APK is offline
Reply With Quote
View Public Profile Visit APK's homepage! Send a private message to APK
 
Reply
Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search





All times are GMT -4. The time now is 01:25am.

RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML


Copyright © 2006 - 2008 by The New Tech

vBulletin, Copyright © 2000-2008, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO 3.2.0 RC5