Yathosho

11th October 2010 12:30 UTC

Favor to ask
hi guys!

as i'm currently working on the new pimpbot 4, i'd like to ask you for your help. i've attached a small tool that will go through all your installed presets to detect which APEs you're using.

it would be extremely helpful if you could post your results in this thread. this tool will most likely take forever, depending on how many preset you have installed. so, in the worst case this will take a bit longer than forever.

once completed, all results will be displayed in its status window. you can right-click this window and copy the results to your clipboard - and paste them in your answer to this post.

thanks for your help!

QOAL

11th October 2010 15:32 UTC

3 hours later...

AddBorder=199 (270)
BufferBlend=19 (21)
ChannelShift=721 (850)
ColorMap=2903 (4542)
ColorReduction=95 (103)
ConvolutionFilter=2260 (3817)
FramerateLimiter=33 (34)
GlobalVarMan=86 (90)
Multiplier=423 (503)
MultiFilter=115 (120)
NegativeStrobe=0 (0)
Normalise=7 (8)
Picture2=64 (80)
RGBFilter=0 (0)
ScreenReverse=0 (0)
Texer=175 (263)
Texer2=868 (3817)
TransAuto=65 (65)
Triangle=87 (700)
VfxAVIPlayer=0 (0)
VideoDelay=136 (170)
Completed

Yathosho

11th October 2010 16:03 UTC

here are my statistics so far. the reason why i'm really interested in this shows in the last two columns: different authors use different apes are used. ultimately, i'm going to use the results to speed up the ape-detection in the upcoming pimpbot version. the more likely an ape is, the sooner it gets detected, the quicker the remaining detection runs.

unfortunately, you can also see how unreliable the detection is. sometimes it's enough to write the name of the ape into a comment and the detection will work (some authors solved this better than others.) texer is especially tricky, have a look at the results in the last two columns!

Grandchild

11th October 2010 22:33 UTC

AddBorder=291 (407)
BufferBlend=49 (53)
ChannelShift=923 (1100)
ColorMap=4175 (6913)
ColorReduction=241 (263)
ConvolutionFilter=3331 (6301)
FramerateLimiter=116 (123)
GlobalVarMan=113 (119)
Multiplier=620 (744)
MultiFilter=177 (188)
NegativeStrobe=0 (0)
Normalise=111 (135)
Picture2=54 (66)
RGBFilter=0 (0)
ScreenReverse=0 (0)
Texer=144 (236)
Texer2=1252 (5088)
TransAuto=491 (498)
Triangle=332 (1309)
VfxAVIPlayer=2 (3)
VideoDelay=325 (388)

The Earthquaker

12th October 2010 12:46 UTC

AddBorder=55(77)
BufferBlend=5(5)
ChannelShift=156(182)
ColorMap=657(1010)
ColorReduction=20(26)
ConvolutionFilter=525(1123)
FramerateLimiter=3(3)
GlobalVarMan=54(73)
Multiplier=103(110)
MultiFilter=41(44)
NegativeStrobe=0(0)
Normalise=11(14)
Picture2=35(51)
RGBFilter=0(0)
ScreenReverse=0(0)
Texer=44(87)
Texer2=319(1811)
TransAuto=39(39)
Triangle=51(379)
VfxAviPlayer=0(0)
VideoDelay=34(53)

QOAL

12th October 2010 21:28 UTC

Yathosho, as this app is so slow I decided to code an app in C++ to do that same, but much quicker.

Example of current output:

     - APE sniffer -

 Scanning...

 Finished!
 Scanned 6353 presets in 1 second(s)
 Found "Jheriko: Global\0" 92 times in 88 presets.
 Found "Misc: AVSTrans Automation\0" 67 times in 67 presets.
 Found "Render: Triangle\0" 702 times in 89 presets.
 Found "Acko.net: Texer II\0" 3819 times in 870 presets.

Yathosho

13th October 2010 11:17 UTC

"Holden03: Convolution Filter"
"Color Map"
"Holden04: Video Delay"
"Jheriko: Global"
"Multiplier"
"Virtual Effect: Addborders"
"Misc: AVSTrans Automation"
"Render: Triangle"
"Channel Shift"
"Color Reduction"
"Misc: Buffer blend"
"VFX FRAMERATE LIMITER"
"Trans: Normalise"
"Picture II"
"Texer"
"Acko.net: Texer II"
"Jheriko : MULTIFILTER"
"VFX AVI PLAYER"
"Negative Strobe"
"Screen Reverse"
"RGB filter"

as you can see, many of them are easy to fool (especially ucd's). i was wondering about the slow speed before, compared to how much quicker search-in-files in notepad++ is. the old version of pimpbot used a script, that was equally slow but more unreliable. for the new version, animaether wrote me a new function. i consider him a nsis pro, but i still wonder about the terribe detection rate.

i guess the best solution would be a custom nsis plugin. if you can think you can turn your app into one, i'll gladly use it for pimpbot (i'd prefer a plugin without dependancies such as net framework etc.)

Yathosho

13th October 2010 12:30 UTC

as i said before, texer(1) is especially unreliable to detect, as all texer2 will also be found when searching for texer. i tried (integer) subtraction, but as you can see in my table of results, in one case this produced a result of zero (could also mean a negative value!)

that was likely caused by the mentioning of the the word "Texer" or by a texer-bitmap including the name.

sucky, sucky!

QOAL

13th October 2010 13:27 UTC

Thanks for the list!

I wouldn't of thought NSIS would be close to as fast as a native app anyway, I'm sure the code you use is very well written though.

Right now I have zero idea on how to make a nsis plugin, but I'll have a look. (if I never mention it again then things didn't go well - I am just winging this)

By adding \0 (null) to the end of my search strings I made the check far more accurate (probably could match something at the very end of a comment/code block though.)

Anyway, I've attached a zip with APE Sniffer and the code for it.
Place it in a folder and it will recursively scan for AVS files and then spit out APE_Sniffer.txt with the results.

Warrior of the Light

13th October 2010 14:12 UTC

AddBorder=434 (612)
BufferBlend=32 (37)
ChannelShift=1241 (1466)
ColorMap=4925 (7583)
ColorReduction=253 (273)
ConvolutionFilter=3661 (6650)
FramerateLimiter=91 (95)
GlobalVarMan=169 (182)
Multiplier=710 (827)
MultiFilter=302 (310)
NegativeStrobe=0 (0)
Normalise=55 (68)
Picture2=127 (183)
RGBFilter=0 (0)
ScreenReverse=2 (7)
Texer=257 (457)
Texer2=1711 (7379)
TransAuto=162 (162)
Triangle=209 (1126)
VfxAVIPlayer=2 (5)
VideoDelay=373 (475)
Completed

QOAL

13th October 2010 19:32 UTC

Okay, I have something working.

Basically call 'apesniffer::scan "DIR"' it'll return a string of 1 and 0, 1 means that APE is used. The string is in the same order as your checkers output (and the same as mine).
I thought that would be the easiest and most compact way of telling you what are used.
It should also be faster than my checker too, as it only needs to find an APE is needed once to stop checking for that APE.

Simple test.nsi included (just remembered it has relative path in there for the plugin location, you'll need to fix that, I would but I'm going out now, sorry) along with the source (which is probably quite fugly).
I'm annoyed at the file size, but I guess that is what I get for using the STD library (I think?).
Anyway, hope it helps you out.

Yathosho

13th October 2010 20:19 UTC

the speed is amazing, will implement this asap.

think you can support subfolders as well? while it's no problem to solve this within nsis, it feels more right if the plugin did this (also to avoid double-scans.)

licensing might also be an issue. i haven't decided yet, but i think i will stick with a bsd-like license (basically that's "do what you want") for pimpbot 4. it's not important for me, but it might be for google code.

QOAL

13th October 2010 23:40 UTC

Yay. I was a bit worried over what to return, but no complaints(?) so that's cool.
(It would be nice to get it's size down, 200k for that is a bit silly)

It does scan subfolders, doesn't it? It's the same code as the standalone sniffer I made, that scanned folders recursively.

Hmm licensing, well I assumed that any of the code snippets I used are in the public domain and I'll agree to whatever license is best for you.
Yes, google code doesn't like you mixing licenses in single repos.

Yathosho

14th October 2010 07:29 UTC

Originally posted by QOAL
It does scan subfolders, doesn't it? It's the same code as the standalone sniffer I made, that scanned folders recursively.

Originally posted by QOAL
Hmm licensing, well I assumed that any of the code snippets I used are in the public domain and I'll agree to whatever license is best for you.
Yes, google code doesn't like you mixing licenses in single repos.

Yathosho

14th October 2010 09:19 UTC

just for the record: it didn't scan all my presets before as the plugin-dir detection isn't flawless. the plugin itself works, i'm currently implementing it to pb4.

QOAL

16th October 2010 16:53 UTC

Right, I couldn't stand the fact the dll was so big so I made it work without using STD and I went back to C. (Source should just be a drop in replacement for the example plugin)

DLL size has gone from 283KB to 8.5KB.

I don't think anything else has changed.

Yathosho

16th October 2010 21:31 UTC

your previous test script gave me this:

101111001100000110000Ûôv0.T
Completed

QOAL

16th October 2010 22:23 UTC

Hmm okay, I think I know what would cause that and I've added a fix in the attached version.
Checks if the path given ends with a / or \, and appends one if not.
I couldn't find any other way to make it spit out all zeros.

I hope that fixes the issue. :S

Yathosho

16th October 2010 22:35 UTC

working again! :)

Yathosho

4th April 2011 13:53 UTC

i guess it's more difficult to build something like this for texer images, videos or gvm files?

QOAL

4th April 2011 20:56 UTC

You want a program that goes through all the presets and checks for .bmp, .avi and .gvm files?
Or you'd like it to list the required file names? (I guess it would need to ignore the default texer images)

I would need to see if I can alter the binary search code, so it can find the extension and then find the first null byte before it and use that as the file name -- which is easier said than done for me, probably.

I'm not sure how to return a list like that to NSIS, which I assume is the end goal here.
Doing it as a comma separated string could easily be way too long.
Can a plugin just keep pushing strings to the stack?

Yathosho

4th April 2011 22:34 UTC

Originally posted by QOAL
You want a program that goes through all the presets and checks for .bmp, .avi and .gvm files?

---

QOAL

5th April 2011 18:28 UTC

Okay so...

I have made progress / might of hit a hole in one with this.

Usage is like so:
avsresources::listresources "$1\avs\QOAL\" "$1\avs\"
${Do}
Pop $R0
DetailPrint $R0
${LoopUntil} ${Errors}

First parameter: Dir to search for AVS in
Second parameter: Resource dir

Each string you pop contains the name of a filename, verified to exist, relative to the resource dir passed.

Apparently as a plugin, you can just keep on pushing strings, which you can then just pop off in a while loop. (Which does work with 1000's of strings, hooray)
Of course, I can also output the file names to a text file too. (Disabled right now)

Couple of things that I noticed while doing this:
One APE prepends(?) the bitmap file name with ð?, rather than NULL (Removed internally)
Lots of possibly old references to files are left in presets, leading to the plugin finding lots of names that have the start cut off. (All files are checked to see if they exist, and are otherwise ignored)

The attached zip contains avsresources.dll and a test.nsi (change the plugin path in it!)
The plugin is large this time as I used some STL stuff (Vector stuff), to hold the list of files, so it can be checked for dupes during the scan.

Hopefully this post makes (some) sense!

Edit:
Just realised that having it check the passed dir for the resource files is a bit silly, and probably only works correctly on a main AVS dir. I'll have to add a second parameter?
Also: The file extensions it looks for are: .gvm, .GVM, .bmp, .BMP, .avi and .AVI
Edit 2:
Updated the plugin with a second parameter for the resource dir and updated the post.

QOAL

6th April 2011 12:55 UTC

Dum de dum...

So... why have two separate plugins when you can have just the one?!

Function names have changed, and a new one has been added for a laugh, please see attached.
I also made it so it checks for presets ending in .AVS along with the previous .avs.

(Contains complied DLL, source and an example nsi file)

Yathosho

6th April 2011 12:57 UTC

doesnt work for me, all it does is creating an empty textfile. my next question wouldve been whether this scans recursively? if not, could you add another parameter for that?

edit: my post was regarding the first version, have to test the new one

edit2: avstools is now listing resources and already works recursively - nice!

QOAL

6th April 2011 13:14 UTC

Yeah, I only noticed this morning that what I posted yesterday had the code to check parameters for trailing slashes (and append one if not) removed by accident - and it had spitting out to a file still enabled. Whoops!

But what a difference a day makes. :)

The apesniffer in AVS tools should, hopefully, return a string that is as long as the number of APEs (before it was 32 chars, which probably contained memory from other stuff at the end)

And yes, everything is recursive.

Yathosho

7th April 2011 11:07 UTC

uploaded a first beta using your plugin

Yathosho

7th April 2011 11:15 UTC

Originally posted by QOAL
Also: The file extensions it looks for are: .gvm, .GVM, .bmp, .BMP, .avi and .AVI

QOAL

7th April 2011 12:44 UTC

Here we go: :)

Yathosho

9th April 2011 22:57 UTC

i've tested avstools on a couple of scenarios and it's not always working. to reproduce my problem, download this file and put the resource files in your avs directory.

avs 1
detects only the bitmaps in the texer, texer 2, picture and picture 2 preset. i've also added two presets using videos and two more using the svp/uvs loader (i didn't know until know the svp plugin also handles uvs files, never heard of those).

avs 2
contains a preset using a picture, but although present in my avs directory, the bitmap never gets detected (the picture effect worked in my first example).

QOAL

10th April 2011 22:14 UTC

Hmm, I'm really not quite sure what problems you're getting.
I've gone through both of the avs folders presets and checked the result compared to what is listed in the files (I haven't actually looked at them with AVS), and what's in the Res folder, and no matter where the folders are (different disks, deep or shallow) it still works correctly.

I don't want to doubt you, but are you sure you're sending the correct resource dir?

my brain is shot for the night now, I should probably give you my msn so it can be nailed down, which I will send now.

Yathosho

12th September 2011 14:07 UTC

would it be possible to get a unicode build of this plugin as well? sooner or later, i will switch to the unicode version of nsis so i thought i better ask now that you still remember how the plugin works ;)

QOAL

14th September 2011 11:38 UTC

I had a attempt at this, but I just don't know what I'm doing when it comes to ansi to unicode conversion.
I'm really not that good at c/++, I just wing it.
I know my way around the plugin code still and I set up it up to use the unicode stuff and did some changes to TCHAR and std::basic_string<TCHAR> but...

"Number of AVS presets: 0??Ѩ 저鰢樀鑞谰럼ℂꀀ山럼젂鰢쑄럿礂띊??h
APEs used: 〰〰〰〰〰〰〰〰〰〰䐰ￄʷ䩹皷⟼棜
Resource files required:
〰〰〰〰〰〰〰〰〰〰䐰ￄʷ䩹皷⟼棜
Completed"

:rolleyes:


So yeah, I'll just attach the source for the ansi version for now. :S

Yathosho

15th September 2011 08:09 UTC

there are currently 5 nsis plugins not available for unicode nsis, at least two of them can not be integrated using the CallAnsi plugin. i guess a unicode build of pimpbot could take a looong time to be realized. it's said as i have some unicode language files that i can't convert to ansi without losing information (e.g. cyrillics).

Yathosho

14th October 2011 10:49 UTC

out of interest, what are you using to compile your plugin?

QOAL

14th October 2011 11:38 UTC

I'm using Code::Blocks with GCC (I'm used to that set up due to previous open source work).
I didn't bother including any project files in the zip because I assumed very few people would use that set up.
(I'm also still not interested in spending hours flapping about in a naive attempt to produce a unicode version)

Yathosho

14th October 2011 13:52 UTC

well, for a second i thought i might take a look at it and make it work for unicode. i doubt i would even get as far as compiling, as don't know any c++ :)

Yathosho

7th November 2011 14:26 UTC

okay, it looks like i found a workaround. pimpbot itself will still be built using the ansi version of nsis, but it will create unicode installers. that way i don't rely on avstools in unicode.

will post a working version in the next couple of days.