Topic: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

My Universal BetaNews FileForum Templates: (TEST)

****IMPORTANT: I believe these templates WILL WORK for the VAST MAJORITY of
                          applications listed at BetaNews FileForum; located at:
                          http://fileforum.betanews.com/. This is the FIRST version
                          of my templates and they have been adjusted for their new site
                          design. IT IS A WORK IN PROGRESS and I will post improvements
                                 as they develop.
wink

                          There are TWO VERSIONS of the template; the FIRST is for all
                          applications that are NOT HOSTED by SourceForce and the SECOND
                          is for applications that ARE HOSTED by SourceForge. It is YOUR
                                 RESPONSIBILITY
to determine which to use by clicking on the apps
                          download link to determine the host location.

****NOTE: These templates use the application's NAME embedded in the
                     application INFO page URL
as designated by BetaNews FileForum.
                 In other words, go to the application's INFO page at BetaNews FileForum
                 and copy the NAME that appears within the URL. For example, for
                 Spybot - Search & Destroy, at BetaNews FileForum:
                 hxxp://fileforum.../detail/Spybot-Search-Destroy/1043809773/1
                 the NAME you would paste at the FIRST prompt from the template
                 would be Spybot-Search-Destroy and the ACTUAL NAME of the
                 application Spybot - Search & Destroy at the SECOND prompt
                 from the template. These templates use BetaNews FileForum's SEARCH
                     ENGINE
which should make the apps fairly impervious to their cryptic
                 URL changes!

1) There are two user defined global variables used, download_url={download_url}
    (used as an alternative to {url}) and root_folder=YOUR Download Destination
    ROOT Folder (for example C:\Latest-Apps) and two local variables, {find}
    and {redir1}.

2) Ketarin (.0.9.9.22) released January 18, 2009 or later is required as this revision
    uses the new 'placeholder' feature implemented by Florian.

3) The template for applications NOT HOSTED by SourceForge follows:

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <DownloadBeta>AlwaysDownload</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <VariableChangeIndicator />
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <HttpReferer>http://fileforum.betanews.com/detail/{find}</HttpReferer>
    <Variables>
      <item>
        <key>
          <string>BNFileForumName</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent><placeholder name="BetaNews FileForum Name ['Their Name' for App within App Page URL (i.e. TreeSize)]" /></TextualContent>
            <Name>BNFileForumName</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>find</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=results[ ]found[ ]for.*?A HREF="http://fileforum.betanews.com/detail/).+?(?=")</Regex>
            <Url>http://fileforum.betanews.com/search?s={BNFileForumName:replace:-: :}</Url>
            <Name>find</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;="main_file_details".*?&lt;[a-z][\d]&gt;.*?)\d[.].+?[ ]??(?=[ ]+[\r\n]|[ ][\r\n]|[\r\n])</Regex>
            <Url>http://fileforum.betanews.com/detail/{find}</Url>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>download_url</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=meta[ ]http-equiv="refresh".*?URL=)http://.+?(?=")</Regex>
            <Url>http://fileforum.betanews.com/download/{find}</Url>
            <Name>download_url</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <Category>[INSERT CATEGORY HERE]</Category>
    <SourceType>FixedUrl</SourceType>
    <PreviousLocation />
    <DeletePreviousFile>false</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath>
    <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
    <Name><placeholder name="Application Name" /></Name>
  </ApplicationJob>
</Jobs>

4) The template for applications HOSTED by SourceForge follows:

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <DownloadBeta>AlwaysDownload</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <VariableChangeIndicator />
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <HttpReferer />
    <Variables>
      <item>
        <key>
          <string>BNFileForumName</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent><placeholder name="BetaNews FileForum Name ['Their Name' for App within App Page URL (i.e. TreeSize)]" /></TextualContent>
            <Name>BNFileForumName</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>find</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=results[ ]found[ ]for.*?A HREF="http://fileforum.betanews.com/detail/).+?(?=")</Regex>
            <Url>http://fileforum.betanews.com/search?s={BNFileForumName:replace:-: :}</Url>
            <Name>find</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;="main_file_details".*?&lt;[a-z][\d]&gt;.*?)\d[.].+?[ ]??(?=[ ]+[\r\n]|[ ][\r\n]|[\r\n])</Regex>
            <Url>http://fileforum.betanews.com/detail/{find}</Url>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>redir1</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=If[ ]your[ ]download.*?)http://.+?(?=")</Regex>
            <Url>http://fileforum.betanews.com/download/{find}</Url>
            <Name>redir1</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>download_url</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=You[ ]are[ ]downloading.*?)http://.+?(?=")</Regex>
            <Url>{redir1}</Url>
            <Name>download_url</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <Category>[INSERT CATEGORY HERE]</Category>
    <SourceType>FixedUrl</SourceType>
    <PreviousLocation />
    <DeletePreviousFile>false</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath>
    <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
    <Name><placeholder name="Application Name" /></Name>
  </ApplicationJob>
</Jobs>

TO USE THEM:

1) Define the user defined global variables under 'Settings' in Ketarin to YOUR preferences
2) Edit the template externally to reflect these modifications (if you chose
     not to use mine)
and change the <TargetPath> line to YOUR preferences
3) Then:
    a) Import it (Obviously)
    b) Enter the information needed as requested as a result of the 'placeholder'
        prompts... it will ask for the application URL's NAME as described above and
        the actual application's name
    c) Modify the Category to your preferences
    d) Change the 'Advanced Settings' to your preferences
    e) VERIFY...VERIFY...VERIFY that ALL regex matches WORK and
        MAKE ADJUSTMENTS to them as necessary!

****NOTE:

IF YOU WISH TO CREATE A TEMPLATE from one of YOUR XML exports:

1)It is important that the Guid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" string be
   removed from the XML export to use it as a template, otherwise you'll just keep
   overwriting the previously saved application entry that was created using the
   imported template
2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" />
3) The <LastUpdated> line must be changed to <LastUpdated xsi:nil="true" />
4) Remove the entire <PreviousLocation> line or replace it with <PreviousLocation />
5) Create 'placeholder' entries as needed (see THIS thread)

I hope somebody can use it... feel free to modify it in any way you wish for your use or make suggestions.

CybTekSol

Re: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

CybTekSol - this is wonderful, I have just implemented it with both types and it worked flawlessly (apart from Eventum as this requires a MySQL dev account despite being listed at SF, there will no doubt be others...)

A few observations though, (may help others)

you state

"root_folder={Insert YOUR download destination ROOT folder here} (for example C:\Latest Downloads)"

I found if I included my download path in the {} it was prefixed with the Ketarin install folder BEFORE the path and caused an error. The root_folder=C:\Latest Downloads ie WITHOUT the {} works without problems

you also state

2) The <DownloadDate> line must be changed to <DownloadDate xsi:nil="true" />

again on my system even after a successful download and export this value is the same/not updated (all others are valid as expected) could this be a bug in Ketarin, can you reproduce/test?

and finally the

<TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath>

is an excellent little tip..

I will be trying a {root_folder}\{category}\{sub_cat} approach for real geek value lol (TODO: maybe some of these could be integrated into a drop down menu within Ketarin)

I suspect it's answered elsewhere but where are the 'share this application on-line' stored/sent do you know?

This is really the start of being able to add support for BetaNews File forum to Ketarin if you ask me (Softpedia anyone?)

Ready Rolled Example

PSPad_editor - NOT SF HOSTED - 1.94MB * Cat is set to 'Developer Tools' you need to create this or change to suit

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <DownloadBeta>AlwaysDownload</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <VariableChangeIndicator />
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <HttpReferer>http://fileforum.betanews.com/detail/{find}</HttpReferer>
    <Variables>
      <item>
        <key>
          <string>BNFileForumName</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>Textual</VariableType>
            <Regex />
            <TextualContent>PSPad_editor</TextualContent>
            <Name>BNFileForumName</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>find</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=results[ ]found[ ]for.*?A HREF="http://fileforum.betanews.com/detail/).+?(?=")</Regex>
            <Url>http://fileforum.betanews.com/search?s={BNFileForumName:replace:_: :}</Url>
            <Name>find</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;="main_file_details".*?&lt;[a-z][\d]&gt;.*?)\d[.].+?[ ]??(?=[ ]+[\r\n]|[ ][\r\n]|[\r\n])</Regex>
            <Url>http://fileforum.betanews.com/detail/{find}</Url>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>download_url</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=meta[ ]http-equiv="refresh".*?URL=)http://.+?(?=")</Regex>
            <Url>http://fileforum.betanews.com/download/{find}</Url>
            <Name>download_url</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <Category>Developer Tools</Category>
    <SourceType>FixedUrl</SourceType>
    </ PreviousLocation>
    <DeletePreviousFile>false</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath>
    <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
    <Name>PSPad Editor</Name>
  </ApplicationJob>
</Jobs>

Re: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

somerandomhash wrote:

A few observations though, (may help others)

you state

"root_folder={Insert YOUR download destination ROOT folder here} (for example C:\Latest Downloads)"

I found if I included my download path in the {} it was prefixed with the Ketarin install folder BEFORE the path and caused an error. The root_folder=C:\Latest Downloads ie WITHOUT the {} works without problems

1) I did not intend for the 'curly brackets' to be included in the definition of the variable... just that its contents was what defined the variable. Sorry for the confusion. I will probably remove them from my template posts as I can see where they could be misleading users. wink

somerandomhash wrote:

Again on my system even after a successful download and export this value is the same/not updated (all others are valid as expected) could this be a bug in Ketarin, can you reproduce/test?

2) Great observation... I had not picked up on this. Out of my 687 apps, the <DownloadDate> data is only included in 7 of them in my 'export all' xml! I don't know if it's a bug or not... sure seems like it. Florian, do you have any ideas on this issue? Could it be related to inserting <DownloadDate xsi:nil="true" /> in the templates? It appears to me that Ketarin is no longer storing this data as the last entry I have is 12/31/08...

somerandomhash wrote:

I suspect it's answered elsewhere but where are the 'share this application on-line' stored/sent do you know?

I'll let Florian answer this one... I'm not sure, but I suspect to a database on one of his servers.

somerandomhash wrote:

(Softpedia anyone?)

3) I have developed a template for Softpedia but chose not to post it as their site has so many inconsistencies it was almost impossible to refine a 'universal' regex scrape. At best, I got to a 50-70% reliabilty... not quite ready for the masses. However, we would welcome that addition if someone can donate the time and effort to achieve a more reliable version. I had to lighten up as my wife voiced her opinion regarding my long hours. sad

Re: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

1. The root_folder=C:\Latest Downloads ie WITHOUT the {} works without problems
- Yeah, that's the way to go. If you have problems with spaces, remember you can use "" arround path, 8.3 shortfilenames or DOS environment variables.

2.Could it be related to inserting <DownloadDate xsi:nil="true" /> in the templates?
- I've noticed this for a while, but if you use Filehippo "template" <DownloadDate> is not updated either, so I would say this is not a template problem. What is updated is the <LastUpdated> tag, e.g., <LastUpdated> 2009-02-03T18:26:28.7522851</LastUpdated>, and this is the data Ketarin displays in main window. It doesn't seem to be causing any problems.

3. <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath> is an excellent little tip...
- This is highly flexible, you can do anything with it. Some of my target paths:
{target}\{category}\
{target}\{category}\{appname}\
{target}\{category}\{appname}_{version}.{url:ext}
{target}\{category}\{url:basefile}.{url:ext}
{target}\{category}\{url:basefile}_{version:replace: :.}.{url:ext}
{target}\{category}\wlx_{appname}_{version}.{url:ext}
{target}\{category}\wcx_{appname}_{version:replace: :_}.{url:ext}
{target}\{category}\{appname}_{version:replace:Vwr_Port:.}.{url:ext}
{target}\{category}\{appname}_{version:replace: (Build :.}.{url:ext}
{target}\{category}\{appname}_{version:replace:PortablePreRel:.}.{url:ext}

4. I got to a 50-70% reliabilty...
- Workaround is to import template and do some minor tweaks to regex prior to update. This works for me even in sites where we can't get 100% reliability. I am yet to create a template I'm not using. smile

Re: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

The 'share this app' is posted via RPC calls to http://ketarin.canneverbe.com/rpc

Available XMLRPC methods

# ketarin.GetApplications()
# ketarin.GetSimilarApplications()
# ketarin.GetUpdatedApplications()
# ketarin.GetApplication()
# ketarin.SaveApplication()

Re: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

FranciscoR wrote:

Workaround is to import template and do some minor tweaks to regex prior to update. This works for me even in sites where we can't get 100% reliability. I am yet to create a template I'm not using. smile

AGREED! I ALWAYS add a new app using one of my templates, even for generic 'one app from a site' additions using my 'generic' template. It reduces redundant entry of data and gives me a starting point from which a minimal amount of tweaking is required! wink

Re: CybTekSol's BetaNews FileForum Templates (Uses THEIR Search)

I made a slight tweak to this template today to adjust for 'BetaNews FileForum's' change in their naming conventions for apps as it appears they now are using hyphens in their app URL names instead of underscores.