Topic: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

My LATEST Universal Softpedia.com Template (TEST):

****PLEASE TEST it and give me some feedback on this one if you refine it or have
       significant problems citing specific examples.

****NOTE: This template uses the application's INFO page URL.

1) There are two user defined global variables used, download_url={download_url}
    (used as an alternative to {url}) and root_folder=Insert YOUR download destination
     ROOT folder here
(for example C:\Latest-Apps).

2) The latest version of Ketarin (1.0.3) released February 24, 2009 or later is
    required to overcome their site changes for 'user-agent'.

3) The template follows:

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <DownloadBeta>Avoid</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <VariableChangeIndicator />
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <HttpReferer>{redir1}</HttpReferer>
    <Variables>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>&lt;title&gt;Download.*?(\d.*?|\d.*?[ ]Build[ ]\d.*?|\d.*?[ ]Beta[ ]\d.*?)(?:\s\/\s|\s-\s)</Regex>
            <Url><placeholder name="Application Info Page URL from Softpedia" /></Url>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>redir1</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=download_bkg.gif.*?&lt;a href=").*?(?=")</Regex>
            <Url><placeholder name="Application Info Page URL from Softpedia" /></Url>
            <Name>redir1</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>download_url</string>
        </key>
        <value>
          <UrlVariable>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=DOWNLOAD LOCATIONS.*?&lt;table width=".*?&lt;a href=").*?(?=")</Regex>
            <Url>{redir1}</Url>
            <Name>download_url</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand />
    <Category>[INSERT CATEGORY HERE]</Category>
    <SourceType>FixedUrl</SourceType>
    <DeletePreviousFile>false</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath>{root_folder}\{category}\{appname:replace: :_}_v{version:replace: :_}.{url:ext}</TargetPath>
    <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
    <Name><placeholder name="Application Name" /></Name>
  </ApplicationJob>
</Jobs>

To use it:

1) Define the user defined global variables under 'Settings' in Ketarin to YOUR
    preferences
2) Edit the template externally to reflect these modifications (if you choose not to use
    mine) and change the <TargetPath> line to YOUR preferences
3) Then:
    a) Import it (Obviously)
    b) Enter the information needed as requested as a result of the 'placeholder'
        prompts... it will ask for the application's DOWNLOAD page URL (NOT the information
        page URL) and the application's name
    c) Modify the Category to your preferences
    d) Change the 'Advanced Settings' to your preferences
    e) VERIFY...VERIFY...VERIFY that ALL regex matches WORK and
        MAKE ADJUSTMENTS to them as necessary!

I have had a 80-90% success rate with this template; in instances where it failed, I simply tweak the regular expression. wink

CybTekSol

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Already in my to-do list big_smile... but you have <HttpReferer> with {redir} and only a {redir1} variable ?

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

but you have <HttpReferer> with {redir} and only a {redir1} variable ?

A typo during an external edit... corrected now... THANKS for catching it! wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Nice indeed,

20-02-2009 12:32:34: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/E … 30373.html'
20-02-2009 12:34:04: Update started with 1 application(s)
20-02-2009 12:34:08: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/E … 30373.html'
20-02-2009 12:34:10: Everest: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.com/dl/69b0c8 … ate500.zip'
20-02-2009 12:34:11: Everest: Checking if update is required...
20-02-2009 12:34:11: Everest: Update required, 'D:\Programas\Test\everestultimate500.zip' does not yet exist
20-02-2009 12:34:13: Everest: Replacing {version} in '{version}' with '5.00.1664 Beta / 5.00.1650 Free Trial'
20-02-2009 12:34:13: Update finished

Will I actually DL Beta or Trial ?

Last edited by FranciscoR (2009-02-20 12:37:55)

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Trial. {version} could be adjusted... ?

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

Trial. {version} could be adjusted... ?

Go for it! Suggestions?

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Not as speaking... big_smile maybe later ? Nice template ! wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

@FranciscoR,
This is an example of one of the Softpedia inconsistencies we discussed in the past. This particular example would require some major regex tweaking... OBVIOUSLY! The 80-90% success rate is still better than starting from scratch with every app there though... know what I mean? wink I don't ever see an infallible template for their site. sad

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

{download_url} seems to be working just fine... I'll be back with some suggestions for {version}. wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Made a small temporary tweak to the {version} regex... looking forward to your input.

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Nope, now I have beta version with a trial DL

20-02-2009 13:28:38: Update started with 1 application(s)
20-02-2009 13:28:40: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/E … 30373.html'
20-02-2009 13:28:42: Everest: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.com/dl/9850f0 … ate500.zip'
20-02-2009 13:28:54: Everest: Checking if update is required...
20-02-2009 13:28:54: Everest: Update required, 'D:\Programas\Test\everestultimate500.zip' does not yet exist
20-02-2009 13:28:57: Everest: Replacing {version} in '{version}' with '5.00.1664 Beta'
20-02-2009 13:28:57: Update finished


3dsMax {version} is not 100% either

20-02-2009 13:30:43: Update started with 1 application(s)
20-02-2009 13:30:46: 3dsMax: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/3 … 15976.html'
20-02-2009 13:30:48: 3dsMax: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.ro/dl/aeb999d … wnload.exe'
20-02-2009 13:30:50: 3dsMax: Checking if update is required...
20-02-2009 13:30:50: 3dsMax: Update required, 'D:\Programas\Test\Autodesk3dsMax2009_ENU_TrialDownload.exe' does not yet exist
20-02-2009 13:30:51: 3dsMax: Replacing {version} in '{version}' with '3ds Max 2009 11.0 Free Trial'
20-02-2009 13:30:51: Update finished


It needs more time... no pressure from me. I'll be back after some testing on {version}. smile

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

As is, it's a very, very, very good template and I must say it already is one of my personal favorites. {version} is good enought to track updates, and I am amazed on how easily Softpedia allowed you to manage the page redirection... I don't have such help on my VersionTracker template, but {version} is so much easy... there's no perfect web design(er) ! wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

Nope, now I have beta version with a trial DL

I just took a quick stab at it when I made that small tweak as my time is limited... didn't have a chance to verify the match between version and DL... I should have known they would be reversed! wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Me2, running out of time right now. But I do understand the SIZE of your problem... I think that for the moment I will be using 'Last Updated:' row, e.g.  'November 3rd, 2008, 14:48 GMT', properly customized, to track versions. Easy to-do and less unpredictable.

P.S. You're using HTTP referer for any specific reason ? Seems to work fine without it.

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

P.S. You're using HTTP referer for any specific reason ? Seems to work fine without it.

If my memory serves me correctly, it was an issue for a couple of the apps I get from there for some freaky reason, so I left it in the template... can't remember the specifics.

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

This is just my personal preference, but until they include a somewhat standardized version format, I will be using ([a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)? . It works for me with every Adobe and Autodesk app and plugin I tested with, and allows me to solve problems like Adobe Photoshop Lightroom 2.3 RC Build 534370 / 2.2 Build 523352.

20-02-2009 16:49:31: Update started with 1 application(s)
20-02-2009 16:49:33: Everest: Replacing {redir1} in '{redir1}' with 'http://www.softpedia.com/progDownload/E … 30373.html'
20-02-2009 16:49:35: Everest: Replacing {download_url} in '{download_url}' with 'http://download.softpedia.com/dl/fb8af7 … ate500.zip'
20-02-2009 16:49:36: Everest: Checking if update is required...
20-02-2009 16:49:36: Everest: Update required, 'D:\Programas\Test\everestultimate500.zip' does not yet exist
20-02-2009 16:49:37: Everest: Replacing {version} in '{version}' with 'February 20th, 2009, 08:12'
20-02-2009 16:49:37: Update finished


I'm quite happy with template, thanks a lot CybTekSol. smile

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

This is just my personal preference, but until they include a somewhat standardized version format, I will be using ([a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)? . It works for me with every Adobe and Autodesk app and plugin I tested with, and allows me to solve problems like Adobe Photoshop Lightroom 2.3 RC Build 534370 / 2.2 Build 523352.

I'm quite happy with template, thanks a lot CybTekSol. smile

I'm glad you could use it. wink Would you please take my template, make changes that you have implemented to suit your needs and post it to this same thread to give other users better coverage of Softpedia apps? Thanks in advance...

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Like I said, it's *only* my personal preference. If a user wants to use last updated date instead of file version, all he/she has to do is to copy that regex into the {version} variable. But your template is working great, let users experience it and decide, if a request is forward I will do that. Still, a community input would be most useful to improve it... Let it rest now, no worries, we cannot be too demanding with DL sites such as this. Well done again. wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

If a user wants to use last updated date instead of file version, all he/she has to do is to copy that regex into the {version} variable.

Agreed... I think the average user will not have any problems doing that. Users can create two different versions of the template on their own with a simple copy/paste replacement of the {version} regex with this:

([a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)?

either within an external editor, since there are no converted characters (new users see more on this below) or within Ketarin and exporting it as template. When confronted with a situation like this at Softpedia (where multiple versions and download locations are combined), they can import the alternative template which captures the 'Last updated' date and then tweak the {download_url} by inserting an additional anchor from the app's download page for the download location displayed ABOVE the location they actually want to use, such as

Softpedia Secure Download \(RO\).*?

inserted which would change the {download_url} scrape from:

(?<=DOWNLOAD LOCATIONS.*?<table width=".*?<a href=").*?(?=")

to:

(?&lt;=DOWNLOAD LOCATIONS.*?&lt;table width=".*?Softpedia Secure Download \(RO\).*?&lt;a href=").*?(?=")

and in the case of "Everest Ultimate Edition" would capture the third download url (the Beta version).

I must caution new users regarding pasting text directly into a template with an external editor as certain characters are converted on export from Ketarin. For example: If you paste the regular expression above into Ketarin it would be exported to a template as:

(?&lt;=DOWNLOAD LOCATIONS.*?&lt;table width=".*?Softpedia Secure Download \(RO\).*?&lt;a href=").*?(?=")

Notice the < character is converted to &lt; (its hypertext equivalent). This is the case for several other characters as well and can result in template import errors if you are not careful. Just 'food for thought' to give new users a head start! wink If you are unsure, the safest method is to paste the expression into Ketarin first, then export, and work with the template in an external editor afterwards.

CybTekSol

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

CHANGES at Softpedia on 2-24-09 have apparently broken this template. sad

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

CybTekSol wrote:

CHANGES at Softpedia on 2-24-09 have apparently broken this template. sad

The latest version of Ketarin (1.0.3) released February 24, 2009 solves the issue of their site changes for 'user-agent'. Flo had a QUICK fix for this! big_smile

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Now using

<Regex>(?:(?&lt;=\&lt;title\&gt;Download.*?<placeholder name="Stable release ?" value="\/.*?"  />\s)(\d+[\.\d+]+?.+?)(?=\s[\-\/\[]<placeholder name="Version format is 1.x.x ?" value="?"  />)<placeholder name="Capture date ?" value="|(?:[a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)?" />)</Regex>

to capture version, based on my SF template. Works with all my SP jobs, probably will do the same with the {download_url} variable. Or not. smile

-----------------------

Last edited by FranciscoR (2009-05-02 18:42:10)

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

FranciscoR wrote:

Works with all my SP jobs, probably will do the same with the {download_url} variable. Or not. smile

Looks promising FranciscoR... I'll do a little experimenting when I have the time. wink

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

I have a restricted universe here, that's no miracle for sure. 'Stable release' should be used only when you are presented multiple versions in a format like version #1/version #2 (#2 is usually stable, whereas #1 is beta). If #3 and #4 exist, adjust regex. 'Version format' excludes 'free trials' stuff from capture, but remenber using it will also exclude 'builds', 'betas', etc. from capture (i'm using it for stable releases). As for date, I could just remove it but maybe tomorow i need it, so i prefer to delete it upon pop-up. smile

-------------------------

Last edited by FranciscoR (2009-05-03 18:23:08)

Re: CybTekSol's LATEST Universal Softpedia.com Template (TEST)

Try this one instead. You need to escape ( and ) with the "Paste link text" placeholder: 'Softpedia Secure Download \(US\)' or 'Softpedia Secure Download \(RO\)'.

<?xml version="1.0" encoding="utf-16"?>
<Jobs>
  <ApplicationJob xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <DownloadBeta>Default</DownloadBeta>
    <DownloadDate xsi:nil="true" />
    <VariableChangeIndicator />
    <CanBeShared>true</CanBeShared>
    <ShareApplication>false</ShareApplication>
    <ExclusiveDownload>false</ExclusiveDownload>    
    <HttpReferer><placeholder name="Download Page URL" /></HttpReferer>
    <Variables>
      <item>
        <key>
          <string>version</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>false</RegexRightToLeft>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?:(?&lt;=\&lt;title\&gt;Download.*?<placeholder name="Stable release ?" value="\/.*?"  />\s)(\d+[\.\d+]+?.*?)(?=\s[\-\/\[]<placeholder name="Version format is 1.x.x ?" value="?"  />)<placeholder name="Capture date ?" value="|(?:[a-z]+?\s\d{1,2}[a-z]{2}\,\s\d{4}\,\s\d{2}\:\d{2})(?:\sGMT)?" />)</Regex>
            <Url><placeholder name="Info Page URL" /></Url>
            <Name>version</Name>
          </UrlVariable>
        </value>
      </item>
      <item>
        <key>
          <string>download_url</string>
        </key>
        <value>
          <UrlVariable>
            <RegexRightToLeft>true</RegexRightToLeft>
            <VariableType>RegularExpression</VariableType>
            <Regex>(?&lt;=href\=")(\w+\:\/\/[\w][\w\-?\w.:]+\/?[\w\.?=%&amp;=\-/,\)\(]*)(?=".*?<placeholder name="Paste link text"  />)</Regex>
            <Url><placeholder name="Download Page URL" /></Url>
            <Name>download_url</Name>
          </UrlVariable>
        </value>
      </item>
    </Variables>
    <ExecuteCommand>{rem} copy /y {file} {shared}</ExecuteCommand>
    <ExecutePreCommand>{rem} del /q {shared}\{appname}*.*</ExecutePreCommand>
    <Category><placeholder name="Category" value="Backup-Benchmark-Clean-Compression-Grep-Internet-Multimedia-Misc-Office-PDF-Recovery-Registry-Scrapbook-Security-Test-Virtualization" /></Category>
    <SourceType>FixedUrl</SourceType>
    <DeletePreviousFile>false</DeletePreviousFile>
    <Enabled>true</Enabled>
    <FileHippoId />
    <LastUpdated xsi:nil="true" />
    <TargetPath><placeholder name="Save to" value="{target}\{category}\" /><placeholder name="Class" value="Diagramming\DL_Manager\Extensions\Graphics\Network\Notepad\Plugins\ScreenCapture\Test\Uninstall\USB\"/><placeholder name="Rename" value="{appname}_{version}.{url:ext}" /></TargetPath>
    <FixedDownloadUrl>{download_url}</FixedDownloadUrl>
    <Name><placeholder name="AppName" /></Name>
  </ApplicationJob>
</Jobs>

Last edited by FranciscoR (2009-05-03 18:17:12)