DSM-G600, DNS-3xx and NSA-220 Hack Forum

KyleK · 2008-05-24 21:26:58

Many Transmission users have been requesting a way to download stuff automatically with the use of RSS feeds, like uTorrent does, for example.

Since the devs consider such functionality as bloat and won't implement it, I went ahead and programed my own tool called Automatic.

It's a stand-alone program written in C that runs in the background and checks a given RSS feed at an interval to check for new downloads.
You can specify what to download via regular expressions.
The app can just download the torrent files, or, if you have Transmission running, add the torrents directly to Transmission.

Latest version: v0.8.2 ffp-0.5 ffp-0.7-arm ffp-0.7-oabi Source

Install it using funpkg:

Code:

$ funpkg -i Automatic-0.8.2-1.tgz       # ffp-0.5
$ funpkg -i Automatic-0.8.2-arm-1.txz  # ffp-0.7-arm
$ funpkg -i Automatic-0.8.2-oabi-1.txz  # ffp-0.7-oabi

Dependencies:
* libcurl
* libxml-2.0
* PCRE

The package contains a configuration file which is placed at /ffp/etc/automatic.conf-sample. Save a copy as automatic.conf and change the options as to your likings.

EDIT: Updated to v0.1.1
* Fixes a bug with parsing the configuration file.

EDIT: Updated to v0.1.2.
* The config file has a new option "transmission-home". Set it to the configuration folder of Transmission (default: /mnt/HD_a2/.transmission).
* This is a change in response to [DrAg]'s problem. Now it's not necessary anymore to set the environment variable TRANSMISSION_HOME.

EDIT: Updated to v0.1.3.
* Should compile on normal Linux distributions now as well.
* Added option "torrent-folder" so you can decide where torrents will be saved
*Fixed a problem with the statefiles which only becomes apparent after quite a lot of downloads

EDIT: Updated to v0.1.3.1.
* Fixed a crash when certain environment variables weren't set on a system

EDIT: Updated to v0.1.4.
* Fixed a crash when a torrent download was unsuccessful (Partly fixes mudapw's problem)

EDIT: Updated to v0.2.
* Support for Transmission 1.3+: The torrents are added to Transmission via HTTP/JSON, no need to save the file to disk and add via transmission-remote.
* Support for multiple RSS feeds
* Removed logging to file for now, it was causing problems. You can still log though, take a look at the automatic.sh startup script

EDIT: Updated to v0.2.1.
* Fixed a rather fatal segfault bug that occured as soon as the download history went full. I'm surprised noone reported this one so far.

INFO 1: There are new options in automatic.conf-sample, and some have been removed. I strongly advise to use it as a template and create a new configuration file (you can copy over your patterns from the old one).
INFO 2: Automatic now assumes by default that you use Transmission 1.3+. If that's not the case you have to change the setting in automatic.conf

EDIT: Updated to v0.3.
* Fixed: Some feeds with secure HTTPS URLs could not be downloaded
* New: If you have multiple feeds, the state file will now remember as much items as there are in all feeds combined
* Changed: Automatic now uses the PCRE library instead of POSIX regex for regular expression parsing.
This now allows for exclusions as well.

INFO 1: The change of the regular expression library may require you to update your defined patterns. The old library recognized whitespace characters ("spaces") as any character. So you could use a pattern like "This TV Show", even though the actual name didn't contain any whitespaces.
This won't work anymore with PCRE. Now if you want to define a whitespace in your pattern, you need to use the regular expression '\s'. If you want to define any character, use the simple '.' (dot). So now the above example needs to read "This.TV.Show".

INFO 2: The switch from POSIX regular expressions to PCRE was made to allow for exclusion patterns. Say that you want to download something automatically, but only of one specific type. So, for example, you don't want to download the 720p version of some video. You can now use this pattern:

Code:

"(?!.*720p)Some.Video"

This will tell Automatic to download any torrent with the name "Some.Video" except the version that also has "720p" somewhere in its name.

You can find more information about regular expressions at http://www.regular-expressions.info/ or http://www.pcre.org

You can test all of your regular expressions with the excellent Regex Coach: http://weitz.de/regex-coach/

EDIT: Updated to v0.3.1.
* Fixed: Automatic didn't follow redirects when downloading torrents or RSS feeds

EDIT: Updated to v0.3.2.
* Fixed: Problems with the download history when Automatic was downloading many torrents in one big swoop
* Added: New option "start-torrents". When set to "yes" (default), torrents will start to download after being added to Transmission
When set to "no" they added, but won't start (Transmission will show them as "Paused")

Due to the new option the automatic.conf-sample included in the package changed. If you want to use the new option, you will have to add it to your personal automatic.conf.

As always, feedback and suggestions are always welcome. When you think you found a bug, please provide a log file so I can trace the error back.

EDIT: Updated to v0.4.
* Fixed: Transmission 1.61 broke compatibility (CSRF protection). This is fixed now
* Added: New config option "upload-limit": Set a per-torrent upload limit for each torrent added to Transmission via Automatic
* Added: New command-line option "-o/--once": Quit Automatic after checking feeds and adding torrents once. This is useful when running Automatic as a cronjob

EDIT: Updated to v0.4.1.
* Fixed: Automatic forgot to save its state since Transmission 1.71

EDIT: Updated to v0.4.2.
* Added: Automatic can now send notifications to iPhones via the Prowl service. Check http://prowlapp.com for more information.
* Added: Instead of sending torrents to Transmission, you can now use an external tool/script to deal with the torrent file (thanks to Oolan Zimmer for the patch)
* Fixed: URLs containing semicolons were parsed incorrectly
* Changed: Modified some web code to deal with "infinite timeouts" as reported by magnazan.
EDIT: Updated to v0.4.3.
* Changed: Made some code more generic to make it work with other types of files (.e.g NZB files) as well.
EDIT: Updated to v0.4.4.
* Fixed: Crash when a closing double-quote was missing inside the 'patterns' option
* Changed: Increased the buffer size for the 'patterns' option. Now you can specify 4 times the patterns you had before :)
EDIT: Updated to v0.5.
* Fixed: Automatic didn't like Transmission torrent IDs greater than 128.
* Added: New command-line parameter '-l|--log' to enable logging to a file of your choice, even in daemon mode
EDIT: Updated to v0.5.1.
* Fixed: Torrent filenames extracted from the HTTP response header were always one character too long.
EDIT: Updated to v0.6.
* Added: Support for private trackers/RSS feeds
* Added: Specify separate download folders per pattern/filter
See this post for documentation of these new features.
EDIT: Updated to v0.6.1.
* Fixed: Parsing of the :COOKIE: identifier within a feed URL didn't always work.
EDIT: Updated to v0.6.2.
* Fixed: The config parser choked on multiple newlines at the end of a config file.
EDIT: Updated to v0.6.4.
* Added: New commandline option "--append-log|-a" to not overwrite the log from a previous session.
* Fixed: Determining filename from "Content-Dispositon" might include stray double-quote.
EDIT: Updated to v0.6.5.
* Fixed: Updated Prowl API URL to prowlapp.com
* Fixed: Curl redirect issue with some RSS feeds.
* Fixed: Allow UTF8 characters in regular expressions.
EDIT: Updated to v0.6.6.
* Fixed: Issues with UTF8 characters in the config file.
* Fixed: Automatic would try to send Prowl notifications even if an invalid key was specified.
EDIT: Updated to v0.6.7.
* Fixed: Issues with websites expecting a HTTP UserAgent.
* Fixed: Ignore multi-option options (filter, feed) with multiple suboptions on the same line in the config file.
EDIT: Updated to v0.7.0.
* Added: Bind filters to a certain feed using a feed ID.
* Changed: Store RSS GUID in history file instead of item URL.
EDIT: Minor change to 0.7.0.
* Fixed: Fixed erroneous automatic.conf-sample
EDIT: Updated to v0.7.1.
* Fixed: Problems with assigning a feed ID to feeds & filters.
EDIT: Updated to v0.7.2.
* Fixed: Downloading a torrent from a feed caused a crash in some rare situations.
EDIT: Updated to v0.7.3.
* Fixed: Feeds with Magnet URIs now work correctly with Automatic.
EDIT: Updated to v0.8.0.
* Added: URL-rewriting (see this post for details)
* Added: Break very long patterns or URLs over multiple lines (see this post for details).
* Added: Reload settings when receiving a SIGHUP signal.
* Fixed: Automatic can now deal with HTTP servers that send their data in encoded form (e.g. gzip).
EDIT: Updated to v0.8.1.
* Fixed: Download history wasn't always properly restored after a reboot.
EDIT: Updated to v0.8.2.
* Fixed: Possible segfault with invalid RSS feeds
* Fixed: Possible segfault with very long download URLs or magnet URIs

Last edited by KyleK (2015-12-20 20:22:49)

vedeja · 2008-05-25 02:09:35

KyleK, it installed without problem. I added the RSS feed URL to the .conf file (inside double quotes) and attempted starting.

Code:

# /mnt/HD_a2/ffp/start/automatic.sh start
Starting /ffp/bin/automatic
Error parsing config file: Success

Last edited by vedeja (2008-05-25 02:10:23)

KyleK · 2008-05-25 06:55:14

Strange error message.
Can you provide the config file?

vedeja · 2008-05-25 22:52:34

KyleK wrote:
Strange error message.
Can you provide the config file?

I did'n change the original automatic.conf ohter than added my feed url.

KyleK · 2008-05-26 04:14:40

Did you make sure to save the file Linux-style, i.e. with only newline characters for linebreaks (instead of newline+carriage return)?

You can send me a PM if you don't want to attach the config directly to the thread.

EDIT: Found the problem and fixed it. You can download the new version 0.1.1 here.

Last edited by KyleK (2008-05-26 04:32:41)

SilentException · 2008-05-26 18:45:28

nice one kylek! are the sources available for download?

KyleK · 2008-05-26 18:52:01

Not yet. I want to do clean it up a bit and maybe add some commenting before releasing the source. It's not gonna take long though.

[DrAg] · 2008-05-27 14:17:19

Thanks KyleK,

I like your new tool. Still having some problems to get it working.

1. I added a RSS feed
2. added a pattern
3. at the moment the new torrent get downloaded tot ffp/tmp/*.torrent

How can I add the torrent directly to transmission-remote / daemon so the torrents get downloaded to a directory as soon a they appear at the RSS-feed? (I left the automatically download option to YES in the config file)

KyleK · 2008-05-27 15:17:24

If the option "use-transmission" is set to "yes" in the configuration file, Automatic will issue the following command after it has downloaded a torrent:

Code:

transmission-remote -a <path-to-torrent>

You can check if the torrent has been added by typing "transmission-remote -l" in the shell.

I can see this fail if transmission-remote does not know where the Transmission home directory is.
I suggest you add the following line to your fun_plug script:

Code:

export TRANSMISSION_HOME=/mnt/HD_a2/.transmission

[DrAg] · 2008-05-27 15:22:03

Thanks, it works!

There was no association for TRANSMISSION_HOME so transmission-remote failed to work.

Pejn · 2008-05-27 18:38:26

good work kylek, this is something i want.. ill try it out when ive updated to funplug 0.5

KyleK · 2008-05-27 19:15:55

New version 0.1.2 is out. I updated the first post.

Last edited by KyleK (2008-05-27 19:23:03)

KyleK · 2008-06-05 19:54:34

I packaged the source code of the project, link is available in first post.

KyleK · 2008-06-24 19:29:51

New version 0.1.3 is up and the links in the first post are updated.
I also put the source code on its own little web page with additional information on how to compile and use it on other systems.

Feedback is - as always - appreciated.

Have fun!

SilentException · 2008-06-24 19:48:19

thanks for the updated version and source code fix, will try it out this weekend when i'm home!

Pejn · 2008-06-25 02:48:42

Thanx for new version i tested it but getting error "Segmentation fault" when i try starting, dont know what that means, but version 1.1.2 works, ill try more tomorrow but now sleep

KyleK · 2008-06-25 02:59:13

Uh, Segmentation fault is not good at all. Please run it like this:

Code:

automatic -f -v 3 -c /ffp/etc/automatic.conf >automatic.log 2>&1

and send me the output.
Your automatic.conf would help me as well.

Last edited by KyleK (2008-06-25 03:26:00)

[DrAg] · 2008-06-25 11:34:59

Same problem here, (sorry ) logfile output:

Allocated 32 bytes (0x17008)
Allocated 28 bytes (0x17078)
Segmentation fault

Last edited by [DrAg] (2008-06-25 11:35:13)

KyleK · 2008-06-25 12:45:45

Problem fixed in v0.1.3.1
Automatic crashed because it was looking for the "TEMPDIR" environment variable, which you guys apparently haven't set.

In this case I strongly suggest you use the option "torrent-folder" in the automatic.conf, because otherwise Automatic will save torrents to /tmp by default, and we all know that /tmp is filled up pretty quickly

Alternatively, you can set TEMPDIR to /ffp/tmp, where you shouldn't have any space problems. Put this in either fun_plug or /ffp/etc/rc:

Code:

export TEMPDIR=/ffp/tmp
export TMPDIR=/ffp/tmp

[DrAg] · 2008-06-25 15:07:41

Thanks for the quick fix. Have some trouble starting, the program seems to point to wrong dir by default: better may be /ffp/etc/automatic.conf

Error parsing config file: Cannot find file '/etc/automatic.conf'

The program's works perfect. Maybe add the torrent-folder = "" by default, makes it more easy to configure. And some times it not clear you have to use "" for the parameters in the config.

Thanks for all the hard work!

KyleK · 2008-06-25 15:29:04

I chose /etc/automatic.conf as default because the tool is supposed to work on normal Linux distributions as well.
I updated the start/automatic.sh file though with the proper switches, so you shouldn't see any problems with that.

Actually I think you can omit the quotes, they're only needed if there would be whitespace within the URL.
But you're right, I should be consistent about their use in the sample config. I've updated the package with a new automatic.conf

d98jh · 2008-07-12 19:35:16

Hi! I'm using Automatic 1.3.1 which works great except for one small thing, I can't get it to start at boot.
I've "chmod a+x automatic.sh" but it still won't start. What have I missed? If I recall correctly version 1.3.0 started at boot without any problems...

Thanks!

mudapw · 2008-07-16 23:11:04

Hi KyleK, thanks for this great little thingy!

Everything works great, but at this one site I have a problem with downloads, is it because it has a ; in the url?

url = "http://www.X.net/rss.php?passkey=X&username=mudape&direct:COOKIE:uid=X;pass=X"

The log says:
[getHTTPData] response code: 302
[getHTTPData] Failed to get 'http://www.X.net/details.php?id=21998&hit=1' [response: 302]
Segmentation fault

According to site the wrong file gets downloaded because there's an error with the user/pass
url works in uTorrent. Any ideas? I tried \; but that didn't change anything, I'm a noob at this...

ps. In automatic.sh line 11 is
automatic_flags"-c /ffp/etc/automatic.conf"
should be
automatic_flags="-c /ffp/etc/automatic.conf"
right?

Cheers

Attachments:

automatic.log, Size: 5,591 bytes, Downloads: 1,603

KyleK · 2008-07-16 23:25:09

Since you apparently use a rss feed for some private torrent site, the remote server responds with a redirect message, which currently Automatic does not understand.
I'll have to look into this.
What worries me more is the segmentation fault, that should never ever happen.
I'll let you know when I know more.

PS: the semicolon shouldn't matter.

d98jh · 2008-07-22 12:59:49

d98jh wrote:
Hi! I'm using Automatic 1.3.1 which works great except for one small thing, I can't get it to start at boot.
I've "chmod a+x automatic.sh" but it still won't start. What have I missed? If I recall correctly version 1.3.0 started at boot without any problems...

Thanks!

KyleK: Any input on my problem? Thanks in advance!

DSM-G600, DNS-3xx and NSA-220 Hack Forum

Announcement

#1 2008-05-24 21:26:58

[REL] Automatic for funplug-0.5

Code:

Code:

#2 2008-05-25 02:09:35

Re: [REL] Automatic for funplug-0.5

Code:

#3 2008-05-25 06:55:14

Re: [REL] Automatic for funplug-0.5

#4 2008-05-25 22:52:34

Re: [REL] Automatic for funplug-0.5

KyleK wrote:

#5 2008-05-26 04:14:40

Re: [REL] Automatic for funplug-0.5

#6 2008-05-26 18:45:28

Re: [REL] Automatic for funplug-0.5

#7 2008-05-26 18:52:01

Re: [REL] Automatic for funplug-0.5

#8 2008-05-27 14:17:19

Re: [REL] Automatic for funplug-0.5

#9 2008-05-27 15:17:24

Re: [REL] Automatic for funplug-0.5

Code:

Code:

#10 2008-05-27 15:22:03

Re: [REL] Automatic for funplug-0.5

#11 2008-05-27 18:38:26

Re: [REL] Automatic for funplug-0.5

#12 2008-05-27 19:15:55

Re: [REL] Automatic for funplug-0.5

#13 2008-06-05 19:54:34

Re: [REL] Automatic for funplug-0.5

#14 2008-06-24 19:29:51

Re: [REL] Automatic for funplug-0.5

#15 2008-06-24 19:48:19

Re: [REL] Automatic for funplug-0.5

#16 2008-06-25 02:48:42

Re: [REL] Automatic for funplug-0.5

#17 2008-06-25 02:59:13

Re: [REL] Automatic for funplug-0.5

Code:

#18 2008-06-25 11:34:59

Re: [REL] Automatic for funplug-0.5

#19 2008-06-25 12:45:45

Re: [REL] Automatic for funplug-0.5

Code:

#20 2008-06-25 15:07:41

Re: [REL] Automatic for funplug-0.5

#21 2008-06-25 15:29:04

Re: [REL] Automatic for funplug-0.5

#22 2008-07-12 19:35:16

Re: [REL] Automatic for funplug-0.5

#23 2008-07-16 23:11:04

Re: [REL] Automatic for funplug-0.5

#24 2008-07-16 23:25:09

Re: [REL] Automatic for funplug-0.5

#25 2008-07-22 12:59:49

Re: [REL] Automatic for funplug-0.5

d98jh wrote:

Board footer