Plugins ======= .. toctree:: :maxdepth: 2 -------- Overview -------- The fuglu core does nothing except receiving mails from postfix and sending them back. All functionality is written in plugins which can be enabled or disabled at will. Fuglu provides plugins for the most common mail filtering requirements, but if some functionality is missing, it is easy to add new plugins without knowing all of fuglu's internals. ---------------- Types of plugins ---------------- Scanner Plugins ............... Scanner plugins are the most important type of plugins. They do the actual mail content filtering. Plugins are run in the order specified in the ``plugins=`` configuration option. Each plugin returns an action for the message: * **DUNNO** : This plugin decides not to take any final action, continue with the next plugin (this is the most common case) * **ACCEPT** : Whitelist this message, don't run any remaining plugins * **DELETE** : Silently delete this message (The sender will think it has been delivered) * **DEFER** : Temporary Reject (4xx error), used for error conditions in after-queue mode or things like greylisting in before-queue mode * **REJECT** : Reject this message, should only be used in before-queue mode (in after-queue mode this would produce a bounce / backscatter) If one of the plugins returns something other than *DUNNO*, all remaining scanner plugins are skipped. If all plugins return DUNNO, the message is accepted and re-injected to postfix. Prepender Plugins ................. Prepender plugins run before the scanner plugins and have the ability to alter the list of scanner plugins to be run. This can be used for example to have different plugins run for incoming or outgoing mails or to skip whitelisted messages. Appender Plugins ................ Appender plugins are run after the scanner plugins when the message has already been re-injected into postfix (or rejected or deleted or...) so they don't increase your scanning time . They are mostly used for statistical tasks, updating counters in a database etc. -------------------- Plugin configuration -------------------- Each plugin has its own configuration section in ``/etc/fuglu/fuglu.conf`` or any ``*.conf`` in ``/etc/fuglu/conf.d``. The section is usually named like the plugin itself. For example, the Spamassassin Plugin's Name is ``SAplugin``, so it would search for a ``[SAPlugin]`` config section. Suspect Filters ............... SuspectFilter are special rule files used by many fuglu plugins. Often they define actions a plugin should take based on a message header or suspect tag. The format is : `` `` ```` can be any of... * a email header name, e.g. ``Received``, ``To``, ``From``, ``Subject`` ... also supports '*' as wildcard character * ``mime:headername`` to get mime Headers in the message payload eg: ``mime:Content-Disposition`` * one of fuglus builtin names to get envelope data: * ``envelope_from`` or ``from_address`` (the envelope from address) * ``from_domain`` (domain part of envelope_from) * ``envelope_to`` or ``to_address`` (envelope to address) * ``to_domain`` (domain part of envelope_to) * a ``message Tag`` prepended by the @ symbol, eg. ``@incomingport`` * ``body:raw`` to match the the decoded message body (only applies to text/* parts) * ``body:stripped`` or just ``body`` to match the the message body (only applies to text/* parts), with stripped tags and newlines replaced with space (similar to SpamAssassin body rules) * ``body:full`` to match the full body ```` is either a standard python regular expression or a numeric comparison applied to ````. if the regex is not enclosed by forward slashes, be sure not to include any whitespace in the regex, it must be one continous string. To match whitespace, use ``\s``. all regexes that are not enclosed in forward slashes are automatically ``case insensitive`` and support multiple lines (re.DOTALL is enabled, so the newline character is matched by a dot). with slashes, the regex flags can be defined manually, eg. :: Subject /hello world/i match hello world, case insensitive If regex is a numeric comparison the format must be ```` op must be one of ``<, >, <=, >=, ==, !=`` it also supports ``= and <>`` for equals resp unequals. The numeric value can be either integer or float. If the comparison is agains a nun-numeric value the result will always be False. ```` depends on the plugin that reads this header filter file, some don't need arguments at all. Please refer to the plugin documentation. Filter files are automatically reloaded if you change anything, you don't need to restart fuglu. Example: :: #normal header test Subject hello Hello in the subject! MIME-Version ^1\.0$ Mime Version is 1.0 #builtin special fields to_domain (\.)?fuglu\.org$ Sent to fuglu.org or any subdomain envelope_from ^sender@example\.com$ #match a tag from a previous plugin @SAPlugin.report MISSING_HEADER #wildcard X-Spam-* .* a X-Spam- header exists #decoded body text parts body Viagra #full body body:full ^--SPAMMY-MIME-BOUNDARY #mime-headers mime:Content-Type ^application\/x-msdos-program$ see :ref:`debug-suspect-filter` for info on how to debug suspect filter files on specific messages. Template Variables .................. Some plugins allow you to create templates (errormessages in bounces, reject reasons, ...). Here is a list of commonly used template variables that should be available in those templates. Plugins may not support all those variables and can add more variables not listed here. If unsure, check the plugin documentation. * ``${from_address}`` : Envelope Sender Address * ``${to_address}`` : Envelope Recipient Address * ``${from_domain}`` : Envelope Sender Domain * ``${to_domain}`` : Envelope Recipient Domain * ``${timestamp}`` : Unix Timestamp when the message was received * ``${subject}`` : Message subject * ``${date}`` : Current date * ``${time}`` : Current time * ``${blockinfo}`` : Reason why this message was blocked (available in the attachment plugin) ------------------------- Plugins included in Fuglu ------------------------- .. include:: includedplugins-autogen.txt Extra plugins ............. Note, in addition to the plugins included in the fuglu distribution there are additional contributed plugins available in the fuglu-extra-plugins repository: https://gitlab.com/fumail/fuglu-extra-plugins ------------------------ Writing your own plugins ------------------------ Assuming you know python basics, writing plugins for fuglu is very easy. All you have to do is create a new class which extends from ``ScannerPlugin``, override ``__str__`` to provide a nice human readable name and override ``examine`` to do the actual work of your plugins. ``examine`` should return one of the action codes above (DUNNO, DEFER, DELETE, ....) and optionally a reason for the action. (e.g. ``return DEFER, 'please try again later'`` In plugin you usually only have to import things from :ref:`fuglu.shared` , so it's probably a good idea to get familiar with that module. This is a quick example of how your plugin code skeleton would look like: :: from fuglu.shared import ScannerPlugin,DUNNO class DemoPlugin(ScannerPlugin): """Copy this to make a new plugin""" def __init__(self,config,section=None): ScannerPlugin.__init__(self,config,section) #config example #self.requiredvars={ # 'maxsize':{ # 'default':'1337', # 'description':'Maximum message size', # } #} def examine(self,suspect): #config Example #maxsize=self.config.getint(self.section, 'maxsize') #debug example #self._logger().debug('hello world from DemoPlugin') #PUT PLUGIN CODE HERE return DUNNO First of all, you need a few imports. ScannerPlugin (so you can extend from it), and possible return values for your Plugin, ``DUNNO`` might be enough depending on what your plugin does. :: from fuglu.shared import ScannerPlugin,DUNNO in ``__init__`` you only call BasicPlugin's ``__init__`` for now. This sets ``self.config`` and ``self.section`` on the object where you later can read config options (eg. ``self.config.get(self.section,'somevalue')``. Do **NOT** load the plugin configuration here. ``__init__`` is only called once when fuglu starts up. Always load plugin config in ``examine``. :: def __init__(self,config,section=None): ScannerPlugin.__init__(self,config,section) then code your ``examine`` method. You will have to work with :ref:`api-Suspect`, which is a representation of the message being analyzed. The suspect has ``tags`` that are read and written by plugins. You can tag a message as virus, as spam, define your own tags, read tags from previous plugins... it's probably a good idea to look at the Suspect class in ``fuglu.shared`` to get a list of things you can do with the suspect. Common Tasks ("API" FAQ) ........................ Define configuration options for your plugin -------------------------------------------- In order to make 'lint' and 'fuglu_conf' work with your plugin it should tell the core what config options it expects. this is done by creating a dictionary named 'requiredvars' in the plugins init: Example: :: def __init__(self,config,section=None): ScannerPlugin.__init__(self,config,section) self.requiredvars={ 'host':{ 'default':'localhost', 'description':'hostname', }, 'username':{ 'default':'root', }, 'password':{ 'default':``, 'confidential':True, }, } This would tell fuglu that your plugin has three config options: host, username and password. The 'dict of dicts' uses your config option name as key for the outer dict. The inner dict supports the following keys: * ``default`` - default value, used if the option is not specified in the config file * ``section`` - config section to check. by default fuglu assumes that the plugin reads its own config section. override this if your plugin requires a config option from a different plugin or from the main config * ``confidential`` - set this to True if fuglu_conf should treat this option confidential and redact it in 'fuglu_conf -n' (passwords etc) * ``validator`` - function that should be called to validate if the configured value is valid. the function will receive the value as argument and must return True or False * ``deprecated`` - mark a config option as deprecated Read the config --------------- Configs in fuglu are stored in ``/etc/fuglu/fuglu.conf`` (or any .conf file in /etc/fuglu/conf.d ) in ini-style format. Your plugin gets its own section named like the plugin class. Example: :: [DemoPlugin] maxsize=10382 You can then read the config in your plugin with the usual methods from a python ConfigParser object ( http://docs.python.org/library/configparser.html ) :: maxsize=self.config.getint(self.section, 'maxsize') Important: always load the configs in ``examine`` (and not in ``init`` !). Reading the config in init breaks loading default values and prevents on-the-fly config reload Get the message source ---------------------- Use ``suspect.get_source()`` to get the message source. The maxbytes option allows you to get only a part of the source. Reading the whole source can slow down scanning of big messages. Add headers ----------- :: suspect.add_header(headername,headervalue) By default, headers are added to the message shortly before it is re-injected into postfix. add ``immediate=True`` to add the header immediately, so other plugins can see it. Replace headers --------------- Here's an example on how to change the subject of a message. Note: It's generally not recommended to change existing headers, this breaks DKIM. :: msgrep = suspect.get_message_rep() msgrep.replace_header('Subject', 'Scanned: '+msgrep['Subject']) suspect.set_message_rep(msgrep) Write to the log ---------------- Your plugin has a _logger method that returns a standard python logger object, so you can use the info / debug / error / fatal methods. Example: :: self._logger().info('hello world from DemoPlugin') Write debug info ---------------- To make the plugin write special debug info when ``fuglu_debug`` is used, use: :: suspect.debug('debug info from DemoPlugin!') Make plugin '--lint'-able ------------------------- By default, ``lint()`` only validates the plugin's configuration settings from ``self.requiredvars``. You can override ``lint()`` to do more stuff. * use simple ``print`` in this method, no logging stuff. * if you override ``lint()`` you should run ``self.check_config()`` to test the configuration * ``lint()`` must return True or False Example of a plugin that would check if an imap account is available: :: def lint(self): allok=(self.check_config() and self.lint_imap()) return allok def lint_imap(self): try: M=imaplib.IMAP4(self.config.get(self.section, 'host')) (type,data)=M.login(self.config.get(self.section, 'user'),self.config.get(self.section, 'password')) if type!='OK': print('Could not login to imap review account: %s',data) return False return True except Exception, e: print "Could not login to imap host:%s - Error %s"%(self.config.get(self.section, 'host'),e) return False Use the 'SuspectFilter' ----------------------- SuspectFilters are a common way for all plugins to perform an action based on conditions defined in a filterfile . These files are automatically re-loaded if changed. * import SuspectFilter from fuglu.shared * define a config variable for your plugin which holds the name of the filter file (not strictly required, you could hardcode the path) * create a plugin property for the filter :: from fuglu.shared import ScannerPlugin,SuspectFilter [...] class MyPlugin(ScannerPlugin): def __init__(self,config,section=None): ScannerPlugin.__init__(self,config,section) self.requiredvars={ 'myrulesfile':{ 'default':'/etc/fuglu/mypluginrules.regex', 'description':'Filter file for my plugin', }, [...] } self.filter=None In ``examine`` create the filter if necessary :: if self.filter==None: self.filter=SuspectFilter(self.config.get(self.section, 'myrulesfile')) Run the filter in examine: ``(match,arg)=self.filter.matches(suspect)`` ``match`` is a boolean, telling you if one of the rules matched `arg`` is an additional argument which have been appended to the filter rule in the config. lets say, the filter rule reads ``to_address example@fuglu.org hello world!``, you would get match=True and arg='Hello world!' if the message is sent to example@fuglu.org :: (match,arg)=self.filter.matches(suspect) if match: if arg!=None: self.logger.debug("""we got a match with arg %s"""%arg) else: self.logger.debug("""We got a match without an arg""") else: suspect.debug("no rule matches") Use the sql extension --------------------- TODO (DBFiles, sqlalchemy connections) Debugging ......... Get a stacktrace ---------------- if something went wrong you should see a stacktrace in the fuglu log (/var/log/fuglu/fuglu.log) with fuglu >=0.6.0 you can also get the most recent exceptions with the following command: :: fuglu_control exceptionlist Debug the plugin while fuglu is runnig -------------------------------------- run ``fuglu --console`` to enter an interactive python console after fuglu startup. Your plugin is then available via the list ``mc.plugins`` Debug a plugin without running fuglu ------------------------------------ ``plugdummy.py`` is a tool that makes plugin development and testing much easier by creating a minimal fuglu environment for the plugin to run. it doesn't require a running fuglu or postfix. it will create a dummy suspect, call the plugin's examine method and print the result (and debug output). the generated input messag is stored as: ``/tmp/fuglu_dummy_message_in.eml`` if your plugin modified the message source, the resulting message can be found at ``/tmp/fuglu_dummy_message_out.eml`` ``plugdummy.py`` is located in the ``develop/scripts`` directory. simple usage: assuming your plugin file ('myplugin.py') is in ``/usr/local/fuglu/plugins`` you can run ``plugdummy.py `` :: #./plugdummy.py myplugin.ExamplePlugin INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml INFO:root:Running plugin: ExamplePlugin INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world! INFO:root:Result: DUNNO INFO:root:Suspect a7babc1e4cfe49c36710065966e6ed0a: from=sender@fuglu.local to=recipient@fuglu.local size=231 spam=no virus=no modified=no tags={'virus': {}, 'spam': {}, 'highspam': {}} Advanced usage: run ``plugdummy.py --help`` to get a list of all options Examples: Running plugins from a different directory :: ./plugdummy.py -p /tmp/ myplugin.ExamplePlugin Change sender / recipient the '-s' and '-f' options change the envelope sender/recipient. -r can be specified multiple times to simulate a multi-recipient message :: ./plugdummy.py -s me@example.org -r you@example.net myplugin.ExamplePlugin INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml INFO:root:Running plugin: ExamplePlugin INFO:fuglu.plugin.ExamplePlugin:me@example.org greets you@example.net: hello world! INFO:root:Result: DUNNO INFO:root:Suspect 423e02e1461cd1c314ac9a409176c4f4: from=me@example.org to=you@example.net size=221 spam=no virus=no modified=no tags={'virus': {}, 'spam': {}, 'highspam': {}} Adding headers to the input message :: ./plugdummy.py -h 'subject:yo! whassup' myplugin.ExamplePlugin [...] cat /tmp/fuglu_dummy_message_in.eml Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: sender@fuglu.local To: recipient@fuglu.local Subject: yo! whassup Date: Fri, 01 Jun 2012 12:58:34 -0000 hello, world! Adding tags: :: ./plugdummy.py -t 'mytag:myvalue' myplugin.ExamplePlugin INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml INFO:root:Running plugin: ExamplePlugin INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world! INFO:root:Result: DUNNO INFO:root:Suspect 168268d6ff2c2748454183efcb554242: from=sender@fuglu.local to=recipient@fuglu.local size=231 spam=no virus=no modified=no tags={'spam': {}, 'virus': {}, 'mytag': 'myvalue', 'highspam': {}} Setting a config option: :: ./plugdummy.py -o 'greeting:go away!' myplugin.ExamplePlugin INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml INFO:root:Running plugin: ExamplePlugin INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: go away! [...] Setting the message body: :: #set body ./plugdummy.py -b 'hi there, whassup!' myplugin.ExamplePlugin #read body from file ./plugdummy.py -b bla.txt myplugin.ExamplePlugin #read headers & body from eml file ./plugdummy.py -e /tmp/bla.eml myplugin.ExamplePlugin Running a interactive console in the dummy enrivonment: :: ./plugdummy.py -c myplugin.ExamplePlugin INFO:root:Input file created as /tmp/fuglu_dummy_message_in.eml INFO:root:Running plugin: ExamplePlugin INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world! INFO:root:Result: DUNNO INFO:root:Suspect 3cf496cbe2a1097abc37ebda5a645cd2: from=sender@fuglu.local to=recipient@fuglu.local size=231 spam=no virus=no modified=no tags={'virus': {}, 'spam': {}, 'highspam': {}} Fuglu Interactive Console started pre-defined locals: {'config': , 'suspect': , 'result': 0, 'plugin': } >>> plugin.requiredvars {'greeting': {'default': 'hello world!', 'description': 'greeting the plugin should log to the console'}} >>> plugin.examine(suspect) INFO:fuglu.plugin.ExamplePlugin:sender@fuglu.local greets recipient@fuglu.local: hello world! 0 >>> config.set('ExamplePlugin','greeting','Greetings, Earthling') >>> suspect.from_address='me@example.org' >>> plugin.examine(suspect) INFO:fuglu.plugin.ExamplePlugin:me@example.org greets recipient@fuglu.local: Greetings, Earthling 0 >>> Deploying Plugins ................. * If there is no ``plugindir`` set in fuglu.conf yet, define a new directory for custom plugins. eg ``/usr/local/fuglu/plugins``. * Copy your plugin file to this directory * Depending on the type of your plugin, add it to the plugin/prependers/appenders config option. Eg. if your scanner plugin class is ``MyHeloPlugin`` in the file myplugin.py you would add ``myplugin.MyHeloPlugin`` to the ``plugins`` config * If your plugin reads configuration entries, make sure those are present in fuglu.conf or in a custom conf-file in /etc/fuglu/conf.d * Run ``fuglu --lint`` to check if fuglu is happy with your new plugin * (Re)start fuglu