en:architecture

Architecture

The Mentat system has been designed as a distributed modular system with an emphasis on its easy extendability and scalability. The core of the system reflects the architecture of MTA system Postfix. It consists of many simple modules/daemons, each of which responsible for performing a particular task. This approach enables smooth parallelization and extendability. All modules use the same core service framework, thus making implementing new modules an easy task.

The original Mentat’s design presupposed features and tools enabling to collate and share security information. This function has, however, later been taken over by a twin project Warden with slightly humbler ambitions and simpler but ultimately better design. At present, the Warden system has profiled as a single communication channel for sharing security information and the Mentat system as a tool for streamlined security information processing. Mentat’s source codes still contain some remains of protocols and components for data sharing between remote nodes.

Technical background

Implementation languages Perl, Python (the aim is to write any new code in Python, and gradually re-write the whole system in it)
Database MongoDB
Data model IDEA
Git repository git clone [uid]@homeproj.cesnet.cz:mentat (so far only internal)
Package format deb, tar (installation guides)

Current system architecture

The diagram below provides an overview of the existing architecture of the Mentat system.

Aktuální stav architektury systému Mentat

The Mentat system consists of tools allowing processing events both in real time and retrospectively over a particular period of time. At present, the following modules for real time processing are available:

  • mentat-inspector
    This feature enables the processing of IDEA messages based on the result of given filtering expression. There is a number of actions that can be performed on the message in case the filtering expression evaluates as true.
  • mentat-enricher
    This feature enables the enrichment of incoming data through the following sequence of tasks: IDEA notification validation, resolving target abuse’s contact (for the reporting purposes), detection of event’s specific type (to enable notification formatting). Implementation of further operations is planned: hostname/ip resolving, passive DNS, geolocation, …
  • mentat-storage
    This feature enables to store incoming IDEA notifications in a database (MongoDB).

Most features enabling retrospective event processing are based on regularly re-launched scripts (i.e. crons). At present, the following features enabling retrospective event processing are available:

  • mentat-statistician
    This feature enables statistical processing of events over a given self-defined period. At present, the feature is preset to five-minute intervals. For each of these intervals, it determines the frequency of events according to detector type, event type, IP address etc. These statistical reports are stored in a separate database and can later support an overview of system’s operation, provide underlying data for other statistical reports or for the creation of dictionaries for a web interface.
  • mentat-reporter-ng
    This feature enables to distribute periodical event reports directly to end abuse contacts of responsible network administrators. More information about the reporter can be found at reporter’s website.
  • mentat-briefer
    This feature is similar to the above described reporter. It provides periodical summary reports on system’s statues and reports sent.
  • mentat-backup
    A configurable feature enabling periodical database backups. At present, a full backup of system collections (users, groups …) is created once a day while event collections are backed up incrementally.
  • mentat-cleanup
    A configurable feature enabling periodical database cleanups.
  • mentat-precache
    A configurable feature enabling data caching, in particular of various dictionaries for web interface.
  • hawat-negistry
    A feature enabling data synchronisation between Negistry and Mentat’s system database. It synchronises abuse groups and address blocks assigned to them.

The last important components of the system are administrative interfaces:

  • hawat
    A web interface for the Mentat system. The interface enables in particular to search through the event database and sent reports, system statistics and overviews and to configure the entire system and the reporting algorithm in particular.
  • hawat-cli
    CLI interface for system administrators enabling the automation of certain acts relating to the administration of the Mentat system.
  • mentat-controler
    A script enabling to control particular deamons/components on a given engine.

Current component architecture

As mentioned above, all system features, including continuously running deamons or periodically launched scripts, use a simple implementation framework which ensures all common actions:

  • Configuration loading and validation;
  • Deamonisation;
  • Log initialisation;
  • Database abstract layer;
  • Abstract layer for working with IDEA messages;
  • Statistical data processing;
  • WHOIS queries, DNS resolving;
  • Formatting and report distribution.

All continuously running deamons operate as ‘pipes’, i.e. the report enters on one side, the deamon performs relevant operations and the report reappears on the other side. To facilitate report exchange between individual deamons, alike in MTA Postfix, the file system and queues implemented by means of files and directories are used. Thus, all deamons alike use the predefined feature Mentat::Processor which ensures correct, easy and configurable configuration upload, log setting, deamonisaton, launches the processing using event service, correct ending at the end, etc. When implementing a new deamon, one only needs to configure the processing; everything else is provided for automatically, including the selection of a report from the queue and subsequent upload into the queue of another deamon in the processing chain.

Last modified: 20.03.2017 09:35