Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
en:architecture [10.09.2018 15:18] – mach@cesnet.czen:architecture [01.04.2025 10:38] (current) – Switch links from Redmine to Gitlab ph_cesnet.cz
Line 1: Line 1:
 ====== Architecture ====== ====== Architecture ======
  
-The //Mentat// system has been designed as a distributed modular system with an emphasis on its easy extendability and scalability. The core of the system reflects the architecture of MTA system [[http://www.postfix.org/|Postfix]]. It consists of many simple modules/daemons, each of which responsible for performing a particular task. This approach enables smooth process-level parallelization and extendability. All modules use the same core framework built on top the [[https://alchemist.cesnet.cz/pyzenkit/doc/production/html/manual.html|PyZenKit]] framework, thus making implementing new modules an easy task. +The //Mentat// system has been designed as a distributed modular system with an emphasis on its easy extendability and scalability. The core of the system reflects the architecture of MTA system [[http://www.postfix.org/|Postfix]]. It consists of many simple modules/daemons, each of which responsible for performing a particular task. This approach enables smooth process-level parallelization and extendability. All modules use the same core framework built on top the PyZenKit framework, thus making implementing new modules an easy task. 
  
 Mentat itself does not have any network communication protocol for receiving events or messages from the outside (however nothing stops you from implementing your own module). Instead it relies on the services of [[https://warden.cesnet.cz/en/index|Warden]] system, which is the security information sharing platform. Mentat itself does not have any network communication protocol for receiving events or messages from the outside (however nothing stops you from implementing your own module). Instead it relies on the services of [[https://warden.cesnet.cz/en/index|Warden]] system, which is the security information sharing platform.
Line 10: Line 10:
 ^ Database | [[https://www.postgresql.org/|PostgreSQL]] | ^ Database | [[https://www.postgresql.org/|PostgreSQL]] |
 ^ Data model | [[https://idea.cesnet.cz/|IDEA]] | ^ Data model | [[https://idea.cesnet.cz/|IDEA]] |
-^ Git repository | ''git clone https://alchemist.cesnet.cz/mentat/repo.git mentat'' | +^ Git repository | ''git clone https://gitlab.cesnet.cz/713/mentat/mentat.git mentat'' | 
-^ Ticket system | [[https://homeproj.cesnet.cz/projects/mentat|Mentat@homeproj.cesnet.cz]] | +^ Ticket system | [[https://gitlab.cesnet.cz/713/mentat/mentat|Mentat @ gitlab.cesnet.cz]] | 
-^ Package format | deb, tar [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/installation.html|(installation manual)]]| +^ Package format | deb, tar [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/installation.html|(installation manual)]]| 
-^ Documentation | [[https://alchemist.cesnet.cz/mentat/doc/production/html/manual.html|link]] |+^ Documentation | [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/manual.html|link]] |
  
 ===== Current system architecture ===== ===== Current system architecture =====
Line 19: Line 19:
 The diagram below provides an overview of the existing architecture of the //Mentat// system. The diagram below provides an overview of the existing architecture of the //Mentat// system.
  
-{{ ::mentat-architecture.png?nolink |Aktuální stav architektury systému Mentat}}+{{ ::mentat-architecture.png?nolink |Current Mentat system architecture}}
  
 The implementation language is strictly [[https://www.python.org/|Python3]] with no attempts whatsoever to be The implementation language is strictly [[https://www.python.org/|Python3]] with no attempts whatsoever to be
Line 25: Line 25:
  
 The //Mentat// system consists of tools allowing processing events both in real time and retrospectively over a particular period of time. At present, the following most important modules for real time processing are available: The //Mentat// system consists of tools allowing processing events both in real time and retrospectively over a particular period of time. At present, the following most important modules for real time processing are available:
-  * **mentat-inspector.py**\\ This module enables the processing of [[https://idea.cesnet.cz/|IDEA]] messages based on the result of given filtering expression. There is a number of actions that can be performed on the message in case the filtering expression evaluates as ''true''. The most common and useful usecases are message classification, verification, filtering or conditional procesing branching. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-inspector.html|(more information)]] +  * **mentat-inspector.py**\\ This module enables the processing of [[https://idea.cesnet.cz/|IDEA]] messages based on the result of given filtering expression. There is a number of actions that can be performed on the message in case the filtering expression evaluates as ''true''. The most common and useful usecases are message classification, verification, filtering or conditional procesing branching. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-inspector.html|(more information)]] 
-  * **mentat-enricher.py**\\ This module enables the enrichment of incoming [[https://idea.cesnet.cz/|IDEA]] messages with additional information, like resolving target abuse`s contact (for the reporting purposes), geolocation and ASN resolving. Implementation of further enrichment operations is planned and custom enrichment plugins are supported (hostname/ip resolving, passive DNS, …) [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-enricher.html|(more information)]] +  * **mentat-enricher.py**\\ This module enables the enrichment of incoming [[https://idea.cesnet.cz/|IDEA]] messages with additional information, like resolving target abuse`s contact (for the reporting purposes), geolocation and ASN resolving. Implementation of further enrichment operations is planned and custom enrichment plugins are supported (hostname/ip resolving, passive DNS, …) [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-enricher.html|(more information)]] 
-  * **mentat-storage.py**\\ This module enables to store incoming [[https://idea.cesnet.cz/|IDEA]] messages in a database ([[https://www.postgresql.org/|PostgreSQL]]). [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-storage.html|(more information)]]+  * **mentat-storage.py**\\ This module enables to store incoming [[https://idea.cesnet.cz/|IDEA]] messages in a database ([[https://www.postgresql.org/|PostgreSQL]]). [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-storage.html|(more information)]]
      
 Most modules enabling retrospective event processing are based on regularly re-launched scripts (i.e. **crons**). At present moment the following modules enabling retrospective event processing are available: Most modules enabling retrospective event processing are based on regularly re-launched scripts (i.e. **crons**). At present moment the following modules enabling retrospective event processing are available:
-  * **mentat-statistician.py**\\ This module enables statistical processing of events over a given self-defined period. At present, the feature is configured to five-minute intervals. For each of these intervals, it determines the frequency of events according to detector type, event type, IP address etc. These statistical reports are stored in a separate database and can later support an overview of system’s operation, provide underlying data for other statistical reports or for the creation of dictionaries for a web interface. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-statistician.html|(more information)]] +  * **mentat-statistician.py**\\ This module enables statistical processing of events over a given self-defined period. At present, the feature is configured to five-minute intervals. For each of these intervals, it determines the frequency of events according to detector type, event type, IP address etc. These statistical reports are stored in a separate database and can later support an overview of system’s operation, provide underlying data for other statistical reports or for the creation of dictionaries for a web interface. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-statistician.html|(more information)]] 
-  * **mentat-reporter.py**\\ This module enables to distribute periodical event reports directly to end abuse contacts of responsible network administrators. More information about the reporter as a service provided by [[https://www.cesnet.cz/?lang=en|CESNET, a.l.e]]. can be found at official [[https://csirt.cesnet.cz/cs/services/mentat|Mentat service]] webpage. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-reporter.html|(more information)]] +  * **mentat-reporter.py**\\ This module enables to distribute periodical event reports directly to end abuse contacts of responsible network administrators. More information about the reporter as a service provided by [[https://www.cesnet.cz/?lang=en|CESNET, a.l.e]]. can be found at official [[https://csirt.cesnet.cz/cs/services/mentat|Mentat service]] webpage. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-reporter.html|(more information)]] 
-  * **mentat-informant.py**\\ This module is similar to the above described reporter. It provides periodical summary reports on system’s statuses and reports sent. It is most useful for system administrators or for target abuse contacts as status overview. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-informant.html|(more information)]]+  * **mentat-informant.py**\\ This module is similar to the above described reporter. It provides periodical summary reports on system’s statuses and reports sent. It is most useful for system administrators or for target abuse contacts as status overview. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-informant.html|(more information)]]
  
 Little bit on the side is a big collection of utility and management scripts and Little bit on the side is a big collection of utility and management scripts and
 tools that attempt to simplify repeated dull tasks for the system administrator. Some of the most useful ones are following: tools that attempt to simplify repeated dull tasks for the system administrator. Some of the most useful ones are following:
  
-  * **mentat-controller.py**\\ A script enabling to control all configured deamons/modules on a given server.[[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-controller.html|(more information)]] +  * **mentat-controller.py**\\ A script enabling to control all configured deamons/modules on a given server.[[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-controller.html|(more information)]] 
-  * **mentat-backup.py**\\ A configurable module enabling periodical database backups. At present, a full backup of system tables (users, groups …) is created once a day while [[https://idea.cesnet.cz/|IDEA]] event table is backed up incrementally. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-backup.html|(more information)]] +  * **mentat-backup.py**\\ A configurable module enabling periodical database backups. At present, a full backup of system tables (users, groups …) is created once a day while [[https://idea.cesnet.cz/|IDEA]] event table is backed up incrementally. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-backup.html|(more information)]] 
-  * **mentat-cleanup.py**\\ A configurable module enabling periodical database and filesystem cleanups. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/bin_mentat-cleanup.html|(more information)]]+  * **mentat-cleanup.py**\\ A configurable module enabling periodical database and filesystem cleanups. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/bin_mentat-cleanup.html|(more information)]]
  
 The last important component of the system is a web user interface: The last important component of the system is a web user interface:
-  * **Hawat**\\ Customizable and easily extentable web user interface based on [[http://flask.pocoo.org/docs/1.0/|Flask]] microframework. [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/hawat.html|(more information)]]+  * **Hawat**\\ Customizable and easily extentable web user interface based on [[http://flask.pocoo.org/docs/1.0/|Flask]] microframework. [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/hawat.html|(more information)]]
  
  
 ===== Module architecture ===== ===== Module architecture =====
  
-As mentioned above, all system modules, including continuously running deamons or periodically launched scripts, use a simple common framework called +As mentioned above, all system modules, including continuously running deamons or periodically launched scripts, use a simple common framework called PyZenKit, which ensures all common features:
-[[https://alchemist.cesnet.cz/pyzenkit/doc/production/html/manual.html|PyZenKit]], which ensures all common features:+
  
   * Application life-cycle management.   * Application life-cycle management.
Line 102: Line 101:
 ==== Web interface architecture ==== ==== Web interface architecture ====
  
-The web interface for Mentat system is called [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/hawat.html|Hawat]] and it is built on top of the great [[http://flask.pocoo.org/docs/1.0/|Flask]] microframework. However the *micro* in the name means, that to make things more manageable and+The web interface for Mentat system is called [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/hawat.html|Hawat]] and it is built on top of the great [[http://flask.pocoo.org/docs/1.0/|Flask]] microframework. However the *micro* in the name means, that to make things more manageable and
 easier a suite of custom tools had to be implemented to enable better interface component integration. easier a suite of custom tools had to be implemented to enable better interface component integration.
  
-[[http://flask.pocoo.org/docs/1.0/|Flask]] already provides means for separating big applications into modules by the **blueprint** mechanism. This is used very extensively and almost everything in the web interface is a pluggable blueprint.+[[http://flask.pocoo.org/docs/1.0/|Flask]] already provides means for separating big applications into modules by the [[http://flask.pocoo.org/docs/1.0/blueprints/|blueprint]] mechanism. This is used very extensively and almost everything in the web interface is a pluggable blueprint.