The Mentat project has seen a number of changes covering the user interface, infrastructure and API.
Many modifications have been made to the UI in an effort to move closer to the needs of both SOC analysts and users. A new severity level of „info“ has been introduced with a long retry interval (service deprecation warnings, minor configuration issues, …), as continued integration with Auror and vulnerability management. We have added full-text search capability in IDEA Description and its display on the timeline. Support for working with new IDEA fields for login credentials used in an attack or time inaccuracy.
We have introduced event reporting by target for collaboration with proactive and IPS tools. Reports can also now be searched by class, target address, port or detectors. Related reports are now linked, including support for mail threads.
Registration page supports links to company identity and legal documents.
In addition, a new page has been created for detailed IP address information, which serves as an entry point for various tools such as Amfora and OTRS.
Many small changes have also taken place - reporting settings UI streamlining, the new group search API, upgrading libraries, droves of bugfixes and more.
Please, visit the issue tracker for the list for related issues: 2.13.1 2.13.2 2.14
Release statistics: 453 commits, 14952 additions, 12643 deletions
Two big features are finished in this release - rework of the class configuration and replacement of the graphing library.
Until now the class configuration was done partially by inspector rule config, partially by Jinja templates, JSON files and specific Babel based translations on filesystem. Now the class definition is simplified to the definition of the rules, wanted columns in the reports and description/link, and together moved to one place of the web interface.
Classes are now also two tiered - first level is the same as before, where as the second level allows to aggregate by more complex rules.
Together with reporting we are working on filtering interface - we have added the possibility to specify target IPs, protocols and classes in the simplified filter rule definitions. Also, filter notification to relevant group admins is streamlined.
We have embraced the Plotly graphing library instead of NVD3, development of which have stopped to the halt. It was also necessary to rewrite most of the backend graphing code, which was closely tied to NVD3.
As we are now able to import network/group data from external sources, we have added a couple of support scripts for a couple of CESNET related organisations.
We have of course fixed usual slew of the bugs pretty much everywhere (searching by storage time, canceling some forms, access rights problems, filter details, visual problems, and also set of development specifics).
Also, the hosts module was removed (event search and timeline can do much more).
Please, visit the issue tracker for the list for related issues: 2.12.1 2.13
Release statistics: 321 commits, 15625 additions, 11962 deletions
The main visible feature is the revamp of the whole web interface - with the upgrade of the underlying Bootstrap library came a lot of changes. In spite of some more invasive changes (selection lists for example), we tried hard to keep the overall logic and feel the same. Also, underlying active (javascript) client parts have been reworked for better responsiveness and lower latency.
The main visible feature of this release is the redesign of the event view. User now shouldn't need to go into the JSON view, as most of the popular fields and event parts are shown directly. This includes structured display of attachments and their various types of data types, parsing and displaying links in References fields, Service related fields and others. Event view now also supports context search and third party services data annotations for hostnames.
Reporting filters gained enhanced possibility of testing before applying and notification of concerned admins about related filter changes, together with a usual bunch of bugfixes (timezone handling, changelog handling, visibility of various fields).
Various modules acquired a lot of bugfixes. Event search got fixes for IPv6 input, limits, report related data, wily whitespace, negative queries and others. Also, incomplete data (as in historical and partially removed) are clearly marked. Timeline is now correctly reflected in „My queries“ and query quotas. Uncategorized data, time marks, some dashboard issues and some specific problematic use cases have been fixed in graph usage. Also group and user management received some love in target mail resolution, permissions, group admin assignment and fixing some searching issues.
On the backend part, Negistry-like JSON API has been implemented for integration with tools already using it. Also mailing API is now unified across various modules and libraries.
Please, visit the issue tracker for list for related issues: 2.11.1 2.12
Release statistics: 312 commits, 12763 additions, 11291 deletions
Quite a bunch of features, improvements and fixes have accumulated in the new release.
The new important feature goes hand in hand with companion part on Warden side – the credibility of detectors. The functionality allows to leverage client reliability knowledge for report generation.
The timeline queries are now reworked to run only for the visible tab, not for all the data, shortening latency considerably. Tabs are also cached on the client, avoiding round trip to server on showing already received data.
As there is a limit for running query per user, users are now able to manage their running events queries and possibly kill them on their own discretion. The plan is to extend this functionality to all the possibly long running queries in the future.
There are some additions to user interface for more consistency and discoverability, and also loads of bug fixes thereof – datetime picker switch/upgrade, more discoverable buttons for new group/network creation, changelog consistency fixes, user registration consistency fixes, network rank visibility, basic filters modification fix, context search update, dashboards review, search case sensitivity and white space fixes, address validation and more.
A set of timezone fixes have been developed in both event search and timeline, which affected lots of query parts, graph bucket deductions, pregenerated bounds and so on.
Other fixes involve last login computation, precache crashing, better cronjob locking, better handling of mailing errors, fixes for network data import, data typing and validation, tooling fixes and upgrades and so on.
We have also managed to shed a considerable amount of cruft by removing dependencies and upgrading important libraries (and adapting code for new versions), like Flask, WTForms, SQLAlchemy, dnspython, requests, rrdtool, nose2, pyflakes, pylint, sphinx, jquery, moment, grunt and others.
Please, visit the issue tracker for list for related issues: 2.10.1 2.11
Release statistics: 290 commits, 16066 additions, 11416 deletions
Tenth round of improvements in 2.X series is out and brings mostly security and bug fixes, library upgrades, stale code refactorings, but a number of changes warrants new release.
Among security fixes there are mitigations for XSS, fixed enforcement of STS, secure cookies and CSP.
Fixed are a few glitches in basic reporting filters. Also, a bug where in some cases reporting filters were not able to be created is fixed.
A number of bugfixes and refactorings concerning report feedback, encodings, timezones, support scripts, configuration, stale libraries and a number of crashes is now in place.
Also, development pipeline and Vagrant support is vastly improved.
Notes for administrators:
Please, visit the issue tracker for list for related issues: 2.9.1 2.10
Release statistics: 95 commits, 10276 additions, 15241 deletions
This version brings redesign of groups and networks and more granular possibilities of reporting. Network ranges or the networks can overlap (thus report can be delivered to multiple groups), groups can have reporting priority and specify the least severity to be reported to them.
We have removed the possibility of sending original Idea data as attachments in reports, as this nowadays brings nontrivial delivery problems (too big messages, messages marked as spam). Original data are available at dedicated URLs to download. We have also removed some unused reporting settings.
Report detail now also correctly shows IPv6 addresses and real target emails (where it was actually sent to).
We have fixed a lot of issues concerning daemon start and run, database usage, web validation, Jinja compatibility and others.
There is also preliminary work on support for simplified development workflow with Vagrant virtual machines.
Mentat is now ready for PostgreSQL 14.
Please, visit the issue tracker for list of related issues:
https://homeproj.cesnet.cz/versions/104
Release statistics: 194 commits, 6745 additions, 5974 deletions
In this release together with upgrade to PostgreSQL 13 aggregated column indices finally come to fruition and we are able to significantly push down search times of IP address and range based queries from tens of seconds to (usually) subsecond speed. Together with overlapping range aggregation for storage of source/target heavy events it seems we have finally finished our performance goal, which started by switching from MongoDB to PostgreSQL. Toast time. (Even though Mentat 2.8 will run on older PostgreSQL instances, to take advantage on those improvements you have to upgrade to PostgreSQL 13.)
Timeline aggregation framework now prominently replaces Hosts view in the main menu and is now better integrated with Event search.
Also, usual set of bugfixes, UI, API, documentation and framework cleanup went in.
Mentat specific namespace within events was originally _CESNET. As a means to shed company dependencies, the key is now renamed to _Mentat. It is advisable to review the Inspector configuration for rules related to the keys in this namespace and to review possible related homegrown code.
Multiple instances of Inspector are now folded into one with a default ruleset merged in pursue for a simpler default configuration. If you use the default configuration, you can just use new default Inspector and Controller configuration (which installation from Debian packages does for you). If you've made local changes, you might review new distribution configuration files and also decide to merge.
Please, visit the issue tracker for list of related issues:
https://homeproj.cesnet.cz/versions/103
Release statistics: 88 commits, 5463 additions, 3780 deletions
This release brings major improvements in our Timeline search module. It now enables greater search condition customization capabilities, the same as our Events search module. The search forms are almost identical, which provides users ability to quickly jump from one results page to another with same search conditions. We have also improved search performance of the Timeline module by performing data aggregations and calculations in database instead of in the application. Sadly, not all of the previous aggregation calculations are now supported, because the data were not available directly in our database model. We might bring them back in the future.
This release also lays the groundwork for future abandonment of including report data as email attachments. In the future email reports will contain only links, which can be used to obtain full data, or the users will have the option to use web version of our reports (link is also included in the email). We are encountering issues with misconfigured mailers or too aggressive email filters, which prevent our reports from being successfully delivered. Also the email format is very restrictive and we are unable to present all necessary information in clear form. You should, after all, use only 80 characters per line, and that is not much. Also the use of CSV format for data attachments is now deprecated and will be removed in one of the future releases.
We have also focused on squishing some annoying bugs and a lot of invisible man hours went into writing better tests for our web interface codebase, so that we can have some peace of mind and produce better releases.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/97
Release statistics: 184 commits, 31.397 additions, 18.237 deletions
This release brings in further improvements to our reporting component. Reports are now templated according to the classification of each reported event to provide recipients with most important information relevant to that event class. This new feature is fully configurable for administrators of Mentat system, soon user manual will be provided.
Additionally lot of work went into database optimizations. First the PostgreSQL was upgraded to latest version 12. Next we have increased the amount of possible paralel queries by separating stored IDEA BSON to different table. We have implemented basic DoS prevention mechanism by limiting number of queries each user may execute at any given time. We have also increased the speed in which IDEA events are stored into database by using bulk inserts.
The MaxMind IP geolocation service recently changed its policies for accessing their free databases, so we have addressed this issue as well as the change with access policies to CESNET PassiveDNS service.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/93
Release statistics: 92 commits, 8.494 additions, 4.476 deletions
We have just released Mentat version 2.5.0. This release brings in major improvements in reporting component. Online reports are now more interactive and integrated into other parts of the system, there are context actions available for each node. On top of that there is a simple feedback button available for each address in each report section, so that users may provide their feedback more comfortably. System Mentat is now capable of enriching displayed information with data from third party services like DNS, PassiveDNS, NERD, WHOIS and GeoIP. There is a new module available currently for system administrators that attempts to display all available information for single IP address. Group membership management was simplified to enable group managers to more easily add or remove members and even activate new user accounts. Additionally we have also managed to squash quite a few bugs.
It is also worth noting, that this version also attempts to speed up the database searching by using aggregated IP ranges to narrow down the number of searched rows even more.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/86
Release statistics: 85 commits, 17.480 additions, 7.706 deletions
We have just released Mentat version 2.4.0. This release completely changes the installation procedures when installing from Debian packages and also attempts to simplify the necessary bootstrap procedures for novice developers. The Debian packages now preconfigure custom Python virtual environment and the whole Mentat system is then installed into that environment using native Python package management. This approach greatly simplifies the installation procedure, we can now install more recent Python packages for you without breaking your system. Additionally lot of work went into making the whole project executable from within the cloned git repository, which should simplify the development process for novice developers. Additionally we have also managed to squash quite a few bugs.
Please take special attention to our installation manual to make yourself familiar with new environment and to make sure your installation is in working order.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/85
Release statistics: 150 commits, 10.912 additions, 19.405 deletions
We have just released Mentat version 2.3.0. After quite long gestation period this release brings brand new timeline visualisations for event and reporting dashboards. For system administrators there is now a new module available called Timeline, which provides results similar to that of event dashboards with the difference that the result is calculated directly from event database. These calculations are very expensive, so this feature should be considered as experimental for now and that is the reason why it is currently accessible only to the administrators. Depending on the size of the selected network, time window and result set the calculations may take minutes.
There are also some improvements under the hood. The event database migration mechanism was implemented to enable further database schema improvements. Also the JavaScript charting library underwent first part of major design overhaul.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/83
Release statistics: 110 commits, 16.114 additions, 13.074 deletions
We have just released Mentat version 2.2.0. This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the JSON API. To enable access to the API from arbitrary scripts and applications a new authentication mechanism based on API keys was implemented. Currently the administrator must generate the API key for the user.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/82
Release statistics: 52 commits, 6.746 additions, 4.723 deletions
We have just released Mentat version 2.1.0. This release focuses on resolving most important bugs and issues, that were discovered after production deployment. There are some database query performance optimizations and couple of new features were implemented as well, the most visible being the web interface dashboard for system administrators and better integration of item changelog within the web interface.
Please visit our ticket tracking system for more in-depth information about this release:
https://homeproj.cesnet.cz/versions/81
Release statistics: 87 commits, 21.196 additions, 5.532 deletions
We have just released Mentat version 2.0.7. This version contains mostly bugfixes and stability improvements, we recommend upgrading ASAP.
We have just released Mentat version 2.0.0. Please read the documentation on how to perform migration from previous production release.
Please visit our ticket tracking system for more in-depth information about this release: