Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
en:news [04.02.2019 13:53] – mach@cesnet.cz | en:news [01.04.2025 10:30] (current) – Switch links from Redmine to Gitlab ph_cesnet.cz | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== News ====== | ====== News ====== | ||
- | <note important> | + | <note important> |
+ | |||
+ | ===== Mon Dec 9 2024 - v2.14.0 ===== | ||
+ | |||
+ | The Mentat project has seen a number of changes covering the user interface, infrastructure and API. | ||
+ | |||
+ | Many modifications have been made to the UI in an effort to move closer to the needs of both SOC analysts and users. A new severity level of " | ||
+ | management. We have added full-text search capability in IDEA Description and its display on the timeline. Support for working with new IDEA fields for login credentials used in an attack or time inaccuracy. | ||
+ | |||
+ | We have introduced event reporting by target for collaboration with proactive and IPS tools. Reports can also now be searched by class, target address, port or detectors. Related reports are now linked, including support for mail threads. | ||
+ | |||
+ | Registration page supports links to company identity and legal documents. | ||
+ | |||
+ | In addition, a new page has been created for detailed IP address information, | ||
+ | |||
+ | Many small changes have also taken place - reporting settings UI streamlining, | ||
+ | |||
+ | Please, visit the issue tracker for the list for related issues: | ||
+ | [[https:// | ||
+ | [[https:// | ||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 453 commits, 14952 additions, 12643 deletions// | ||
+ | |||
+ | ===== Wed Jun 26 2024 – v2.13.0 ===== | ||
+ | |||
+ | Two big features are finished in this release - rework of the class | ||
+ | configuration and replacement of the graphing library. | ||
+ | |||
+ | Until now the class configuration was done partially by inspector rule | ||
+ | config, partially by Jinja templates, JSON files and specific Babel based | ||
+ | translations on filesystem. Now the class definition is simplified to the | ||
+ | definition of the rules, wanted columns in the reports and description/ | ||
+ | and together moved to one place of the web interface. | ||
+ | |||
+ | Classes are now also two tiered - first level is the same as before, | ||
+ | where as the second level allows to aggregate by more complex rules. | ||
+ | |||
+ | Together with reporting we are working on filtering interface - we have | ||
+ | added the possibility to specify target IPs, protocols and classes in the | ||
+ | simplified filter rule definitions. Also, filter notification to relevant | ||
+ | group admins is streamlined. | ||
+ | |||
+ | We have embraced the Plotly graphing library instead of NVD3, development | ||
+ | of which have stopped to the halt. It was also necessary to rewrite most of | ||
+ | the backend graphing code, which was closely tied to NVD3. | ||
+ | |||
+ | As we are now able to import network/ | ||
+ | we have added a couple of support scripts for a couple of CESNET related | ||
+ | organisations. | ||
+ | |||
+ | We have of course fixed usual slew of the bugs pretty much everywhere | ||
+ | (searching by storage time, canceling some forms, access rights problems, | ||
+ | filter details, visual problems, and also set of development specifics). | ||
+ | |||
+ | Also, the hosts module was removed (event search and timeline can do | ||
+ | much more). | ||
+ | |||
+ | Please, visit the issue tracker for the list for related issues: | ||
+ | [[https:// | ||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 321 commits, 15625 additions, | ||
+ | |||
+ | ===== Wed Mar 20 2024 - v2.12.0 ===== | ||
+ | |||
+ | The main visible feature is the revamp of the whole web interface - with the upgrade of the underlying Bootstrap library came a lot of changes. In spite of some more invasive changes (selection lists for example), we tried hard to keep the overall logic and feel the same. Also, underlying active (javascript) client parts have been reworked for better responsiveness and lower latency. | ||
+ | |||
+ | The main visible feature of this release is the redesign of the event view. User now shouldn' | ||
+ | |||
+ | Reporting filters gained enhanced possibility of testing before applying and notification of concerned admins about related filter changes, together with a usual bunch of bugfixes (timezone handling, changelog handling, visibility of various fields). | ||
+ | |||
+ | Various modules acquired a lot of bugfixes. Event search got fixes for IPv6 input, limits, report related data, wily whitespace, negative queries and others. Also, incomplete data (as in historical and partially removed) are clearly marked. Timeline is now correctly reflected in "My queries" | ||
+ | specific problematic use cases have been fixed in graph usage. Also group and user management received some love in target mail resolution, permissions, | ||
+ | |||
+ | On the backend part, Negistry-like JSON API has been implemented for integration with tools already using it. Also mailing API is now unified | ||
+ | across various modules and libraries. | ||
+ | |||
+ | Please, visit the issue tracker for list for related issues: | ||
+ | [[https:// | ||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 312 commits, 12763 additions, 11291 deletions// | ||
+ | |||
+ | ===== Fri Jun 30 2023 - v2.11.0 ===== | ||
+ | |||
+ | Quite a bunch of features, improvements and fixes have accumulated in the new release. | ||
+ | |||
+ | The new important feature goes hand in hand with companion part on Warden side – the credibility of detectors. The functionality allows to leverage client reliability knowledge for report generation. | ||
+ | |||
+ | The timeline queries are now reworked to run only for the visible tab, not for all the data, shortening latency considerably. Tabs are also cached on the client, avoiding round trip to server on showing already received data. | ||
+ | |||
+ | As there is a limit for running query per user, users are now able to manage their running events queries and possibly kill them on their own discretion. The plan is to extend this functionality to all the possibly long running queries in the future. | ||
+ | |||
+ | There are some additions to user interface for more consistency and discoverability, | ||
+ | |||
+ | A set of timezone fixes have been developed in both event search and timeline, which affected lots of query parts, graph bucket deductions, pregenerated bounds and so on. | ||
+ | |||
+ | Other fixes involve last login computation, | ||
+ | |||
+ | We have also managed to shed a considerable amount of cruft by removing dependencies and upgrading important libraries (and adapting code for new versions), like Flask, WTForms, SQLAlchemy, dnspython, requests, rrdtool, nose2, pyflakes, pylint, sphinx, jquery, moment, grunt and others. | ||
+ | |||
+ | Please, visit the issue tracker for list for related issues: [[https:// | ||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 290 commits, 16066 additions, 11416 deletions// | ||
+ | |||
+ | |||
+ | ===== Thu Aug 11 2022 - v2.10.0 ===== | ||
+ | |||
+ | Tenth round of improvements in 2.X series is out and brings mostly security and bug fixes, library upgrades, stale | ||
+ | code refactorings, | ||
+ | |||
+ | Among security fixes there are mitigations for XSS, fixed enforcement of | ||
+ | STS, secure cookies and CSP. | ||
+ | |||
+ | Fixed are a few glitches in basic reporting filters. Also, a bug where in | ||
+ | some cases reporting filters were not able to be created is fixed. | ||
+ | |||
+ | A number of bugfixes and refactorings concerning report feedback, encodings, | ||
+ | timezones, support scripts, configuration, | ||
+ | crashes is now in place. | ||
+ | |||
+ | Also, development pipeline and Vagrant support is vastly improved. | ||
+ | |||
+ | Notes for administrators: | ||
+ | |||
+ | * Because of nasty and hard to track situations, lower case (for case insensitivity) is now enforced in user names. If case | ||
+ | |||
+ | * Basic reporting filters combining operator now defaults to AND and as previous OR combination of basic rules was nonsensical, | ||
+ | |||
+ | * As mostly redundant library-like Vial tree is now refactored and simplified back into Hawat, please take care to review mentions of " | ||
+ | |||
+ | Please, visit the issue tracker for list for related issues: [[https:// | ||
+ | |||
+ | //Release statistics: 95 commits, 10276 additions, 15241 deletions// | ||
+ | |||
+ | ===== Tue Mar 22 2022 - v2.9.0 ===== | ||
+ | |||
+ | This version brings redesign of groups and networks and more granular possibilities | ||
+ | of reporting. Network ranges or the networks can overlap (thus report can be | ||
+ | delivered to multiple groups), groups can have reporting priority and specify the | ||
+ | least severity to be reported to them. | ||
+ | |||
+ | We have removed the possibility of sending original Idea data as attachments in | ||
+ | reports, as this nowadays brings nontrivial delivery problems (too big messages, | ||
+ | messages marked as spam). Original data are available at dedicated URLs to download. | ||
+ | We have also removed some unused reporting settings. | ||
+ | |||
+ | Report detail now also correctly shows IPv6 addresses and real target emails (where | ||
+ | it was actually sent to). | ||
+ | |||
+ | We have fixed a lot of issues concerning daemon start and run, database usage, web | ||
+ | validation, Jinja compatibility and others. | ||
+ | |||
+ | There is also preliminary work on support for simplified development workflow with | ||
+ | Vagrant virtual machines. | ||
+ | |||
+ | Mentat is now ready for PostgreSQL 14. | ||
+ | |||
+ | Please, visit the issue tracker for list of related issues: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 194 commits, 6745 additions, 5974 deletions// | ||
+ | |||
+ | ===== Fri Aug 13 2021 - v2.8 ===== | ||
+ | |||
+ | In this release together with upgrade to PostgreSQL 13 aggregated column indices finally come to fruition and we are able to significantly push down search times of IP address and range based queries from tens of seconds to (usually) subsecond speed. Together with overlapping range aggregation for storage of source/ | ||
+ | |||
+ | Timeline aggregation framework now prominently replaces Hosts view in the main menu and is now better integrated with Event search. | ||
+ | |||
+ | Also, usual set of bugfixes, UI, API, documentation and framework cleanup went in. | ||
+ | |||
+ | Mentat specific namespace within events was originally _CESNET. As a means to shed company dependencies, | ||
+ | |||
+ | Multiple instances of Inspector are now folded into one with a default ruleset merged in pursue for a simpler default configuration. If you use the default configuration, | ||
+ | |||
+ | Please, visit the issue tracker for list of related issues: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 88 commits, 5463 additions, 3780 deletions// | ||
+ | |||
+ | ===== Wed May 20 2020 - v2.7.0 ===== | ||
+ | |||
+ | This release brings major improvements in our **Timeline** search module. It now enables greater search condition customization capabilities, | ||
+ | |||
+ | This release also lays the groundwork for future abandonment of including report data as email attachments. In the future email reports will contain only links, which can be used to obtain full data, or the users will have the option to use web version of our reports (link is also included in the email). We are encountering issues with misconfigured mailers or too aggressive email filters, which prevent our reports from being successfully delivered. Also the email format is very restrictive and we are unable to present all necessary information in clear form. You should, after all, use only 80 characters per line, and that is not much. Also the use of CSV format for data attachments is now deprecated and will be removed in one of the future releases. | ||
+ | |||
+ | We have also focused on squishing some annoying bugs and a lot of invisible man hours went into writing better tests for our web interface codebase, so that we can have some peace of mind and produce better releases. | ||
+ | |||
+ | Please visit our ticket tracking system for more in-depth information about this release: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 184 commits, 31.397 additions, 18.237 deletions// | ||
+ | |||
+ | |||
+ | ===== Mon Feb 3 2020 - v2.6.0 ===== | ||
+ | |||
+ | This release brings in further improvements to our reporting component. Reports are now templated according to the classification of each reported event to provide recipients with most important information relevant to that event class. This new feature is fully configurable for administrators of Mentat system, soon user manual | ||
+ | |||
+ | Additionally lot of work went into database optimizations. First the PostgreSQL was upgraded to latest version 12. Next we have increased the amount of possible paralel queries by separating stored IDEA BSON to different table. We have implemented basic DoS prevention mechanism by limiting number of queries each user may execute at any given time. We have also increased the speed in which IDEA events are stored into database by using bulk inserts. | ||
+ | |||
+ | The MaxMind IP geolocation service recently changed its policies for accessing their free databases, so we have addressed this issue as well as the change with access policies to CESNET PassiveDNS service. | ||
+ | |||
+ | Please visit our ticket tracking system for more in-depth information about this release: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 92 commits, 8.494 additions, 4.476 deletions// | ||
+ | |||
+ | |||
+ | ===== Tue Sep 3 2019 - v2.5.0 ===== | ||
+ | |||
+ | We have just released Mentat version **2.5.0**. This release brings in major improvements in reporting component. Online reports are now more interactive and integrated into other parts of the system, there are context actions available for each node. On top of that there is a simple feedback button available for each address in each report section, so that users may provide their feedback more comfortably. System Mentat is now capable of enriching displayed information with data from third party services like DNS, PassiveDNS, NERD, WHOIS and GeoIP. There is a new module available currently for system administrators that attempts to display all available information for single IP address. Group membership management was simplified to enable group managers to more easily add or remove members and even activate new user accounts. Additionally we have also managed to squash quite a few bugs. | ||
+ | |||
+ | It is also worth noting, that this version also attempts to speed up the database searching by using aggregated IP ranges to narrow down the number of searched rows even more. | ||
+ | |||
+ | Please visit our ticket tracking system for more in-depth information about this release: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | //Release statistics: 85 commits, 17.480 additions, 7.706 deletions// | ||
+ | |||
+ | |||
+ | ===== Fri May 24 2019 - v2.4.0 ===== | ||
+ | |||
+ | We have just released Mentat version **2.4.0**. This release completely changes the installation procedures when installing from Debian packages and also attempts to simplify the necessary bootstrap procedures for novice developers. The Debian packages now preconfigure custom Python virtual environment and the whole Mentat system is then installed into that environment using native Python package management. This approach greatly simplifies the installation procedure, we can now install more recent Python packages for you without breaking your system. Additionally lot of work went into making the whole project executable from within the cloned git repository, which should simplify the development process for novice developers. Additionally we have also managed to squash quite a few bugs. | ||
+ | |||
+ | Please take special attention to our [[https:// | ||
+ | |||
+ | Please visit our ticket tracking system for more in-depth information about this release: | ||
+ | |||
+ | [[https:// | ||
+ | |||
+ | <note warning> | ||
+ | |||
+ | //Release statistics: 150 commits, 10.912 additions, 19.405 deletions// | ||
===== Mon Feb 4 2019 - v2.3.0 ===== | ===== Mon Feb 4 2019 - v2.3.0 ===== | ||
Line 11: | Line 251: | ||
Please visit our ticket tracking system for more in-depth information about this release: | Please visit our ticket tracking system for more in-depth information about this release: | ||
- | [[https://homeproj.cesnet.cz/versions/83]] | + | [[https://gitlab.cesnet.cz/713/mentat/ |
<note warning> | <note warning> | ||
+ | |||
+ | //Release statistics: 110 commits, 16.114 additions, 13.074 deletions// | ||
| | ||
===== Thu Nov 28 2018 - v2.2.0 ===== | ===== Thu Nov 28 2018 - v2.2.0 ===== | ||
- | We have just released Mentat version **2.2.0**. This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the [[https://alchemist.cesnet.cz/ | + | We have just released Mentat version **2.2.0**. This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the [[https://713.gitlab-pages.cesnet.cz/ |
Please visit our ticket tracking system for more in-depth information about this release: | Please visit our ticket tracking system for more in-depth information about this release: | ||
- | [[https://homeproj.cesnet.cz/versions/82]] | + | [[https://gitlab.cesnet.cz/713/mentat/ |
+ | |||
+ | //Release statistics: 52 commits, 6.746 additions, 4.723 deletions// | ||
Line 31: | Line 275: | ||
Please visit our ticket tracking system for more in-depth information about this release: | Please visit our ticket tracking system for more in-depth information about this release: | ||
- | [[https://homeproj.cesnet.cz/versions/81]] | + | [[https://gitlab.cesnet.cz/713/mentat/ |
+ | |||
+ | //Release statistics: 87 commits, 21.196 additions, 5.532 deletions// | ||
Line 37: | Line 283: | ||
We have just released Mentat version **2.0.7**. This version contains mostly bugfixes and stability improvements, | We have just released Mentat version **2.0.7**. This version contains mostly bugfixes and stability improvements, | ||
+ | |||
===== Fri Jul 27 2018 - v2.0.0 ===== | ===== Fri Jul 27 2018 - v2.0.0 ===== | ||
- | We have just released Mentat version **2.0.0**. Please read the [[https://alchemist.cesnet.cz/ | + | We have just released Mentat version **2.0.0**. Please read the [[https://713.gitlab-pages.cesnet.cz/ |
Please visit our ticket tracking system for more in-depth information about this release: | Please visit our ticket tracking system for more in-depth information about this release: | ||
- | [[https://homeproj.cesnet.cz/versions/74]] | + | [[https://gitlab.cesnet.cz/713/mentat/ |