Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
en:news [20.05.2020 11:25] – [Wed May 20 2020 - v2.7.0] mach_cesnet.czen:news [01.04.2025 10:30] (current) – Switch links from Redmine to Gitlab ph_cesnet.cz
Line 1: Line 1:
 ====== News ====== ====== News ======
  
-<note important>Please take special attention to our [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/installation.html|installation]] and [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/upgrading.html|upgrading]] manuals to make sure your installation is in working order.</note>+<note important>Please take special attention to our [[https://713.gitlab-pages.cesnet.cz/mentat/mentat//html/_doclib/installation.html|installation]] and [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/production/html/_doclib/upgrading.html|upgrading]] manuals to make sure your installation is in working order.</note>
  
 +===== Mon Dec 9 2024 - v2.14.0 =====
 +
 +The Mentat project has seen a number of changes covering the user interface, infrastructure and API.
 +
 +Many modifications have been made to the UI in an effort to move closer to the needs of both SOC analysts and users. A new severity level of "info" has been introduced with a long retry interval (service deprecation warnings, minor configuration issues, ...), as continued integration with Auror and vulnerability
 +management. We have added full-text search capability in IDEA Description and its display on the timeline. Support for working with new IDEA fields for login credentials used in an attack or time inaccuracy.
 +
 +We have introduced event reporting by target for collaboration with proactive and IPS tools. Reports can also now be searched by class, target address, port or detectors. Related reports are now linked, including support for mail threads.
 +
 +Registration page supports links to company identity and legal documents.
 +
 +In addition, a new page has been created for detailed IP address information, which serves as an entry point for various tools such as Amfora and OTRS.
 +
 +Many small changes have also taken place - reporting settings UI streamlining, the new group search API, upgrading libraries, droves of bugfixes and more.
 +
 +Please, visit the issue tracker for the list for related issues: 
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/28|2.13.1]]
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/29|2.13.2]]
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/30|2.14]]
 +
 +//Release statistics: 453 commits, 14952 additions, 12643 deletions//
 +
 +===== Wed Jun 26 2024 – v2.13.0 =====
 +
 +Two big features are finished in this release - rework of the class
 +configuration and replacement of the graphing library.
 +
 +Until now the class configuration was done partially by inspector rule
 +config, partially by Jinja templates, JSON files and specific Babel based
 +translations on filesystem. Now the class definition is simplified to the
 +definition of the rules, wanted columns in the reports and description/link,
 +and together moved to one place of the web interface.
 +
 +Classes are now also two tiered - first level is the same as before,
 +where as the second level allows to aggregate by more complex rules.
 +
 +Together with reporting we are working on filtering interface - we have
 +added the possibility to specify target IPs, protocols and classes in the
 +simplified filter rule definitions. Also, filter notification to relevant
 +group admins is streamlined.
 +
 +We have embraced the Plotly graphing library instead of NVD3, development
 +of which have stopped to the halt. It was also necessary to rewrite most of
 +the backend graphing code, which was closely tied to NVD3.
 +
 +As we are now able to import network/group data from external sources,
 +we have added a couple of support scripts for a couple of CESNET related
 +organisations.
 +
 +We have of course fixed usual slew of the bugs pretty much everywhere
 +(searching by storage time, canceling some forms, access rights problems,
 +filter details, visual problems, and also set of development specifics).
 +
 +Also, the hosts module was removed (event search and timeline can do
 +much more).
 +
 +Please, visit the issue tracker for the list for related issues: 
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/27|2.12.1]]
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/26|2.13]]
 +
 +//Release statistics: 321 commits, 15625 additions,  11962 deletions//
 +
 +===== Wed Mar 20 2024 - v2.12.0 =====
 +
 +The main visible feature is the revamp of the whole web interface - with the upgrade of the underlying Bootstrap library came a lot of changes. In spite of some more invasive changes (selection lists for example), we tried hard to keep the overall logic and feel the same. Also, underlying active (javascript) client parts have been reworked for better responsiveness and lower latency.
 +
 +The main visible feature of this release is the redesign of the event view. User now shouldn't need to go into the JSON view, as most of the popular fields and event parts are shown directly. This includes structured display of attachments and their various types of data types, parsing and displaying links in References fields, Service related fields and others. Event view now also supports context search and third party services data annotations for hostnames.
 +
 +Reporting filters gained enhanced possibility of testing before applying and notification of concerned admins about related filter changes, together with a usual bunch of bugfixes (timezone handling, changelog handling, visibility of various fields).
 +
 +Various modules acquired a lot of bugfixes. Event search got fixes for IPv6 input, limits, report related data, wily whitespace, negative queries and others. Also, incomplete data (as in historical and partially removed) are clearly marked. Timeline is now correctly reflected in "My queries" and query quotas. Uncategorized data, time marks, some dashboard issues and some
 +specific problematic use cases have been fixed in graph usage. Also group and user management received some love in target mail resolution, permissions, group admin assignment and fixing some searching issues.
 +
 +On the backend part, Negistry-like JSON API has been implemented for integration with tools already using it. Also mailing API is now unified
 +across various modules and libraries.
 +
 +Please, visit the issue tracker for list for related issues: 
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/25|2.11.1]]
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/24|2.12]]
 +
 +//Release statistics: 312 commits, 12763 additions, 11291 deletions//
 +
 +===== Fri Jun 30 2023 - v2.11.0 =====
 +
 +Quite a bunch of features, improvements and fixes have accumulated in the new release.
 +
 +The new important feature goes hand in hand with companion part on Warden side – the credibility of detectors. The functionality allows to leverage client reliability knowledge for report generation.
 +
 +The timeline queries are now reworked to run only for the visible tab, not for all the data, shortening latency considerably. Tabs are also cached on the client, avoiding round trip to server on showing already received data.
 +
 +As there is a limit for running query per user, users are now able to manage their running events queries and possibly kill them on their own discretion. The plan is to extend this functionality to all the possibly long running queries in the future.
 +
 +There are some additions to user interface for more consistency and discoverability, and also loads of bug fixes thereof – datetime picker switch/upgrade, more discoverable buttons for new group/network creation, changelog consistency fixes, user registration consistency fixes, network rank visibility, basic filters modification fix, context search update, dashboards review, search case sensitivity and white space fixes, address validation and more.
 +
 +A set of timezone fixes have been developed in both event search and timeline, which affected lots of query parts, graph bucket deductions, pregenerated bounds and so on.
 +
 +Other fixes involve last login computation, precache crashing, better cronjob locking, better handling of mailing errors, fixes for network data import, data typing and validation, tooling fixes and upgrades and so on.
 +
 +We have also managed to shed a considerable amount of cruft by removing dependencies and upgrading important libraries (and adapting code for new versions), like Flask, WTForms, SQLAlchemy, dnspython, requests, rrdtool, nose2, pyflakes, pylint, sphinx, jquery, moment, grunt and others.
 +
 +Please, visit the issue tracker for list for related issues: [[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/23|2.10.1]]
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/21|2.11]]
 +
 +//Release statistics: 290 commits, 16066 additions, 11416 deletions//
 +
 +
 +===== Thu Aug 11 2022 - v2.10.0 =====
 +
 +Tenth round of improvements in 2.X series is out and brings mostly security and bug fixes, library upgrades, stale
 +code refactorings, but a number of changes warrants new release.
 +
 +Among security fixes there are mitigations for XSS, fixed enforcement of
 +STS, secure cookies and CSP.
 +
 +Fixed are a few glitches in basic reporting filters. Also, a bug where in 
 +some cases reporting filters were not able to be created is fixed.
 +
 +A number of bugfixes and refactorings concerning report feedback, encodings,
 +timezones, support scripts, configuration, stale libraries and a number of
 +crashes is now in place.
 +
 +Also, development pipeline and Vagrant support is vastly improved.
 +
 +Notes for administrators: 
 +
 +  * Because of nasty and hard to track situations, lower case (for case insensitivity) is now enforced in user names. If case     conflict situation appears in your case, you will find redundant users with _case_conflict suffix - please review conflicting users to find and enable the correct one.
 +
 +  * Basic reporting filters combining operator now defaults to AND and as previous OR combination of basic rules was nonsensical, please review whether your basic filters do what you expect. As new combination is stricter, you won't lose any events from reports, however more of them can get through the filters.
 +
 +  * As mostly redundant library-like Vial tree is now refactored and simplified back into Hawat, please take care to review mentions of "vial" in your configuration (if any, replace with "hawat").
 +
 +Please, visit the issue tracker for list for related issues: [[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/22|2.9.1]] [[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/20|2.10]]
 +
 +//Release statistics: 95 commits, 10276 additions, 15241 deletions//
 +
 +===== Tue Mar 22 2022 - v2.9.0 =====
 +
 +This version brings redesign of groups and networks and more granular possibilities
 +of reporting. Network ranges or the networks can overlap (thus report can be
 +delivered to multiple groups), groups can have reporting priority and specify the
 +least severity to be reported to them.
 +
 +We have removed the possibility of sending original Idea data as attachments in
 +reports, as this nowadays brings nontrivial delivery problems (too big messages,
 +messages marked as spam). Original data are available at dedicated URLs to download.
 +We have also removed some unused reporting settings.
 +
 +Report detail now also correctly shows IPv6 addresses and real target emails (where
 +it was actually sent to).
 +
 +We have fixed a lot of issues concerning daemon start and run, database usage, web
 +validation, Jinja  compatibility and others.
 +
 +There is also preliminary work on support for simplified development workflow with
 +Vagrant virtual machines.
 +
 +Mentat is now ready for PostgreSQL 14.
 +
 +Please, visit the issue tracker for list of related issues:
 +    
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/18]]
 +
 +//Release statistics: 194 commits, 6745 additions, 5974 deletions//
 +
 +===== Fri Aug 13 2021 - v2.8 =====
 +
 +In this release together with upgrade to PostgreSQL 13 aggregated column indices finally come to fruition and we are able to significantly push down search times of IP address and range based queries from tens of seconds to (usually) subsecond speed. Together with overlapping range aggregation for storage of source/target heavy events it seems we have finally finished our performance goal, which started by switching from MongoDB to PostgreSQL. Toast time. (Even though Mentat 2.8 will run on older PostgreSQL instances, to take advantage on those improvements you have to upgrade to PostgreSQL 13.)
 +    
 +Timeline aggregation framework now prominently replaces Hosts view in the main menu and is now better integrated with Event search.
 +    
 +Also, usual set of bugfixes, UI, API, documentation and framework cleanup went in.
 +    
 +Mentat specific namespace within events was originally _CESNET. As a means to shed company dependencies, the key is now renamed to _Mentat. It is advisable to review the Inspector configuration for rules related to the keys in this namespace and to review possible related homegrown code.
 +    
 +Multiple instances of Inspector are now folded into one with a default ruleset merged in pursue for a simpler default configuration. If you use the default configuration, you can just use new default Inspector and Controller configuration (which installation from Debian packages does for you). If you've made local changes, you might review new distribution configuration files and also decide to merge.
 +    
 +Please, visit the issue tracker for list of related issues:
 +    
 +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/17]]
 +
 +//Release statistics: 88 commits, 5463 additions, 3780 deletions//
  
 ===== Wed May 20 2020 - v2.7.0 ===== ===== Wed May 20 2020 - v2.7.0 =====
Line 14: Line 195:
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
          
-[[https://homeproj.cesnet.cz/versions/97]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/16]]
  
 //Release statistics: 184 commits, 31.397 additions, 18.237 deletions// //Release statistics: 184 commits, 31.397 additions, 18.237 deletions//
Line 29: Line 210:
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/93]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/15]]
  
 //Release statistics: 92 commits, 8.494 additions, 4.476 deletions// //Release statistics: 92 commits, 8.494 additions, 4.476 deletions//
Line 42: Line 223:
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/86]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/13]]
  
 //Release statistics: 85 commits, 17.480 additions, 7.706 deletions// //Release statistics: 85 commits, 17.480 additions, 7.706 deletions//
Line 51: Line 232:
 We have just released Mentat version **2.4.0**. This release completely changes the installation procedures when installing from Debian packages and also attempts to simplify the necessary bootstrap procedures for novice developers. The Debian packages now preconfigure custom Python virtual environment and the whole Mentat system is then installed into that environment using native Python package management. This approach greatly simplifies the installation procedure, we can now install more recent Python packages for you without breaking your system. Additionally lot of work went into making the whole project executable from within the cloned git repository, which should simplify the development process for novice developers. Additionally we have also managed to squash quite a few bugs. We have just released Mentat version **2.4.0**. This release completely changes the installation procedures when installing from Debian packages and also attempts to simplify the necessary bootstrap procedures for novice developers. The Debian packages now preconfigure custom Python virtual environment and the whole Mentat system is then installed into that environment using native Python package management. This approach greatly simplifies the installation procedure, we can now install more recent Python packages for you without breaking your system. Additionally lot of work went into making the whole project executable from within the cloned git repository, which should simplify the development process for novice developers. Additionally we have also managed to squash quite a few bugs.
  
-Please take special attention to our [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/installation.html|installation]] manual to make yourself familiar with new environment and to make sure your installation is in working order.+Please take special attention to our [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/release/html/_doclib/installation.html|installation]] manual to make yourself familiar with new environment and to make sure your installation is in working order.
  
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/85]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/12]]
  
 <note warning>In this version of Mentat system support for migration from MongoDB to PostgreSQL was dropped.</note> <note warning>In this version of Mentat system support for migration from MongoDB to PostgreSQL was dropped.</note>
Line 70: Line 251:
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/83]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/11]]
  
 <note warning>In the next version of Mentat system a support for migration from MongoDB to PostgreSQL will be dropped. If you have not yet upgraded from version 1.x to 2.x, please do so now.</note> <note warning>In the next version of Mentat system a support for migration from MongoDB to PostgreSQL will be dropped. If you have not yet upgraded from version 1.x to 2.x, please do so now.</note>
Line 79: Line 260:
 ===== Thu Nov 28 2018 - v2.2.0 ===== ===== Thu Nov 28 2018 - v2.2.0 =====
  
-We have just released Mentat version **2.2.0**. This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the [[https://alchemist.cesnet.cz/mentat/doc/production/html/_doclib/hawat_plugin_events.html|JSON API]]. To enable access to the API from arbitrary scripts and applications a new authentication mechanism based on API keys was implemented. Currently the administrator must generate the API key for the user. +We have just released Mentat version **2.2.0**. This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/release/html/_doclib/hawat_plugin_events.html|JSON API]]. To enable access to the API from arbitrary scripts and applications a new authentication mechanism based on API keys was implemented. Currently the administrator must generate the API key for the user. 
  
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/82]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/10]]
  
 //Release statistics: 52 commits, 6.746 additions, 4.723 deletions// //Release statistics: 52 commits, 6.746 additions, 4.723 deletions//
Line 94: Line 275:
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/81]]+[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/9]]
  
 //Release statistics: 87 commits, 21.196 additions, 5.532 deletions// //Release statistics: 87 commits, 21.196 additions, 5.532 deletions//
Line 106: Line 287:
 ===== Fri Jul 27 2018 - v2.0.0 ===== ===== Fri Jul 27 2018 - v2.0.0 =====
  
-We have just released Mentat version **2.0.0**. Please read the [[https://alchemist.cesnet.cz/mentat/doc/production/html/manual.html|documentation]] on how to perform [[https://alchemist.cesnet.cz/mentat/doc/production/html/migration.html|migration]] from previous production release. +We have just released Mentat version **2.0.0**. Please read the [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/release/html/manual.html|documentation]] on how to perform [[https://713.gitlab-pages.cesnet.cz/mentat/mentat/master/html/_doclib/migration.html|migration]] from previous production release. 
  
 Please visit our ticket tracking system for more in-depth information about this release: Please visit our ticket tracking system for more in-depth information about this release:
  
-[[https://homeproj.cesnet.cz/versions/74]] +[[https://gitlab.cesnet.cz/713/mentat/mentat/-/milestones/8