Skip to main content
Loading...

Distributed search - the ability to search for selected nodes (other RuSIEM / RvSIEM servers).

Distributed search allows not to aggregate events from a set of remote objects with weak channels, but to have the necessary data on request.

A query to several nodes allows you to glue the data into a single answer and search multiple separate objects at once. For example, an operator in a central office can search a query for 20 or more branches of a company, although physically it is 20 different regional servers.

For example, in the SOC there is no need to transmit all events. However, the operator in SOC can make a complex request for selected companies "from which hosts was the appeal to ip 8.8.8.8". In this case, the operator selects for which companies he makes a request. In the saved query, the operator can also save a pool of companies in order not to make a choice every time, but to receive a ready-made selection.

Distributed search works in both the commercial version of RuSIEM and the free redistributable RvSIEM free.

Distributed search:

  • allows operators without centralization of events to search for a variety of branches and companies
  • see the full picture of what is happening in the infrastructure in a single request
  • provides savings on the disk subsystem and communication channels
  • allows you to build an extensive architecture of collecting and analyzing events, which has no restrictions

Configuring Distributed Search

Distributed search is possible only with the version Elasticsearch starting with 5.6.

The current installed version can be viewed using the command from the console:

dpkg --list | grep elasticsearch

You can upgrade Elasticsearch version to version 5.6 with data loss (only accumulated events will be lost) with the command:

/opt/rusiem/database/support/migrate_es_from_1.7_to_5.4.sh

To migrate from version 1.7 to 5.6, contact technical support mailto:support@rusiem.com

Setting up a distributed search is done manually in several stages:

  1. firewall configuration
  2. Elasticsearch configuration
  3. web interface configuration

Firewall configuration

To be able to make a request for a remote node (Elasticsearch, where the rusiem-database package is installed), the tcp / 9300 ports must be opened. By default, ports are closed for information security reasons.

For example, to have from node A request events on node B, on node A:

  • edit the firewall configuration /etc/init.d/firewall.sh by adding the lines:

iptables -A OUTPUT -p tcp -s $EXTIP --dport 9300 -d 172.16.0.125 -j ACCEPT
iptables -A INPUT -p tcp -d $EXTIP --sport 9300 -s 172.16.0.125 -j ACCEPT

where 172.16.0.125 - node B ip

or to connect to any remote node (outgoing only):

iptables -A OUTPUT -p tcp -s $EXTIP --dport 9300 -j ACCEPT
iptables -A INPUT -p tcp -d $EXTIP --sport 9300 -j ACCEPT

  • on node B, you must also allow incoming connection from node A by editing the configuration of firewall /etc/init.d/firewall.sh on node B, adding the lines:

iptables -A INPUT -p tcp -d $EXTIP --dport 9300 -s 172.16.0.22 -j ACCEPT

iptables -A OUTPUT -p tcp -s $EXTIP --sport 9300 -d 172.16.0.22 -j ACCEPT

where 172.16.0.22 - node А ip.

Attention! It is strictly not recommended to open incoming connections for all, specify only a dot - which ip is allowed to access the data. Otherwise, anyone can access the data!

After changing the firewall configuration, you need to apply the changes with the /etc/init.d/firewall.sh start command

Elasticsearch configuration

To configure Elasticsearch:

1. Make sure the Elasticsearch version is not lower than 5.6 with the dpkg command --list | grep elastic

2. In the configuration file /etc/elasticsearch/elasticsearch.yml set the parameters:

network.host: 0.0.0.0

http.port: 9200

3. In the configuration file /etc/default/elasticsearch set the parameter:

MAX_LOCKED_MEMORY=unlimited

4. After configuration changes, restart the service with the service elasticsearch restart command

5. The netstat -vpntlu command | egrep '9200 | 9300' make sure that the elasticsearch service started successfully and is available at 0.0.0.0:

tcp        0      0 0.0.0.0:9200            0.0.0.0:*               LISTEN      10690/java
tcp        0      0 0.0.0.0:9300            0.0.0.0:*               LISTEN      10690/java

Attention! After the restart of the service, there is some delay in its full recovery, depending on the node's hardware resources (from seconds to a couple of minutes).

If something does not start, you can see the reasons with command: tail -f /var/log/elasticsearch/rusiem.log

In the case of migration from version 1.7 - it is necessary to install the correct version of Elasticsearch (5.x) in the "Settings" web interface!

Web interface configuration

Setting up the web interface is done only after setting up the firewall and Elasticsearch!

To configure, you first need to add in the web interface in the Settings -> Nodes - your remote nodes (in the example - B) section.

Make the settings as in the screenshot below, changing your ip and names.

After making changes to the nodes, save and click the "Update remote node settings" button. This button must be pressed after saving the settings and if there is a connection with the remote node! The button passes the parameters to the local Elasticsearch data of the remote node through curl.

If there is a connection with a remote node, in the "Events" section it will be possible to select the remote node at the top of the header, as in the figure below.

If the node is not available, it will be highlighted in gray in the selection list.

Also, in the saved query, it is possible to determine which nodes to search in this view so as not to select a list of nodes each time.

The selected Local node or the missing node selection means that the search is performed only by the local node (or cluster specified in the settings of the web interface).

Selecting for a node other than Local means that the search is performed only on that selected node.

Selecting by several selected nodes means that the results for several separate Elasticsearch nodes (or clusters) will be displayed (glued).