QRadar can operate in a master-slave deployment model. The single master is the console, which manages configuration updates for all managed hosts (slaves) in the deployment set. The console can read and write to the Postgres database, while all managed hosts have read-only database access. After you make any configuration change in the console, managed hosts must reflect the same settings. The process of sending changes can be incremental or full. The most common option (Deploy Changes) includes both database replication updates (through hostcontext) and flat-file configuration changes. Full Deploy Changes also includes the full configuration set and restarts QRadar services.
Hostcontext service and its role in deploying
The hostcontext process is the primary process that runs on each managed host and on the console. This service controls all the core qradar processes. On the contrary, it does not control Tomcat, imq or postgres. Nevertheless, hostcontext also executes deploy requests from the console and reports deploy status updates in the Deployment Model on the Console. The hostcontext also controls downloading configuration sets when changes occur and executes “replication” processes each minute. Replication is different from deploying changes because it happens automatically and it tracks only changes in Postgres database but not with the configuration files. The Hostcontext reports also the status of the HA (High Availability) peer if a Managed Host is running in HA.
Any time you have a problem deploying changes, the first thing to check is whether hostcontext is running on the affected Managed Host. You can quickly check the status of all hostcontext instances running on each Managed Host using the following command:
/opt/qradar/support/all_serversh.sh -C "systemctl status hostcontext"
Please keep in mind that the -C switch runs the command on the Console as well as on the other hosts in the Deployment Model Managed Hosts. In the screenshot below, I use only the standalone All-in-One Console Appliance type 3199
What actually is “Deploy Changes” in the Deployment Model
When the Admin tab detects changes that need to be sent to managed hosts in the Deployment Model, it shows a notification in the Admin tab. Despite the number of changes have done up to the moment (one or more changes), you can not select only one change from others to process, but all of them have to push out from the Console during deployment.
QRadar stores any changes made on the Console in the “staging” area. This triggers a notification that the user needs to deploy the changes. Copying changes over to the “runtime” or “deployed” area of qradar (proper tables in postgres, /opt/qradar/conf/ directory on each host) occurs after clicking the appropriate button. Finally, hostcontext tells the components to reload or restart.
A user may initiate the Deploy Changes Action or Full Deploy Changes from the main UI interface. You can also do a full deploy of changes from the command line using do_deploy.pl script. Using the same script, deploying occurs automatically after system installation or patching. It is also executed during the Add Managed Host action.
Please keep in mind that if you are still using a version prior to 7.3.1, a service interruption is expected during full deploy changes. Since the introduction of the ecs-ec-ingress service, this is no longer a problem.
Execution of the “Deploy Changes” process step by step
Tomcat creates an archive of changed configuration files, which applies to all managed hosts. The archive contains the results of comparing files in the staging and deployed folders and all their subfolders. We call this archive a Globalset. Each Globalset has two configuration-set archives: one for Full Deploy and one for Incremental Deploy. There are also similar archives for QVM, applicable only to separate QVM components. Tomcat also creates a globalset_list.xml file that contains checksums for each package and the token of the current deployment. In the file
/store/configservices/configurationsets/IP.deploymentToken.txt
We can find recently installed archives configurationset on each Managed Host.
When a Managed Host hostcontext (Console is also a Managed Host from this perspective) finds that a new global set is ready, it downloads the globalset_list.xml file. Based on that, Managed Host knows which archive to process. MH downloads it and unzips in the /store/configservices/deployed/GLOBALSET/ folder.
After that, the Managed Host runs local transformers. These ones will create new final configuration files for the host in the /store/configservices/deployed/LOCALSET/ folder. Finally, hostcontext copies LOCALSET/* files to /opt/qradar/conf/ on the Managed Host and notifies all components of the change. It also reports the current deployment token to the Console and lets it know the process was successful.