How does it work:
we have two kind of servers which upload data to erp5:
1. Reference server, which's data is considered as correct and used as a reference
2. Server to check, which's data is need to compare with reference server's data, if there has difference, we consider this server is in bad state and modified by someone
Here is an overview:
1.Server
With this understanding of digital signatures, the UEFI "Secure Boot" technology consists of a collection of keys, categorized as follows:
- Platform Key (PK)
- Key Exchange Key (KEK)
- Whitelist Database (DB)
- Blacklist Database (DBX)
On a system with Secure Boot enabled and configured, each of these items will contain the public portions of public/private key pairs. The keys are used to authorize various components of the firmware and software.
- The Platform Key (PK) establishes a trust relationship between the platform owner and the firmware (UEFI BIOS) by controlling access to the KEK database. There is a single PK per platform, and the public portion of the PK is installed into the system, typically during production at the OEM. The private portion of the PK is necessary for modifying the KEK database.
- The Key Exchange Key (KEK) database establishes a trust relationship between the firmware and the OS. The KEK consists of a list of public keys that can be checked against for authorization to modify the whitelist database (DB) or blacklist database (DBX). There can be multiple KEKs per platform. The private portion of a KEK is necessary for modifying the DB or DBX.
- The whitelist database (DB) is a list of public keys that are used to check the digital signature of a given firmware or software. To discuss the DB, let's assume the system is booting and is about to execute the bootloader for selecting an OS to boot. The system will check the digital signature of the bootloader using the public keys in the DB, and if this bootloader was signed with a corresponding private key, then the bootloader is allowed to execute. Otherwise, it is blocked as unauthorized.
- Conversely, the blacklist database (DBX) is a list of public keys known to correspond to malicious or unauthorized firmware or software. Any software signed with a corresonding private key from this database will be blocked.
XXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXX
MORE
2.ERP5:
2.1 Data Structure
Here is the 4 most used data object: Compute Node, Data Stream, Data Array, Data Mapping
2.1.1 Compute Node
It present a object which have data come, in our case, it present server, for each server, we need to create one data product
2.1.2 Data Stream
When server start, it uploads data into Data stream object, one server can have only one data stream, which means data inside grows from time to time.
each complet data is separeted by beginning_date and end_date.
The blow example log contains two complet data
{beginning_date: "2022/11/17 16:42 CET"}
{"path": "/sysroot/usr/bin/", "stat": {"st_dev": 2050, "st_ino": 402653315, "st_mode": 16877, "st_nlink": 2, "st_uid": 0, "st_gid": 0,
"st_rdev": 0, "st_size": 24576, "st_blksize": 4096, "st_blocks": 80, "st_atime": 1642709435, "st_mtime": 1666710210, "st_ctime": 1666710210, "st_atime_ns": 0,
"st_mtime_ns": 616307360, "st_ctime_ns": 616307360}}
.......
.......
{"path": "/sysroot/usr/sbin/dhclient-script", "stat": {"st_dev": 2050, "st_ino": 91798, "st_mode": 33261, "st_nlink": 1, "st_uid": 0, "st_gid": 0,
"st_rdev": 0, "st_size": 14380, "st_blksize": 4096, "st_blocks": 32, "st_atime": 1664810268, "st_mtime": 1622091588, "st_ctime": 1664810268, "st_atime_ns": 0,
"st_mtime_ns": 0, "st_ctime_ns": 943505665}, "hash": {"sha256": "a30068639e634f11ea66377effcbc946c3d2a18155df24713b5b2c8205af9980", "sha512":
"1175c8ddf37f0d9b6a16c48390e609e054d8e284ccaba763dcb15152eb2c25a6d7a66f33fd71edea52d4d1b75e8ce95c42171a618ff6e73827e36d389d8b66d7"}}
{"path": "/sysroot/usr/sbin/ip", "stat": {"st_dev": 2050, "st_ino": 91791, "st_mode": 41471, "st_nlink": 1, "st_uid": 0, "st_gid": 0,
"st_rdev": 0, "st_size": 7, "st_blksize": 4096, "st_blocks": 0, "st_atime": 1664810268, "st_mtime": 1612568099, "st_ctime": 1664810268, "st_atime_ns": 0,
"st_mtime_ns": 0, "st_ctime_ns": 899505666}, "target": "/bin/ip"}
{end_date: "2022/11/17 16:45 CET", end_marker: "fluentbit_end"}
.
.
{beginning_date: "2022/11/18 10:42 CET"}
{"path": "/sysroot/usr/bin/", "stat": {"st_dev": 2050, "st_ino": 402653315, "st_mode": 16877, "st_nlink": 2, "st_uid": 0, "st_gid": 0,
"st_rdev": 0, "st_size": 24576, "st_blksize": 4096, "st_blocks": 80, "st_atime": 1642709435, "st_mtime": 1666710210, "st_ctime": 1666710210, "st_atime_ns": 0,
"st_mtime_ns": 616307360, "st_ctime_ns": 616307360}}
.......
.......
{"path": "/sysroot/usr/sbin/dhclient-script", "stat": {"st_dev": 2050, "st_ino": 91798, "st_mode": 33261, "st_nlink": 1, "st_uid": 0, "st_gid": 0,
"st_rdev": 0, "st_size": 14380, "st_blksize": 4096, "st_blocks": 32, "st_atime": 1664810268, "st_mtime": 1622091588, "st_ctime": 1664810268, "st_atime_ns": 0,
"st_mtime_ns": 0, "st_ctime_ns": 943505665}, "hash": {"sha256": "a30068639e634f11ea66377effcbc946c3d2a18155df24713b5b2c8205af9980", "sha512":
"1175c8ddf37f0d9b6a16c48390e609e054d8e284ccaba763dcb15152eb2c25a6d7a66f33fd71edea52d4d1b75e8ce95c42171a618ff6e73827e36d389d8b66d7"}}
{"path": "/sysroot/usr/sbin/ip", "stat": {"st_dev": 2050, "st_ino": 91791, "st_mode": 41471, "st_nlink": 1, "st_uid": 0, "st_gid": 0,
"st_rdev": 0, "st_size": 7, "st_blksize": 4096, "st_blocks": 0, "st_atime": 1664810268, "st_mtime": 1612568099, "st_ctime": 1664810268, "st_atime_ns": 0,
"st_mtime_ns": 0, "st_ctime_ns": 899505666}, "target": "/bin/ip"}
{end_date: "2022/11/18 10:46 CET", end_marker: "fluentbit_end"}
2.1.3 Data Array
Data Array contains data from data stream after transforming, each completed data will be converted to one data array, it means one data stream can have several data arrays
Actually we get only path and sha256 from data stream
There has 3 ways to store those datas in data array:
2-D Array with basic dtype(int, float, string....), but we need compare to the reference data to find which path has wrong sha256, compare 2-D array is slow and not efficient
| path |
sha256 |
| sysroot/usr/sbin/dhclient-script |
a30068639e634f11ea66377effcbc946c3d2a18155df24713b5b2c8205af9980 |
| ....... |
....... |
1-D Array, but with complex dtype, but it's not supported by wendelin
| (sysroot/usr/sbin/dhclient- script, a30068639e634f11ea66377effcbc946c3d2a18155df24713b5b2c8205af9980) |
| |
still 1-D array, but this time we map complex value into a simple integer, 1 is the mapped value which represent one (path, sha256), which this structure, the comparaison is very efficient.
we finally use 1-D array with mapping, to archive this, we use data mapping object
XXXXXXXXX put algorithm complexity here
2.1.4 Data Mapping
Data mapping map complex data, it take complex object like: (1, test, 3.4) then return a single integer value
2.2 Data Processing
Data Processing is triggered by alarm, there has 2 kind of server data to handler
2.2.1 Reference server data
For such data, we only change state to processed and archive previous one if we upload several times
2.2.2 Normal Server data
First we search reference data with the same distribution to compare with it, if there has difference, maybe it's a package updated, in this case, a new data array is generated which contains those differences
Then we search next reference data to compare, the reference data is sorted by Sort index of distribution category Sort index.