[ANN][ARGUS]|ARGUS|| ACCOUNTING AND AUDITING PLATFORM ON THE BLOCKCHAIN |

Quote from: Uncurlhalo on March 03, 2017, 03:12:39 PM

ARGUS DEV UPDATE 1.0.1

ARGUS DATABASE CLIENT

Using network flow data for network operations, performance and security mangaement is a large data problem, in that we're talking about collecting, processing and storing a large amount of data. Modern Blockchain databases management (MBDM) technology works very well to assist in processing and mining all that data. Some thought and engineering, however, needs to be done to get the most benefit.
When using MBDM technology to support flow data auditing and analysis, the primary issue is database performance and data arrival rates. Can the RDBM keep up, inserting the flow data from the sensors and also serve queries against the data? For most companies, corporations and some universities and colleges, using an MBDL like MySQL to manage and process all the primitive data from their principal argus sensors (or netflow data sources) works well. Many database systems running on contemporary PC technology (dual/quad core, 2+GHz Intel, 8-16GB memory, 1+TB disk) can handle around 500-1000 argus record insertions per second, which is around 50-85M flows per day or around 70GB flow data/day), and have plenty of cycles to handle many standard forensic queries to the data, like "show me all the activity from this network address in the last month".
For larger sites, such as member universities of Internet2, where the flow record demand can get into the 20-50K flows per second range, there are databases that can keep up.

Code:

ra -r netflow.file

When reading from the network, argus clients are normally expecting Argus records, so we have to tell the ra* program that the data source and format are Netflow, what port, and optionally, what interface to listen on. This is currently done using the "

Code:

-C [host:]port" option.

Code:

ra -C 9996

If the machine ra* is running on has multiple interfaces, you may need to provide the IP address of the interface you want to listen on. This address should be the same as that used by the Netflow exporter.

Code:

ra -C 192.168.0.68:9996

Code:

thoth:tmp carter$ ra -r /tmp/ra.netflow.out    StartTime  Proto      SrcAddr  Sport   Dir      DstAddr  Dport SrcPkt DstPkt SrcBytes DstBytes
12:34:31.658    udp 192.168.0.67.61251     ->  192.168.0.1.snmp        1      0       74        0
12:34:31.718    udp 192.168.0.67.61252     ->  192.168.0.1.snmp        1      0       74        0
12:35:31.848    udp 192.168.0.67.61253     ->  192.168.0.1.snmp       10      0      796        0
12:35:31.938    udp 192.168.0.67.61254     ->  192.168.0.1.snmp        1      0       74        0
12:35:31.941    udp  192.168.0.1.snmp      -> 192.168.0.67.61254       1      0       78        0
12:35:31.851    udp  192.168.0.1.snmp      -> 192.168.0.67.61253      10      0      861        0

thoth:tmp carter$ racluster -r /tmp/ra.netflow.out    StartTime  Proto      SrcAddr  Sport   Dir      DstAddr  Dport SrcPkt DstPkt SrcBytes DstBytes
12:34:31.658    udp 192.168.0.67.61251     ->  192.168.0.1.snmp        1      0       74        0
12:34:31.718    udp 192.168.0.67.61252     ->  192.168.0.1.snmp        1      0       74        0
12:35:31.848    udp 192.168.0.67.61253     ->  192.168.0.1.snmp       10     10      796      861
12:35:31.938    udp 192.168.0.67.61254     ->  192.168.0.1.snmp        1      1       74       78

Using a database for handling argus data provides some interesting solutions to some interesting problems. racluster() has been limited in how many unique flows it can process, because of RAM limitations. rasqlinsert() can solve this problem, as it can do the aggregation of racluster(), but use a MySQL table as the backing store, rather than memory. Programs like rasort() which read in all the argus data, use qsort() to sort the records, and then output the records as a stream, have scaling issues, in that you need to have enough memory to hold the binary records.

	Author	Topic: [ANN][ARGUS]\|ARGUS\|\| ACCOUNTING AND AUDITING PLATFORM ON THE BLOCKCHAIN \| (Read 43113 times)
This is a self-moderated topic. If you do not want to be moderated by the person who started this topic, create a new topic.