According to Gartner's Group IT Glossary:

"Network intelligence (NI) is an enabling technology that allows communications service providers (CSPs) to capture subscriber-, service- and application-level awareness contained in network traffic. This information is analyzed and exposed for integration with other applications in the back office, allowing CSPs to apply granular policies to influence customer experience and adapt to dynamic shifts in application and service usage. The solution is based on nonproprietary hardware and software platforms and can be used by CSPs on any network."

Well, almost true, but sorry Gartner, why are you limiting this to CSPs (Communication Service Providers) ?

Take a look at the following example charts:

Company Network OverviewCompany Traffic Overview ELK Dashboard

In this simple ELK dashboard, we have an immediate overview of what's happening overall in a company network. There's a Transmission (BitTorrent) client downloading stuff, people are spending time on Social Networks, there are some storage flows (no NFS, no CIFS, no FTP, no iSCSI, maybe Dropbox or AWS S3 ?), there's somebody using TOR but the Internet pipe is not so much used.

Company Security OverviewCompany Security Overview ELK Dashboard

In this other ELK dashboard we see that there are Scans on the internal network (do you have network probes everywhere ?), in the detail there are a lot of SYN, XMAS and NULL scans, there are some contacts with IPs with Bad Reputation (according to Talos checklists), there is an average 50/100 authentication requests (these are due to the portscans), and the average Risk Index is about 15/30% (yes, this is happening in Fl0wer 1.3, now available).

Wanna some more insight ?

The new Threat Index ViewThe new Threat Index View in the OSS Python Interface.

Obviously, if you want, you can drill in and have all the required detail from your analytics tool, hopefully integrated with all other information sources that your analytics can receive, like firewall logs, IDS logs and System logs.

Do you think it takes big bucks to have such detail ? I've done it on a 600E refurbished workstation (a 2012 Fujitsu Celsius R570-2 took on eBay), a lab network, the free ELK stack and a disruptive low-price Network Intelligence product: Fl0wer.

According to an article by Anton Chuvakin from Gartner's Blog (sorry, didn't ask for permission but hey, it's from Gartner's blog, neither Google did :-D !):

These are some of the reasons why a security data lake project fails:

  • Dirty data – you throw stuff in and then cannot use it; a #1 “fail-cause” (great story about it)
  • Trouble with collecting data – SIEM vendors spent 10+ years debugging their collectors for a reason…
  • Trouble with accessing data – data went in – plonk!- and now nobody knows how to get it out to do analysis (great story here)
  • No value beyond collection – the data lake was created and filled with data, so it is there just in case, but any subsequent project phases stumbled
  • No value beyond keyword search – data lake was created to enable advanced analytics, but ultimately delivered only basic keyword search of logs
  • No threat detection value – this happened when somebody hired a big data company to build a security data lake; they build all the plumbing and said “ah, security use cases? you do it!” and left
  • Failure to conceptualize and define the security analytics use cases – OK, we will now detect threats… OK, how? Well, nobody knows and no time to experiment. And see #1 – dirty data
  • Security analytics use case design much harder then expected
  • Much higher bar for analytics and big data expertise talent requirements and failure to acquire said talent.

Silk, flowtools, product's integrated Netflow/IPFIX collectors and many more OSS tools are great learning instruments to understand the basics of NetFlow/IPFIX technology and get some insights, but entering contextless information in your security data-lake, as you see, is likely to make you lose a lot of time instead of giving you added value.

TLDR: It is useful to know that there was a UDP flow on port 53 to IP but it is much more useful to know that there was a DNS flow to the Google DNS server in the United States in violation of Company Business policy that only involves the use of well-defined internal DNS servers, do not you agree?

Fl0wer gives you this added value, enabling you to integrate it with your favorite Analytics tools like ELK or Splunk in the way you prefer, using open standards with extreme high performance.

With Fl0wer, developing the above dashboards in ELK took less than an hour on "not so current" hardware. Further, having the data directly inside ELK (but probably with Splunk too) provides you with the x-pack plugins, by means of which you can add machine-learning and alarms (in addition to what you can do directly with Fl0wer).

Seamless integration with your favorite analytics tool, in combination with machine-learning tools (integrated in ELK) and cleaned-up data are a guarantee for success.

What is your vision about Network Security: Do you think the cost of time that your security analyst puts you at your disposal is free, or is it wiser to deal with things more intelligently ?