— Log management as a λrchitecture OpenSource UseCase#*

The incontournable logs aggregation Hortonworks, MapR and myDistrib ( 2.6.X) has been continued.



Milestones

Oldschool grep to get into logs.

logstash as my "λ 1 en 2" : #1 elasticsearch, #2 webHDFS.

...
output
{
  elasticsearch
  {
    hosts => ["jbd-vm01:9200"]
  }
  webhdfs
  {
    host => "jbd-vm01"
    port => "50070"
    path => "/test/lstash-00/%{+YYYY-MM-dd}/%{host}/%{hdfs}"
    user => "hduser"
    codec => line { format => "%{message}" }
  }
}
...

#1 Elasticsearch world, plugins Architecture xN.

#2 Grep 2.0 to investigate everywhere, whenwhere, for datascientist only. Hadoop World, Architecture #*xN.

TODO#1 MapReduceV2 past & (no:-)futur ?

From ASM, C/C++, SDK32/MFC, JAVA, 2D/3D, PHP, Androïd, HTML5 to Pig Latin, jruby, gem by the way Shell(s), PERL, IDL, OCCAM, LISP, PROLOG, APT...

$ gem build logstash-output-webhdfs.gemspec
$ sudo gem install logstash-output-webhdfs-2.0.4.gem -f
$ gem list --local | grep webhdfs
logstash-output-webhdfs (2.0.4)
$ logstash-2.3.2/bin/logstash-plugin install logstash-output-webhdfs
Validating logstash-output-webhdfs
Installing logstash-output-webhdfs
Installation successful
$ logstash-2.3.2/bin/logstash-plugin list | grep webhdfs
webhdfs
TODO#3 Dev langages from XIXième to XXième.

and grok to always end with vi, ZZ.

...
filter
{
  grok
  {
    match => { "message" => "%{TIMESTAMP_ISO8601:date} %{LOGLEVEL:level} %{JAVACLASS:class}%{NOTSPACE} %{GREEDYDATA:texte}" }
  }
  mutate
  {
    add_field => { "hadoop" => "%{path}" }
    add_field => { "hdfs" => "%{path}" }
  }
  mutate
  {
    gsub => [ "hadoop", ".*hduser-", "" ]
    gsub => [ "hadoop", "-jbd.*$", "" ]
    gsub => [ "hdfs", "^.*logs/", "" ]
  }
}
...

myDistrib 2.6.X network security chalenge continue with VBox tunnings. @ myToolBox 2.7.X

and a few advanced all-road tunningS for Linux-eux.

...
04:33:08.661455 Changing the VM state from 'RUNNING' to 'GURU_MEDITATION'
...
Status: CORRUPT -> The filesystem under path '/' is HEALTHY
...
WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hduser OPERATION=...Finished - Failed TARGET=RMAppManager RESULT=FAILURE...
...
# @JBD patch logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/java
...
# cf doc !!! message_format => "%{message}"
...

TODOs

The incontournable logs aggregation Hortonworks, MapR and myDistrib 2.6.X to be continued #1, 2.7.X TODO#2.