Kibana and ElasticSearch over Haproxy logs

One of the most interesting things about open source, is the capability of being able to deploy software that can compete with tools more emblazoned or “enterprise” grade appliance.

One of these software is HaProxy which, with some tuning and doc reading, can easily sustain 20k-30k connection per seconds on a 1GB dual core virtual machine.

Of course, during testing phase and during live you would need to see haproxy logs, in particular error logs but also the access one; since the amount of data is rather big, you’d prefer having another server that has to do with write on the disk and let haproxy only deal with load balancing.

This can be easily done with haproxy due to his inner ability to use a remote logging daemon without having to use the local one.

This can be easily achieved on the conf file by adding under global section:

log "syslogip"
log-send-hostname "haproxy hostname"

This will redirect to facility local2 of remote every haproxy log.

After you have done this simple step, one question arise: how to parse those logs and get some useful statistic?

What “database” should i go for writing all that data?

This is where tools like elasticsearch come helping you.

There’s an extension to make your rsyslog talk with elasticsearch so it’s all pretty much trivial to install and configure.

Start from a template like this one to configure how to send your info to elastic search (this one suppose your elastic search is on localhost)

module(load="omelasticsearch") # for outputting to Elasticsearch

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index" type="list") {
    constant(value="logstash-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog" type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
local2.* action(type="omelasticsearch" template="plain-syslog" searchIndex="logstash-index" dynSearchIndex="on")

#this is to avoid local2 being processed in other log rules
& ~

Why we use this template? because we want to adhere to logstash format, in case we want to play with that tool or use one of his output plugin.

At this point you can use Kibana for general looking to those logs.

A Wonderful world with ansible

Ansible is a beautiful tool to help you manage a lot of linux server.

It follows the philosophy that orchestrating servers should be dead simple rather than complex as writing software.

Here some quick and simple use for ansible

update apt cache and install python-requests:

ansible all -u myuser -s -m apt -a "pkg=python-requests state=installed update_cache=yes"

copy /tmp/myfile to any host :

ansible all -u myuser -s -m copy -a "src=/tmp/myfile dest=/tmp/"

Of course things can be made a bit more complex depending your usage

From Screen to Tmux

If you ever worked on long task you’ll end up having the need of detaching from your terminal and come back the next day, without loosing your output.

This is where screen and tmux come handy, by making you able to detach your console and logging out.

Lately i’ve been trying to migrate from screen to tmux, mostly because tmux looks more advanced and modern from screen.

Thanks to this page it’s quite easy to migrate from screen to tmux

here’s a quick reference for you all.

The formatting here is simple enough to understand (I would hope). ^ means ctrl+, so ^x is ctrl+x. M- means meta (generally left-alt or escape)+, so M-x is left-alt+x

It should be noted that this is no where near a full feature-set of either group. This – being a cheat-sheet – is just to point out the most very basic features to get you on the road.

Action tmux screen
start a new session tmux OR
tmux new OR
tmux new-session
screen
re-attach a detached session tmux attach OR
tmux attach-session
screen -r
re-attach an attached session (detaching it from elsewhere) tmux attach -d OR
tmux attach-session -d
screen -dr
re-attach an attached session (keeping it attached elsewhere) tmux attach OR
tmux attach-session
screen -x
detach from currently attached session ^b d OR
^b :detach
^a ^d OR
^a :detach
rename-window to newname ^b , OR
^b :rename-window
^a A
list windows ^b w ^a w
list windows in chooseable menu ^a “
go to window # ^b # ^a #
go to last-active window ^b l ^a l
go to next window ^b n ^a n
go to previous window ^b p ^a p
see keybindings ^b ? ^a ?
list sessions ^b s OR
tmux ls OR
tmux list-sessions
screen -ls
toggle visual bell ^a ^g
create another shell ^b c ^a c
exit current shell ^d ^d
split pane horizontally ^b “
split pane vertically ^b %
switch to another pane ^b o
kill the current pane ^b x OR (logout/^D)
close other panes except the current one ^b !
swap location of panes ^b ^o
show time ^b t
show numeric values of panes ^b q

Killing in the name of

Using psmisc in python i’ve found this function quite useful in many projects i’ve been working lately.

Basically it will kill all the process older than one hour that match “name”

All those try except are just to avoid permission denied on kill or on name check.

def killing_in_the_name_of(name):
    now = time.time()
    for p in psutil.get_process_list():
        exe = ""
        try:
            exe = p.exe
        except Exception,e:
            pass
        if name in exe:
            exetime = now - p.create_time
            if exetime > 3600.0:
                try:
                    p.kill()
                    print "killed %s ( %d s of execution time)"%(exe,exetime)
                except Exception,e:
                    print e

A suitable vimrc for python

Since i find myself in a lot of trouble when going to a new linux system due to misconfigured vim i thought

i might share with you my tipical default configuration for vim that is good for developing in python

here are the main lines, since i use a dark background i like a lot the “background=dark” feature

set compatible
syntax on
set background=dark
if has("autocmd")
  au BufReadPost * if line("'\"") > 0 && line("'\"") < = line("$")
    \| exe "normal g'\"" | endif
endif

set showcmd             
set showmatch           
set incsearch           

autocmd FileType python set tabstop=4|set shiftwidth=4|set expandtab " Python
autocmd FileType make set tabstop=8|set shiftwidth=8|set noexpandtab " Makefile
autocmd FileType man set tabstop=8|set shiftwidth=8|set noexpandtab " Man page (also used by psql to edit or view)
autocmd FileType calendar set tabstop=8|set shiftwidth=8|set noexpandtab

I hope this will help you too