New Weekly Email

I started a weekly newsletter to share some of the more interesting links I find over the course of the week. You can subscribe here http://eepurl.com/bNRrFX

Posted in Uncategorized | Leave a comment

Increasing security of openldap password hashing

I’ve been trying to figure out how to change the hashing method of openldap from SHA to something more secure (specifically SHA2). After spending hours trying to custom compile it with a special overlay I found out how to use the unix crypt command. Of course, the instructions were for slapd.conf, not for olc.
It’s pretty straightforward, just add the following 2 lines to slapd.d/cn=config/olcDatabase={-1}frontend.ldif

olcPasswordHash: {CRYPT}
olcPasswordCryptSaltFormat: "$6$%.12s"

This will of course break all of your passwords, so be careful.

Posted in Uncategorized | Leave a comment

RHEL 7 and Network Manager DHCP Issues

So at $work we’ve decided to start using DHCP for assigning addresses to our severs. I’ll write up a post on why, but I assure you, it’s a good enough reason.
Any way, I noticed the other day that my webserver was consistently losing the connection to our MySQL database. Looking into the logs, it occurred every time DHCP went to renew the lease.


Sep 18 08:20:59 xxxxx dhclient[652]: DHCPREQUEST on eno16780032 to 10.21.x.x port 67 (xid=0x68397b65)
Sep 18 08:20:59 xxxxx dhclient[652]: DHCPACK from 10.21.3.2 (xid=0x68397b65)
Sep 18 08:20:59 xxxxx NetworkManager[595]: address 10.21.x.x
Sep 18 08:20:59 xxxxx NetworkManager[595]: plen 24 (255.255.255.0)
Sep 18 08:20:59 xxxxx NetworkManager[595]: gateway 10.21.x.1
Sep 18 08:20:59 xxxxx NetworkManager[595]: server identifier 10.21.x.x
Sep 18 08:20:59 xxxxx NetworkManager[595]: lease time 3600
Sep 18 08:20:59 xxxxx NetworkManager[595]: nameserver '10.21.x.x'
Sep 18 08:20:59 xxxxx NetworkManager[595]: nameserver '10.21.x.x'
Sep 18 08:20:59 xxxxx NetworkManager[595]: domain name 'xxxx.lcl'
Sep 18 08:20:59 xxxxx NetworkManager[595]: (eno16780032): DHCPv4 state changed bound -> bound
Sep 18 08:20:59 xxxxx dhclient[652]: bound to 10.21.x.x -- renewal in 1636 seconds.

Digging around on the net, I wasn’t able to find anyone that had my problem, however did find something similar. https://mail.gnome.org/archives/networkmanager-list/2015-June/msg00018.html It looks like the new NetworkManager and dhclient with a kernel module now resets the routing table, which of course drops all connections. I haven’t found a solution yet, but I want to avoid going back to static addresses.

Posted in Uncategorized | Leave a comment

Resizing root partions in VMWare

Just a quick script to resize a root partition in linux live so I can remember. I’ll be writing up a explanation of all the steps later this week.

echo “1” > /sys/class/scsi_device/0\:0\:0\:0/device/rescan
echo “1” > /sys/class/scsi_device/2\:0\:0\:0/device/rescan
fdisk /dev/sda
partx -l /dev/sda
partx -v -a /dev/sda
vgextend /dev/vg_00 /dev/sda3
lvextend -l +100%FREE -r /dev/vg_00/lv_root

Posted in Uncategorized | Tagged , | Leave a comment

Quick redis queue size nagios plugin

I’ve been running into some issue with buffers filling up in my logstash setup (more to come later on that) so I’ve been adding a redis buffer. I still had that fill up once so I whipped up this script to check the length of redis queue.

Script

Posted in Uncategorized | Tagged , | Leave a comment

Data Science Tools/Tips

I was recently talking data science with a friend who wanted some good links and ideas on how to train non programmers on how to do data analysis.
First thing is to determine what type of analysis is needed, visualization, correlation, causation, classification. Then you just start exploring. A few links came to mind,

http://www.amazon.com/Data-Science-Scratch-Principles-Python/dp/149190142X

http://www.datacommunitydc.org/blog/2013/07/python-for-data-analysis-the-landscape-of-tutorials

http://kenanbek.me/post/guide-to-python-data-mining-data-analysis-and-data-visualisation-with-tutorials-and-samples/

Posted in Uncategorized | Leave a comment

Piwik and Load Balancers

Ran into an issue with Piwik not seeming to deal with load balancers and the X-Forwarded-For header. The documentation for setting up this relativly common thing is a little lacking in the official documentation, and I found it’s a bit prickly. I finally found the instructions here but it neglects to mention that it needs to be exactly this way. To summarize though, the steps are as follows;
1. add the line proxy_client_headers[] = "HTTP_X_FORWARDED_FOR" (note the HTTP, all caps, and underscores
2. On your LB (or proxy) make sure that the ip header is X-Forwarded-For. (note the proper caps, and the dashes)

Posted in Uncategorized | Tagged , , | Leave a comment

Logstash and WSO2 Carbon Logs, dealing with Java Stack Traces

So, now that I’ve got my WSO2 Cluster setup, I get to diagnose issues. The biggest problem is that when trying to work through a cause I’ve got to look at half a dozen log files spread across half a dozen machines. Centralized logging is the solution of course.

I prefer logstash and kibana because they’re free and very configurable. They are suprisingly easy to setup, and once you find a few tools (grok debugger) easy to configure. The biggest problem is that most logs are only a single line. However, WSO2 has the nasty habit of dumping java stack traces in it’s log all the time. Luckily Logstash has the Multiline filter to help with that. Configuring Multiline is a bit of a pain, so here is the config I’m using.

input {
 syslog {
  port => 514
  type => "syslog"
 }
}

To make life easier, I just use rsyslog for everything, one thing I didn’t realize is that syslog automatically applys a syslog grok, and truncates the message.

filter {
    if "_grokparsefailure" in [tags] {
        grok {
            type => "syslog"
            match => ["message", "%{SYSLOG5424PRI}%{TIMESTAMP_ISO8601} +(?:%{HOSTNAME:syslog5424_host}|-) %{SYSLOGPROG}%{GREEDYDATA:messagebodysyslog}"]
            match => ["message", "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} +(?:%{HOSTNAME:syslog5424_host}|-) %{SYSLOGPROG}%{GREEDYDATA:messagebodysyslog}"]
            remove_tag => ["_grokparsefailure"]
        }
        if "_grokparsefailure" not in [tags] {
            mutate {
                replace => ["message","%{messagebodysyslog}"]
                remove_field => ["messagebodysyslog"]
            }
        }
    }
}

This section just parses the syslog portion out of anything not caught by the input syslog filter. Notice the 2 matches, for some reason I have some rsyslog messages coming in with one time format, and others with a different one. Somtimes even from the same machine

filter {
    if "wso" in [program] and "multiline" not in [tags] {
                grok    {
            match => [ "message", "TID\: \[%{INT}\] \[%{WORD:product}\] \[%{TIMESTAMP_ISO8601:logdate}\] +%{LOGLEVEL:level} \{%{DATA:classname}\} - %{GREEDYDATA:messagebody22}"]
                        remove_tag => ["_grokparsefailure"]
                }
        if "_grokparsefailure" not in [tags] {
                    mutate {
                            replace => ["message","%{messagebody22}"]
                remove_field => ["messagebody22"]
                    }
        }
    }

}

Here is the wso2 parser, it gets almost all versions of wso2 messages, except stack traces of course

filter {
    if "wso2" in [program] {
        multiline {
            pattern => "(Uncaught exception.+)|(([^\s]+)Exception.+)|(([\s]+)at.+\))|(.+\.Exception)"
            stream_identity => "%{logsource}.%{@type}"
            what    =>"previous"
        }
    }
}

The stack trace, it only catches if it’s a wso2 message, and any line that matches the pattern regex gets stored and merged

filter {
    if "wso2" in [program] {
        mutate {
            replace => ["type","wso2_carbon"]
            add_field => ["logsource","%{syslog5424_host}"]
            remove_field => ["@originalmessage"]
            remove_tag   => ["_grokparsefailure"]
        }
    }
}

Clean up the messages

output {
 elasticsearch_http {
  host => "10.21.3.48"
 }
}

Output to elasticsearch

Posted in Technical | Leave a comment

Important NFS setting when using OSX Clients

I’ve been getting permission denied errors when connecting to nfs shares from my OSX system. According to logs on the server, it was authenticating fine, and no errors were showing up. Turns out, OSX uses a nonstandard port. So you need to add insecure to the export options.

Posted in Technical | Tagged | Leave a comment

Friday’s Cool Tools May 23rd

In an attempt to post more regularly I’m going to start a new entry going over some of the neat tools I’ve found over the week.

I’m a huge fan of Notational Velocity style apps, when I can use a Mac one of the first things I setup is NVAlt which is a fantastic program in and of itself. However, I’m also a huge fan of vi/vim, as well as stuck on a Windows system at work. I was thrilled to discover nvim which brings a similar environment to vi and the command line.

https://github.com/cwoac/nvim

nvim screenshot

I also have been using AsciiFlow quite a bit recently to do quick diagrams for emails. This is an awesome web tool that lets you create ASCII Diagrams.

http://asciiflow.com

asciiflow screenshot

Posted in Uncategorized | Tagged , , , | Leave a comment