BGINFO - A Posh Recreation
Recently I have been building a lot of Windows Servers in different environments - one
The TLDR
Grumpy Admin, plays around in PowerShell trying to send Historic Windows Events to his ELK stack and when he gets frustrated that it’s not exactly working, he googles and lifts someone else’s code off GitHub to make it work – ergo happy Grumpy Admin!
Main Post
Grumpy Admin here, I was asked to implement a logging system – so I decided to implement an ELK stack. I know of it, and played around but this implementation was going to be running in parallel to our production logging solution with the hopes that once configured correctly and proved to be stable. ELK can take its place and the primary logging solution for the company.
Naturally, prior to implementation, I produced a working ELK stack that worked like a charm for our Cisco ASA firewall logs. So once, my boss, was happy with it. I decommissioned it and awaited permission to implement a production ELK stack. Grumpy Admin gets frustrated waiting for management to ok things, but like a good Admin, I understand the change control process… Shame our users don’t. I won’t drift in to BOFH territory today!
Finally I’m allowed to build what will be my production ELK stack, as it turns out it will be another “proof in concept” rig, and another rebuild will be required! Grrr…. Grumpy Admin is niffed at this because, as well as the CISCO logs, the boss wants the ELK stack to take a copy of all the Windows logs. He got this idea from Grumpy Admin.
I was trying to sell the power of Logstash and mentioned you can throw anything at it really, and you can store it and parse it and visualise it! He joked and said not windows logs you can’t…. I think my boss already knew you could, but the challenge was already laid down!
After a few hours, building the ELK stack and then getting the CISCO ASA to spit syslog messages at it, and some time showing how cool Kibana is. I set to work on getting the Windows Events logs inside the ELK!
After a bit of research, I decided to configure my Windows Servers to use NXLog which forwards events. To make it easy, I will use NXLog to pre-format them in to JSON.
So with NXLog and changes to my LogStash Config done, within the hour, I was getting Windows Event Logs displayed in Kibana and some nice dashboards for the boss.
Then the boss, who coffee nearly ended up in his face, asked, where yesterday’s Windows Events Log for XXXX Server.
I am sure he wasn’t listening when I described what a forwarder does to him! Anyway this got me thinking, wouldn’t it be good if we have a method of capturing all the previous events stored in the event log and sending them to ELK.
So I thought, I would try!
We know that PowerShell can get the Event Logs, we know that we can convert things to JSON in PowerShell, so this seems like a good marriage. Not that in my current, view point I think any marriage is good!
For example, we could use (i’m going to limit my results to the newest 1 result for easy)
get-eventlog -logname system -newest 1
Then we can pipe that output to the convertto-json cmdlet
get-eventlog -logname system -newest 1 |convertto-json
Or we could even use
Get-WinEvent -LogName system -MaxEvents 1 |convertto-json
Which is little bit more gritty but I don’t like it really! So going to stick with get-eventlog for this.
But how do we get the JSON encoded event log to ELK. I don’t think there is in built PowerShell way! I would love an output-tcpport cmdlet, but there isn’t one… this makes me grumpy!
so let’s write one shall we – We start simple, and once it’s working, go back and make it nice and friendly with prama and data input validation etc, just like a good PowerShell Developer.
function output-tcpport {
$ip="192.168.10.53"
$port="514"
}
Excellent, a good start, coffee time!!!! Burp!
Right we now have an IP Address and Port, and that makes….. a socket!!!! so let’s create a socket using .net system.net.sockets . I am sure you come across this before J
However, let’s check this MSDN article out for more information
https://msdn.microsoft.com/en-us/library/system.net.sockets(v=vs.110).aspx
function output-tcpport {
$ip="192.168.10.53"
$port="514"
$socket = new-object System.Net.Sockets.TCPClient($ip,$port)
}
Now let’s have look at our socket, using Grumpy Admins over used tried and tested get-member investigation methods….
$sockets|get-member (inside the function due to scope)
As you can see quite a few things that can help us here – the thing that strikes me instantly to help me solve this little problem is! Stream. As shown before we can use the .net system.io.streamwriter class to write to a stream.
(Here is hoping I finished and posted the blog I was working on that used the system.io.streamwriter class) If I didn’t just get grumpy with me! I know my partner does, when I forget to do things or start something and never finish it off, I have like 20-30 partial blog posts that I never seem to finish off. Today I’m determined to get a blog post out!
Firstly I need to get the TCP stream in to a variable so we can use it! Variable, object you know what I mean for them pedantic types out there!
$tcpstream = $socket.getstream()
function output-tcpport {
$ip="192.168.10.53"
$port="514"
$socket = new-object System.Net.Sockets.TCPClient($ip,$port)
$tcpstream = $socket.getstream()
}
Next we need to write to the stream, which sends it to the networking stack and over the wire!
Let’s investigate how to do this shall we?
Lets go old skool, and google and hit MSDN for the information we seek
https://msdn.microsoft.com/en-us/library/system.io.streamwriter(v=vs.110).aspx
As I can see, there a writeline, this might be useful. But we need to create a streamwriter object. But as you know that is simple as this 🙂
$streamwriter = new-object System.IO.StreamWriter($tcpstream)
And then we can use it like this!
$streamwriter.WriteLine($content)
With the $content being our JSON encoded Windows Event Log:) so we can do some quick tests, lets quickly throw something in to $content. After all, Grumpy Admin loves to test stuff out!
$content = Get-EventLog -LogName system -Newest 1 |convertto-json $content
Of course we will then need to close the tcpstream and the socket one we are finished with it!
$tcpstream.close() $socket.close()
Now to test our $tcpstream.writeline($content) fingers crossed
Excellent, there stuff inside ELK! Brilliant, not quite perfect due to parsing errors but meh! I’ll take the small victory!
So let’s recap
function output-tcpport {
$ip="192.168.10.53"
$port="514"
$socket = new-object System.Net.Sockets.TCPClient($ip,$port)
$tcpstream = $socket.getstream()
$streamwriter = new-object System.IO.StreamWriter($tcpstream)
$content = Get-EventLog -LogName system -Newest 1 |convertto-json
$streamwriter.WriteLine($content)
$tcpstream.close()
$socket.close()
}
My logstash config doesn’t like the format of JSON I suspect etc. But the message is getting there! So I am not worried about this at the moment, let have a look at my elk config…
So I going to open a new Logstash TCP listener on port 515 and change the type – so that I just get the raw message by-passing all my mutate code for my NXLog forwarders.
tcp {
port => 515
type => "posh-log"
codec => "json"
}
Don’t forget I need to change
$port="515"
Right – still not working as expected let’s do a Google search and see if someone else has come up with this – Grumpy Admin, loves trying to work shit out himself, but chances are someone smarter has already solved your problems.
The top hit on Google!
https://blog.rootshell.be/2015/08/24/sending-windows-event-logs-to-logstash/
I read his blog post, and understand his methods, slight different to mine but the concept is the same, then I head over to his GitHub and take a look at his code to see what I can shamelessly steal to try and get my solution working!
So he has a complete script and function… looking at the code, I can see some things he is doing, he is converting the JSON to a single line using and controlling the depth to 3 levels. This could be the thing that is messing me in my attempt at this problem!
$x = $event | ConvertTo-Json -depth 3 $x= $x -replace "`n",' ' -replace "`r",''
He is also flushing the writer buffer after each event, which is good, I’m sure I would of done that too, honest. He has written his code as a decent enough function with param’s and input validation. All aspect I wanted to do!
Basically doing what I was trying to do but in a more graceful way – I am a lazy admin, as you know. Some he has the solution to my problems with his additions my code would look like this:
function output-tcport {
$ip="192.168.10.53"
$port="514"
$socket = new-object System.Net.Sockets.TCPClient($ip,$port)
$tcpstream = $socket.getstream()
$content = new-object -TypeName PSObject
$streamwriter = new-object System.IO.StreamWriter($tcpstream)
$content = Get-EventLog -LogName system -Newest 1 |convertto-json
$txt = $content -replace "`n",' ' -replace "`r",' ' -replace ' ',''
$streamwriter.WriteLine($tx)
$streamwriter.Flush()
$tcpstream.close()
$socket.close()
}
I understand the code he wrote, so I am actually happy, so I am just going to lift his work! I’m not claiming credit for it, so all is good right! This is just experimenting at the moment really! All good learning.
$LOGSTASH_SERVER = "192.168.10.53"
$LOGSTASH_PORT = 514
Function Dump2Logstash {
param (
[ValidateNotNullOrEmpty()]
[string] $server,
[int] $port,
$jsondata)
$ip = [System.Net.Dns]::GetHostAddresses($server)
$address = [System.Net.IPAddress]::Parse($ip)
$socket = New-Object System.Net.Sockets.TCPClient($address, $port)
$stream = $socket.GetStream()
$writer = New-Object System.IO.StreamWriter($stream)
# Convert the existing data into a JSON array to process events one by one
$buffer = '{ "Events": ' + $jsondata + ' }' | ConvertFrom-Json
foreach($event in $buffer.Events) {
echo "Processing:"
echo $event
# Convert to a 1-line JSON event
$x = $event | ConvertTo-Json -depth 3
$x= $x -replace "`n",' ' -replace "`r",''
$writer.WriteLine($x)
$writer.Flush()
}
$stream.Close()
$socket.Close()
}
And let’s run that and give it a try! right from his example!
$starttime = (get-date).addhours(-1)
$data = Get-WinEvent -FilterHashtable @{logname="*"; starttime=$starttime} | ConvertTo-Json -depth 3
Dump2Logstash $LOGSTASH_SERVER $LOGSTASH_PORT "$data"
A sea of red…This makes me grumpy, was hoping it would just work and I can move on, not a good sign but let carry on and have a look at what is going on.
$data is empty
So I will put our own data string in there, we know that already works!
$data = get-eventlog -LogName Application -Newest 10 |ConvertTo-Json -Depth 3
And let’s run that!
Excellent that worked – we can get mass events up to Logstash – my filter in Logstash isn’t quite right, it doesn’t parse the date correctly but I can fix these later – the concept works. Needs fine tuning but it works!!!
Why I didn’t just Google and steal, and be done with it in the first place I don’t know, but then how do you learn! Where is the fun if everyone just gives you the answers!
Anyway – my whole ELK stack needs work and to be rebuilt before handed over as a production box! So I’m off out for lunch!
Hazzy