LUG Meeting (GnuPlot, InfluxDB, Grafana) 04-06-2015

Okay, here we are again, back from spring break. Linux user— the log— GNU and Linux User Group at the University of Illinois Urbana-Champaign. Let's go ahead and move into the group updates. So I've been working on documentation here for all our services, so this will hopefully get you up and running so you can actually use it. I'm just using Markdown, so it's real simple. Here's the pages, all linked from the main README. You can be like, oh, I want to check out the GitLab. And then there's the actual— here's the host, and here's how to add your keys, and here's how to set up your configuration so that you can pull from the host.

Here's how to use HTTPS.

So all of this stuff is documented here. If you want to use our Tor proxy service, here's how you can set up your local system to actually tunnel through that. Here's, uh, go over the website. We also have a— I don't know if I mentioned this. We do have our own package repository now, a cacher and actual repository where we have packages. So this is how you set up your machine. And I do have it now so that it's publicly accessible. So if you want to have your own packages hosted here that you want to have on your machines and you don't want to have to compile it or carry it around with you, just copy to our server and then follow this documentation on how to run it. You drop it in a directory and you run this command essentially. And then you can actually add that, our repository to your machine and you can apt-get install your stuff from wherever you are from our system.

So that's kind of nice.

So anyway, that's the documentation.

There's a bunch more.

That's the documentation of what I'll be talking about tonight too, Graphite and Grafana. We do have a new system and this is grafana.gnulog.org. It's also influxdb.gnu.log.org. They're all CNAMEs and they point to the actual host docker1.gnulog.org. So all three refer to each other, and it's our first Docker system, and it's running a container for Grafana and a container for InfluxDB. And we'll get into that into the talks today. Also, RMS Roundup, we were going to have a discussion about the, uh, Orchard's, uh, installment talk. Anyone have any comments on that? Um, I think, um, having been in the car with him for 4+ hours One thing I could say is that he doesn't like to talk about technology. I don't know if he gets asked those same questions a lot. Like I said in the past, he would just like reset the URL for a particular question such as, "What is your favorite Linux distribution?" or "What do you use?" or "What is your particular view on this?" like systemd, for example. Those were all shut down fairly quickly. But I think to be fair to him, it was annoying in the car because it kind of hampered discussion. But to be fair to him, I guess in a sense, it's probably best that he, being at his level or his stature, not giving favoritism to any particular distribution or to the things such as systemd. I think he actually said he didn't have any knowledge on it, so he didn't want to comment, which is fair. So those are fine. Granted, there are definitely some intricacies with his personality that made the trip, um, uncomfortable. What'd you say?

Did you pick him up in Chicago?

Yeah, we went to pick him up in Chicago. He stayed with his friend, um, which is interesting. Then he had us run a few errands for him, which we were expecting to get back. Like, I was expecting to do a half day of work, but it ended up being a full day because we're gonna take him to the grocery store. We had to help him find a place to repair his camera in downtown Chicago, uh, or not downtown, but some of the busy areas of Chicago. There's cars all the way behind us, and he's like, go slower, slower, so we could actually see the the actual, um, the, uh, the extra places on this—

they're actually—

sorry, the actual stores for camera paper. Like, looking— there's one down here somewhere. Waylon was going so slow, and he kept going so slow that the people behind us were honking. Like, it was like cars behind him during Chicago on a busy road. It was, it was, it was funny. And then he requested to go to a bookstore, which I enjoyed. That was a Jane Addams Downtown Champaign. You guys are playing—

that's awesome.

It is awesome.

Really cool.

There is tons of good stuff in there. Like, I was looking in the philosophy and political science section.

There was all kinds of cool stuff.

I'll definitely go back there.

Me too.

So that was one of the benefits of the trip, was having him recommend to you where to do that. Other things, his personality was— I know there was some speculation on the internet whether he has Asperger's or not. I went around, I looked at some talks and stuff online. And I'm inclined to believe that's probably true. But I do appreciate him to actually come speak for us. And he gave a great talk. And so I loved the Emacs thing at the end.

You guys, that was really funny.

So it was an experience. I'm glad I got to meet the guy. It was a lot of work too, though.

But yeah.

Yeah, I think that concludes RMS-RAN unless you guys have anything to say about it. What did you guys think about the whole experience?

I mean, there's little points that I didn't always agree with what he said, but the things he's accomplished by being so hard-edged about everything, that he couldn't have done what he did without being so idealistic and unbending. So it's phenomenal. So I really enjoyed getting to see him talk.

Yeah, it was really cool to meet him in person.

Yeah, I'm glad we were able to do that.

So that was fun.

That's the roundup. We're going to go ahead and move into the meeting sections. First off, Kitematic is a Docker GUI that joins the Docker family. So essentially, there's a company that created this thing that allows you to have a GUI for Docker on OS X. And I was actually going to hopefully play with it over break, but I did not get a chance to. But it looks really interesting. I went and pulled up the website. Sorry guys if I'm making some noise. I'm very, very hungry.

I say Simon, I meant Sam, sorry.

So let's take a look at the webpage here. So yes, sysgui, and they have a picture of it here, so you can actually quickly and easily run Docker containers on OS X. And since it's not native, it's where the kernel doesn't have namespace and cgroups. It actually just runs them in a virtual machine, kind of like PuTTY Docker, but gives a prettier interface for it. VirtualBox 3.2.6 was released. Let's take a look at that. I don't know if there's anything particular. Some bug fixes. Looking for features here.

Looks like all bug fixes.

Yep, maintenance release right here. So FYI. Linux containers, userland tools 1.1.1.1 was released. So they have, you can actually fuse filesystem from templates. Oh, proc/sys/net is now writable. Interesting. Few other options here. TTY issues with attachment, etc. Yeah, I've been wanting to do more with LXC tools. I'm mostly using Docker for my runtime. I have used it in the past, but I definitely want to get my program ILLiX supported for this too. So it relies on more than one runtime. Another thing is SSH. OpenSSH 6.8 was released, and one notable thing was— or two notable things. One is that use_dns now defaults to no. And they actually support two keys now. You can do it, you can set up with two public keys. And where is that at?

One second.

I didn't go through this whole list, but, um, search for two. Yes. So remember when public keys have been used for authentication and refused to accept previously used keys? But it also allows authentication methods equal public key that are common public keys to require users to authenticate using two different public keys. So maybe this could be a use case where you have a shared public key among your team, and then you have a private one. So it's just two levels of authentication. Also, just a guess, I don't know if anybody's using this, but this is brand new. It just came out. So that's a use case that I can immediately think about whenever I see this. But it's something that's interesting.

GitHub. Ah, there's your soup. Hey man, how's it going? Good.

So next we have GitHub announced they have PDF viewing on their interface now. So if there is a PDF in there, you can see in this repository there has one. You click on it and it will actually load in the preview section. That's kind of nice. So you don't have to download it and check it out. There was this Unix clone called Coherent that was open source. So that's kind of interesting. I'd never heard of it before. I don't know if it's— I don't know if it was a System V clone. I'm not real sure. But you can check out the source here. It's all in this tarball at Nessus Software. So you check that out if you're interested in it. Also, TrueCrypt. The audit was completed. You guys remember when there was some compromise in TrueCrypt, on their website anyway, apparently, and there was also speculation that maybe TrueCrypt is indeed backdoored or something. But this cryptographer right here actually did— worked on the audit himself, and I guess the big takeaway here is the too long, don't read. TrueCrypt appears to be a relatively well-designed piece of crypto software. So the NCC audit found no evidence of deliberate backdoors or any severe design flaws that would make the software insecure in most instances. So that's good to hear because a lot of people were depending on that piece of software. If you're interested in more information, do check out this blog post. So this tool I came across on, I think, an email list or mailing list somewhere, but it's called HowDoI. It gives you instant coding answers via the command line. So you just give it some strings and it tries to print how, like in a database, how you think it would do something. How do I format the date in bash?

And it gives you an example of how to format it. Also, yes, this one's funny.

How do I create a tar archive?

It gives you an example using exclude and creating the archive and naming it. So that's kind of cool.

Do check this out. I'm not sure if it's in any package repositories, but it's on GitHub right here. So the installation—

Is it just a big self-contained database or does it actually access your own pages?

It's a Python script.

Question? Pass.

Where's it pulling it from? Search URL? Oh, it looks like it queries the website. This website. Well, no, that's for a group. Wait.

How do I? No.

Oh, is it actually pulled from— yeah, it looks like it pulls from— see if I can— no Python there. I do think you can see this in this quick, brief example. But it looks like it pulls from Google. Yeah. Yep, so it's using region string. So it's going to grab whatever is in the results. That's kind of cool. I wonder if it's random each time. I don't know. Need a further look at that.

We'll go ahead and move on. So that's kind of cool. Do check that out.

Nothing for ProcPeriod, nothing for SH section. So I was playing around, there was a discussion on Facebook of how to hide a process on the system if you want to do like a malicious binary. So if you compromise the system, how do you hide stuff? And I was kind of like, well, how can we do this without writing a program in C without, you know, trying to exploit some vulnerability? Are detached from the process table or whatever. Something more simple than something like that. And I was like, well, I started looking through the bash manual page and the exec has a specific option. Let's go ahead and go over here.

Bash exec. And we can see— that's too big.

We can see that So exec by itself, it replaces the current process. In the case of the shell, it will replace the shell with the new process. We're looking for the argument. I wish there was a way to do highlighting or make changes in the cursor color. But I'm looking for— 'cause I can hardly see it on here. I'm looking for this -a. If you look at the actual definition of it, I assume I see -a.

It's the shell's pass name as the 0th argument.

Okay, right, so the 0th argument is the program name. So what we can do is, just to demo this, let's go grab what I came up with. So this, the attack here is you're on a system, you want to cat everything in /etc, you want to get all the configuration files including password and shadow, right? And then you want to, since most systems have Netcat installed, you want to take that data and pass it to Netcat. Netcat, it will then connect to— if you replace this localhost, this is my example, with the actual server that you want to get to, like your server on this port, and then you would like, you know, with the netcat listener on that other host, your host, netcat -l like 4444, and then you use the redirection operator to write it to a file, and all that stuff that you cat will then be written to a file for that. What this actually does is replaces all this with -i, don't mind me, so you can You might have to— you'll have to put something in here and make it sleep 10.

And then we'll do pgrablf. Tim, what happened this time?

Yeah, that sleep is not complete.

That was—

that's not the time.

Why did that do that?

Should be sleeping and then it should be echoing that, or that kind of idea to there.

Yeah, hold on.

Let's just do it a different way.

Let's do this. Let's pretend just to demo it. That should— no, no, no.

What the hell's wrong? I did this a bunch of times.

Hold on. Let's see if I can open my history.

What does it mean, "add a space"?

Well, I think they're doing different things. The one with sleep in it is substituting a file.

OK. Wait, is this— The other one's just standard redirection. OK. Mm. Oh, damn it. So that's replacing the shell. And— oh, yeah, that was bad. That was a bad move.

I literally replaced the shell. That worked that time. You waited 10 seconds and it replaced the shell.

Okay, so. Sorry, guys.

I did this like 4 weeks ago and haven't tested since, so forgive me if I don't remember exactly how this works.

Solarize dark. Pop that up. Boom.

We're going to get this though. All right, so this one will demonstrate— this should demonstrate it.

Oh wait, I think I'm gonna have to do something like this.

Well, either way, I don't have to do it. I don't have to be tricky like that. I should be able to do it.

Well, no, no, no. Yeah, that's gonna kill it. Here, I can just demo it like this. If I can type fast enough. There it is, okay.

That's what I wanted to show you. You don't have to remember the details, but so—

ah, goddamn it.

Because that window died, right? Because we replaced the process of the shell with what we exec'd. And we exec'd sleep 10, so it literally removes that process from memory and replaces this new one in its place. And it's similar to the— pretty much the same thing as the execve system call. And in this particular case, we're in a new window when we ran that, which is what I should have been doing, which I think is why it didn't work. I don't know why it didn't background, but it actually called the process don'tmindme, which is the zeroth argument I passed using the -a option. And then, so instead of saying sleep 10, it says don'tmindme 10. So with this you can hide processes. However, you will not be able to hide the actual arguments after 0. So 1 through whatever will still show in the process table. So I was trying to find a way to do that. I didn't find a good one besides using anonymous pipes, which we'll talk about in just a second because that's what all that other crap is. If you don't understand, like this right here, this generates an anonymous pipe using process substitution. And what that looks—

should look like, I think, if I echo cat, say, I don't know, let's give me some files. What do I got in my directory? Do I have test? T test.

Okay, so that shows you how this, oh, I didn't realize my terminal, sorry. I was looking at this one.

You can't really see that, guys. Now you can.

Okay, so now what's actually happening here is process substitution. The process here that's generated cat test is then given its own file descriptor and the file— the output is then stored in the file descriptor and the file descriptor actually becomes the path that is then passed to echo, right? So in this way, we're not really doing anything interesting because we're just echoing it. But if it was the actual file, like we can't— because echo doesn't work with files, right? You have to do it with cat and then that will actually cat the value of whatever's in that of that file. So this is a case of— this is how process substitution works. So you can actually— one example of process substitution, getting a little off track, that is useful is doing remote diffs. So if you want to check out— yeah, if you want to check out the diff of a file in this one system and a file in another system, you would use this. And can I give you an example? Do I have any systems that I can access right now?

You could just do a local diff too.

That's true. So let me show you how you would do it though. So you would do diff. I like these options, which is the don't worry too much about whitespace. And also, instead of the less than, greater than to show you which parts are which file, use plus and minus. I find it's easier to read. Then you would do— process substitution here, SSH to my host here, right? And then you would actually get the file from there and then you would do another one and the result is— say, hand me one. Actually, I could probably demo this. Yeah, Valboa. So, we want to do— we're going to cat the file /etc/ssh/sshd_config. I think this will work.

I think I have a good setup.

So, cat /etc/sh/sshd_config and what actually happens here is whatever is the output of this process, that is then stored in a file descriptor via the anonymous pipe. So a new one is generated. It starts usually dev/fd61 and I think this one over here for the second will then be 62 and the output of that which is the actual sshd_config file will be stored forward and accessible by that pipe. So if we do this, logs in both systems.

Once it takes both.

Yep, boom, we got the diff. So it actually works. So we SSH into both systems. I public key set up and you can see that the first file was, it actually jumped up to 63 and the second one was 62 and the difference between the SSH deconfiguration on those systems were This file, the second system had allowgroups nsmset and a blank line and a comment. And this one had, well, that was the difference. That was not available in the other one. That was in the NSM one. So that's an example of how you could use process substitution and named pipes for solving things. And there was an excellent article which unfortunately I forgot to comment, but named pipes that—

oh gosh. It might be this dude.

Yes, this article is awesome. If you check this out, I should have put this in the notes, but he shows how instead of doing something complicated like this, you can just use named pipes. And watch, yeah, he actually creates named pipes. So this is different from the regular pipes where you're passing data through. Gives you the create using mkpipeo the pipe. It's just a file that has, you can see, the attribute P for pipe. And then you can work with those. So he gives an example of using the two. And here he's actually just passing the file descriptor from the anonymous pipes here. And then it cuts down on the program quite a bit compared to the one above. And his big thing to take away was speed. Based on the data he was working with, which I think was biological data, is that Instead of using a parallel process substitution, if you can just write it to a file, you can actually write to the pipe and it's actually— or writing to a file is slower. You should write it to the pipe instead. So that's just something. We get a little off track. We'll go ahead and move on to the next thing. Oh, and apparently I have that up there and I forgot I put it there.

Whoops.

Anyways, we already covered that one. Going on to Container City. We're talking about the latest containers. I wrote an article. Okay, so I'm working on the I/O paper. We actually finished it up and we turned it in to the XSEDE conference, waiting for review and hopefully it gets published. But whenever I was doing my testing with Docker, I noticed that when I was running hundreds and hundreds of containers doing some particular task, they would fail using the device mapper storage backend, which is the default in Ubuntu. And I did some digging around and it turns out there's actually this big bug, basically a race condition, which when you dev device mapper in Docker,. It actually occurs because since Docker is statically linked, like, you know, it doesn't rely on any dynamic executables, so the runtime linker is not really called to look all that stuff up. You can see I'm using an LDD in Docker, so it's not dynamic. It actually has— it can't have the code for udev all in that one binary. So that's one thing they skipped off, which I think because it was probably impractical. And second though, I mean, because using the dynamic— because of this bug, I couldn't do my test. So to run 100 or 1,000 containers over and over and graph the data, I had to figure out a way to get it working with our device mapper backend because that's what I'm using for iLib. It's the only backend in Docker that, at this time that I'm aware of, that allows you to set the size of the disk for each container. Otherwise, someone could log into the container, type yes, and pipe it to a file and fill up your entire system and the host becomes unusable. So with device mapper, you can set like dm— base size equal like 10 gigabytes, for example.

That's the default.

It won't get any larger than that. So I had to figure out how to do this, and some suggestions on the mailing list were like, build it dynamically. So I had to figure out how to build it dynamically, which is a pain in the ass. But luckily I've had enough heads and did some S-tracing and I could figure it out. But then I wrote an article on it. So if anybody's interested in trying out Docker and building it dynamically, you can. And essentially what I had to do was, you know, I installed Golang, the libdev, mappers, a few other libraries. And then what I had to do—

I hate that, it's really annoying. What I have to do is—

the big pain of this was— I don't know why that's not allowing me to zoom in— but the big pain of this was I had to compile btrfs from source. So the filesystem write driver from source. So I needed these libraries. It took me a long time to figure out this because the documentation wasn't really too up to date. At least I remember I had some trouble with it, but eventually got it to work. And because Docker kept building for that, so that was a big thing. And I had to, I think I, you know, one time I had to modify the source to actually make sure that this value was set because in this one upstream one it was not set, so that didn't work. So you needed, as I say, you needed, or it's not set in v3.19rc2 version. If it doesn't exist, you need this other one that has it. But essentially building it after you have all the dependencies is real simple. You get clone Docker right here. You see the directory and then you just do autogopath=1 ./hack make.sh dynamic binary. It actually builds the dynamic binary and all you have to do is copy it over top of Docker into /usr/bin/docker, /var/lib/docker, /etc/docker/init, and /usr/bin/docker init. And at that point it works. If you do docker info, you can actually see it says udev sync support is true. In this case, it's actually using the udev libraries directly. And then it doesn't have that race condition because it can call its functions which actually mitigate this issue. So just, that was a big thing for me because it took a long time to figure it out, but I eventually got it working, so that's cool. Then all our tests worked and I could make pretty graphs, which I'll show you guys later. Look, some are pretty. They're not the greatest. All right, moving into the next section, Vim vicinity. So here's a few cool things you can do in Vim. Let's go ahead and do this.

So, f file. Yeah, this will be fine.

So you can do split windows. Lucian, I think you may have talked about some of them.

I can't remember. I think I talked about splitting, maybe.

Yeah. Well, this one we can do— let's see— star.txt, vim -o. And -o, you can actually open file already in split windows. So let's just open up a few of these. augustgraph.txt. SkyPod.txt and, well, that's fine. You can see by start Vim already opens them up in vertical split, right?

So that's kind of cool. If you want— let's go a little slower.

So -O just means the following arguments are all files to open?

Yeah, so these -O, capital O is vertical split for all the files that we this is after, and the lowercase O, which I'm doing right now, will do a horizontal split. So you can see it horizontally now. So you can add just multiple after. And to switch between them, you do Ctrl+W and then you use the keys here. Oops, I didn't mean to open that one. So J, K go up and down, right, and H and L go left and right. We'll demo that with the other one. But first, before we do that, I keep hitting wrong keys. Let's go ahead and try it without that. So we're going to open Vim by itself. We're going to do Vim, we're going to do vertical split, VSP, and then what we're going to do is we're going to specify a file. I'm just going to use this right here. It opens up, right? And then I'm going to do Ctrl+W and L to move to the window on the right, and then I'm going to do another VSP, and then I'm going to open a second file, Menorius graph image right here, and you can see that's there. You can keep going. You can throw a horizontal in there too. SP for horizontal. And we can do install— what do I have—

this, Tex.

That's horizontal split. And I can do another down. Of course, this gets too ugly and unwieldy. But yeah, you can divide them up like that. It's really easy. That's a cool thing. Also, so that was these two items. This other one that I've been using is reformat lines by width. So I'm working on— I'm co-authoring another paper, and this girl had— it's in LaTeX— this girl had lines that were—

I don't—

pretty much the entire paragraph was on a single line, and my editor went, "Oh, I can't read it." I had to scroll over, right? It was ridiculous. I was like, I need— and this was done for an entire paper, so I was like, I cannot read it like this. So I had to figure out a way to do this, and one way to do it or there might be multiple ways, but this is the way I do it. You set text width to 80, so let's go ahead and open a file. This could be good for modifying any type of text document. Do I have anything that I would consider? Yeah, see that article I wrote? So, speaking of which, so I'm working on an article for 2600 Magazine right now, which it should be, it got accepted, it should be coming out anytime soon, because in April is when they do the new season. So, but if you're interested, check it out, it's an article on Eyelet. Here's the draft I didn't— I wrote in Vim. So, you can see these lines are longer than 80 characters, so I'm going to do setText width equals 80. And then what I want to do, I want to use V to highlight the lines I want to change, and I use G— whoops.

I'm sorry, not PG, VQ. That's what I was looking at.

So VQ. So I press, I would do V to highlight my lines and then press Q. GQ. I'm thinking, gosh, I'm feeling, um, and you can see it formatted all. If I want to do the whole file now to 80, let's accept.

Let's just do this.

GQ, GQ, and you can see that now I'll format to 80 lines. I can take it back and now you can see that they're longer than 80 lines.

Something like this one right here.

So that's kind of nice. Let me just do one more example. If we set the line to 20, very, very small. I want to highlight this paragraph. Do PAP for paragraph and GQ, and you can see that it formatted it 20 characters each. So this could be helpful in code too, possibly, if you don't have to worry about whitespace and, you know, I mean, like C, for example.

So that's kind of nice.

It's really nice for HTML.

Oh, yeah.

I guess, yeah. Because I write— like, I have a blog that I just write in HTML.

And it's really useful for that. Cool.

Yeah, I can see each one being useful.

That's a good idea.

Well, does anybody else have anything to present, want to talk about or present on? No worries if you don't.

Just curious.

I found a nice tool. What VPN clients does everyone use on their Linux machines for IllinoisNet?

OK.

I found out— so I used to use the Avast client. And then they got rid of that. So then I used the AnyConnect Cisco one for a while, but it was buggy sometimes. I just discovered that you can use OpenVPN and OpenConnect, and they're available in the Arch repos, so it's built into everything else, and it's way, way easier and more reliable.

You know what, Rob, you should do a— you should add that to our documentation. Yeah, yeah. That might be interesting to some students.

It's like a 5-line script just to create a tunnel. And I like it infinitely better than any connection I've ever connected. So I'll add that to—

yeah. I tried using the Open Connect.

That's what I use, yeah.

OK. Yeah, yeah.

Open Connect and OpenVPN. And do you have trouble with it?

It's kind of flaky, but I've never really had too much luck with VPN. The PPT one works pretty well. Back when I was poor, I still used it. I'd love to try your scripts. Yeah, yeah.

It's super easy and it's been very stable.

So, yeah.

All out of that tonight.

Hey, does anybody else want to talk about anything so I can eat my bite of sushi?

Today I learned a new bash thing, which is really low-level, really easy, but I've never had a need for it. And that's assigning default values for or variables where you can get them as an argument to the script. So I never knew that before.

Like the one—

like you say variable name equals $1 or a default number, a default value that doesn't exist.

Oh, right, right, with the curly braces?

Yeah, the notation is dollar sign curly brace. You can do $1. First starting, or one, first starting, and then colon, minus sign, and then the default value.

Yeah, just to talk, I got an example. I've used a couple of those. So here's a program, a shell script I wrote for analyzing different types of logs and comparing it to intelligence feeds. And here's an example right here. So the default is if malware host URL If the user— this is what the user specifies, it passes in by arguments. If that is not set, then it defaults to the actual Malhost default, which I already have set in the script. So default, and that is right here, that variable right there. So if that was not set, then it's going to use the user specified. Or if user specified is not set, it's going to use the default. Is that essentially what you do then? Yeah, yeah.

I was doing it not for like keywords, but just But yeah, and that was really useful for some glue code that I wrote this week.

Right, yeah, cool. That's definitely useful to know.

I started writing a replacement for ls last night, but I couldn't get it ready in time.

Oh, really?

I'll probably show it next week. Cool.

Now, do you do anything different with it?

It's just like it's easier to see file permissions and stuff.

OK. You're in the back?

Yeah. Of course. I figured. Yeah.

Hey, by the way, what was the reason you were telling me that you used— you like 10.4? What was 10? Was that right?

Oh, it's because I have a lot of PowerPC stuff.

Oh, OK. And Leopard's really slow on PowerPC stuff.

OK.

I couldn't remember if there was something specific about the tools that it gave you that you liked.

No, I've just been using it really— I've been using it since it came out.

OK.

I actually got a machine now, a 10.4.

I think it's a G4 iMac or G5.

Yeah, let me know if you have any questions about getting stuff to work. I've been using it for a while.

Cool. Anybody else learn anything new over break?

Surely you guys learned something over break.

I've been learning Lisp.

It's fun.

Oh, nice. It's a functional language, right? It's a functional programming language.

Uh, it's a multi-paradigm. It's like, uh, so you can be— it's got full-blown object-oriented, uh, it's got full-blown— that's got a lot of capacity to it.

What's that? Was it Lisp?

I'm learning Common Lisp.

Okay.

Yeah, so I'm not very good at it yet, but, uh, it's really fun.

Do you know yet how similar, uh, like, Antilisp to, like, do Lisp?

I don't.

I'm just doing—

That's cool, though.

Yeah.

It's a lot different than anything I've done before. So you think a lot of different things.

I have a book on Lisp.

And I was looking at it. And I was like, whoa. Some funky stuff. It's pretty cool, though.

That's one thing I wish Stallman had talked about.

I know he's a huge Lisp fan.

Oh, is he?

Yeah, he's got a page on his website that talks about it.

I think that would be a good question to ask him.

Well, back to that, I say that he's not talking about technology, at least the way we're talking about it. I think I wish I now asked more questions about his personal interests. It seems he, like, he was interested in science fiction, and he's really into politics now. So I'd be curious to, you know, pick his brain on some of that stuff. Didn't come to me at the time, but I think that would have been— he would have been more approachable for that kind of—

those kind of things.

I already started using Blue Orders actually.

Oh really?

Yeah, so, and I might have actually— so I issued a ticket a couple of days ago. It might actually get to you eventually, but I think it has a number of file descriptor limit set to 32,000 something, and I actually asked it to like increase, if it's possible to increase the number.

Okay.

Yeah, like, and I— the response, the initial response that I got so far is like, no, you can't.

Do you know who said it?

Uh, I think it was, um, Brett, Greg, or Mark, Matt, maybe.

I don't know. I actually went through like multiple people and Yeah, that won't actually go to me because I'm just security, but, um, they may talk about— we have a ticket meeting tomorrow, we'll review all the tickets. Yeah, they may come up there and I'll be funny if I saw it.

What are you using Blue Waters for?

So, uh, I'm doing a research from one of the professors in the computer science department. Basically what we're doing is, um, you know how like the supercomputer jobs are all like batch and like they're all kind of scientific workloads? But the research that we're doing is like, if you're trying to kind of adopt like a distributed key-value store, which is like interactive, so it has to like respond in a matter of like milliseconds. So in reality right now, it's kind of feasible to use like a supercomputer for these cases. But our assumption is that maybe companies like Google or Facebook, they're like just too rich. So because they already own multiple data centers, if they build like a new data center. And if it's cost-effective, if it's enough cost-effective, they might actually choose to build like a supercomputer instead of like a regular commodity, like just connecting multiple commodity servers.

So that might be actually suitable for some other use case, like interactive, the key-value tables.

But we're kind of just testing the blue waters to evaluate the possibility.

Cool. How are you liking your experience with it?

Actually, so first of all, I had to just compromise to use the CCM mode instead of sending the job using the app run.

Mm-hmm. Yeah.

So I'm kind of having a hard time if it's even possible to port my binary to whatever that's compatible with. Not CCM.

Like, what's the— what kind of—

the distro the CCM is running, is it supposed to be—

I don't know, I actually don't do much on the server. We just kind of monitor the traffic to it and stuff and log in.

Yeah, so like the CCM is like, like, uh, the actual Linux, but the most of the workloads that run on Pool of Orders is like, it's like a specialized compiled version very, like, very specialized for, like, the supercomputers. So the Linux environment in the supercomputer doesn't look anything close to, like, a regular Linux. It doesn't have any kind of binary that I would normally use. So, like, so I just, like, went with the easy way of—

like, the file system is completely different?

Yeah, even the file system is like It's called Lobster file system. It's like a parallel file system where you have this bunch of thousands of storage nodes, which take like a thousandth of my file. So whenever I ask for this portion of the file, it actually has to a bunch of different servers, which actually stores the data. And then gives back to me whenever all of those storage servers are acknowledged. The request. So the advantage of that is that you have like a huge I/O throughput, but the disadvantage is like obviously like the latency is very— it's like kind of unpredictable. There, there is like some, some sort of like spike in latency.

If anybody's interested in parallel computing, um, like every month or two months, NTSA hosts like a workshop. And they have instructors that will actually give you access to Blue Waters, a couple nodes. And then you can actually— they'll show you how to use like OpenMPI, Message Passing Interface Library, and OpenACC and instruct you how to do basic HPC concepts and how to distribute your program over multiple nodes. So that's cool. And it's free. A lot of it's free. And they'll provide you lunch too. So you get a benefit of that as well. Also, there's a project called XSEDE. Extreme Science Environment for Discovery and something, but I'm like funded for that kind of. And they have— you can actually just go on, I believe go on their website and register and you can get access as a student to nodes and actually do your own stuff. So check this out. Extreme Science Engineering Discovery Environment. And check out the services. And you can actually get an allocation on it if you're a student, I believe, to run programs if you want to get hands-on experience with a supercomputer.

All right.

Moving into Smalltalk here. So, anybody here use gnuplot to do any graphing?

You have? You done?

I had a hard time working with gnuplot. I finally just moved on to like the Matplotlib, the Python. Okay, I think I still need to know how to use like a new— I don't, I honestly don't know like why people like use it over like Python or anything like that.

But well, if you— I mean, I can't, I'm not like an expert on it or anything, not even close. But I'm working on this paper, how to do all the evaluation of the software, and I ended up— good, I ended up being my best friend for a couple weeks. So I'll kind of go through my experience with that. It's not— for basic stuff, it's really nice. So we'll kind of talk about that.

Let's go ahead and run it now.

So gnuplot is, you know, GNU version of a plotting program. It's a really nice one. You can make really nice graphs in it, but you have to know quite a bit about it. There's a learning curve for sure. And there's a free book online though if you're interested. gnuplot. Yeah, this right here, I believe this is— I thought this was free.

Maybe it's not.

Yeah, maybe it's not. But anyway, that is one you can get. Sorry, I'm losing my train of thought. Okay, so let's go to my IOLIT testing directory. Talking about my dataset. So we're going to go to the startup test. So this is our test container startup times. And I have some files here. So essentially there were some shell scripts I wrote to write values to a file. And let's take a look. This is an Amazon EC2. So we can look at this and you can probably tell this is the load average from uptime, uptime, right? So I just did uptime in a for loop every time we're in a container. And the first column, of course, at that time is the 5-second, the 15-second, or the 1-minute, the 15-minute, or—

wow—

1-minute, 5-minute, 15-minute average. So I want to graph one of them or all of them. So I wrote a gnuplot script. So in gnuplot, you have configuration files. You just tell it, hey, what do you want to do? What is the output? X and Y label. And where the data is. In this case, you most of the time now see— really, you can see graphing this only takes 5 lines. So here I'm going to graph it to PostScript. I want to use color. The output is going to be to this container startup EPS file. And I want to give it the y-axis to label Docker container startup time, x-axis the number of independent trials. And then this is where the important part is. The plotline gives the data. So we're going to give it— we're going to put in that catch file that we just looked at. We're going to say using the first field, the first column, and points is the type of graph we want to do.

It's just little points.

And we want to give a title for that line. So how we're going to run this, we just do gnuplot and then the name of the file, plot.cfg. Just runs it.

It generates—

got the output. Now the file that we generated was EPS. Take a look back in the plot script. And it's that container startup EPS. Let me look at the screen.

Excuse me. And I have a graph.

So, and you can see that from the graph, the startup time for 0 through 100 containers was all less than 1 second and less than 0.5 seconds. So the startup time using this Docker was very fast. So we just did a startup container, ran the command uptime 100 times, and then figured out the time for each one of those. So it was kind of useful. We were trying to prove a claim that's often made on the internet that we haven't really seen substantiation for. For example, CoreOS's website where it says container start in milliseconds. So we didn't doubt it. We figured it was true, but we had to verify it, and then we did we did here.

So that's an example.

You can also modify it to do more things. So let's just change a few things. Let's change PostScript to PNG. Right?

We want to get a PNG file now.

And we change this.

Oops.

PNG. And do that. And we run the same plot program. Whoops. Unrecognized terminal option. Oh yeah.

PNG.

The PNG one does not have the color.—

argument here. Why am I hitting— okay.

So now we do it. And now it generates the PNG, .png. And now we have the same version of the graph in a PNG file. You can do other formats. So let's do one more example. We're going to graph the load average for 5 minutes.

So—

You just change 2 to that and then plot it, and we'll actually have the 5-minute average. Whoops.

Open container.png.

And this one is the 5-minute average, right? So we can do multiple. We'll just go to a different one to show you that, though. We can take a look here.

Let's go back. Let's go to— ooh, storage types. Do I need a graph here? Nope, I did not. Okay, go back.

Alright, now we're gonna go to— what's a good one? Let's do concurrent. This test, take a look at the plot file. And we're actually graphing 3 separate data sets or data points. All from the same file, concurrent2.txt. So the only difference really from the last one is that we added 2 more plot lines. And you don't do plot per line. It's one plot here and then 3 lines, and you have to break—

line break—

or use the backspace bar to tell it to properly—

it's not going to be on different ones. This will fail.

So here we actually graph load minute for— our load average for 1 minute, 5 minutes, and 15 minutes. Real simple. So we'll go ahead and quit out. We're going to run gnuplot and use that. Now it's generated, and we can take a look at a file I generated and that was containerdensity.eps for PostScript. And you can see there is the graph for that. So fairly simple so far if you're just working with— it's pretty much all we're talking about is working with a few columns. You don't really need to do any math on the columns like median or sum or whatever. It's fairly simple. You just give it the file, tell it where the data is in the column, and it graphs each one.

How do you specify how the data was saved, like tab separated?

Oh, good question.

So, oops.

Where can we see the data container density?

Oh, I'm sorry.

Current?

Yeah, sorry. I'll show you.

You can change it.

So by default, it's whitespace separated. So you got 5 columns here. You can do any one of those. It probably failed this because of the slash in there.

But you can have it handle that if you need to.

So yeah, so there's an option in gnuplot to tell you the data, what the separator is. So you can use different ones. And I'll show that next. Well, in a second. So let's go back here. Let's mess with it a little bit more. So lines, right? So this time we're using lines instead of points. Let's try, well—

Does line mean the first set of data, like the first column?

First field.

First field. So yeah, one, the field, the start of the, yeah, the first field, the second field, the third field. So white value, white space, second field, white space, third field. Yep. And then what we're going to do right here is we're going to change this to points and do the same thing.

And oh, she kept it open. And is it the same thing? I should have, uh, I don't know why that one— points line, let's try a different one. Maybe that's the—

no, container— oh, I'm sorry, that's the wrong file.

That's why. Container density 2.

There's our points, right? So that's lines or points instead of lines. So let's keep it open, let's play with a few more. So we're gonna do—

let's do lines_points, I think.

Yes. It may be the same thing as points, actually.

One second.

I forget them all. There's a way to view, um, uh, oh yeah, here's another one. We want to do impulses.

It's a fun one. Ah, damn it.

Oh, I know what I did. I'm sorry. With PostScript, you have to reconvert it every time.

That's why.

Though that was the last one that it worked.

Yeah.

So there is impulses. There's a few different other ones. I forget the command.

Is it list?

There's a way to— so by the way, gnuplot's interactive. So if you just want to do it, you can do it all in here. There is a way to list the type.

Line styles, I think that's it. Line styles. Line styles. List line styles. Oh, I'll do plot. A new plot list line styles. Wow.

Yeah, there is an option. I've done it. I was looking through it.

That's not it. Um, no, that's not it.

Set line list type, maybe. I don't know. I actually don't even know the length. Yeah, so anyway, we'll worry about that. But you can do it. I don't use it enough to know it off the top of my head. So let's go to another graph now though. We got a more advanced one I think here somewhere.

So let's go to New and—

Yeah, so if you—

oh, I think I got this.

So one thing is you can see how it's a pain in the ass. You have to open the Plotter config file up every time, change a few parameters, run GNU Plot again. So I just make a shell script out of it and pass it options, right? So here is me doing that. So here are the actual— all of them available. Boxes, steps, impulses, line points, points and dots. And let's say usage plot. And you can just pass it a here doc with each one.

So, give me an example. So plot.sh.

So we're gonna say 1. We're gonna do points.

And we wanna do PNG.

Oh, you can't do color with that.

We'll do— oh yeah, let's use the—

I don't wanna edit that.

That needs to be edited anyway.

So what we're going to do is we're going to— and here's Brian just talking about the data file separator. So by default, there's whitespace. Here it's going to be using commas. So now I have the CSV file. So if you want to graph that, you can— I'll just show you perf100, 1 is the rounds, and collect D. And if you want to go to load, for example, and these are all comma-separated values now. We're going to graph those. And you can see that I'm viewing 3 files from there from memory: 100 users, 200, and 500.

And we want to change—

actually, we don't even need to specify that one, do we? So let's go back to the script—

I'm sorry, file.

And what we're going to do here is we're just going to give it a round essentially, I think. And then I'll do the default.

So 1. Oops. Plug in the stage.

1. Feeling color.

Where you at? Remove you. Okay. Now do it. Okay.

Now let's do this again. Again, sorry. I haven't looked at this stuff in a while. Let's go ahead and remove this to test.

test.png. And we're working with this file now.

So, plot it.

Test.png.

There's the file. Memory consumption. Or no, this is system load, process cutoff point, right? So now let's edit it again. Or we can actually specify something different. You can see your options. So let's just generate the same thing. We're gonna use dots this time.

Dots. Oh. Oh yeah. I should have test that condition.

Now you can see the dots. They're really hard to see there. Let's do another one. Let's do one steps.

Steps.

And then we can do—

sir, do you need something? I have a meeting. I have this room from 9:00 PM. And the next 10 minutes, I can still.

Oh, OK.

It's OK. We are going to break. It's done, so until people show up, you can use it. Okay, thank you. I gotta hurry then. Blame that on Shane.

He's the one that books the room. Anyway, there's a few more. So we'll go ahead and I'll show a few other ones that I've built. I'll show that I've generated with different types. And then finally I'll show a shell function. So here's one where you do a multigraph, or you do two separate ones, you just have two different plot sections in the graph. Here's one, this one's using the impulses, that kind of looks cool I think. And then this one's using the lines, this one is using the lines points, this one is using just the points, and this one's using the steps. So that's kind of nice, I did graphs for each one. And then finally, we'll move into—

Does this support LaTeX for the legends and everything?

I don't know.

Yeah, I simply don't know.

And I wrote a shell function so we can do declare f function name plot. So we wrote a shell function to actually pipe data into gnuplot from stdin. So this way you can write little scripts and do it. And give me an example here, see if I got anything.

Oops. Well, that sucks.

I'm using all that stuff. So let's just do— so for i in 1 through 100, echo i, um, grab the do, whoops, and it's going to do that, pipe that right into plot, and you got that. So you can do little things like that. You can do stuff with your system metrics. I can do uptime, I suppose, and graph that data. You want to parse it. It's going to get ugly because there's nowhere everything is, but, um, You can parse all that out. You can do multiples of that and then pipe it there. I guess I can do—

I don't know what else is good. Oh, PID maybe?

That might break text. Yeah, so of course you can grep SP and then the line that said PID.

I think that'll do it.

I must have, yeah, there we go. For this process ID. So you can do stuff like that, whatever you want. I just have this so I can make it easy on myself. That concludes the gnuplot talk, though. That's cool to mess with. Highly recommend checking that out. Now we're getting into our big talk here on InfluxDB and Grafana. This is our new systems. I worked over spring break. So first off, I'll GraphLinet is awesome for making graphs look really pretty. So how we're doing this is we're running InfluxDB. InfluxDB is a time series database, right? So NoSQL. It is for storing time data very efficiently. You can reduce the number of columns and store everything in rows, and you can aggregate things and break the rows to get a performance speedup. So that's kind of how that works. So this is available. I can give you the password if anybody wants to play with it. You do have to proxy through our— you follow the documentation of logging through a proxy server to actually get access to this particular interface.

We are not running HTTPS.

Okay, so I'm gonna go ahead and log in. And it's actually talking to the API on port 8080. So it runs a few different ports. 8083 is by default the web interface. 8086 is the API, but we— I moved it to 8080 because from publicly accessing the stuff in the ACM network, I'm actually not— there's filters in the firewall somewhere. I don't know if it's sites or whatever, but I had changed to 8080 and it allows me through.

So connect. Whoops.

And we have one database here now. And what we can do is we can query the data in the database. So the data that we actually store in here is from Collectd. Collectd is a really cool— we talked about it before, but it is a system metrics or stats tool that's written in C. It's good for high performance. It's just a great way of doing system stats.. And all it does, it just gives you the data. You have to figure out how to graph it yourself. So you install the package. You can set the configuration files where to put it. And then you have plugins, output plugins, to say what to do with them. In the GNU plot data, you saw the CSV files. Those were syscollectd outputting to CSV as an output module. Also, there are other things it can do though. And one of the things it can do is it can store data in InfluxDB. And we are using the InfluxDB output module. So just to quickly break off here, let me show you. So log has Puppet up front. We already talked about this. So you're just going to see quickly how all our systems get a hold of this file. /vm/modules/puppetd/files, or cert—

yeah, log.

So every system that we put up on our VM infrastructure sends their Collectd data to collectd.gnu.org and influxdb.gnu.org. So we're actually running two graphing systems. We're running Graphite, which is running on the Collectd system, and we're running Grafana, which is running on the InfluxDB system. So they're all getting the same copies of data. And it's sending data for syslog, battery, CPU, disk, Ethernet, memory, etc.

And then the actual server— is this the server one?

Yeah, we'll actually write that data and put the server right into a different database. In this particular one, it writes into the Graphite database. Graphite uses Whisper. So we'll talk about that a little later. That's just another way to store it. We're running InfluxDB in this case using the container. And we can actually query our data. So let's do that. Select star from— I'll pick a host, docker memory memoryused. Limit 5. We get 5 results back and then we have a graph of memory usage, right? So really nice. Graph CPU-0 CPU-user limit 100, right? And then we have a graph of 100 values. So you can do that. It makes it really nice. So this is cool. It's useful and you can do all this stuff for all these different values. However, It's not the prettiest. So this is where we turn to Grafana. And we're actually using the new alpha version of Grafana. And let's go ahead and check it out. Grafana, like a new Log.org, you can actually go to this. You guys will have to request an account from me though. And I'll show you what we're going to do first. So first, I built out dashboards. So these are the dashboards I built out. And doesn't that shit— doesn't that look nice? That is sweet looking. So this is system load for all our systems running on our ZenProject system. So you can go, you can put your mouse over top and it tells you the load for each one. And you can zoom in on one. So for Hammy, click on that and it just says load for Hammy. And for each particular one. And this is for processes running for every VM that we have. So as soon as you fire up a new VM, all the plugins are already set up by Puppet and all that. So all this data is already in front of here.

You can begin querying it immediately.

Then we do memory used here.. And we got swap usage at this point. And then I got this one's not—

I'm gonna redo that one.

Let's take a look at what it looks like inside.

So we can go to— which is an interesting one? Let's just do this one. Edit this. Edit.

So you can see here we're matching anything. This is a regular— used to be the regular expression. Matches anything that matches load and load is a value for all the systems we're collecting. And then we say we're going to match, we're going to do like a WHERE, where ds_name equals short-term. We'll just use the short-term load. And then, which is essentially, if we go back to the previous query, ds_name is writing. Or that wasn't the right—

no, I want load.

Select value, load, load, limit 1.

Be fine.

You can see that DS name has different values: long-term, short-term, and midterm.

You can't see all those, but if we do 20, you should be able to see enough.

So short-term, midterm, and we're just saying where the value is— where DS name equals short-term.

Limit 20.

Yes, now we just got the short-term load averages. So you have that, and then you have all the x-axis and stuff, and then you have the display styles, and this is easy stuff. So I can change this really easily to— right there, it's no longer a bar graph, right, real simple. Now it is again.

Lines, points.

It's ugly, it's not really usable. But you got all these different options you can play with. And then you can do like stacking percentages. I don't know how to use all this stuff yet, but they do adjust. So that's more of a stacked view. That's kind of nice.

Percentages.

I don't see if there's any difference yet. But so this is kind of how it works. We can kind of play with a new one real quick. So let's go to overview, or Hammy, we want Hammy. So you can just create these different dashboards. So here's the system load average for Hammy. Just the host is the mid— or the short-term, mid-term, and long-term graph.

Let's add a new one.

So add row. Click this. Add panel.

Graph.

And then we click on this.

We hit edit.

And then we wanna do something. So I don't know, CPU. So we can start our expression here. And the cool thing about this, it auto-finds and completes it for you. So for different hosts. So I just want to grab— let's do Docker. Let's do everything from Docker 1, all the CPU stuff.

I want it to look pretty.

But okay, that's everything right there.

A lot of lines.

At that point you can see it all in there. It's not really useful at this point because it's so zoomed in or zoomed out and you won't see the lines unless you actually select a like, then you can see the details. But there's different ways to handle that, and I'm not really good enough with it yet to know how to do all that.

Whoops.

Yeah, so you can see how it's really nice. And then we can do— so let's just do nice. You click off, and then it graphs that. And, okay, so that's essentially—

Graphana.

It's really nice to do all these things. And we'll have one for, uh, we'll have one, I'll build out these dashboards more and more. You're more than welcome to try it out if you'd like. Now I wrote some shell functions so I can do it from the command line. So with the API, you can just use curl, right? So HTTP, I have this in the documentation. influxdb.gru.log or even Graphana, they're just CNAMEs, the same host.

8080, DB is collecting.

If you also want to store your own stats up there, We can create a new database for you and you can put your own stuff up there. And we say we're going to authenticate user root password, which is a variable so it wouldn't be on video. And the actual query we want to use. So here we're selecting star from docker1, get the CPU, I/O stuff. And we're just going to get that value out. And this data is in JSON. Oh no. Why is that not right?

I don't want to echo that value because it's going to show the password on the screen. DB—

semicolon doesn't matter, right? I noticed you put that in.

No, it only— which is weird. It only matters when you have a WHERE clause. But other than that, no, it doesn't.

Good eye, though.

Yeah, what is that? Oh, I know why. Because we killed off that shell earlier. Exact. That was the one I had stored and I didn't have it exported to my environments. Oh well, I don't care. Everybody, the password is, you can all use it, gnulog-influxdb. So if you all want to try it out, there you go. I can change it later too. So there you go. There's all the values. So I printed out 10 of them.

Ugly.

So I'm using jq, which is a way to do parse JSON from the command line. I wrote a little another function because jq is really ugly to mess with, to actually just do— well, it's just a shell function, jq. I guess I can declare that and show you that. jfield, and it just prints— honestly, I don't even know how to read this. Here's an array, the 0th part, and then everything underneath that in the subarray And then whatever value I pass into it.

So these are going to be the fields.

So jField, in this case, I think it's on 2.

Nope, it's not on 2.

It's on 1. Oh, that's timestamp.

3, 4. There it is.

It's 4. And I do have a bug in there I need to change.

But I'll get it fixed.

But you can see that— there we go.

I can plot it now.

Hold on. What are these times? Damn it. Damn it. What?

Surely you would have—

nope.

I thought I fixed this, but apparently I didn't. Okay, anyways. Anyway, I thought I had fixed this. But anyways, you can see that— so something's getting— when I actually am making the curl request, some value's getting changed. Screwing the whole thing up. But you can use this to pipe into gnuplot.

So I should try to get over here. Yeah, it's like, damn it, I thought I solved this.

It was working earlier. I wonder if I did something else. Anyways, so this is how I was doing this and then I was putting it into gnuplot. Let's change it to a different one. Let's do docker1 memory, memory used.

Let's see if we can get it to go. And let's set it to 5. Okay. And jfield. That was the 2 I thought. Ah, there it is. We'll get lucky maybe.

I'm waiting to change back to those numbers. Well, this is a disappointment, guys. I apologize. I need to work on that function apparently some more. But what we can do is just take a look.

Oh, let's just cat these into that thing. You can see the same thing, right? Um, plot. Uh, yeah, there it is right here. This graph— all the other ones failed, so they're empty. Okay, got the point. Got it.

So you can see here's the graph of that. So I would do my own little graph by piping it into it for fun. I'll have to figure out what I did wrong in that. Any questions about this?

I don't know anything about databases, but the InfluxDB is one type of database and when you query it with that script, you can converts it to a JSON object and then you extract data from that object?

Yeah, so what happens is Grafana, so the collectee data gets stored in InfluxDB. Whenever you go to Grafana and say graph something, it then connects to the InfluxDB database by the API and pulls out the data that you requested and puts it in the graph. And whenever I'm here in those examples, I was querying the InfluxDB database directly. So it's just, it has a, you know, a web-based API where you can just use curl or Python and import URL into or whatever and make your queries and get the results back. And then you can just handle them. So, did I answer it?

Yeah, yeah, yeah. Okay.

The other thing we have using the same data is Graphite. So it's another tool. You can all access this right now, by the way. It's not password protected or anything. But if you wanna check out our systems, You can go to one, click on the system and drill down. This is kind of how I go back and forth between the two because this one actually shows the folder view of the data from CollectD. So I'll go into one of these and then I'll try to figure out how to use the query in Grafana, for example, if I don't know what's available. And this one we're going to do load, for example.

And oops.

Long-term, mid-term, and short-term. I can do it right there. You can just click on this. That one's really nice. So this is cool if you were having projects and you need data graphs and stuff for school presentations or whatever you're working on.

So that's kind of nice.

So that's Graphite, and that stores it in a Whisper database. It has its own thing. So we're doing both of these at this point in time.

And we'll probably continue to do that because they're both useful in different ways.

I can't get over how good-looking this is.

How much overhead does collectd have?

Very minimal.

It is very minimal. It is, like I said, written in C. Now, you specify your interval. I think by default it's 10 seconds. So every 10 seconds it pulls all the stuff on the system and sends it. But it's written in a way that it's very efficient about the polling. So every time it pulls, you don't see a spike real quick. Compared to like some of the tools like maybe Munion where it's 5-minute intervals and it runs all these shell scripts which take a long time because they're slow. You know, you can see this spike sometimes on that. But yeah, Collectd is made to be very efficient. All these machines are about to run Collectd and hosts with 512 megabytes of RAM and 1 CPU without any issues. So it's pretty minimal.

Is it—

did you install it with like the Brewpacks?

I don't know if it's on OS X, but these are all on our Linux hosts. It's in Ubuntu and Debian, so apt-get install collectd will install it. And by default, it will write data out for you. Like, you pretty much just need to see where you want to put it. And like, if you just want to have the CSV stuff so it's easy with gnuplot, you can do that. So it's pretty easy to use. Literally, it's like editing 3 lines and you have data. You say, "What was my hostname?" or "What do you want to show?" Uncomment an output plugin, or if you want to send it across the network, uncomment the network portion and put the server you want to send the data to. So it's really nice. Go back to the thing.

Is that it?

I believe at that point we did. I can't think of anything else. Yeah. Graph on it is very sweet. I highly recommend checking that out. If you guys want to play with this, let me know. And this one I'm doing a stack graph for number of processes running per host. And Hammy would typically have the most because it's the VM server.

If I click on that, it actually drills down for the particular— just for the host. And so it's kind of nice.

But I don't know, I really like that. Well, that's all. That's Graphing 101 and how we're using it in the log and stuff. So I hope we'll get more data and do some more cool queries.

Does everyone in here use ImageMagick a lot?

I've never really used it.

It's one of the most useful tools for me day to day because I do a lot of data analysis. So maybe I can do a small talk on that sometime.

That'd be cool to see. Do you use the command line version of the tool? Yeah, I'd like to see that.

Yeah, it's really good.

Yeah, there's a lot of—

instead of doing open, you use display and it's way faster than using menu and stuff like that. And it's got a lot of things.

Yeah, I'd definitely like to see that work.

What's the name of the tool?

What's that?

ImageMagick.

You can get it from Purdue. And it's great for converting. Converting images, resizing, all kinds of stuff.

OS X has something, I think, SOX or something.

Or no. I don't know. OS X has some image thing.

Image converter? Yeah.

I-C-O-N-V?

That might be it. There's something that's like—

it's probably not as advanced as ImageMagick, but it's— I don't know. I see a lot of people talking about it. Thanks. Yeah, thank you.

How's this coming out, guys?

Yeah, yeah.

That concludes one proceeding.

Yeah, yeah, it's very nice.