COBOL

So I haven’t really posted in the last month, mainly because I haven’t done anything interesting. I’ve been writing some Perl code using Catalyst and Mason. But that’s not terribly exciting.

I am however, taking a Mainframes course at RIT this quarter. It promises to be relatively exciting. In it, we get to learn the intricacies of z/OS, and how to manage one of those beasts. That’ll be fun. Surprisingly, there’s a programming project towards the end of it (it’s surprising because it’s a Networking and Sysadmin course, we don’t program much). And since I’m learning how to administer and use mainframes, I thought to myself, “Hell, let’s do this project in COBOL.” So here I sit, trying to find a decent COBOL tutorial on the internet.

If anyone finds one, please send it to me. It’d be nice.

Posted in Life, Projects, Systems Administration at March 13th, 2010. No Comments.

Google App Engine, Twitter4J and OAuth

So I’m writing a Twitter app in Java on Google App Engine right now with my friend, Dave Bright. We’re using Google Web Toolkit for our frontend, and App Engine for hosting and database and such. Twitter4J is our Twitter library of choice.

Since we ran into a spot of trouble using OAuth with the Twitter4J library, I decided to give a bit of a brief tutorial on how to get it working on App Engine. There is actually very different than the code in my as I do some more error checking, and need to do some other stuff with my datastore.

Twitter4J OAuth Overview

  1. Set the consumer key and secret for your twitter object with setOAuthConsumer(key, secret)
  2. Get a request token with getRequestToken(callback_uri)
  3. Persist the request token somewhere, we’ll need the original later when we need to get the access token.
  4. Redirect the user to the URL returned by getAuthorizationURL()
  5. Get the persisted request token, setOAuthConsumer again.
  6. Call getOAuthAccessToken(request_token, oauth_verifier) to get an access token
  7. Persist that token with the user data, you’ll need this everytime you want to authenticate.

Some incomplete example code

Please note, this is not code I actually have in production anywhere. It’s original code just for the blog.

public String authenticate() {
TwitterFactory tf = new TwitterFactory();
Twitter twitter = tf.getInstance();

// This is actually really important, as Twitter WILL NOT send you back the oauth_verifier if
// you don't provide the callback URI, even if you provide a callback on their site
twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
RequestToken rtoken = new RequestToken(CALLBACK_URI);

// This is a special class that just relates the token to the RequestToken object in the app engine
// datastore. The token string itself is the key.
OAuthTemp temp = new OAuthTemp(rtoken.getToken(), rtoken);

// Our persistence manager is named pm
pm.makePersistent(temp);

return rtoken.getAuthorizationURL();
}

The second bit of the code gets called when the user is returned to the page. You need to get the oauth_token and the oauth_verifier GET variables and pass them to this function.


public void authenticatePartII(String oauth_token, String oauth_verifier) {
TwitterFactory tf = new TwitterFactory();
Twitter twitter = tf.getInstance();

// Get the temp object out of the datastore, and get the old RequestToken
// Yes, it MUST be the old RequestToken. Or at least, have all the same parameters
// as the old RequestToken.
Key k = KeyFactory.createKey(OAuthTemp.class.getSimpleName(), oauth_token);
OAuthTemp temp = pm.getObjectById(OAuthTemp.class, k);
RequestToken rtoken = temp.getrtoken();

// You MUST provide the original RequestToken and the oauth_verifier passed to you by Twitter
twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
AccessToken ac = twitter.getOAuthAccessToken(rtoken, oauth_verifier);

// You really don't need the RequestToken information after you got your AccessToken, so
// delete it.
twitter.setOAuthAccessToken(ac);
pm.deletePersistent(temp);

// However you store user data, you ought to associate the AcessToken object you
// just got with your twitter user, and pop it in the datastore. I'll leave that bit up to you.

That’s all there really is to it. The Twitter4J library isn’t the best documented library, and just looking at the Javadocs and the Desktop-centric code examples isn’t really that helpful. I hope this clears up some confusion on how to get Twitter4J working with OAuth.

Posted in Projects at March 13th, 2010. No Comments.

Useless programs are the best programs

A couple of weekends ago, a couple of friends and I were joking about making a useless Debian utility called apt-echo. So that weekend, I sat down and wrote it in Perl. I also went and made a Debian package, complete with relevant documentation. I never really put it anywhere.

So I got bored and decided to make a really an awesome web 1.0 page for apt-echo, as well as off it for download. I even set up a github repo for it. If you want to see the site, it’s right here.

So go ahead, install it. Maybe someone will find it useful one day. It’s slightly faster than aptitude search, and it has few dependencies. Send me an e-mail if there are bugs. Or if you like it. Or if you found a way for it to be useful.

Posted in Projects at January 31st, 2010. No Comments.

My major is too easy…

This weekend, I had absolutely nothing to do. I didn’t have any homework, studying, projects or the like because my major is far too easy. So I got bored and wrote a complete Brainfuck interpreter in Perl.

The interpreter was the easiest; it took me a morning to get that working albeit without support for nested parens. After that, I decided to write a text to Brainfuck converter, which (not surprisingly) turns text into executable Brainfuck code. Maybe if I get bored again, I could write a module to output the equivalent Perl or C code given some Brainfuck code.

This was my first real project using object oriented Perl. In all honesty, I like that Perl’s object system allows for many different types of objects (objects can be scalars, arrays, or hashes) and how flexible that is. Unfortunately, I didn’t get to exploit that flexibility in my code. It’s still cool though.

What I wrote is 2 Perl modules that make up the linked list that store data generated by Brainfuck code, a Perl module that interprets and runs the code, and a Perl module that translates text to executable Brainfuck code. I know I could have used a pre-built linked list, but I thought it was more fun to write my own.

The biggest challenge was definitely writing the translation code. I could’ve done it the easy way, and just output x amount of pluses for the character code of each letter, but I decided that was the lamest way to write Brainfuck. Instead, I looped through each character in the phrase, and added some nodes in my Brainfuck code that corresponded to the tens place of each character code. Then, when outputting a character, my Brainfuck code would change the data pointer to the node that was set to the tens place of the character code of the desired character, and increment or decrement it to the proper value.

A more ideal way to do it (to generate cleaner Brainfuck, at least) is to set each of those prep nodes to the nearest multiple of 20 for each character code, and then use whichever node is closest to the desired value. Maybe if I feel like doing more with this, then I’ll add that feature (probably not).

In addition to writing all this, I also made a GitHub account. My GitHub has all the above code, so you can clone the repo if you want to see what the code looks like (it’s actually really easy to read Perl).

This whole idea came from an idea on CSH. A guy here decided to publish a “Hello World of the Week” article in our newsletter, and to combat him I wrote a Brainfuck version of Hello World. This contest grew, and other people joined (albeit humorously) with such contributions as a boot loader that printed out Hello World, Hello World in White Space, and Hello World in Microsoft Word. My contribution (also in my git repo) generated valid Brainfuck code for the phrase Hello World, and then ran it through my interpreter (aka Hello World in 425ish lines).

Posted in Life, Projects at January 18th, 2010. No Comments.

omirssi: An Omegle irssi plugin

Sometimes I really hate going home. I’m on Christmas break, and I’ve spent the majority of my time just watching movies in my basement. I got really bored, so yesterday morning while I was watching “Juno” (love that movie), I wrote an Omegle script for irssi. I just made up a google code page for download, so have at it irssi users. Maybe I’ll get around to adding it to the big list of irssi scripts sometime soon.

Posted in Uncategorized at December 25th, 2009. No Comments.

E-mail me!

Got my mail server up and running (finally). Now you can e-mail me at will@worrbase.com.

Posted in Uncategorized at December 14th, 2009. No Comments.

Perl, Apache2::Request, and uploading files

So I started writing a netboot server for floor. I decided to start writing the web interface first, and learn myself some mason, web perl, and database-y things. Also found some things out about the process that were a little undocumented/misdocumented (now a word).

The biggest issue (so far) was file uploads. So here’s a quick tutorial on how to do file uploads with perl, with mason.

In index.html (or wherever your form HTML code lives):

<form method="post" action="upload.mas" enctype="multipart/form-data">
  <input type="file" name="file_name" /><br />
  <input type="submit" value="Submit" />
</form>

In upload.mas:

<%init>
use Apache2::Upload;
# $r is an Apache2::RequestRec, not an Apache2::Request like some places say it is
my $req = Apache2::Request->new($r);
# file_name refers to the form name you had in your HTML
my $upload = $req->upload('file_name');
my $fh = $upload->fh;
open OUTFILE, ">$where_ever_you_wanna_save_the)file";
binmode $fh;
binmode OUTFILE;
print OUTFILE $line foreach my $line (<$fh>);
close OUTFILE;
close $fh;
</%init>

In your httpd.conf:

APREQ2_READLIMIT 2G # Sets the max size of your user's uploads to 2GB

I intentionally skipped over error handling code and all of the httpd.conf config for the sake of brevity (I did say brief tutorial). Since I need to allow for larger file sizes than Apache does by default (64MB), I spent quite some time looking for that Apache directive.

Hopefully this saves someone some time.

Posted in Projects, Systems Administration at December 7th, 2009. No Comments.

OpenBSD pf vs Linux iptables: A Comparison

This weekend, I decided it would be a good idea to turn my Linux router/firewall into an OpenBSD router/firewall. Clockfort recommended it, so I decided to grab backups of my iptables rules, install OpenBSD, and learn pf.

I learned iptables about a year ago, when I first built this router/firewall. It acts like those little home gateways that you get; it does NAT, DHCP, DNS, etc. I’ve also used iptables to firewall my desktop and laptop (though those rule sets were significantly simpler than the firewall).

Configuration

The first notable difference between pf and iptables: pf has a config file! It also has variables, lists and tables that you can manually populate which ease configuration. You can even include other config files in case you need to split your config for whatever reason. When you’re done modifying the config file, just call pfctl -f /etc/pf.conf and it’ll load that rule set and start filtering.

iptables doesn’t have any of that. iptables are primarily populated through the iptables command. You can use iptables-save and iptables-restore to save and load iptables rules from a file. The file is basically a bunch of iptables commands with the iptables bit omitted. Another alternative is to write a bash script that loads your iptables rules one by one. That option gives you the benefit of using variables so that it’s trivial to change IPs or similar.

I used iptables-save and restore to configure my iptables rules, and just wrote rules to that file in a similar syntax when I wanted to reconfigure parts of my firewall.

Rule Processing

The next most obvious way they differ is how they process rules. iptables has various tables, each with different chains that packets traverse, whereas pf just processes packets straight down the config file.

With pf, packets traverse the flat pf.conf file. Even if a packet matches a rule, it continues to process the packet all the way down the configuration file. Only if a rule contains the “quick” option does pf stop processing and take action before hitting the end of the rule set. If a packet makes it all the way to the end of the config file, the last action specified from a rule that matched that packet is taken.

With iptables, packets are processed by various chains in different order, depending on the source and destination in the packet. For example, normal outgoing packets are processed by the OUPUT chain on the filter table. Various rules within that chain may cause processing to hop over to a different set of rules on a user-defined chain, or might take action on a packet. When a packet matches a rule description, processing on that chain stops immediately, and the action is taken.

Dynamic Modification

pf really just owns iptables here. To dynamically update your rules with iptables, you just write new rules on the fly. If you want to do a bunch, you would write a bash script to delete some rules, and write new ones.

pf, you can change tables, variables, lists and anchors on the fly. Anchors are a really cool feature of pf. They are basically sub-rulesets that have names. So if you define an anchor somewhere in your ruleset, you can call pfctl and totally redefine the rules within that anchor. You can even write anchors to files, and load them from different files.

Packet Filtering

pf is pretty simple concerning what it can filter for. It can filter based on protocol, TCP flags, source IP, destination IP, interface, and port. There is also some slightly more advanced filtering, like antispoof, unicast reversing, and passive operating system finger printing. For the majority of situations, this kind of firewall control is fine.

iptables can do all of the same stuff. However with iptables, you can load all sorts of modules that do far more intensive filtering than pf. You can filter based on state (no, you can’t filter based on state in pf), where they are (geoip), time, statistics, ToS, and much more. There are so many target extensions for iptables, it is ridiculous. And if that isn’t enough, you can pass the packet into userspace and write a script to filter it further there.

Performance

pf is fast. I said before that you can’t filter packets in pf based on state. That doesn’t mean that pf isn’t a stateful firewall. It definitely recognizes state, as it passes packets that are part of an established connection without even processing them with pf. This means the majority of your packets skip your firewall rules entirely. This isn’t nearly as insecure as it sounds, since most iptables rulesets pass packets that are part of an established state anyway. It definitely has the distinct advantage of making it faster though.

With iptables, all of your packets pass through all of your rules. This can really slow things down, especially if you have complicated rulesets. If you use all sorts of crazy iptables modules, that will slow it down pretty heavily too. And if you pass the packet into userspace for further processing, it will slow it down even more.

Conclusion

pf and iptables are both great firewalling solutions, but cater to people of different needs. pf is ridiculously fast, but lacks some of the more avanced features of iptables. Since my router/firewall box doesn’t really need those advanced iptables features, and since it needs to be fast, I’m gonna stick with pf for now.

Posted in Networking, Systems Administration at November 14th, 2009. 1 Comment.

mars_nwe and the Great IPX Battle: Part I

So lately I’ve been trying to get an IPX test network going between my NAS and my desktop. The idea was to make my NAS an emulated NetWare server, and then make my desktop a client using the Linux IPX tools.

IPX setup was trivial.  Getting a NetWare emulator set up is almost impossible now.

First off, the ipx.h that ships with Debian stable right now is broken. Flat out broken. I have tried incessantly to get various IPX programs to compile against it, but they always throw errors. Next quarter, when I have more time due to easy classes, I might try and figure out what’s wrong. Fact of the matter, I don’t have time now.

This problem was pretty limiting, as the only mars_nwe packages I could find were source, or (horribly broken) rpms. However, I managed to find a mars_nwe Debian package here. It installed cleanly, and started up after a bit of configuration. Right after it started up, it failed to open a critical library, libdb.so.2.

With a little apt-file magic, I found the library in the Debian repos: libdb1-compat. Great news: it was compiled against the wrong version of glibc. So no dice on Debian just yet. It refuses to compile Gentoo whatsoever because the mars_nwe Makefile chain horribly confused.

The provided mars_nwe Makefile reads a configuration file for all the options to mars_nwe (they have to be compiled in; it can’t be reconfigured later). After that, it runs the current Makefile through the C preprocessor, generates a new Makefile, and then begins the compilation process. The best part of the Makefile that it generates is that the syntax is wrong. Just wrong. Make just spits out errors everywhere. So I grabbed the Makefile.o it generated and started correcting the syntax errors and cut out the bit that generates that Makefile. Thing still won’t compile. Shelving that until after my CS project is all done.

I’ll be so happy when I’m done with that project. I’ll actually be able to work on all the projects I wanna get done this year.

Posted in Networking, Systems Administration at November 6th, 2009. 2 Comments.

Google Wave

Yeah, that’s right. Just got an account. Well, by just, I mean I got one on Friday. Don’t even ask for invites, because they were gone by Saturday morning. Also, if you don’t know what Wave is, you should really go to this site and watch the video. I know it’s long, but it’s well worth it.

So Google Wave is pretty awesome. It’s ridiculously buggy, but it works well enough if you aren’t using it in a production environment. I have yet to try out any of the multimedia stuff beyond embedding YouTube videos in Wave, but I imagine it works just as well as everything else does. I’ll probably talk more about that when I’ve played with it.

What I’ve really been playing with is the Wave API, in Python of course. The jist of what I’ve been reading is that the Java API isn’t as up to par as the Python API. Also, I hate Java, and I hate writing code in Java.

The Wave API is built off of Google App Engine, and has all the same limitations as App Engine. My roommate and I kinda half started working on an NNTP gateway to Wave. Unfortunately, the limitations App Engine sets makes this a little more difficult than we’d like (no sockets or anything of the sort).

The one thing I’ve come across in the API that I don’t like, is that you *must* include the entire Python Wave module in the app directory that you upload to App Engine. Not a huge issue, but definitely a bit of an annoyance. The only real problems I see are the possibility for people to modify the version of the Wave API they upload, and that the version of the Wave API they upload is outdated.

Another issue with using App Engine as the platform for all of the Wave code and such, is that you *must* use Python 2.5. App Engine doesn’t work with anything higher than that. So even though the Wave development site says you can use Python 2.5 or above, the above part is really a lie.

The good news: Google Wave won’t forever be reliant on App Engine. Sometime in the future, they will be allowing Apps from anywhere. This is will be awesome, and I can’t wait.

Oh, and I had other productive things I was gonna do this weekend, but they all got shot to hell when I got my Wave account.

Now you can wave me at ay1244@googlewave.com

Posted in Projects at October 26th, 2009. No Comments.