So I’ve been really busy the last few months. REALLY BUSY.
But, I’ve also had the chance to learn a lot of new things. One of the more interesting things I’ve learned lately (as in, the past week) is that Build Forge Agent communication can be routed through a reverse proxy. This isn’t officially support by IBM, mind you, but it does seem like it can be done. And quite easily at that.
I had used Wireshark to look at the Build Forge Agent protocol a couple of times just so I could understand how it worked. This helped me tremendously when it came time to request firewall rules in different environments. I never saw anything that looked like the Console needed to really know what Agent it was talking to. Instead, the traffic appeared to be very simple and to the point: “Hey Agent! I need you to do X, Y, and then Z. Oh, and use this account while you’re at it!”
So we ran into a situation where I was only allowed one hole through a particular firewall. To clarify, this one whole meant one source server and one target server. If you know anything about Build Forge, you know that means you can only get to one Build Forge Agent on the other side of the firewall. This was a major bummer for us because we already had three agents installed on the other side with plans for many, many more. But the security gods had spoken and one hole it was.
All hope looked lost until we started tossing around the idea of a proxy server. I knew we couldn’t use a normal forward proxy since we’re basically talking about routing telnet traffic here and that just isn’t forward proxy routable (nothing is built into the protocol to allow for that like HTTP). So, reverse proxies came to mind. If we could open multiple ports to this one server and then have multiple reverse proxy listeners that would route the traffic on the other side of the firewall this just might work.
The first step was to find a simple reverse proxy. We couldn’t use an HTTP proxy because, again, this isn’t HTTP traffic and HTTP proxies alter the traffic. We needed the raw TCP packets to be passed through. I knew I could write something like this myself in Java quite easily but I wanted to see if someone else had done any work before
wasting investing a lot of time on this. After trying a few things, a Google search on “java simple tcp proxy” gave me what I wanted. And what exactly did I want? The Grinder TCP Proxy.
This thing is nice. Very easy to use and quick to test out. I set up the proxy for simple TCP routing and voilá, it worked! The Build Forge Console was able to communicate with my personal PC running The Grinder which routed the traffic on to a real Build Forge Agent. I even ran several concurrent jobs through the reverse proxy connection and it didn’t miss a beat. Well, this was very promising. But what about SSL support? We had to use SSL for our implementation and that would be a deal breaker there.
While The Grinder’s TCP Proxy can do a basic SSL “man in the middle” approach (not an attack per se, but the concept is the same). I attempted to use this but it failed because it tried to connect to the Agent with SSLv2 rather than TLSv1 (SSLv3). Well, my own implementation might be able to address this. Then I wondered…hmm, could just routing the TCP packets work? I know it shouldn’t because the Common Name on the SSL certificate wouldn’t match a reverse DNS lookup on the origin IP, but I wasn’t completely sure that Build Forge Agents actually did their due diligence to confirm the origin IP was associated with the Common Name on the certificate. It was worth a shot.
OK, I’ll admit, I’m lying. I messed up initially and forgot to pass the TCPProxy class the “-ssl” option at first and everything worked. Only afterward did I realize that SSL traffic worked through The Grinder in pure TCP routing mode! It was once I used “-ssl” (like I meant to originally) that I found the SSLv2 issue mentioned above.
So, pure TCP routing not only worked for regular connections, it also worked for SSL connections too. Great! On a slightly not so great note, IBM Rational Build Forge Agents (and the console too) don’t actually reverse ping the origin IP of SSL communication to verify that the reverse DNS name retrieved matches the Common Name on the certificate. This doesn’t compromise the encryption since a “man in the middle” can’t do much with encrypted traffic; however, I’m still a bit shocked to say the least. Perhaps IBM’s approach here is to just use SSL for encryption and not truly for client/server verification. Still, I find that a bit odd, especially considering one of the Build Forge Agent paramters is ssl_verify_client. Even with that set to “true”, raw TCP routing through a reverse proxy didn’t cause an SSL alert.
So we’re now working on “productionizing” this and writing our own simple TCP router. It will listen on multiple ports and forwards them to multiple agents on the other side of the firewall. We might also look at putting in a true SSL “man in the middle” which decrypts the traffic (like a normal agent would) and then begins a new SSL channel to the real Agent (like a normal console would). This would prevent any issues later if IBM decides to start performing reverse DNS lookups for Common Name matching to origin IP. It never hurts to plan ahead.
Alright, I’m headed to bed now.