Dash docset for Python 2.2.1 (i.e. Jython for Weblogic / Websphere)

I use Dash quite a bit, so I just spent a little bit of time creating a docset from Python 2.2.1 documentation. This old Python version matches the Jython implementation shipped with Oracle WebLogic ("WebLogic Scripting Tool", or WLST) and IBM WebSphere.

To install it in your Dash, just click on this link:dash-feed://https%3A%2F%2Fraw.githubusercontent.com%2Ftoyg%2Fpy221dashdocs%2Fmaster%2Ffeed.xml

The source script is in my GitHub repo, and you can manually download resulting packages on the Release page.

As tempting as it is, the idea to repackage webapp-specific documentation (e.g. for connect(), startEdit() etc) is a non-starter due to Oracle and IBM being quite trigger-happy with their copyright lawyers.

Python and cmd.exe on Windows - a world of pain.

As I mentioned a few days ago on Hacker News in a Ruby thread, CPython support for Windows is, overall, extraordinarily good for a runtime with clear Unix roots. This said, occasionally you'll eventually hit a wall and find yourself cursing Guido & Bill under your breath. Yesterday was one of those times for me.

I'm currently working on a project using Python 3.3.5 on Windows 2008 r2, building a program that will talk to Weblogic 10.3 via the Jython-powered WLST interface (it actually does more than that: it leverages Jython to also instantiate several complex Java classes, launch VBS scripts and so on and so forth). In order to correctly set up WLST/Jython, I have to launch a batch file which in turns calls several other batch files in order to set up all sorts of environment variables. These are all pure-DOS batch files doing very little except creating or reading environment variables, but they're nested two or three levels deep from the entry-point batch.

For some reason, when I launched this batch with Popen(), variables were not set correctly. In fact, it looked like "call" statements in the batch were just silently ignored. I tried using shell=True, and it made no difference whatsoever. I put it down to some weird cmd.exe behaviour; tried to switch extension from .bat to .cmd and things started to move a bit more (so much for all those posts saying there is no difference between the two) but still some stuff wouldn't work, so I eventually settled for reimplementing the whole batch chain in Python (which is terrible and will likely bite me a year down the line as the version of Weblogic changes, but beggars can't be choosers).

The most frustrating thing, however, was that opening an interactive pipe to test and do exploratory programming was just too difficult. There are a lot of examples out there talking about .send(), .communicate() and stdin=subprocess.PIPE, but nobody seems to mention what I experienced: as soon as you call communicate() on a cmd.exe launched with Popen(), all pipes are closed and there is no obvious way to reopen them. I don't think this is due to cmd.exe, because the last output I got was always ">More ?"; I think this is just CPython being too eager to clean up.

Luckily, I found a solution in WinPexpect, a fork of Pexpect that actually deals with Windows weirdness. Processes launched with winpexpect.winspawn() actually keep their stdin pipes up long enough for me to figure out enough stuff to fully re-implement the batch chain.

The result is a Python script three times as long as the original batch and likely to break the first time Oracle changes a line here or there. It will do for now, but the experience left a sour taste in my mouth, so to speak; cmd.exe is a crappy shell, but it shouldn't be that hard to open a long-running prompt-like process piping stuff to it. If I'm missing an obviously-better solution, please let me know and I'll happily blog about it, because clearly Google and DuckDuckGo need to know about it.

Encrypting and Decrypting SQLDeveloper 3 Passwords

Some Oracle products are fairly sweet, let's be honest. One of them is the revamped SQLDeveloper, which has finally caught up (mostly) with MSSQL Management Studio.

One of its best features is the ability to import and export list of connections via XML (right-click on Connections to find the relevant menu). The resulting file is very readable, and hence easily manipulable. The only opaque item is the encrypted password, but it turns out that they are not particularly hardened. This is what you have to do to be able to manipulate them for fun and profit.

1. Get Jython, or your favourite choice of JVM dialect that can work with jars. I picked Jython because 1) it's Python! and 2) Oracle ships it with most products, under oracle_common\util\jython.
2. Load ojmisc.jar and db-ca.jar. These can be found in different places depending on your SQLDeveloper version.
3. Import oracle.jdevimpl.db.adapter.DatabaseProviderHelper. That class has the two methods you need: goingOut (i.e. encrypt) and comingIn (i.e. decrypt).

So here's a complete Jython script for the Windows version of SQLDeveloper:

# set this to the path where you extracted SQLDeveloper
SQLDEV_ROOT = r'C:\sqldeveloper' 
# here's the real stuff
import sys
from os.path import join
sys.path.append(join(SQLDEV_ROOT,r'sqldeveloper\extensions\oracle.datamodeler\lib\ojmisc.jar'))
sys.path.append(join(SQLDEV_ROOT,r'sqldeveloper\modules\oracle.adf.model_11.1.1\db-ca.jar'))
from oracle.jdevimpl.db.adapter.DatabaseProviderHelper import goingOut as encrypt, comingIn as decrypt

if __name__=='__main__':
    print "Encrypted 'password': " + encrypt('password')
    print "Decrypted 'password': " + decrypt(encrypt('password'))

IntelliJ PyCharm 2.0 and Jython 2.2 don't really go together

UPDATE: Dmitry Jemerov from IntelliJ responded, explaining what the situation is. TL;DR: 2.2 is simply too old, other features might come if there is demand. I've amended the post to reflect this.

Let me preface this rant by saying that I've been happily using JetBrains PyCharm for a few months, and it's certainly one of the best Python IDEs out there. The price is ridiculously low and if you're serious about Python, buying PyCharm is one of the best investments you can make. It can be used for free for 30 days, so you really should give it a shot.

This said, if you happen to work with Jython 2.2, you'll probably want to use something else. The claim that Jython is fully supported as a runtime, while literally true, is somehow stretched is only valid for 2.5+.

Let's say you work on Windows, and you have your Jython installed under C:\jython-2.2 (yes, it's damn old, but it's still the most widely-deployed release out there -- just ask IBM and Oracle).

You create a new project in PyCharm, then go to Settings -> Python interpreter, remove the preconfigured CPython runtime, then click on Add and point to your C:\jython-2.2\jython.bat. Bang, "SDK is not valid". The list of library paths, which supposed to be automatically generated by the IDE picking up the environment configuration, is now empty.

Still, PyCharm should be smart enough to parse arbitrary .py files in specific directories, right? So let's click on Add... to point to C:\jython-2.2\Lib, then OK.
Now let's create a .py file, "from pprint import pprint", the module is recognised; "Run" the script, output is correct, life is good: Jython is indeed supported as a Python runtime.

Ok, let's do some file I/O, "import os"... uh, os is not recognised as a valid module. Same for sys. Apparently, they are somewhat special in Jython and are implemented directly into the main jar, so PyCharm can't see them for autocompletion or any other smart feature. I don't know what else is "special" in Jython 2.2, but I'd rather not have to find out.

Which brings us to the main shortcoming of PyCharm as a Jython IDE: it simply won't recognise or parse any Java jar. This is somewhat surprising, considering how the program is basically a spin-off of IntelliJ IDEA, a Java IDE, and is completely built on Java. In fact, it shares the codebase with the Python Plugin for IDEA. One would think PyCharm would be ideally suited to the task of handling the "Python on Java" mesh that is Jython, but alas that's not the case. A quick search on the IntelliJ forum brings up recent posts stating that full Jython support for autocompletion is simply not on the cards; Jython is supported as a runtime and nothing more. In fact, the Python plugin for IDEA probably handles a Jython setup better than PyCharm, and that's not going to change any time soon. The main target for PyCharm are clearly Django/web developers, not integrators. UPDATE: see this blog post for more details on the real situation.

This state of things is a bit saddening. I don't know if this is a way for JetBrains to avoid cannibalizing its main cash-cow (IDEA), or simply a commercial oversight; the fact is that we have a product, ideally positioned to completely own a niche, which simply refuses to do so and actually delivers a second-rate experience. I hope JetBrains will re-evaluate their stance at some point, because it's a bit of a shame really.

VBScript, PowerShell, Python, IronPython, Jython, or... ?

In my job, more and more I find that we end up building distributed architectures, i.e. multi-server installations where services are routinely spread around 20+ Windows boxes, plus the occasional Unix. These services have specific startup and shutdown sequences and dependencies, so they can't just be set to Automatic startup; we usually provide batch files to manage them, but this is quickly becoming ugly and unreliable as the number of machines go up every year. It's also tricky to test actual service availability -- some are binary-based, some are HTTP based, some are Weblogic services, etc etc etc... So I'm investigating alternatives.

The first, natural choice for me was obviously Python: it does everything I need, it's flexible etc. However, distributing scripts to customers is a bitch; either you compile everything with cx_freeze (crashy) or py2exe (no python 3! no 64bit bundling! party like it was 2004! have fun tracking down which un-redistributable DLL or manifest you need on each release...), or you drop an entire environment and teach the customer what python is -- not ideal.

The traditional "native" approach to these problems in the Windows world is VBScript. It's fairly flexible, doesn't need to be deployed on Windows Win2003+ (yes, we still deal with loads of Win2003 servers), documentation is extensive and there are plenty of resources out there.
The problem with VBScript, apart from the ugly syntax and pseudo-OOP quirkiness, is that it's clearly seen as legacy by Microsoft. Year after year, running scripts becomes more cumbersome, security checks increase and new technologies don't expose the necessary interfaces. Does it make sense, in 2011, to invest time and effort building solutions that are, de-facto, already obsolete?

So we come to PowerShell, Microsoft's "future of scripting", which must be the less intuitive shell I've ever had the pleasure to deal with. I simply can't get my head around the way it deals with input and output; it doesn't seem to have reference assignment, so you have to retrieve an object on every line before you can use it; the syntax seems to combine the worst of Perl and Bash, and it quickly becomes unreadable. Also, deploying it on anything older than Vista has to be done manually and has to be justified to customers.

I honestly can't see a good solution here. I keep looking at IronPython, but its infrastructure baffles me and I wouldn't know where to start redistributing programs (I don't use Visual Studio). It's clearly a second-class citizen in the .Net world, with all that it entails.

Maybe Jython? After all, the products we install will drop JREs absolutely everywhere, so I could leverage that. I'd like to avoid going full-Java if I can, I hate the whole byzantine toolchain and I'm not really up to speed with post-1.4 changes; plus, there's always a degree of customization required in each environment, so I'd like to keep stuff as easily-editable as possible.

Please feel free to drop any suggestions in comments, I could really do with them!

django on jython

You can now run Django on Jython, as announced by Leonardo Muñoz. Lovely. I really should go back hacking django a bit.