Remote Data Rendering
I've grown a bit weary of this pattern:
- Generate data on remote machine
- Generate and save images on a remote machine
- SFTP or SCP images to local machine
- Display images
It sure would be great if we could collapse 2-4 into a single item. The issue I've been running into, however, is that X11 forwarding simply doesn't work that well. VNC provides a slightly better solution, but both of these are clunky and difficult to synchronize with other windows. Ideally, I'd want to be able to spawn an ssh tunnel and an IPython session, and then use that to control remote data, plotting in local windows.
Of course, the raw data could be brought back -- an sshfs mount, for instance. But as long as my data lives on the super computer, why not let the images get created there, as well?
To that end, I used the IPython MultiEngineClient interface to set up an ipengine, an ipcontroller, and then to interact with them. By creating a NonLocal data object that pushed new parameters and pulled back new buffers to display locally, I was able to drop this in as a replacement for the plotting engine in the pan and scan image display. So now, I can directly pan and scan locally without having to pass images manually, without having to generate smaller data products and pass those, and without having to copy any data anywhere. The code to run it is relatively simple, and at the end, it all pops up in a nice GUI.
I'm not sure this is precisely what IPython's remote interface was designed to do -- and it is somewhat showing, in the responsiveness and lag times. To pass a single array back and forth, between a client and an engine both running on localhost, it costs about a second. So FPS is around 1, which is pretty low, but still doable, and still within the limits of what I am willing to live with. Some profiling suggests that the acquiring of thread locks for blocking communication is where the slowdown comes in, so perhaps there's a place to speed it up there -- the multiple layers of abstraction in IPython may not be doing me any favors, and I may benefit from investigating how to bypass some, or even use a different backend eventually.
But, again: this lets me chew through a lot of data, very easily, and without a lot of hassle. It's a huge net win. Next up is figuring out how to get some of our custom colormaps into Chaco, add a couple convenience buttons on the Traits window for my particular purposes, and then I'll be totally set.
Update: Evidently I can't get Posterous to include code properly. I have no idea why. Apologies.
