- Posts tagged python
- Explore python on posterous
HPCwire: Python Snakes Its Way Into HPC
Interpreted programming languages usually don't find too many friends in high performance computing. Yet Python, one of the most popular general-purpose interpreted languages, has garnered a small community of enthusiastic followers. True believers got the opportunity to hear about the language in the HPC realm in a tutorial session on Monday and a BoF session on Wednesday.
via hpcwire.com
I'm not at SC10 this year, but I am actually giving a seminar today on HPC with Python. That aside, I'm a bit bummed I missed this BoF. For our astro work, we use Python extensively for analysis, and the paper that was up on the arXiv yesterday was in fact all about this.
It's worth noting that David Beazley, the author of SWIG and a luminary in the Python community, has been involved with Python for HPC applications since sometime in the mid 90's. Python for HPC is not new!
Spatial indexing with Quadtrees and Hilbert Curves - Nick's Blog
Spatial indexing is increasingly important as more and more data and applications are geospatially-enabled. Efficiently querying geospatial data, however, is a considerable challenge: because the data is two-dimensional (or sometimes, more), you can't use standard indexing techniques to query on position. Spatial indexes solve this through a variety of techniques. In this post, we'll cover several - quadtrees, geohashes (not to be confused with geohashing), and space-filling curves - and reveal how they're all interrelated.
via blog.notdot.net
Includes a handy Python function for converting to a Hilbert curve.
New Phone
I got a new phone last weekend. I love it. I have especially been loving the android scripting environment (now known at Scripting Languages For Android, or SL4A) which can be found at its Project Page. Tonight I managed to get most of NumPy working on it, as seen in the image below. I had to make some major hacks and workarounds, but I'm going to try to prepare this into a patch to send upstream to the developers. Hopefully it'll work for people other than me, in my emulator...
A GUI revisited, once more
A week before I left on my vacation, I spent a week running a conference on how to use our AMR code, Enzo. During this conference, the idea of a GUI for yt came up again, and my old thesis advisor (the head of a project of his own that's a GUI for analyzing Enzo data!) suggested I scale back my ambitions a bit and try to just replicate the GUI he has created, one step at a time -- and his enthusiasm about this was pretty contagious. So I spent a day doing that, and then some isolated work on planes afterward, and I ended up with a very simple Tkinter GUI that can open up multiple slices and look at multiple outputs at a time.
I'm calling it Fisheye, and so far it's still very small -- but I've been really enjoying browsing data and using it for very, very rapid exploration. The other GUIs I've posted about in the past have had their place (and they've all also been very small) and this actually draws on all bits and pieces that were developed for those GUIs! I'm not sure how long I'll keep playing with this on planes and buses, but it's been fun.
The part about this particular pass at it, and it really is only a little pass at it since it's under 400 lines of code, is that it uses Tkinter for the GUI. I've been polling the other yt devs to see where Tkinter is available and I've been pleasantly surprised that it has pretty good penetration at supercomputing centers. So maybe there's something to it.
I know I feel a bit better about it than anything else I've tried -- because it requires no additional dependencies, and it's pretty straightforward to write.
Remote Panner
Last night, using 0MQ I set up the image panner to use a remote renderer. This let me start up a REP/REQ system, where on the remote system (bound to localhost and ssh tunneled) I waited for new requests that describe the limits I wanted to plot, and locally I had the image panning GUI send the new requests and receive the new image. It worked great -- very easy to set up -- except that the arrays it was passing were 2MB each, so the latency to my home network was unbearably high. Today I'll give it a shot at work, see if it's a workable solution.
Nicholas Piël » ZeroMQ an introduction
ZeroMQ is a messaging library, which allows you to design a complex communication system without much effort. It has been wrestling with how to effectively describe itself in the recent years. In the beginning it was introduced as ‘messaging middleware’ later they moved to ‘TCP on steroids’ and right now it is a ‘new layer on the networking stack’.
via nichol.as
IPython's moving to 0MQ for their client/server model. Some elements of yt likely will as well -- specifically the remote push/pull of updated image buffers for the remote pan-n-scan.
Shedding Bikes: Programming Culture And Philosophy
But, seriously, if your first question about a piece of GPL software is, "How can I go about gettin' around the GPL?" Then you just don't get open source at all.
Zed Shaw relicenses his project Lamson under the BSD license. I favor the GPL in my projects, but my projects tend to be much more specific and targeted at a very narrow audience, so any experience I have from that may be inapplicable to the style of projects elsewhere in the "Python Community."
Figuring Out Stereo Volume Rendering
Last week I was approached by a friend and collaborator to prepare some large volume renderings using the aforementioned software volume renderer in yt. In the past we've successfully made very, very large image renderings using yt -- Sam's even made one at 8192^2, although at extremely high resolution like that sometimes the lack of fidelity in the underlying volume renderer shows up; sometimes even artifacts in the AMR grid boundaries, but that's less common. Making the very large volume renderings isn't too bad -- it scales roughly with the number of pixels, but we can dispatch many frames to be rendered at once on a cluster.
There are a couple other, more important things to consider when making the big volume renderings. For starters, the entire structure of volume rendering in yt was not really created to generate a series of images -- only a single image. The idea was that you would prepare a specific image, make it, and move on. However, for this project, I want to do a zoomin, or possibly a more complicated camera path.
Additionally, one of the first things that we did with the volume rendering was silly: we applied no normalization to the output images. That was a mistake, I see now. Part of the reason for this was uncertainty in the correct normalization -- the bias that the user wanted to apply may not be the natural bias from the image. But more than that, because the rendering algorithm itself was some what holistically settled upon (the original implementation, which we used for shell-style renderings, was not a "correct" implementation of alpha blending) a natural mechanism for scaling did not immediately present itself. One likely exists, possibly dependent on the field of view, I simply do not yet know it. This will have to be rectified, because the mechanism used for scaling a set of images will have to be different than the mechanism for scaling an image in isolation, or else frames will jump in brightness during the movie's course.
The final thing that I wanted to change was to add support for stereo rendering. Rather than repeat any of the amazing discussion from Paul Bourke's website, I'll simply direct you there. Everything you ever wanted to know about stereo rendering. (When I was a first year grad student, we actually bought a copy of his site to use locally -- it was our way of showing support for him putting it online, and it also came with a bunch of source code for example applications.) I first attempted to apply the correct method for stereo, where the view direction is parallel and the total view frustum is shifted.
This did not work. In fact, it made me realize that all this time, the yt method for volume rendering is in fact ... not really a volume rendering method inasmuch as it is a planar ray-casting method. Typically when doing volume rendering, there's a perspective applied to the image: the rays all emanate from a single place, creating a frustum. But for yt, we actually set up a single plane of vectors at the back of the volume and advance that forward across the image. This is good and bad; it's good in the sense that it's more clear precisely what is going on. But it's bad in the sense that correct stereo is more difficult. (Of course, on Bourke's page he has a workaround that may work for this, but I have not yet attempted it.) Here's a rough depiction of the different between the two methods.
The upshot is that stereo doesn't seem to work unless you go with the "toe-in" method that can cause eyestrain after a long time and shows visible parallax at the edges. I'm not sure if this is going to be a problem, but because I am not right now eager to rewrite the rendering backend, this is the way it is for the moment.
To set up the stereo rendering, I separated out the rendering mechanism from the objects to be rendered. Previously, there was a single VolumeRendering object that you could create, raycast through, then discard. I created a new camera object that accepted a homogenized volume and would call "traverse" on that volume, feeding a back and a front point. The Volume is then responsible for passing off fixed-resolution grids to the camera, which accumulates an image buffer by calling the ray traversal functions. The front and back points are essentially the only thing needed to know this order, but the camera also stores its three orientation vectors and its position that describe it in 3D space. By separating out these two conceptual objects, we undo some of the "single, carefully constructed image" bias that was in the original volume renderer. (And, we open ourselves up to being able to use the Camera with a hardware volume renderer, should that day ever come.)
So now we have a camera, and it makes images like this:
It's a little dim, but that's a task for another day. The next step is taking that perspective and turning it into a set of stereo images. To do that, I added a new class called StereoPairCamera. It accepts a Camera object and turns it into two camera objects, where the final interocular distance is calculated relative to the image plane width. As I mentioned above, this only operates via toe-in stereo, so it does this in the simplest manner possible: it moves each of the Left and Right cameras by half of the interocular distance away from the original location, and then recalculates a normal vector to point back at the original center. Now we can generate left and right images:
Unfortunately, on my laptop (which is my primary/exclusive work computer) I don't have the ability to view these pairs. To get around that, I wrote a simple stereo pair image viewer in OpenGL and imposed upon my friends that do have a stereo viz wall to test it out -- and after some fiddling with the interocular distance, we got what appeared to be workable stereo pairs.
The full code for generating camera paths as well as stereo pairs is already in yt, but the documentation is still being written; I might also clean up the interface a bit. Additionally, at some point in the future, the issue of toe-in stereo versus correct parallel-frustum stereo will need to be dealt with; the last thing I really want to do is force people to only use a bad method for generating stereo pairs. Hopefully that is something that can be dealt with at a later time.
Thanks to the wealth of resources out there for making this a relatively easy task: the aforementioned Paul Bourke website on stereo pairs, the PyOpenGL and PIL teams for making the image pair viewer easy, and everyone else whose work I've built on to make things like this.
Python Package Index : rpdb 0.1.1
pdb wrapper with remote access via tcp socket
via pypi.python.org
I wrote something like this; it's in yt as of right now. Mine was designed for a different task, though -- debugging MPI tasks interactively. Still, this one's probably a lot more robust.
Remote Data Rendering
I've grown a bit weary of this pattern:
- Generate data on remote machine
- Generate and save images on a remote machine
- SFTP or SCP images to local machine
- Display images
It sure would be great if we could collapse 2-4 into a single item. The issue I've been running into, however, is that X11 forwarding simply doesn't work that well. VNC provides a slightly better solution, but both of these are clunky and difficult to synchronize with other windows. Ideally, I'd want to be able to spawn an ssh tunnel and an IPython session, and then use that to control remote data, plotting in local windows.
Of course, the raw data could be brought back -- an sshfs mount, for instance. But as long as my data lives on the super computer, why not let the images get created there, as well?
To that end, I used the IPython MultiEngineClient interface to set up an ipengine, an ipcontroller, and then to interact with them. By creating a NonLocal data object that pushed new parameters and pulled back new buffers to display locally, I was able to drop this in as a replacement for the plotting engine in the pan and scan image display. So now, I can directly pan and scan locally without having to pass images manually, without having to generate smaller data products and pass those, and without having to copy any data anywhere. The code to run it is relatively simple, and at the end, it all pops up in a nice GUI.
I'm not sure this is precisely what IPython's remote interface was designed to do -- and it is somewhat showing, in the responsiveness and lag times. To pass a single array back and forth, between a client and an engine both running on localhost, it costs about a second. So FPS is around 1, which is pretty low, but still doable, and still within the limits of what I am willing to live with. Some profiling suggests that the acquiring of thread locks for blocking communication is where the slowdown comes in, so perhaps there's a place to speed it up there -- the multiple layers of abstraction in IPython may not be doing me any favors, and I may benefit from investigating how to bypass some, or even use a different backend eventually.
But, again: this lets me chew through a lot of data, very easily, and without a lot of hassle. It's a huge net win. Next up is figuring out how to get some of our custom colormaps into Chaco, add a couple convenience buttons on the Traits window for my particular purposes, and then I'll be totally set.
Update: Evidently I can't get Posterous to include code properly. I have no idea why. Apologies.
