Git-er dunn

So I'm a late-comer to this version control phenom called Git.  Up until basically a couple weeks ago I'd been mostly tied to Perforce for work and for my own private or local small things I'd maintain my own SVN server repo and for single one-offs even used RCS simply because it was quick and dirty and there were emacs came with bindings by default.

Oh, I was aware of Git and did some reading on it over the years and had the interest, just never had a need until recently.  I basically created the need and have been playing with it for a couple weeks now.  The need came about because Perforce licenses cost money and at work if you aren't doing development on something in Perforce for a sufficient amount of time, they "threaten" to pull your license.  Actually it's not a big deal because I can always just get a new one.  Still, I didn't like getting the notification.  More than that though, was this company-wide push to migrate towards Git from Perforce, SVN and the others.  Now that I've been using it, and given what I've read about it, not only do I understand the move but I wholly endorse it and wish I had used it before and pushed for it at my previous job. It might be the novelty and "shiny new toy" effect but I think it's pretty slick, albeit slightly obtuse in learning since it has some significant points of deviation from what I've used before and requires some re-thinking.

Initial highlights for me are:

1. Split location development
2. Promiscuous committing and branching
3. Speed and Offline work

Split location development

The primary "oh, this is actually cool" feeling came when I realized that it may solve my split development-environment problem (at least the mechanical part).  What I mean is, I do/did development on different machines such as laptop and desktop and other remote machine.  I'd usually had some helper scripts and such to rsync things back and forth and keep track of what files I changed in each, always at risk of clobbering some work I had done.  It was really kludgy but necessary.  It was just easier and more convenient to develop on my laptop but I was limited because the gobs of data I needed to run on or the other needed tools/services/libraries were on the servers which my desktop could have access to but that I couldn't bring locally to my laptop.  Currently, I'm experimenting with a Git setup where my desktop pulls and pushes from the main repos on the servers, and my laptop pulls and pushes to my desktop.  So far it seems like a great way to keep things aligned and with commit changes and history preserved, made possible by branching and merges being first class features of the system.

Promiscuous committing and branching

This was new to me in terms of typical work flow.  Since branches and merges were kind of painful in the other version control systems, and because even checking work back in to a central repository with others was a time-consuming process (with regression tests, locking, merging, code-reviews, etc),  I'd basically work on many multiple things in the same checkout and check them in en masse with a single description just so I didn't have to do it as often.  Now I see I can do many local and -- more importantly-- annotated commits which is nice for collecting clusters of work related to the same goal and keeping track of change history.  And as well, I find myself now separating different work goals into different branches, even disposable ones, knowing it is still organized and relatively quick and painless to re-merge especially since it will be with myself only. In effect (if not in actuality) I get to be the repo owner and all of the benefits that entails.

Speed and offline work

I really like that I truly get to work locally.  No waiting on a central repo.  I hate when people spout nonsense that free wifi is everywhere and we're never without a connection.  Even with your own mobile wifi hotspot you don't always have a connection.  Just ride the BART for an hour like I do and see what percentage of that time you have an uninterrupted signal.  It's not that I don't wish I had uninterrupted wifi all the time, but I don't want the lack of it to gate me from being able to be productive when I'm a poor offline user.  Just as important, I really don't always want to be VPN'd into work either, which is extra annoying because I have to dig out and use a password-generating dongle to VPN in.  This is especially frustrating on e.g. BART where interrupted signal can kick me off of VPN every few minutes in some areas. Poo.  The side benefit is that since the whole repo is local and I don't need to consult with the central repo for every little thing, it's really fast.

All of this is well-known to Git users.  There are other bigger philosophical and paradigm shifting things connected to it as well.  But this is what effects me.  I hear that it was a bit more painful and/or confusing to use in the early days so it's probably just as well that I only pick this up now that it has passed the mass adoption phase.  To many I'm sure it's just another version control system, so what, we work fine with the old method, it suits us better, less to learn, the are workarounds to ease the pain points with the old system, it's still improving, etc. etc.  For me, little things like this that sort of optimize some annoyance out of existence or greatly reduce it still excite me.

Python Logging

I've been using the Python logging module for a couple of weeks now, and I want to like it because A) it's a standard module, B) it has some cool features like multiple handlers and hierarchy.  But almost every time I use it I feel like I might as well just write my own logging module suitable for my purposes... because it seems like I have to do that anyways.  The module just seems to require too much scaffolding and setup to use.

Here's what I mean. To do it properly you have to:

• get a logger
• set the verbosity level of the logger
• create a file or steam handler
• create a formatter (the default needs replacing)
• add the formatter to the handler
• add the handler to the logger
• do this all again if you want to mirror to stderr AND to a file (which is why I started using logging in the first place)
• put in code to shut down the logging (makes sure the streams get flushed) and for safety use the atexit module, meaning
• import atexit
• register the shutdown
• add an exception hook so that we can log uncaught exceptions too

This is just a little too much for basic proper use, don't you think?

To be fair, there is a "simple" way to use logging which is to just use the logging module functions "BasicConfig()" and "debug|info|warning|error|etc()" functions without getting a logger for your module.  But it doesn't give the behaviour I want and even they prefer you don't use it in this manner.

What I believe is missing is a set of helper-functions and/or sytactic sugar to handle common tasks.

• let more things like Level and Handlers be put in the argument to getLogger
• automatically wrap common things like a File-like object and filename string into a handler instead of having the need to explicitly make one.
• an at-exit shutdown should be somewhat implicit (maybe an option to turn it off) as well as the option to trap other exceptions

And what I'd like to have for simple operation:

• one line (minus "import") to get a logger for my module with any optional formatting and whatnot.
• one line to configure the root logger with all options, that can deal with an array of logging destinations, that will auto-interpret formatting strings and destinations instead of needing to create sub-handlers and formatters, etc.

Here was my first crack at collapsing all that with two helper functions, but I hate having to add more functions to import for things that should have just been available (yes it's a bit ugly).

def add_to_logging(log,whereto=None,level=10,format="%(levelname)s: %(message)s",dateformat='%Y%m%d_%H%M%S'):
    ''' shortuct to attach a destination to an existing logging object
    logfile can be file or gzip or stream or None(meaning stderr) '''
    if whereto is None: whereto = sys.stderr
    if isinstance(whereto,(str,unicode)):
        fp = opener(whereto,'w')
    else:
        fp = whereto
    fh = logging.StreamHandler(fp)
    fh.setLevel(level)
    if format: 
        formatter = logging.Formatter(format,dateformat)
        fh.setFormatter(formatter)
    log.addHandler(fh)

def setupLogging(logname=None,rootname='',timestamp=False,consoleLevel=20):
    ''' shortcut to set up a dual stderr/logname LOGGING stream 
    default level for file is DEBUG, for console is INFO
    (set consoleLevel to 0 to turn off console)
    SEE PYTHON LOGGING DOCUMENTATION FOR LOGGING BEHAVIOR
    returns a logging object'''
    import atexit
    logger = logging.getLogger(rootname)
    logger.setLevel(logging.DEBUG)
    dateformat='%Y%m%d_%H%M%S'
    # change the formatting if timestamp
    fmtstring = "%(levelname)s: %(message)s"
    if rootname is not None and rootname != '':
        fmtstring = "%(name)s:" + fmtstring
    if timestamp:
        fmtstring = "[%(asctime)s:%(name)s:%(lineno)s:%(levelname)s] %(message)s"
    # add a file if specified
    if logname:
        assert isinstance(logname,(str,unicode))
        #logging.basicConfig(filename=logname,format=fmtstring,dateformat=dateformat)
        add_to_logging(logger,logname,format=fmtstring,dateformat=dateformat)
    # add a console
    if consoleLevel!=0:
        add_to_logging(logger,sys.stderr,format=fmtstring,level=consoleLevel,dateformat=dateformat)

    # cleanup and exception handling
    atexit.register(logging.shutdown)
    # the following will capture exceptions to the logs as well
    sys.excepthook = lambda *x: logger.error('Uncaught Exception',exc_info=x)
    return(logger)

In the above "opener()" is a separate function I have that wraps opening a filename, file object, pipe, or what have you depending on the input and optionally with encoding.  Sometimes I miss how easy that is dealt with in Perl.

Emacs, you ugly nerd

I once saw Emacs described as a "thermonuclear text processor" and I find that description fitting.  Emacs is a beast.  It has a really steep learning curve.  But much the same way the *nix in general can have a steep learning curve, after you use it for a long enough time you grow to appreciate the raw power available over the other offerings.  Some even take it up a level "Emacs IS my operating system".  There are too many quirks with that for me to accept on that part though.  I know I can run a shell terminal in Emacs but I never do, mostly because of how the environment seems to get screwed up just enough to turn me off.

Anyhow, I sometimes can't believe how utterly craptastic it is make Emacs look pretty by way of fonts and such.  It's a freaking mess.  And there is the invariable quirk with it as well.  Thankfully the internet comes to the rescue.  What I love most about the web coupled with a good search engine is that I often enough am able to find the workaround to a problem I'm having.  Not easily some of the time but it puts a whole host of things within the realm of possibility.  Most recently I wanted to correct the annoying issue of Emacs not using the default font when opening a new frame via C-x 5 2.  Found the snippet of code I needed to add to my .emacs file almost immediately.

Only thing I really regret about using Emacs as much as I do is never having learned Lisp.  It's always been on my someday list.

FF4

I installed Firefox 4 partly because I simply wanted the speed improvements and partly because of newest-gadget-syndrome.  I'm pleased to report that I am digging the loading speed improvement immensely.  As for the better handling of memory, even with the feature where it will unload unvisited tabs after some time and it can be set to not load all tabs at startup, I find myself again running into the issue of FF taking up way to much memory and slowing to a crawl.

The fatal flaw with FF4 was that they didn't go far enough and just up and swallow the BarTab add-on, the first or second most critical addon in my FF arsenal.  I was really dismayed that BarTab hadn't been updated to work with FF4 yet.  I tried the tweaks and suggestions to get some of that functionality that's built-in to FF but it just doesn't hold water.  I need the ability for tabs to explicitly be unloaded exposed, preferably with a rightclick menu and a menu item to "unload all other tabs".  When FF gets slow, this has helped a lot with BarTab.

Thankfully, I just found a hack to get BarTab mostly working with FF4, and it must be treated as Beta.  There's already the issue where tabs don't load automatically when clicking on them but pressing F5 is the just price I will pay to have the ability to unload tabs.  We'll see though, maybe it will get too annoying.

On a side note, I get really annoyed in the discussions from people imposing their view that you should never have more than a few tabs open.  There's merit to the notion that you shouldn't keep more tabs than you need open, but there are lots of times where I do need many open simply so that I do not have to constantly look for previously opened sites.  Case in point, I have a bunch of tabs open to a set of Python modules and related info.  I just need the page loaded once so I can click on the tab to look something up, I don't want to have to close it and load it up again when I need to look it up again.

Tramp at work

One major change in workflow that I had to get used to with the company change a few years ago was doing most of my work and development through remote terminals.  Try as I might to do at least some work on my local machine there was just no escaping that it was just more convenient to do my script writing and edits and tests on the remote machines.  A shell is a shell so I can manage: windowless emacs and vim for editting, and screen sessions for stuff.  But I hate the lag introduced working remotely, I hate not being able to exploit having remote drives mountable to my local machine, I hate not being able to exploit my local tools and environment.  All of this could go away partly if they just enabled sshfs so that I could at least access some of the data for development purposes (much to much to xfer locally and there's policy against it).  But alas, they are unwilling to enable it, even if there's no good reason.  It's not like i would be able to access anything more than I could anyways.

So one thing I tried before was at least being able to use my local emacs editor to edit remote files with Tramp.  I recall having to munge my commandline prompt to get it to work and that it was a little clunky and so I gave up.  But I've tried recently tried it again and it went a lot more smoothly to my surprise.  It even came with my latest Ubuntu installation (finally updated after a year+ of laziness) vs my downloading and installing it myself previously.  So it was trivial to try again and I was pleasantly surprised.  Saving and loading has the lag still but everything else gives me the snappiness of a local machine (save for when I have a network interruption).  I guess I'm going to try this method out until I come up with a more satisfactory method of doing more work locally without all the constant syncing.

Unfortunately what got me thinking of this again was wanting to try another editor to edit Python.  As much as I like (or rather am used to) emacs and vim, Python feels like it require a lot more editor support to do right and to overcome annoyances.  So I've been wanting to move to an IDE so I'm testing out Eclipse and the remote access plugin, but so far I am thwarted by it's need to have SFTP enabled remotely or another server running remotely in order to work.  Not having root there I just don't have the time or energy to work around trying to hammer that together.  I'm bugged to no end that there is no easy facility for installing packages as a user without root.

No vertical screen split for J00

I got quite happy when I learned there was a version of Gnu Screen that let you do vertical splits.  I always have 2 screens side by side (emacs and shell) both distinct ssh sessions with distinct instances of screen.  Had this worked it would have saved me some minor inconvenience since I'd only need to log in once and not have to select between the two detached screen sessions.

I said "had it worked".  Actually it did work, but there is a side effect that is a deal breaker and it makes me really sad.  The fatal flaw is that I mouse select for cut/paste a LOT and the selection does not respect the border between the vertical split.  Yes, there is the keyboard option for cut/paste but that's insufficient.

vbox

A couple years ago I was using vmWare to have linux running on my work laptop.  It did it's job but for some reason felt just clunky enough to where I just ended up using putty terminals instead.  They're light and quick even if a bit featureless and I got accustomed to running emacs without a mouse.  vmWare was one of those things that I installed and didn't want to touch again because I recall it being a little confusing, particularly finding the right product.  They had a bunch, they all required some kind of registration, it wasn't always clear to me which one I was to download.  Eventually I just gave up.  Not that it was bad, just that it wasn't something I needed to spend time on.

A year ago or so I saw some positive press on Virtualbox (prior to Sun being acquired by Oracle).  I gave it a whirl (coinciding with my new laptop purchase).  My impressions were quite positive. It seemed less complicated.  Of course, the REAL reason I gave it a try was because the reports were that there was finally a VM that could run COMPIZ.  Back when I had my own Gentoo workstation at the office, I grew addicted to Beryl/Compiz.  It made work just a little more fun, and I'm also one of those people who's productivity is increased by multiple desktops.  But if I'm gonna stare at a screen all day, why not make it cool too right?  (and I get a little pleasure from the wow factor in other people who saw me at work)

Anyhow since then I've been using it more often.  I normally work on remote servers because I have little choice (that's where the data is) but for development It really helps to do things locally, especially when I'm on the road with intermittent internet access, or when I know that the servers will be down, or when they're running really slow, etc. etc.  Mostly though, I prefer having a full environment than just putty terminals.

So for months, I've been seeing this popup whenever I launch vbox that there is 4.0.4 out but I never made the upgrade.  Upgrades terrify me because when it's important, something almost certainly will go wrong.  But finally the mood struck me and I gave it a try.  And it failed!  It was start installing and then hang there doing nothing.  Meanwhile, it uninstalled my old version.  Horrors!!!  Panic!!!  Did a bunch of searches, tried several things, had to reboot countless times (primarily because installing vbox turns off the network, very annoying).  After many hours of following what turned out to be the wrong post, I found another that simply stated you turn off DropBox.  Boom! It worked quickly and cleanly.  If I hadn't found that post I don't know what I would have done.  I completely forgot about DropBox, it auto starts (another rant) and it interferes with the network changes.  It's completely unrelated to vbox so I don't know how long it would have taken for me to make the connection.  It never fails, something always goes wrong.

As I mentioned I have some trepidation about upgrading software but I figured that since the upgrade was already months old it might be safe, and what's more I finally got around to moving my home directory to it's own partition; making me a little more cavalier about making OS modifications now.  I used to simply let the home directory exist along with the OS because with limited drive space I've been burned before about how big to make each partition.  But that's less of a concern now so I finally got around to it.  Need to do it on my other computers now tho.  Now that I have it, I can finally play around with other linuxera with more impunity and vbox makes that a lot easier to do.

BarTap for Firefox

Truth be told, one of the reasons I have recently gotten into the habit of suspending my laptops instead of shutting down completely is so that I don't have to suffer through reloading Firefox with the gazillion tabs I kept open but didn't want to close for one reason or other.

But finally this Firefox app just crossed my desk to solve that problem and I love it.  It has shot to the top 3 of must haves, maybe tied with Tree-style tab.  I agree with the comments to the app, it should have been included with Firefox by default.

You complete me

Being the lazy typist, like most programmers, I use command/file-name completion A LOT. I probably press "tab" more than I do "return". That works fine for the commands and for the file arguments.
However, there are a few command-line programs that are collected into toolkit style and operate on the form:

command subcommand subargs

It's sometimes a useful thing to do to wrap a bunch of little but related commands into a single command line, especially when the subcommands would otherwise share a bunch of code and procedures. Sadly, completion doesn't fill in choices for the secondary command; how can it? So one has to remember what choices there are and type the whole thing. This also applies to commands where the argument is another command, string, or filename that may not exist yet, as in makefiles.

But no more! Found another little *nix gem that's been in existence for who knows how long but lost on me because I never bothered reading the complete bash man page. I was alerted to it during my dig on Ruby Rake tips. Basically it concerns being able to write your own bash completion. Way cool!

In the makefile (and rakefile) example, it's very convenient to type "make [tab]" and get a list of possible targets of make. And in my toolkit example, I can type "command [tab]" and get a list of possible subcommands and a completion when it's unique. Very handy. It can save seconds at a time and has some good "wow" potential since I haven't seen anyone else in my group use this feature. I don't feel so bad that I missed this little time-saving gem all these years, but I wish I had known of it earlier.

Screen gems

So there's this *nix tool called "screen" that I know has been around a long time and I've seen it before and in use... but never used myself. I never really felt like I had a compelling reason to use it until this week. Now I wish I had added it to my tools arsenal long ago. It's just really handy. It basically just lets you run multiple shell sessions in the same window, BUT with the ability to detach them and re-attach them elsewhere. Now that's handy. I can run some stuff at home, detach the screen and resume it at work with history and output text and all. The multi-session lets me have fewer ssh windows up as well. At the office I just use kterm which has tabs and I keep multiple (maybe too many) windows up. No real need for screen there. But when remotely logged in, it's just that useful.

Then I thought, back when I was doing a lot of remote work before on a previous Linux box, how come I never used it? I guess it's the nature of the work. Back in grad school (ugh, I keep saying that) I did actual developing on my laptop, which I would rsync back to my workstation. In this instance, I can't really do that since I'm not developing (much) and what I do requires running through many gigs of data that I don't, can't, and prefer not to download. So I'm forced to work remotely through ssh. The other thing is, these jobs can sometimes run for hours, of which I don't necessarily need to be connected to completion... or it gets to late and I want to resume the next day. Screen to the rescue, just detach and exit and then the next day re-attach and continue.

I don't know how I missed such a little gem. I'm such a noob.