Custom file extensions and Espresso 2

A fair number of people write into MacRabbit support asking how to add support for a custom file extension to Espresso (common requests being for LESS or SCSS mapped to CSS, various templating languages mapped to HTML or PHP, etc.). Unfortunately, a number of articles persist online instructing people to modify the built-in language plug-ins that live inside the Espresso application, which is not a good method to accomplish this particular task because you lose your changes when you update Espresso (additionally, mucking around with the internal plug-ins can sometimes lead to really weird behavior; for instance, messing up the built-in CSS support can have wide-ranging implications on Espresso’s syntax coloring for all languages).

Adding support for custom file extensions is extremely easy, but is unfortunately not something you can currently do through the GUI. Here’s how to go about it:

  1. Create a folder named something memorable. For instance, if you are mapping .less to CSS you might call it “LESS-Standin”. Or if you are feeling capricious, perhaps “Snickersnack”.
  2. Drag and drop your new folder on the Espresso Dock icon to create a temporary project window
  3. Within your new project window, create a new file named Languages.xml
  4. Add the following code to your Languages.xml file and save the file:

    <?xml version="1.0" encoding="UTF-8"?>
    <settings>
        <language id="com.yourdomain.less-standin" hidden="true">
            <root-zone>language-root.css</root-zone>
    
            <name>LESS Standin</name>
    
            <detectors>
                <extension>less</extension>
            </detectors>
        </language>
    </settings>
    
  5. Convert the folder into a plug-in and install it (see below)

What you are doing is defining a new language for Espresso that references a pre-existing syntax. The things you will want to customize are the id attribute of the language element, the <root-zone> element, the <name> element, and your list of detectors.

You can find documentation for the available detectors in the Espresso plug-in development wiki, and here is a list of the most commonly-used root-zones that are included as of Espresso 2.1:

  • CSS: language-root.css
  • HTML: language-root.html
  • XML: language-root.xml
  • PHP: language-root.html.with-php
  • JavaScript: language-root.js
  • Markdown: language-root.markdown
  • Python: language-root.python
  • Ruby: language-root.ruby

If you wish for your language stand-in to show up in the View→Languages menu, remove the hidden="true" attribute from the language element. Also note that you can add multiple items within the <detectors> block in order to define multiple file extensions to map to the same language (so in the example above, you could add <extension>scss</extension> in order to map both LESS and SCSS to the built-in CSS parsing within the same plug-in).

Installing your new plug-in

You have a couple of ways you can install your new plug-in:

  1. Simply add a .sugar file extension to the folder you created, and double click to install it. This is a good option if you don’t expect to add support for other file extensions, since it’s straight forward and easy.
  2. Open up the Terminal and execute the following command (replacing the path to point to your new folder):

    cd ~/Library/Application\ Support/Espresso/Sugars
    ln -s PATH/TO/LESS-Standin LESS-Standin.sugar
    

The latter option will create a symbolic link pointing to your folder, which allows you to easily modify your Languages.xml file down the road.

After installing your plug-in with either of the methods above, relaunch Espresso and you should be good to go!

First impressions of Mojang’s Scrolls

When I was but a lad, my aunt introduced me to a little card game called Magic: The Gathering. And down that rabbit hole I went. Magic’s not perfect (and I haven’t purchased a card in years, at this point), but it certainly has provided me with years of entertainment and is a go-to game for some of the people I play games with to this day.

However, as the years went by, people I played Magic with tended to drift apart. College killed the hobby for me for a while, until I moved into a house with a few guys and discovered most of them had played when they were young. Then we graduated college, and now one of those guys is in Texas, another is just getting married, and I’ve got a baby on the way. All of which makes getting together for a game day every so slightly more difficult than when we could walk down the hall and say, “Hey, want to play a game of Magic?”

For years I have been trying one online TCG after another (or collectable card game, as some of them prefer to be called, since not all feature trading), but have been unable to find a fix worth replacing my Magic addiction with. So when I saw the announcements about Scrolls from the makers of Minecraft, I was highly intrigued. A game that isn’t pay-to-win? Tactical board game elements? Cards that transform into little animated characters? Sign me up!

Scrolls is now available for purchase in “public beta” form (whatever that means; frankly, once you start charging money you are providing a product no matter what you call it), and I have been playing it for the past few days to see if it will scratch my CCG itch.

Unfortunately, the short answer is “probably not so much.” While Scrolls is interesting, there are a number of flaws that are embedded deeply enough in the core gameplay that no amount of beta tweaking is likely to fix them.

(On the other hand, keep in mind that this is not really a review of Scrolls, because we are still very early in its lifecycle and Mojang has shown before that they are perfectly happy to charge for a very rough product, and then turn it into a completely different game by the time they proclaim it out of beta. So take these comments with a grain of salt, since they may well only apply to the game at the time of this writing.)

The curse of the starter deck

When you first launch Scrolls (after purchasing it), you will find yourself facing a choice in which starter deck to begin the game with. The choices are growth (lots of creatures), energy (machines and direct damage), or order (fewer creatures, more buffs). I chose growth, because who doesn’t like hordes of beasties?

Unfortunately, the growth starter deck at least is terrible (I can’t speak for the other, but strongly suspect them to be similar). It seems like Mojang opted for decks that provide a solid basis of cards representing the various things you can do with a given faction, rather than focusing the deck on a particular theme (the way I’m used to with Magic preconstructed decks). The good part about this is that once I accrue more cards (a lot more cards) I will have a good basis for deck-building faster.

The bad part about this is that the deck has so little internal synergy my first few games boiled down to more luck of the draw than anything else.

Take, for instance, this card (one of the few cards of which the growth starter deck contains the maximum three copies you could ever need):

Scrolls: Junkyard

At first glance, this appears to be a pretty neat card. It only costs one resource (the prominent top number). It has three defense, so provides a nice little barrier between your idols (the back row you have to protect) and your opponent’s low-power creatures. And every rat you play gets an extra health! Sounds like the perfect first-turn play, to me.

Except that the starter deck contains no rats. (At this point, I’m not sure if there are any rats in the initial batch of cards at all; I’ve never seen one in any of the cards I’ve purchased, or any of the bouts I’ve fought. I assume they must be out there, though, or why does this exist?)

Similarly, one of the “rare” cards included in the starter is a wolf who gains power based on how many other wolves you have in play, but because there are so many non-wolf cards it’s very difficult to get that synergy working correctly.

Taken alone, none of the cards in the starter are bad (I can see a use for most of them, if only I had the cards to support them), but taken together the whole thing plays poorly and with very little synergy.

Interesting decisions abound

Okay, so I was disappointed in the starter, but after playing at least a couple hours daily over the past week (and spending the 2,000 “gold” that the game provides you to start out), I was able to tweak the preconstructed deck just enough that it began to work a bit smoother (good-bye, Junkyard!). At this point, I started becoming familiar with the things that Scrolls does well. In particular, it provides a number of interesting decisions:

  • On a given turn, you can discard one of your cards in order to permanently increase the resources you have to spend on other cards. This leads to fun decisions in the early game (“Do I save this super-expensive, context-specific card in hopes that I can play it at the perfect time later, or transform it into a resource so I can play my low- and moderate-cost cards now?”)
  • On a given turn, you can discard one of your cards in order to draw two additional cards. However, this is mutually exclusive with sacrificing a card for resources.
  • After you get some creatures onto the battlefield, you have to decide where to move them. Most creatures have a cooldown value of at least two, which means the majority of them will only be attacking every other turn. Additionally, creatures can only move one square per turn. This can lead to some tricky tactical choices; do I spend two turns of movement to position my creature so my opponent can’t take it out on his next turn even if it results in a sub-optimal attack? Or do I chump block with it, and hope that something better comes up?
  • Anticipating your opponent. This is part and parcel of the tactical board play, but being able to predict what your opponent will want to attack is key to setting up your own attacks.
  • Timing (both of cards you play and movement) is very important. I have several times screwed myself up by moving a creature, sacrificing a card to draw two more, and discovering had I done that in the opposite order I would have had a far more optimal play.

All of this adds up to solving some of my least favorite parts of Magic: no lack of resources due to bad card draw, no turns where you’re just killing time hoping the one card you draw will make a difference, and no getting stuck with nothing in your hand and no way to refill it.

The flip side of tactical decisions

Unfortunately, there is a downside to all those tactical decisions that I love so much, and that is games that last quite a lot longer. Part of this is simply that you’ve got a lot of cards in your deck (50), so as long as one player doesn’t completely neglect to draw cards odds are good you’ll be able to pull something out to thwart at least some of your opponent’s schemes.

Additionally, because the game is so tactical, if one player gains a sufficient advantage in cards on the board, the end can be effectively decided long before the third idol falls. The card pool helps alleviate this somewhat by offering several things that can unexpectedly flip the game your way (creatures that attack the turn they come out, low amounts of distributed direct damage, etc.), but at least so far most of the games I have played have had a very clear winner by the mid-game and destroying the final one or two idols is mostly busywork as the winning player waits for their cooldowns to drop, which makes the game feel even longer.

Asynchronous play

Scrolls is fully asynchronous; on your opponent’s turn, you will spend your time passively watching whatever they decide to do. On yours, they’ll return the favor. This is extremely common in online CCGs, of course (playing Magic online takes forever, for instance, because both players have to be constantly opting out of responses whenever one player does something; in an asynchronous game this is a non-issue). However, it also reduces your investment in the game because if your opponent wants to take forever pondering the many decisions available to them, you might as well go out to coffee in the meantime.

Plus so far as I can tell, you can’t participate in anything except real-time matches, and the ability to participate in more leisurely games is about the only main upside to asynchronous play, so far as I am concerned.

Putting the “T” in TCG

While designing Scrolls, Mojang clearly was very interested in promoting a trading culture. There are basically three ways to get cards:

  1. Random singletons and packs (buying in a pack guarantees a seven commons, two uncommons, and one rare, whereas singletons are completely luck of the draw) for in-game gold. You get gold for playing matches, be they against other people, the AI, or the special “trials” that pit you against themed opponent AI (you can also “sell” cards back to the system, for a very small return)
  2. Six non-random cards that refresh weekly (two of each rarity) for either gold or “shards” (shards are purchased with actual money)
  3. Trading with other players

Of course you’ll notice that it is not possible to “pay to win” since the bulk of your cards will be purchased with gold (unless you can trade shards for gold with other players, which I have not investigated). This is good, because everyone is theoretically on an even playing field where your card collection is based more on how much time you’ve spent on the game than how much disposable income you have. But it is also bad, because it means that as the game gets older, it will be harder and harder for new players to collect a competitive set of cards without spending truly phenomenal amounts of time, either playing matches for gold or engaging in trades.

For me, this is a solid black mark against Scrolls because spending a bunch of time and energy trading has never been something I am interested in (not to mention the difficulty in figuring out what cards are worth, which requires up-to-date knowledge of the market and meta).

Idoling the time away

Coming away from my first brush with Scrolls, I get the feeling that this is a game that has been released at the wrong time in my life. I suspect that had it been release when I was younger, I would have loved it to pieces and spent far more time and money obsessing over it than I should. However, as I have grown older, my patience with games that demand huge chunks of my time in order to succeed has dwindled. I dislike arbitrary rarity schemes and the massive investment in time trading that they require to accrue a decent collection.

Oddly, although I find Scrolls’ theme to be one of the more unique fantasy worlds I’ve come across in a CCG recently, it ultimately leaves me flat. The idea that a bunch of people are wandering around with a giant collection of magic scrolls on their back just waiting to throw down is even less believable than things like Pokémon (which is saying something). Although the energy faction has some really quirky, interesting creatures, growth and order both feel very ho-hum to me (“wolves, and Vikings, and ducal liegemen…oh my?”).

In short, Scrolls demands a large chunk of my time, but the payout simply does not seem worth it. I will probably continue to try it on and off, and keep a general eye on its development, but at the moment it feels like something to idle away the time while I wait for Hearthstone or one of the other upcoming digital CCGs to fill the hole in my life left by Magic.

Goodbye, Nal

A little under three years ago, my wife and I decided that we wanted a pet, but unfortunately while she loves cats I am deathly allergic. I suggested rats, but they live such short lives that she didn’t want to go that route. We settled finally on a hedgehog, since based on our research it looked like they could live up to five to seven years. After being on a waiting list for several months, we were finally able to go claim our little girl: an African pygmy hedgehog we named Nal (which means “needle” in Norwegian).

The day we brought Nal home, she was not a happy hedgehog. Taking her out of the cage for the first time was a trick (we had to wear gloves, since we hadn’t figured out the art of hedgehog handling yet), and after we got her out she sat in my wife’s lap for a good fifteen minutes, balled up, hissing, and clicking:

Nal's first outing

We tried to tempt her with mealworms, but she was having none of it! Being in a new place with strange new people and smells was absolutely not alright by her. After a long wait, though, she finally uncurled and we started to get to know her properly. By the time we put her back in her cage, she was still tentative about us, but willing to let us see her face:

Nal uncurls

Ever so slowly, Nal started to get used to us. She absolutely refused to be out and about in her cage with us in the room, so we never got to see her run in her wheel (although she pooped all over it enough times that we knew she enjoyed it), but after a year and a half or so she started coming out of her how-dare-you-just-picked-me-up funk quicker and quicker. For the past several months, she sometimes would come out of her cage without even hissing, although she always expressed her displeasure as we took her house off her.

She particularly opened up once we bought our house. For some reason, the move to the new house really made Nal happy, even though it also coincided with her coming down with mites, causing her discomfort and some quill loss (thankfully easily treated, despite a traumatic visit to the vet; Nal may have grown comfortable with us, but other people touching her was not okay in the least).

Nal sleepingNal, we discovered, was possessed of a strong personality. She loved to explore (her jaunts in the out of doors were things she particularly enjoyed), would tolerate petting if we provided a place for her to hide her face, and vastly preferred mountain climbing up human legs to being picked up. She hated baths, and would burrow under any scrap of fabric she could find:

Nal burrowing

Nal anointingFor quite a while we thought the only treat Nal truly enjoyed was any bit of paper or cardboard that she could find and would then self-anoint with. However, almost purely by accident we also discovered that she would go absolutely ga ga for were bits of chicken, fresh corn, peas, or sunflower seeds:

Nal chowing down

Unfortunately, a few months ago Nal had a couple of seizures while we had her out, and when we took her to the vet they x-rayed her and told us it looked like she had a tumor. We started her on a regiment of meds twice a day, but the vet told us we did not have much time left.

Nal would hear none of it, though. We never saw her have another seizure, and she was a perfectly happy hog for long after the vet told us we could expect her to pass on. As a result of taking her out twice a day instead of just once, she grew more comfortable with us than ever and we started developing different routines (I would often serve as jungle gym and preventer-of-escapes-under-the-couch as she ran around on the floor, while my wife would spend more time cuddling with her).

Sadly, about a week ago Nal started to lose her balance, drastically dropped weight, and day by day lost control of her limbs. She finally stopped eating yesterday, and today I came into the office to find her on her side squeaking in distress; it was the first time I had ever seen her out in her cage when I got up in the morning, and the first time I’d ever heard her vocalize that way in distress.

Though betrayed by her body, Nal remained cantankerous and determined to face life on her own terms to the end. We will miss you dearly, Nal. Rest in peace.

Nal

Introducing my book log

I am pleased to introduce a new member to my little family of personal websites:

Ian Beck's Book Log
http://log.beckism.com/

It includes short reviews and complete plot summaries for the books that I am reading, with the goal being to post them as I read them. I created the site for a couple reasons:

  1. I read a lot of science fiction and fantasy, which means I read a lot of sequels. However, except for series that I have re-read countless times because I love them so much, I tend to forget what has happened. My book log is thus a resource for myself to look up the plot summary for books that I don’t want to re-read before I proceed in the series.
  2. I like sharing my opinions about books, but have never found a place conducive to doing so regularly (I’ve done it on this blog in the past, but since I wanted a resource for all the books I’m reading, it would have just polluted Beckism.com and caused me no end of trouble when it came to searching for old books).

In any case, if you enjoy science fiction or fantasy, check it out! Even if not, take a gander; it’s one of my nicer-looking sites, and I’d love to hear what you like (or don’t) about it.

I’m planning to post about some of the technical problems I ran into developing the site and how I addressed them, but I’ll leave that for another day. In the meantime, enjoy a couple of recommendations for great books, and some scathing commentary on the not-so-great ones!

Whither webOS?

My house is littered with webOS devices I no longer use.

My Pre+, the phone that hooked me on webOS as a user and later tempted me into app development, lies facedown on top of a Pre 2 that I received to let me test my app’s usage of webOS 2′s functionality. Nearby a TouchPad gathers dust, its battery long since run down to nothing because I use it so infrequently that moving it the two feet to its inductive charger is simply not worth it. The second TouchPad that I ended up owning through a quirk of fate I finally gave away to a friend who was interested in hacking on app development in his spare time. An inductive “touchstone” charging station lies abandoned on the floor nearby, banished from my desk when I installed my Kangaroo standing desk on top, and several webOS-related charging cables that I used for traveling and as backups are scattered nearby.

When I switched to webOS from my old iPhone, it felt like I was using the future. The inductive charging and reliance on cloud accounts for contacts, calendars, and email permitted me to be have a truly cord-free phone (something that the iPhone still has not accomplished for me, mainly thanks to Apple’s clumsy insistence on iCloud as the One True Cloud Account while paying little more than lip service to alternatives). The card metaphor for switching between apps was not only ridiculously simple to learn, but a joy to use. The interactions and design were reminiscent of iOS enough to be familiar, but with a unique approach that was internally consistent and addressed a lot of the niggling little things I disliked about iOS.

Yet despite how much I loved using and developing for webOS, HP’s vicious mismanagement of the platform forced me to the greener pastures of an iPhone, which was eventually joined by a retina iPad. For me webOS has transitioned from an awesome glimpse of the future into a nostalgic bit of the past.

Not everyone has moved on from webOS, of course. Every so often I receive an email asking for help with my webOS app TapNote; there remain a scattered few of the webOS faithful who have not yet given up hope in the platform. One such user recently asked me if I thought webOS was truly dead, or if I saw any hope of it succeeding now that HP has open sourced it.

This is a tricky question to answer, because while I think webOS as a platform may still have a future I do not think it is a future that is conducive for commercially-minded developers like myself. I knew developing for webOS was a bet with long odds when I first got into it on the Pre+. Now those odds are so long as to be astronomical.

Development requires a user

There are two main situations where it is worth devoting time to developing an app:

  1. You think there are enough prospective users who will buy the app to make your efforts worthwhile
  2. You want to use the app yourself, regardless of sales

Any other reason is going to lead to a poorly maintained app that doesn’t sell well (and if it’s a “scratch my own itch” app, it might not sell well regardless).

For those developers still using webOS, the second motivation can still apply, of course; webOS is ridiculously simple to develop for compared to other platforms (particularly if you have any Javascript experience behind you).

However, the user base on webOS is currently stagnant and declining, and with no new commercial hardware anywhere on the horizon that is unlikely to change in the near future. With no new users and an existing userbase that is not buying a lot of apps, there is not enough financial incentive for developers like myself to devote time and effort to the platform.

Software requires hardware

I think a lot of people’s hopes for a webOS resurgence rest on the dream that a third party will take the open sourced webOS and use it to power their awesome, cutting-edge hardware.

However, outside of some bargain-bin quality tablets or similar, this is unlikely to ever happen. The problem webOS faces is that at this point it is so far behind iOS, Android, and Windows Phone in terms of features that it would take a truly prohibitive amount of work for anyone to make it competitive. As such, it makes very little sense for a commercial entity to license webOS or otherwise use it on their hardware when they could instead use Android or just roll their own feature-light option (which would not saddle them with the downsides that are inherent to webOS, such as serious performance issues without highly optimized hardware/software setups).

It’s true that HP is continuing to develop webOS, but since they discontinued the TouchPad they have been doing very little more than running in place, as far as the external world is concerned. I’m sure that open sourcing it took a whole heck of a lot of work, but the end result is a codebase that drops support for their own hardware (thus effectively consigning the TouchPad to irrelevance even faster) and offers nothing new in the “Community Edition” webOS that TouchPad users continue to leverage.

So what we have is a mobile OS with very little to recommend it to commercial entities because it does not offer anything substantially new anymore (since a lot of the unique webOS contributions to the field like notifications and cards have been mimicked or adapted for the other OSes), is still plagued by performance issues in basic features like scrolling that simply aren’t an issue on the other platforms, and is not compatible with the hardware that the few webOS faithful continue to use.

I would love to be proven wrong on this, of course, but it seems to me that a me-too device mimicking the iPad (or mimicking the loads of Android tablets that have in turn mimicked the iPad) will not sell well, regardless of what operating system it runs. This is why Microsoft is trying to frame their Surface tablet as something different from the current batch of tablets (whether they implemented their difference effectively and can monetize it remains to be seen).

The chimeric promise of open source

“But wait!” says the die-hard webOS faithful. “Now that webOS is open source (or nearly so), anyone can use it!”

Which is true, for a given definition of “anyone”. And that of course is the crux of the matter: who will use webOS now that HP has open sourced it?

In the worst-case scenario, open sourcing the platform will have little to no impact. A few people will play around with it, but then inevitably be drawn to the better-maintained and faster-moving Android. In this scenario, webOS quickly fades to complete irrelevance in the short-term future, and as devices start to fail (figuring on a two to four year window for this, with some statistically irrelevant outliers) the existing userbase quickly erodes down to nothing. HP eventually will quietly drop the project all together, or else retire it into a back room where they never have to think about it again except as a rounding error on their budget.

I hope that doesn’t happen, but all too many prior software projects have languished into complete obscurity in my lifetime for me to discount the likelihood.

In the best case scenario, HP’s continued efforts to open source webOS in the short term are a rocky transition period, and then the platform is more widely taken up by hackers and do-it-yourself-ers, likely installed primarily on hardware intended for Android. It might well gain a foothold in the education and research sectors, too, where quick and easy deployment of code can be more important than speedy UI performance and tight budgets require people to wear both researcher and programmer hats.

Yet even in this scenario, webOS is largely a dead-end for developers like myself because none of these crowds are likely to want commercially distributed software. People comfortable enough with open source to install a mobile OS on a nonstandard device are typically familiar enough with open source’s user-unfriendly interfaces to the point that they would rather find an open source alternative instead (or write one).

What this means for me

I am still trying to figure out where to take TapNote from here, but at least in the short term it doesn’t make sense to pour more development time into it since webOS is transitioning towards a future where I am unlikely to be able to be compensated for my efforts. If I am going to move TapNote forward, I need to find a way to make it stand apart from the crowd of Dropbox text editors that litter iOS and Android.

This makes me really unhappy, since to this day I love webOS and had a blast writing TapNote (which I mostly wrote to scratch my own itch; I never expected it to make much money). Hopefully webOS will find a place in the world, but at the moment I am pessimistic about its immediate future.

Archiving tweets using IFTTT and Dropbox

Update (Sept. 28, 2012): the method for archiving tweets using IFTTT and Dropbox describe here no longer works thanks to Twitter cutting off IFTTT’s access for anything except posting tweets to Twitter. I am looking into alternatives, but don’t know of any drop-in replacements currently.

Justin Blanton recently posted an approach to archiving tweets using plain text and Dropbox. In short, he’s using IFTTT.com (also known as If This Then That, a service that allows you to setup triggers and actions for events in common web services) to append every tweet to a plain text file in his Dropbox account.

In turn, Brett Terpstra took Justin’s IFTTT recipe and modified it to use Markdown formatting.

Now I have added my own spin to the idea by creating a script that I run via Hazel to automatically break the tweets into files by month. You could, of course, run the script using some other method; I just prefer the ease-of-use of Hazel.

Setting up

For this to work, you need three things:

  1. The IFTTT recipe
  2. The archiving script (also available inline below)
  3. A Hazel rule to pull everything together (or some other way to automatically invoke the script and pass it the filename for your initial archive file)

When you have those three things in place, shortly after you publish a tweet it will be appended to a plain text in Dropbox by IFTTT, then subsequently sorted into archival files by month by the archival script. The script also (optionally) expands Twitter’s shortened t.co links into the actual URL you posted.

For those who want a little more hand-holding, here’s specifically how to get all the various pieces lined up.

IFTTT configuration

You need to change a couple things in the IFTTT recipe to make it work for you. In particular, the default folder path (ifttt/twitter) is very uninspired. You also need to change the name of the file to your Twitter username. If you want, you can use a different file extension (like .md).

(Note that it’s entirely possible to archive multiple Twitter accounts using this method, but you will likely need multiple IFTTT accounts; so far as I know it is not possible to link multiple Twitter accounts to a single IFTTT account.)

Once you’ve got the recipe activated in your IFTTT account, post a tweet and make sure that it is showing up in your Dropbox (should happen within 15 minutes, or you can run the IFTTT recipe explicitly).

Archival script

Setting up the archival script will require a little bit of command-line work, but nothing too scary. To get started, you can download the script from GitHub, or create a file called archive-tweets.py in your favorite text editor and copy and paste:

#!/usr/bin/python
# -*- coding: utf-8 -*-

'''
This script parses a text file of tweets (generated by [IFTTT][1],
for instance) and sorts them into files by month. You can run it
manually from the command line:

    cd /path/to/containing/folder
    ./archive-tweets.py /path/to/@username.txt

Or run it automatically using [Hazel][2] or similar. The script
expects that you have a file named like your Twitter username with
tweets formatted and delimited like so:

    My tweet text
    
    [July 04, 2012 at 06:48AM](http://twitter.com/link/to/status)
    
    - - -

And that you want your tweets broken up by month in a subfolder next
to the original file. You can change the delimiting characters between
tweets and the name of the final archive file using the config variables
below.

By default, this script will also try to resolve t.co shortened links
into their original URLs. You can disable this by setting the 
`expand_tco_links` config variable below to `False`.

   [1]: http://ifttt.com/
   [2]: http://www.noodlesoft.com/hazel.php
'''

# CONFIG: adjust to your liking
separator_re = r'\s+- - -\s+'     # IFTTT adds extra spaces, so have to use a regex
final_separator = '\n\n- - -\n\n' # What you want in your final montly archives 
archive_directory = 'archive'     # The sub-directory you want your monthly archives in
expand_tco_links = True           # Whether you want t.co links expanded or not (slower!)
sanitize_usernames = False        # Whether you want username underscores backslash escaped

# Don't edit below here unless you know what you're doing!

import sys
import os.path
import re
import dateutil.parser
import urllib2

# Utility function for expanding t.co links
def expand_tco(match):
	url = match.group(0)
	# Only expand if we have a t.co link
	if expand_tco_links and (url.startswith('http://t.co/') or url.startswith('https://t.co/')):
		final_url = urllib2.urlopen(url, None, 15).geturl()
	else:
		final_url = url
	# Make link self-linking for Markdown
	return '<' + final_url.strip() + '>'

# Utility function for sanitizing underscores in usernames
def sanitize_characters(match):
	if sanitize_usernames:
		return match.group(0).replace('_', r'\_')
	else:
		return match.group(0)

# Grab our paths
filepath = sys.argv[1]
username, ext = os.path.splitext(os.path.basename(filepath))
root_dir = os.path.dirname(filepath)
archive_dir = os.path.join(root_dir, archive_directory)

# Read our tweets from the file
file = open(filepath, 'r+')
tweets = file.read()
tweets = re.split(separator_re, tweets)
# Clear out the file
file.truncate(0)
file.close()

# Parse through our tweets and find their dates
tweet_re = re.compile(r'^(.*?)(\[([^\]]+)\]\([^(]+\))$', re.S)
# Link regex derivative of John Gruber's: http://daringfireball.net/2010/07/improved_regex_for_matching_urls
link_re = re.compile(r'\b(https?://(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".,<>?«»“”‘’]))', re.I)
dated_tweets = {}
for tweet in tweets:
	if len(tweet) > 0:
		# Parse our tweet
		matched_tweet = tweet_re.match(tweet)
		# Replace t.co links with expanded versions
		sanitized_body = re.sub(r'@[a-z0-9]*_[a-z0-9_]+', sanitize_characters, matched_tweet.group(1))
		formatted_tweet = link_re.sub(expand_tco, sanitized_body) + matched_tweet.group(2)
		# Grab our date, and toss the tweet into our dated dictionary
		date = dateutil.parser.parse(matched_tweet.group(3)).strftime('%Y-%m')
		if date not in dated_tweets:
			dated_tweets[date] = []
		dated_tweets[date].append(formatted_tweet)

# Now we have our dated tweets; loop through them and write to disk
for date, tweets in dated_tweets.items():
	month_path = os.path.join(archive_dir, username + '-' + date + ext)
	# Construct our string with a trailing separator, just in case of future tweets
	tweet_string = final_separator.join(tweets) + final_separator
	# Append our tweets to the archive file
	file = open(month_path, 'a')
	file.write(tweet_string)
	file.close()

# All done!

I like to save the archive-tweets.py file in my Dropbox right next to my @ianbeck.txt archival file (makes things easy to keep track of). If you have changed the formatting of the

Next, you need to ensure that the script can be executed. To do so, open /Applications/Utilities/Terminal.app. This example code assumes that you are using the default settings for the IFTTT recipe and have saved the script in the same Dropbox folder, so adjust the path as needed and then run this command in Terminal:

chmod +x ~/Dropbox/ifttt/twitter/archive-tweets.py

You should also create the folder where your monthly archive files will live. By default it should be called archive, but you can use something else if you want.

Lastly, you might want to modify some settings in the script. There are two things you might need to adjust:

  1. If you have modified the delimiter between tweets in the IFTTT recipe, you need to specify what you are using in the script
  2. If you do not want t.co links to be expanded, you need to disable that in the script
  3. If you want your monthly archives in a folder named something other than archive, you need to specify your preferred folder name

You can find the configuration variables on line 34, or search for “# CONFIG“.

Hazel workflow

Now that the main moving pieces are in place you need to setup your Hazel workflow to automatically run the script every so often. Here’s what mine looks like:

Hazel setup

The important bits are having the name start with the “@” symbol, making sure that the subfolder depth is less than 1, and sticking a minimum file size in there (to make sure the script doesn’t get endlessly executed when the file is empty).

Why am I doing this, anyway?

To be honest, most people will probably not care about the fact that Twitter only allows you to access your 3200 most recent tweets. For most of us, tweets are ephemeral; you post it, your friends read it, and that’s that. If you have something you think is particularly clever or worth saving, you can mark it as a favorite and access it whenever you like.

For myself, though, I like having a record of the things that I write, even if it’s something stupid like, “Wow, my balaclava is particularly itchy today.” Why? Because every so often, I remember tweeting something that I need to reference (a link, a prior opinion, etc.), and searching Twitter always fails me. With the above archival setup in place, though, I can easily search for it using the tools built into my computer, and since the archive files are plain text they are as future-proof as I can get, extremely easy to work with, and won’t take up much space in my Dropbox.

Whether having access to your tweets down the road is important to you or not is something you’ll have to decide on your own. If it is, though, this is a pretty easy way to save them despite its geeky underpinnings.

Updates and corrections

July 5, 2012

Dr. Drang points out that @hugovk is the original creator of the IFTTT Twitter-to-Dropbox workflow, not Justin Blanton as I originally thought. Additionally, Dr. Drang offers a script for converting past ThinkUp databases of tweets into plain text format.

Brett Terpstra meanwhile has been hacking away with both my script above and Dr. Drang’s script, and currently has his modifications available on GitHub. I expect he will post his final workflow to his blog once he has them finalized. Good stuff.

July 6, 2012

I have updated the shell script (modified version is on GitHub or inline above) and added the following:

  1. All URLs are now converted to self-linking URLs for ease of parsing as Markdown: <http://wherever.com>
  2. Underscores in usernames are optionally escaped with a backslash in order to avoid italicizing: @some\_name (this is off by default, but you can enable it in the CONFIG section; I just have a friend named @_squark_ and was getting annoyed at his italicized name)

Note that neither of these changes is retroactive, so you will have to modify your existing archive file(s) if you want consistency.

Shuffling files around with Dropbox

One of my abiding problems is how to easily and quickly transfer files between my two computers. For the past several years, I have used an iMac as my work computer, and a MacBook Pro for my personal computer. (This might seem silly to some people since I work at home, and the two computers literally live about six feet away from one another most of the time, but I can’t overstate how much this helps my sanity.)

Recently, the problem with moving files around has been exacerbated because my iMac is stuck running 10.6, and I need 10.7 to test Slicy, so I finally decided to hack something together to make life simpler. (I’ve debated many a time upgrading the iMac to 10.7, but given how bad a performance hit my newer and better-equipped-in-the-RAM-department laptop took when upgrading, there’s no way I would be able to squeeze acceptable performance from the aging iMac.)

After looking at various options, I ended up creating a little workflow using Dropbox and Hazel, because both are tools that I already use. My goal was to create a way to move files to a different computer with a single action without needing the other computer to be awake or on the same network, and without using up my Dropbox storage quota on storing temporary files.

A disclaimer: the following workflow requires two to three bits of software, all of which will cost you if you aren’t already using them: 1) Hazel ($25 at the time of this writing), 2) a Dropbox account with sufficient empty space to move arbitrarily-sized files around (free if you manage to get a ton of referrals, otherwise $100 a year), and 3) optionally FastScripts ($15 at the time of this writing) or some other way to quickly execute an AppleScript. Oh, and a Mac. You could do the same thing on Windows or Linux, but it wouldn’t be as magical without Hazel (unless you could find some Windows or Linux equivalent app).

Setting up

The first step was easy enough; in Dropbox I created one folder for my laptop and one for my iMac (“To Laptop” and “To Desktop”, respectively). Since I rarely access these folders directly, I hid them a ways down in the folder hierarchy in Dropbox to keep them out of the way.

The second step was to setup a Hazel workflow on each computer targeting that computer’s folder. Here’s the laptop’s version:

Hazel workflow

Basically, any time it finds a file in the “To Laptop” folder on Dropbox, it immediately moves it to the Desktop and colors it blue so that I’ll notice it more easily. (You can, of course, move the file anywhere you liked; the Desktop is just convenient for how I work.)

Third, I wanted to be able to send files between computers with a keystroke, so I hacked together a quick Applescript that would take the selected file(s) in the Finder and duplicate them to my target Dropbox folder:

-- CONFIGURE: Set this path to your target folder
set targetDropboxFolder to POSIX file "/Users/MYACCOUNT/Dropbox/Sync/To Laptop/"

tell application "Finder"
	set targetSelectedFiles to selection
	repeat with activeSelectedFile in targetSelectedFiles
		duplicate activeSelectedFile to targetDropboxFolder with replacing
	end repeat
end tell

To use this script, create a new script in AppleScript Editor (/Applications/Utilities/AppleScript Editor.app), paste in the code above, adjust the path to your target Dropbox folder, and save it either in ~/Library/Scripts/ or one of its subfolders.

Personally, I use FastScripts to associate a keyboard shortcut with the script, but there are innumerable other ways to quickly access AppleScripts. Alternatively, you could just create an alias to your “To Other Computer” folder, stick it on the Desktop or somewhere else easy to get to, and then drag and drop things you wanted to move across (just make sure to hold down the option key when you do, or the file will be moved to the other computer rather than copied).

Wishing for an easier way

In a perfect world, I would write an app to take care of this stuff for me instead of relying on a bunch of third-party apps and services, but I don’t really have time to devote to the concept. Ideally, I would prefer not to need to always route through the cloud; if I am transferring a file, and the target device is on the same local network, the file should just be moved across using the local network. For lack of a more elegant solution, though, this workflow functions well, was quick to setup, and has been making me happy. Hopefully it will help a few other folks, too, or at least sparks some ideas for easier transferring of files between computers.

Clicky Web Analytics