Wednesday, December 19, 2012

303

I've discontinued using Blogger.com in preference for my personal blog.

New posts can be found at www.jamesrhys.net/blog

Sunday, November 4, 2012

RabbitMQ - The best friend you never knew

Bin: RabbitMQ Client / Server
Env: Mac OSX (available to all env)
Desc: RabbitMQ

Summary

I've used RabbitMQ for quite a while without actually knowing what it was. It came as the default
message queuer on Celery. All along I thought Celery was the juice but after spending an afternoon debugging a messaging protocol we'd written for AWS, I was surprised to learn that I wasn't taking advantage of a queuer correctly. RabbitMQ is O/S, has primary language bindings in Python and Java (my 2 fav langs for Enterprise), is stable and kicks a whole lot of ass.

Installation instructions for RabbitMQ

This tute is mainly for OSX users but as it's the defacto standard, the same commands and set up should work on all environments.

1) Install RabbitMQ and Erlang (if you don't already have the best lang for doing multi-thread programming)

sudo port install rabbitmq-server
sudo port install erlang

2) Path

One big problem with macports is that the bin installs are frequently all over the place. So I added the following line to my .zshrc (if you don't use zsh or use bash or don't know what I'm talking about add the following lines to ~/.bashrc)

PATH=$PATH:/opt/local/lib/rabbitmq/bin
PATH=$PATH:/opt/local/lib/erlang/bin

and ensure you reload your conf

source ~/.zshrc

3) Run

sudo rabbitmq-server -detached

4) Make sure it's running

ps -A | grep -i "rabbit"

83724 ??         0:01.68 /opt/local/lib/erlang/erts-5.9.2/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /opt/local/lib/erlang -progname erl -- -home /Users/jrm -- -noshell -noinput -sname rabbit@Exhibit-E -boot /opt/local/var/lib/rabbitmq/mnesia/rabbit@Exhibit-E-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/opt/local/var/log/rabbitmq/rabbit@Exhibit-E.log"} -rabbit sasl_error_logger {file,"/opt/local/var/log/rabbitmq/rabbit@Exhibit-E-sasl.log"} -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/opt/local/var/lib/rabbitmq/mnesia/rabbit@Exhibit-E" -noshell -noinput

Note the hostname: 

Intro to RabbitMQ as a queuing service

RabbitMQ is a message broker. The principal idea is pretty simple: it accepts and forwards messages. You can think about it as a post office: when you send mail to the post box you're pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman. The major difference between RabbitMQ and the post office is the fact that it doesn't deal with paper, instead it accepts, stores and forwards binary blobs of data ‒ messages. RabbitMQ, and messaging in general, uses some jargon.

Producing means nothing more than sending. A program that sends messages is a producer. We'll draw it like that, with "P":



A queue is the name for a mailbox. It lives inside RabbitMQ. Although messages flow through RabbitMQ and your applications, they can be stored only inside a queue. A queue is not bound by any limits, it can store as many messages as you like ‒ it's essentially an infinite buffer. Many producers can send messages that go to one queue, many consumers can try to receive data from one queue. A queue will be drawn as like that, with its name above it:
 

Consuming has a similar meaning to receiving. A consumer is a program that mostly waits to receive messages. On our drawings it's shown with "C":


Note that the producer, consumer, and broker do not have to reside on the same machine; indeed in most applications they don't.


Producer sends messages to the "hello" queue. The consumer receives messages from that queue.

Uses of RabbitMQ as a queuing service

If you're new to queueing services, I suggest just setting up a simple to-do list with Celery. Once you're up to speed, here are a few things I use queueing services to perform:

1) Nightly rotations / BI / parsing for logs. It's a lot friendlier than running a script in nice mode or arbitrarily setting a cron execution
2) Distributing tasks amongst workers for continuous processing of blobs
3) RPC's

Sunday, October 7, 2012

Scripting Layouts in Tmux for Mac

Yes, I'm on the band wagon.

Tmux should be the default terminal multiplexer, replacing Screen. It should be viewed as a knock-down-rebuild of the standard, supporting local and remote sessions, fast context switching, key-binding and layout scripting.

I've found it hard to find a decent guide on scripting your conf file, so this post was born. As usual with Unix-compliant terminal tools, it sees itself primarily as a GNU/BSD/Linux application so when working with the Mac platform you need to make a few tweaks to your core conf file.

For the uninitiated, most terminal tools draw their configurations from a dotfile, Tmux's is ~/.tmux.conf.
For the sysadmins (we're looking for one if you like Macs and LAMP (not MAMP)), the global conf files are in /etc/tmux.conf.

The rest of the blog will just walk you through my conf file, noting points for extension where important. My file is the child of an orphaned conf file from the Gentoo lib and the ArchLinux entry, both are valuable references for the advanced user.

  As a preface to the exposition, here are some key points to keep in mind:

  • The Tmux command key is re-bound to the back-tick ( ` )  and the screen access keys are re-indexed to 1. This one pedestrian directive will have your hands wringing in ecstasy. 
  • I have several environments set up and each is atomic. Whilst my work primarily involves LAMP development, my free time frequently involves Python and Java development. Whilst I won't go through the Python and Java set ups as I'm unhappy with the way they're organized and show far too much fluidity across projects, I will show you how to separate your development environments like Eclipse Workspaces.
  • My LAMP environment is very specific. I use my first screen as a focus for debugging and environment issues. It contains tailed Apache and PHP logs, a mysql and bash prompt and a small Vim editor for accessing host files and environment variables. If I weren't so opposed to live edits on development and production servers that aren't channeled through source control / continus integration services I'd have window #2 as a dedicated SSH window.




    Subsequent screens are used as file-oriented slaves. They primarily contain full-screen Vim editors but given how flexible sessions are in Tmux, they can contain bash prompts or tailed logs where relevant.
  • Colour, the most important attribute for some people, should be seen as uniquely specified within tmux. Instead of the GNU approach where you're supposed to memorize hex/octal combinations like it makes sense, Tmux uses the colorXXX scheme. To see all the available colors, run this as a bash script within tmux itself:

#!/bin/bash
for i in {0..255} ; do
    printf "\x1b[38;5;${i}mcolour${i}\n"
done

          All props to cYrus for that one. For the pragmatic, here's a quick and dirty screenie of some popular colors. Please note that in order to test the colour schemes you'll have to stop all instances of tmux

If you're looking for transparent colour schemes, you'll need to run the script for yourself. Tmux gets sticky (not it's fault!) in OS X 10.8 so you'll probably have to manually kill threads when opening new consoles and not seeing your changes.




          P.S - It's so awesome to see colour spelt correctly!

          As for what colors I've chosen, well that's my poor taste innit?

  ## `prefix
  set-option -g prefix `

  #### COLOUR
  set -g default-terminal "screen-256color"

  # default statusbar colors
  set-option -g status-bg colour235 #base02
  set-option -g status-fg colour136 #yellow
  set-option -g status-attr default 10 

  # default window title colors
  set-window-option -g window-status-fg colour244
  set-window-option -g window-status-bg default
  #set-window-option -g window-status-attr dim 15 
  # active window title colors
  set-window-option -g window-status-current-fg colour166 #orange
  set-window-option -g window-status-current-bg default
  #set-window-option -g window-status-current-attr bright
  
  # pane border
  set-option -g pane-border-fg colour235 #base02
  set-option -g pane-active-border-fg colour240 #base01
  
  # message text
  set-option -g message-bg colour235 #base02
  set-option -g message-fg colour166 #orange
  
  # pane number display
  set-option -g display-panes-active-colour colour33 #blue
  set-option -g display-panes-colour colour166 #orange
  
  # clock
  set-window-option -g clock-mode-colour colour64 #green
  
  # Allows use of hotkeys for resizing panes
  bind-key + resize-pane -D 3
  bind-key / resize-pane -L 3
  bind-key - resize-pane -U 3
  bind-key * resize-pane -R 3
  
  # set-option -g default-terminal "screen-256color"
  set-option -g mouse-select-pane on
  set-option -g status-keys vi
  set-option -g bell-action any
  set-option -g set-titles on
  set-option -g set-titles-string '#H:#S.#I.#P #W #T' # window number,program name,active (or not)
  set-option -g visual-bell on
  # Visual styles
  
  set-option -g pane-active-border-fg green
  set-option -g pane-active-border-bg black
  set-option -g pane-border-fg white
  set-option -g pane-border-bg black
  
  set-option -g message-fg black
  set-option -g message-bg green
  
  #setw -g mode-bg black
  
  setw -g window-status-bg black
  setw -g window-status-current-fg green
  
  set -g status-left '#[fg=red]#H#[fg=green]:#[fg=white]#S #[fg=green]][#[default]'
 
  set -g status-right '#[fg=green]][#[fg=white] #T #[fg=green]][ #[fg=blue]%Y-%m-%d #[fg=white]%H:%M#[default]'
  #set -g status-right '#[fg=green]][ #[fg=blue]%Y-%m-%d #[fg=white]%H:%M#[default]'
  
  # 0 is too far from ;)
  set -g base-index 1
  


Monday, May 21, 2012

SVN E2000031 / E155004


Got these errors in the lab today:

$ svn ci -m "Updating repo" .

svn: E155004: Working copy '/mnt/restore/var/siteupdate/svn/adelson' locked
svn: E200031: sqlite: attempt to write a readonly database

Over the weekend, a co-worker updated our SVN repo's for a legacy application from 1.6 to 1.7. Through that process, he ran:

ls repo_directory | xargs `svn upgrade`

The default behaviour of this script changed the permissions on all child files of the repo, so when I came along and tried to check in changes, I got the permissions issues on the .svn files for each repo and the system-wide /var/svn/libs/db/rep-cache.db changes required through the SQLite dependency.

TL;DR

Check the user and group permissions on the .db files for the repo (in .svn/wc.db) and server-wide (in /var/svn/libs/db/rep-cache.db) and ensure that your user and group has the perms required to write to them.

Saturday, April 7, 2012

Brooklyn's Suburbs

For anyone out there looking to move to Brooklyn, here's a map of Brooklyn's suburbs.


I've got two more appointments lined up for tonight: Flatbush and Williamsburg...wish me luck!

Sunday, April 1, 2012

Settling into the new role

So, it's April. The cherry trees are spreading snow flakes across side-walks and sewers and I'm finally starting to break out of the frame at my new job. It's a great team with creative, hard-working, smarty pants and I think we're all learning new things each day.

That said, my O/S work has been atrocious. At the moment I'm commuting 3 hours every day. Whilst I spend most of that time asleep, it's leaving me shattered at night. I need to get into it. So, to start with, I'm going to write an interface for Aptana for bash scripting to try and get my boss back into it.

Monday, January 16, 2012

Dynamically Loading and Unloading Javascript using AJAX and JS - Gabblista

So it turns out there's really no good techniques for unloading javascipt files for Ajax apps. It's going to become very important in the very near future as we approach the Mayan Apocalypse and HTML5 continues to grow in popularity. The current solution is to search the DOM for all scripts (document.getElementsByTagName('script')) and if they match your declared namespace, remove them by calling .removeChilld.

When approaching state-driven application design (a very intuitive choice for old-school Haskell/functional programmers like myself), it's important to be able to manage resources dynamically. Here's how I did it for Gabblista.

1. Design Application States

I sketched up a rough application work-flow in Google Docs with the following states:

  • Intro - 1
  • Menu - 2
  • Statistics - 3
  • Pre-Game - 4
  • Play - 5
  • Post-Game -6
  • GLOBAL - 0

That way I can just create an enum to track application states and loading/unloading events.

2. Design  a resource management framework

Now that I've got a rough idea of what states I want to show, I can start to declare namespaces for MVC design. All my files relating to the intro will be prepended with an int designating their place as application states.





Screencap of my implementation of state-based resource management.
3. Script it

Heavily drawing upon these OLD articles (http://www.javascriptkit.com/javatutors/loadjavascriptcss2.shtml
http://www.bytemycode.com/snippets/snippet/537/) I was able to implement the following script for dynamically loading and unloading stuff


function removejscssfile(filename, filetype){
 var targetelement=(filetype=="js")? "script" : (filetype=="css")? "link" : "none" //determine element type to create nodelist from
 var targetattr=(filetype=="js")? "src" : (filetype=="css")? "href" : "none" //determine corresponding attribute to test for
 var allsuspects=document.getElementsByTagName(targetelement)
 for (var i=allsuspects.length; i>=0; i--){ //search backwards within nodelist for matching elements to remove
  if (allsuspects[i] && allsuspects[i].getAttribute(targetattr)!=null && allsuspects[i].getAttribute(targetattr).indexOf(filename)!=-1)
   allsuspects[i].parentNode.removeChild(allsuspects[i]) //remove element by calling parentNode.removeChild()
 }
}

removejscssfile("somescript.js", "js") //remove all occurences of "somescript.js" on page
removejscssfile("somestyle.css", "css") //remove all occurences "somestyle.css" on page


4. FAQ

I don't accept donations or job offers for my blog entries, but if you'd like to support the blog - just click an ad.

My dev stack consists of the following:

- A Python-based MVC web framework (take your pic)
- GAE (application hosting, data storage)
- GitHub (SCM, derrr)
- A fuck-ton of hand-crafted HTML5, CSS, JS (application logic)
- Adobe CS5 (audio and sound)