To add comments to the posts, you will need to signup for an openid account. There is a set of archives of all the older posts and also a tags cloud.

RSS Atom Add a new post titled:
Summer is here and its BBQ season

It's summer and we've been living in the house for almost a year now and I just got one of these things..., a charcoal kettle starter, got it for 13e from homebase and it now takes me less than 20minutes to start the grill. Being able to reliably start the grill means I can have a BBQ more spontaneously than before. Must watch more videos from http://www.disturbedcooking.com/

Note: I forgot to publish almost a year ago.

Interview with Dublin South FM

I recently participated in a documentary at Dublin South FM about people who have gone through the process of naturalisation. It was shorter than I expected it would be, but it seems they are doing a survey on about 40-50 people. I look forward to hearing the show.

Brewing my own beer!

I've finally made the move to brewing my own beer, Rob got me a Dusseldorf Alt beer kit, which resulted me purchasing enough equipment for brewing.

Pictures to come soon!

100% overnight jump in immigration fee completely unjust, sudden announcement comes without notice or consultation

A reader of this blog pointed me to the 100% overnight jump in immigration fee - I never thought anyone would be interested in the topic of naturalisation and migration to Ireland.

Seriously? 300euro? What gives? I'm at a loss to understand the reasoning for bumping up the fee with such short notice. A bulk of the GNIB card holders are going to be either international students (who are paying full fees, health insurance and are told to have a small sum of money lodged in their bank accounts) or highly skilled migrant workers whose work permits are capped at a "minimum pay level" which means they are probably in the medium to upper tax brackets.

Migrants who are hit with the same levies and taxes that a local is already paying are asked to stump up even more money? Perhaps there should be a divide in society there seems to be one already even though there shouldn't be one.

So the foreigners who are legally working or studying here are forced to carry (unconstitutionally as far as I know) GNIB cards which aren't supposed to be valid forms of ID. Irish people are not required to carry ID, so there is a bit of discrimination happening it's not clear to me what the rules are in this case, but it is clear that it's unfair.

Another kicker to the 300euro GNIB Fee for the GNIB card is the application fee for applying for Citizenship which is 175euro. There is also a 950euro fee for processing the application if is approved.

There's probably a hidden cost in solicitor fees as well for complications that might arise during the application process.

I'm not really complaining about having to pay the fees I agree about being sustainable and delivering a quality of service. The issue here is the lack of transparency, organisation and justification for some of the fees. It's sent me the wrong messages in the past and the system is really sending the wrong signal to migrants.


Related posts:

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

Loving the Brompton

Having cycled in and out of work from our new home for the past month I really don't miss getting the bus. It takes roughly 30mins door to door with my Brompton bike (roughly 6km cycle). Getting the bus means a 5min walk, 5min wait for the bus, 20-45mins on the bus depending traffic, then a another short 5-10min walk to the office. Or else its a 15min walk to the Luas, 20min journey on the Luas, then a further 10-15min walk. Either way public transport requires about an hour of commute time and cycling requires 30mins.

The bike isn't as slow as you might think, but it's not as fast as I would like it. It does however get me from point A to point B which is why I got it. At 30mins of a journey plus I get some exercise it really is a no brainer that should cycle.

The fact that the Brompton is small and folds up means that I can store it in the office or bring it into a supermarket (in one of the big trolleys. It's dead handy even if I do get strange looks at times with my bike in a supermarket.

I'm really getting to like the Brompton and cycling it, however I don't like Dublin roads. The roads around Dublin definitely aren't bike or cyclist friendly.

I think the bike was a good investment overall, I just need to get a proper bike bag and some wet gear next.

The second thing I did with my new Irish Passport (citizenship)

Life goes on with being naturalised and being Irish, it's only been a few months since naturalisation. I've been happily married, now a mortgage owner and with a relatively different job from my previous sysadmin/devops role.

I recently went to a conference in Edinburgh relating to Digital Preservation, Curation and Access - that is Open Repositories 2012 and I got a chance to use my new passport for travel. As good as being Irish is, the UK security network and paranoia just freaks me out, I had lots of fun with travelling back home to Ireland from the the UK and it was not pleasant at all.

So the second time I get to use my Irish Citizenship it ended in a fail with some security guy scanning my ass with a metal detector and giving me a secondary inspection. This was also my first time that I travelled with my new citizenship!


Related posts:

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

Sorting out my Irish passport

Now that I am naturalised, I reckon I should share what a first time applicant whose recently got naturalised needs to do to get a passport.

The application forms can be got from any garda station or postoffice (well most post offices have them). As I had previously mentioned, you will need 4 photographs (two of which need to be signed by your local garda). An original copy of your birth certiticate, a marriage certificate if you are married, your naturalisation certificate, proof of residence (a utility bill of some sort) and existing id to verify you are you.

Make sure you put your name on the passport application whilst making sure it is the same as what is on your naturalisation certificate. Make sure it matches and if like me if your name on your birth certificate is different it is worth ticking the box which makes a note of your birth name in your passport.

It takes about 20works days if you apply at the passport office like I did. I chose to do it at the passport office as I had questions to ask to make sure my application was correct. In hindsight I would have applied via the postal passport express service.

Something that I did do was make a photocopy of the naturalisation certificate and went to the local garda station to put a stamp and signature on the document to state that it is a verified copy. This was just in case if I lost the naturalisation certificate for whatever reason.

You can track the application at http://www.dfa.ie/home/index.aspx?id=2859, you'll get a code from the application so don't worry.


Related posts:

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

I'm now naturalised and Irish!

April the first was the day I got naturalised, according to the INIS website there was 4000 people at the ceremony. It was a mass ceremony, while I appreciated the whole ceremony and that the colours of the state and flags were paraded during the ceremony etc, I would have appreciated it even more if the event had some more Irish in it.

Even though I don't speak Irish, I would have appreciated if at least parts of the swearing in process could have been done bi-lingually in both irish and english.

Upon receiving my certificate I had to make sure everything was correct, and sure enough it was correct. The next step now is to get an Irish Passport then I'm good for travelling around europe with less hassle. I did come across one or two minor issues when I was sorting out my passport application form, I needed original copies of my birth cert, naturalisation cert, some proof of residence and address. Again the whole name thing did come up again. My advice for anyone in a similar position is that they make sure the name to appear on the passport be the same as the one that appears on the naturalisation certificate.


Related posts:

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

24 months and two weeks

So after 24 months and about 2 weeks later my naturalisation application has been approved. Not quite the 6 month wait that a new applicant would have to wait for.

Date Average wait time
2009-06-21 23 months
2010-02-28 24 months
2011-03-08 26 months
2011-05-11 25 months
2012-02-06 23 months
2012-02-23 24 months*

* What I waited for, 24 whole long months.

I have a few more things left to do before it is all official, such as pay the 950 euro fee to register as a citizen. Get an Irish (EU) passport and I will be done! Post paper work, I am going to make sure I register on the voting register. Now that I can make a change, I will at least make the attempt at participating in making changes for the better for this country.


Related posts:

I'm now naturalised and Irish!
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

Getting a Brompton on the Bike to Work scheme

There is a scheme in Ireland called the Bike to Work scheme. Which basically allows people to buy bikes to cycle to work, where the cost of the bike is basically tax free. I decided to avail of this scheme since I will soon be moving out of the apartment that me and my wife are currently living at soon (I hope). I'm preparing to commute into work via cycling (or walking more), I'm going to try and avoid public transport where possible. This is both to save time and money as the cost of public transport is ever increasing. It also means that I get more exercise, I used to do lots of cycling. I stopped, why did I stop? I discovered work!

Anyhow I ended up ordering (in the process of) getting a Brompton, specifically with the M-type handlebars, mud guards & rack, 6 speed gears and in the stock black paint work.

Basically I got a comfortable commuters fold up bike, I'll probably need to get a helmet and some lights when the bike gets ordered and shipped. The total cost was 1136e, it was 280e more than the Dahon Mu P8 which is equivalent to the Brompton that I spec'd up which was what I originally wanted.

I didn't get the Dahon as most of the retailers in Ireland seemed to be dropping the brand for some strange reason (I asked a few sellers, and it seems they are pulling out of the UK & Ireland markets for some reason). Hence Brompton is probably a safer bet. I'm really excited with this as I haven't owned a bike in ages and I've wanted a Brompton for a while now.

I wonder if it's because of the Dahon lawsuit that was happening last year and the company is just restructuring now which is causing all these weird issues.

2012-02 -- Update on long naturalisation times in Ireland

It's time to revisit the average wait times at INIS for citizenship applications.

Date Average wait time
2009-06-21 23 months
2010-02-28 24 months
2011-03-08 26 months
2011-05-11 25 months
2012-02-06 23 months

So I made my application back in early 2010. I had received acknowledgemnt that INIS had received my application around about this time two years ago, since then not much has happened on this front.

I've waited two years for my paperwork to be done. In that time the goverment has changed, I've gotten a few pay cuts and a new job, got married. By the way, the average wait time since I started recording the average wait times is now 24 months. Well it's an average of the averages.

According to the application processing times page on the INIS website, a number of measures have been introduced to reduce the wait times. I suspect the measures that they have introduced apply only to new applicants and that existing applications that are queued up will be dealt with in due time.

It's still a depressingly long wait.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

Usability of the DART and Bus system in Dublin

The DART

Imagine this scenario for the DART, you are a tourist from Germany. You've just seen the book of Kells and decide to get the train out to Dun Laoghaire on the DART, and you are close to Pearse St. DART station. So far so good, you as the tourist buys a return ticket to Dun Laoghaire. You then walk up to the sign trying to figure out which is north and which is south bound, while it's nice to have the sign before you walk up, sometimes you just make a mess of it. So you walk up to the platform and assume that you are on the correct platform. You get on the train, and behold! you are on the wrong train! because you didn't pay attention.

What can the DART do to make the system much more usable? Add a sign with on it, on each platform and at each station pointing to the next station and put the names on it. That would help tourists and users who don't use the system often a lot. It's bad to assume everyone knows which direction is North and which is South and where everything is.

Dublin Bus

Full disclosure, I haven't regularly gotten the bus in a long time now.

While those fancy new digital displays are cool and fancy. You'd think they could have done some useful things like perhaps put maps at all bus stops or at key bus stops if it is too expensive, put a list or a schematic of where the lines are going, where the stops are, but more importantly if there is an intersection of buses mark it clearly on the map or list.

It could even be a simple schematic of a few bus lines with key points of where routes might intersect, cross over. There are times where I want to get a connecting bus to somewhere, but most of the time, I have no idea where to interchange and what routes go where. Listing key stops on the bus stops may be of use to locals who know the area, but for someone just visiting, it's pretty useless without a map.

Also, what is with the the silly stage system of charging users, are they still doing that? I never know how many stages I need to get, and can I be sure that the bus driver is clued in enough to figure it out and not over/under charge me. Why can't there just be a flat rate or zoned rates to simplify things?

Loop lines for buses (and trams and trains)

Another useful thing that could be done, which some commuters might appreciate is to create a loop line. Perhaps do something sensible like create a loop line that connects the two Luas lines? There is now a smart card system, so offer people a discounted ride if they tag on and off the loop line when connecting to the Luas lines. You could apply the same approach with Connolly station and Heuston station. Those old City Imp buses would be perfect to shuttle people around quickly on a few loop lines.

This brings me to the next point, why the Luas wasn't joined in the first place? Seriously, I need to walk 30mins between the two lines? sigh I guess it's just a dream to expect that all these transport systems will work together to provide a better service. It shouldn't be that hard to do a few simple things to provide a better transport network.

New Year's Resolution for 2012

Now a happily married man, I need to make a new year's resolution!

  • Get back into the swing of posting stuff, I have months worth of Windmill Lane graffiti pictures to post.
  • Take more photos!
  • Spend more time doing stuff to make me and the wife happy.
  • Eat more vegetables.
  • Bake more bread.
  • Read more.
  • Learn how to brew beer.
  • Perhaps really learn go when it goes stable.
  • Install Linux on my mac, I'm beginning to really miss Linux and being able to change and fix things as I see fit.

There's probably more stuff, but let's be realistic, I'm lazy :P

I should have done this long ago

I should have done this long ago, I should have separated the technical rants, commentaries and other technical things that aren't personal to me to another blog. I thought it might be an interesting experience to setup a new blog at http://jcftang.github.com, just go there for more random mutterings which you may or may not agree with. It's just an experiment for now to see how much I like or dislike publishing in something other than this.

GNIB 2011 (till 2012) - Here I go again

It's my yearly trip to the GNIB offices to get my stamp to legally allow me to stay in Ireland to work. Though this time it was less frustrating that previous years.

I arrived at 8am on the dot and queued up to get a ticket, I got number 30. Thankfully they were processing approximately 1 person per minute. I got to see a representative of the GNIB by ~8:40, showed him my documentation, paid my 150e in processing fees. I then proceeded to wait till 10:10am before I got my passport and GNIB card back.

Hopefully I won't need to do this next year when (if) I get naturalised.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

Starting a new job as a Software Engineer

After many years of doing work as a Systems Administrator on High Performance Computing Systems and providing support/development work; I have now changed roles to being a software engineer. This will be interesting as I will get to learn new things and expand on what I know already. I'll have to put down those old fashioned tools I used to use and now use more modern ones!

What is best in life?

To crush your enemies, see them driven before you, and to hear the lamentation of their women. Of course getting married to Frau Klein is a close second place, the big day is looming, it's tomorrow. It's also talk like a pirate day too! I must resist the urge to say "yarrr, marry me wench" in place of "yes, I do".

Everything is almost set, there isn't much left to do.

Configuring SLURM on a BGP machine after getting it from IBM
  • Copy over the SLES installation media to the server or front end node so we can install packages to build other packages
  • Install openssl-devel, readline-devel, pam-devel and munge-devel (and libs) to build munge and slurm
  • Configure an LDAP server on the service node then point the frontend node to the LDAP server

Why did we pick slurm?

Since we run slurm on all our other compute clusters, it made sense to have it on this system as well. Probably the best thing is all the accounting logs are sent to the same logging daemon.

Configuring Slurm

Most of the configuration of slurm was pretty straight forward. Since we only have a 1 rack system in work I had only configured it to do DYNAMIC layouts.

The relevant configuration options/lines are shown here. The first time Bluegene/P adminitrator will probably appreciate these snippets of configuration lines. The basic setup is

  • service-node runs the slurmctld, this is also where all the Bluegene/P management software runs.
  • frontend-node runs the slurmd.

For the bluegene.conf file, I generated it by doing smap -Dc -v, then I typed 'save' in the command window.

#
# bluegene.conf file generated by smap
# See the bluegene.conf man page for more information
#
CnloadImage=/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk
MloaderImage=/bgsys/drivers/ppcfloor/boot/uloader
IoloadImage=/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
BridgeAPILogFile=/var/log/slurm/bridgeapi.log
#Numpsets=4 # io poor
Numpsets=16 # io rich
BridgeAPIVerbose=2
BasePartitionNodeCnt=512
NodeCardNodeCnt=32
LayoutMode=DYNAMIC

It's not recommended to run with a DYNAMIC layout mode if you have a big system, but for small systems its probably fine.

The slurm.conf file also requires some tweaks, notably

SelectType=select/bluegene
# COMPUTE NODES
NodeName=bgp[000x001] Procs=2048 NodeHostname=bg-fe State=UNKNOWN
PartitionName=compute Nodes=bgp[000x001] Default=YES MaxTime=12:00:00 State=UP

More documentation can be found at https://computing.llnl.gov/linux/slurm/bluegene.html, luckily the rest of the slurm configuration is quite straight forward and if you are experienced with slurm already there should not be any major problems.

Gotchas

Some of the gotchas that I've come across so far has been that I needed to configure both the frontend and serivce nodes to look up a directory service of some sort. That is both nodes should have the same posix users, this is so that all the accounting information is logged correctly. In my case I chose to use LDAP on the two machines, I did not point our Bluegene/P system to our existing LDAP servers due to some stock configurations that IBM had shipped.

mpirun and its configuration file mpirun.cfg also caused me some minor problems with the suid bits, we tweaked it abit. It's less secure than what it was before but we trust our users to a certain extent.

Some real life usage of slurm-bank

It's been almost two months since we turned on full accounting with just SLURM and our own slurm-bank set of wrapper scripts. It's mostly working as we expected with allocating hours to people and projects.

However we have noticed a few things, messing with the QOS settings in slurmdbd is a global option, you can hose all your jobs across all your clusters if you aren't careful. That said, you could probably implement charge rates for a higher QOS level for users who wish to have their jobs queued up ahead of everyone elses

One of the flip sides of doing full accounting with just SLURM is that fairshare doesn't quite work anymore. Since the decay and reset period is set to one year (or never). The whole system pretty much turns into a FIFO queue where jobs that wait longer gets bumped up. We will need to tweak the settings a bit.

Overall using just slurm without GOLD has been great, things tend to just work faster with out GOLD or MAUI slowing things down with scheduling. Having just SLURM also meant that we now have access to the multi-cluster capabilities for submitting and checking jobs across multiple clusters. So far two thumbs up for SLURM and slurm-bank.


Related posts:

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Astro, Particle, High Energy physics is cool
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Sector/Sphere for OSX

I've yet again revisited sector, this time I have a need. I need some system or some way of storing lots of old user data and have it stored relatively safely. I also did not want to spend much time setting up things and configuring things.

Having set it up I then got annoyed with the fact that I couldn't manage the files from my own mac workstation. So I spent a few hours figuring out what I needed to do to make it work on my mac. It actually turned out to be pretty easy to port over to OSX, the original developers did a pretty good job at making it portable. I've temporarily put up a repo with the changes that I have made.

git clone git://sgenomics.org/jtang/code/sector.git

My changes are based on sector 2.6.a, once you have checked out the code (on OSX) just type make, and it should build automatically if you have xcode installed. Everything thing else is the same as the Linux version once it is installed. The tools all work. I have even tried to setup a mini "cluster" with the OSX version, although I would strongly suggest users not to use OSX as a slave, master or security server for sector unless you know what you are doing.

I would expect that the running the servers on OSX would work, but OSX's default case insensitive system would probably cause you no end to pain. If you really do want to use OSX as a server one should probably make sure that the filesystem backing the servers are case sensitive.

I should probably make an effort to submit the patches upstream, I suspect the current developers might not care too much as their development efforts seem to be moving along much faster now. They might just make it more portable themselves.


Related posts:

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Astro, Particle, High Energy physics is cool
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

IO can be a killer even on small pieces of code

Having spent a week or two at fixing up a code for "Genetic Classification of Populations Using Supervised Learning" was interesting, but I was baffled at why there were constant fopen's and fclose's for loading up an input file that never got altered.

Removing the fopen/fclose calls to point to a piece of memory automagically made the the code run about 20% faster. The code isn't as slow as I thought it was going to be, it runs at quite an acceptable rate now.


Related posts:

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

git-annex and tahoe-lafs as a special remote
Posted

Memory debuggers and garbage collectors for C/C++

Usually I come across relatively well written pieces of code. That is code that is broken up with 50-60 lines per function and not 1000+ line long functions. Recently I was given a code written by a scientist (I think he is a physicist), and it does some rather clever stuff scientifically. However everything is in one big main() function.

The code mostly ran, and occassionally failed with no apparent reason. The first few things I did was to format/indent the code as much as I could and I started to replace all the malloc/free and some of the string functions with wrapper functions to sanitise everything.

It then occured to me that I should probably run the code through valgrind or something similar to debug the malloc/free calls. Traditionally I would have used electricfence during testing, bug fixing and development. I saw this opportunity to see what else is out there. I had known about gc for a while but never really got around to using it, I'm quite happy to say that it's pretty easy to use and it seems okay for doing garbage collection. It is particularly useful but requires a small bit of work with existing codes.

I was however surprised with google-perftools, I had never realised that it was a malloc() replacement and that it provided a bunch of profiling and checking features. Best of all it is relatively easy to use, short and concise. I've yet to read up on the garbage collection features in this library, assuming there is garbage collection. I can see myself using this library a bit more when I write C or when I have to fix someone elses code.


Related posts:

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

git-annex and tahoe-lafs as a special remote
Posted

Impossible Film

I recently came into possession of an old Polaroid camera from one of Kerstin's friends. It's a Polaroid Onestep express camera which takes Polaroid 600 film. Luckily the impossible project manufactures film for the camera.

I bought me three packs of film for the camera and I plan on using some of it for wedding related things. I haven't opened the film yet, I will wait till Kerstin is back before I open and play with the film.

Barcelona 2011 - After hours

I was recently in Barcelona with some colleagues for a meeting/conference which was hosted at BSC otherwise known Barcelona Supercomputing Centre. We ironed out a bunch of issues and to do items with our project which was good.

We had some extra time left over since we were flying out the next morning, and we had plenty of day light cause it was the summer... So some sight seeing was done! We also met up with some of Jose's friends (previous co-workers?) much fun was had.


Related posts:

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems

We used to use Maui and Torque for all our resource management needs in work. Then we had a brief encounter with Sun Grid Engine as well as SLURM. It mostly worked okay, till we decided to introduce GOLD into the mix for various reasons (funding agencies and accountability). GOLD provides all the banking functionality to limit and account for hours spent on projects.

We managed to get SLURM, Maui and GOLD to work together to provide us a relatively reliable resource management system with accounting/banking. Quite frankly Maui's documentation is pretty poor as is GOLD, not only that GOLD tended to be overly complex, slow and clunky. This is probably because the code base is quite old and it was and still is pretty much the go to tool for a lot of people for doing accounting and banking at HPC centres.

We had known that SLURM had some of the capabilities for doing banking and accounting, but it lacked a few features to make it a viable alternative to the existing Maui, SLURM and GOLD setup that we had.

After a few emails with the SLURM developers

and some experimenting, we came to the conclusion that all that was missing were just a few helper scripts and utilities were needed to provide a simple banking system for SLURM.

The end result was slurm-bank, which is just a collection of shell and perl scripts to wrap up some of the existing SLURM functionality along with a description/design on how SLURM should be configured to provide GOLD like functionality.

The scripts are pretty simple and mostly dumb, but they work well enough at my current work place in TCHPC. There's a bunch of wanted features that will probably take more time to flesh out. There will probably be a need for a rewrite at some point when the ideas are more tested and developed. For now we are able to just run SLURM (without Maui and GOLD) at our site. Hopefully the SLURM developers will take on board some of the ideas that we have mashed up and scripted up in a really bad and dodgy way.

To get the code

git clone git@github.com:jcftang/slurm-bank.git

There are tags in the repo that you can checkout, the tags are relatively stable and reliable, as is the stable branch.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

The ultimate sandbox game, ever!
Posted

Astro, Particle, High Energy physics is cool
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Brewing my own Kombucha

I've recently taken to a liking to brewing my own Kombucha.

Ingredients

  • 1 Kombucha culture and starter tea (get this from a friend or online)
  • Sugar (White or brown granulated sugar should be fine) - 100g
  • Clean drinking water (preferably filtered water) - 1litre
  • Plain black or green tea (make sure the tea isn't flavoured or has oils in it, teas such as fruit tea's, earl grey, mint are all bad). I'm using either plain Barry's tea bags or loose tea leaves myself - 2 teabags

Needed utensils

  • One large cooking pot and stirring utensil. I use a metal spoon instead of wood as it is easier to clean with non-antibacterial cleaners.
  • One large glass jar (or a plastic beer/wine brewing bucket). I'm using a 5litre beer brewing bucket where I have cut a large hole on top.
  • One piece of clean cloth

Procedure

  • Make sure everything is clean, but do not use anti-bacterial cleaners. You could just use white vinegar to sterilise everything.
  • Bring water to the boil, once water has boiled, turn the heat right down to the lowest setting.
  • Add the sugar (about 100g to 1lire of water), add the tea bags as well.
  • Stir in the sugar till the sugar dissolves, do not burn the sugar. Once the sugar has been dissolved, turn off the heat.
  • Let the sugary tea mixture cool down to about room temperature. Or if you are lazy leave it to cool overnight. Remove the tea bags and throw them away.
  • Once the sugary tea mixture is cool, add the starter tea and mix it all up.
  • Pour the tea mixture to your brewing jar or container. Then add the Kombucha culture.
  • Cover the top of the jar with a piece of cloth and secure it with a rubber band. Make sure the cloth is dry and clean.
  • Wait about 5-7 days before testing the flavour, I find it takes about 7-8 days for my brews to taste good and fizzy. I'm also using a heating pad to keep the brew at a consistent ~20C.
  • Once you are happy with the taste, bottle the Kombucha. Make sure you save about 10% of the brew and the kombucha culture mother and baby for the next brew. Store the bottled Kombucha in a dark place. Chill it before drinking! I usually keep one baby in a jar separate from my main brewing containers just in case if my cultures develop mold.

Things to watch out for

  • Mold, it will be a different colour to the culture and it will be dry and fuzzy.
  • Make sure the brew is covered with a fine enough cloth to keep out flies and insects.
  • Never use anti-bacterial cleaning agents, it will kill the Kombucha culture
  • Don't let the temperature drop too low or rise too high, keep it around ~20C or else it will either develop mold if the brew hasn't developed enough to kill all the bad things or else it might take more than a week to brew.
New Irish citizens will have to swear oath of fidelity

According to the Irish Times article New Irish citizens will have to swear oath of fidelity, assuming I get naturalised I will now have a ceremony to attend instead of going to a court to be sworn in. This new Minister for Justice isn't too bad I guess, he is doing his job and making things more efficient in general. It's interesting to see that the article quotes some numbers. They have processed 5578 applications and have about 17000 sitting in a queue and they did this in about 2.5 months. At this rate of processing I might actually see some post informing that I have or have not been naturalised.

To refer back to my previous post 2011-05 -- Update on long naturalisation times in Ireland, here is an updated table.

Date Average wait time
2009-06-21 23 months
2010-02-28 24 months
2011-03-08 26 months
2011-05-11 25 months
2011-06-21 23 months

The INIS page for informing people about processing times has also changed to this. Based on what I have, the average wait time is 24 months with a standard deviation of 1.3 months.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

Astro, Particle, High Energy physics is cool
Posted

Windmill Lane graffiti for 2011-06-10

I've been away and busy for the past while, but here's my regular stash of graffiti


Related posts:

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

The ultimate sandbox game, ever!

What's the best sandbox game (or openworld games) around that's cross platform, free, opensource and never gets boring? Is it minecraft because it's written in Java? Is it SimCity, Grand Theft Auto?

Well the ultimate sandbox game ever developed has to be a thing called gcc - Gnu Complier Collection, you can build anything you want with it. All you need is a computer with enough space to install it, a text editor and the manual for the language binding you want to use. You can use a number of languages with gcc such as C, C++, Objective-C, Fortran, Java, Ada, and Go and maybe others as well.

There are no constraints in the worlds that you can create with gcc provided you have enough time and determination. It's the ultimate sandbox game!

If gcc is too lowlevel for you then you might want to try something like haskell, perl, ruby or python. All of which were probably created with gcc anyway.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

git-annex and tahoe-lafs as a special remote
Posted

Making the most of Hugin -- erasing unwanted objects

No this isn't a tutorial, but rather a post on me praising how good Hugin is, and how useful it is.

I was doing my usual thing on Windmill Lane with taking pictures of the graffiti being put up, normally I don't get disturbed by highly annoying people. All I wanted to do was take a few pictures for putting together a nice picture. Sadly some locals got in my way, too bad for them that Hugin is great for digitally removing things from panoramas and the such, this was lucky for me.

Here were the source images:

The final output without the annoying individual, car, and sign post


Related posts:

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

Windmill Lane graffiti for 2011-05-15

Summer is fast approaching if its not already here, the frequency of pieces being put up is quite high now.

The mosaics, I am really getting to like Hugin and this mosaic mode...

The usual stash of Windmill Lane graffitti


Related posts:

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

Astro, Particle, High Energy physics is cool

Sigh TCD (my work place) didn't even get a mention about being a "supercomputing centre" in a recent Irish Times article To infinity, and beyond ... nor did Tyndall and probably a few others weren't mentioned. You got to love it when the media attaches themselves to popular science type topics (or else they didn't do their research). Astro, Particle, High Energy physics is cool again, so is being a weatherman is cool too. At least that's the impression that I get from the article, and yes supercomputers are cool tools too.

2011-05 -- Update on long naturalisation times in Ireland

I just thought it might be good to check INIS again, the average wait time seems to have plateaued. I guess this a good thing, I wonder why the average time went down. I would have expected it to remain at 26months or increase.

Date Average wait time
2009-06-21 23 months
2010-02-28 24 months
2011-03-08 26 months
2011-05-11 25 months

Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Creating a git branch with no ancestry

Here's a recipe for creating a branch in git with no ancestry, no doubt this has been noted on the internet by many already.

  git symbolic-ref HEAD refs/heads/upstream
  git rm --cached -r .
  git commit --allow-empty -m 'initial upstream branch'
  git checkout -f master

Why would one want to do the above? Well to keep two disconnected projects in one git repo, that's why!


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Using gource to visualise projects stored in version control
Posted

git-annex and tahoe-lafs as a special remote
Posted

Using gource to visualise projects stored in version control

I've only dabbled at this in the past with codeswarm but I never got too far. Then I came across gource, I found it pretty easy to use and, it's fast as well. Here's what I did to generate a visualisation of the cports repo that I have:

$ gource --hide filenames,dirnames,bloom --seconds-per-day 1 -720x480 \
--title cports -o cports.ppm
$ ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm -i cports.ppm \
-vpre libx264-default -vcodec libx264 cports.x264.avi

Or if you are lazy you can do this...

$ gource --hide filenames,dirnames,bloom --seconds-per-day 1 -720x480 \
--title cports -o - | ffmpeg -y -b 3000K -r 60 -f image2pipe -vcodec ppm \
-i - -vpre libx264-default -vcodec libx264 cports.x264.avi

The result,

The result wasn't too interesting since the history is pretty linear, and the files in the repo are pretty organised to begin with. So I decided to visualise my irods-hpce2 repo,

I think I will visualise the smeagol electron transport code next with something like this,

gource -a 0.5 -s 0.15 -i 0 --highlight-users --multi-sampling --hide filenames
git-annex and tahoe-lafs as a special remote

I've been using git-annex for a few months now and I have been longing the capability to plug in arbitrary remotes/backends and that support has arrived in the form of special_remotes/hook.

At first I wanted to integrate irods or sector/sphere, then it occured to me that tahoe-lafs would a good match for doing my backups and just managing my data that I want to preserve (with resources that I find available around the office/labs).

Assuming you have a recent copy of git-annex which supports this new special remote hook, you could do something similar to this

git config annex.tahoe-store-hook 'tahoe mkdir tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2 && tahoe put $ANNEX_FILE tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY'
git config annex.tahoe-retrieve-hook 'tahoe get tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY $ANNEX_FILE'
git config annex.tahoe-remove-hook 'tahoe rm tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY'
git config annex.tahoe-checkpresent-hook 'tahoe ls tahoe:$ANNEX_HASH_1/$ANNEX_HASH_2/$ANNEX_KEY 2>&1 || echo FAIL'
git annex initremote library type=hook hooktype=tahoe encryption=none
git annex describe 1d1bc312-7243-11e0-a9ce-5f10c0ce9b0a library

The above assumes that the user understands how tahoe-lafs works and knows how to configure and use tahoe-lafs. In the above example the default tahoe: name space is used. It would probably be a good idea to create a new tahoe-lafs alias/namespace (I think it is called a DIR-CAP in tahoe speak) to store git-annex'd files.

I've just been testing so far with the above, it's a little slow, but for dumping data to a private cloud it's fine. I need to redirect some of the output to /dev/null after I am satisfied with the overall behaviour of the system. So far git-annex has been great for me for archiving and tracking research data and projects which I no longer need on my primary system.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Keeping tabs on Lustre and whamcloud

Over the past year or two Lustre development and releases have largely been slow and un-eventful with a fear that Oracle might kill off the product/project. Luckily that hasn't been the case, with so many large sites running Lustre, everyone just went and formed their own support groups and companies to support and develop Lustre.

This has lead to some pretty nice developments, in particular is whamcloud. I suspect whamcloud will pretty much lead the feature development and release stable enough packages for the community at large which is nice.

For the first time in a while there is a nice set of downloadable RPM's and patched kernels of Lustre which a user can download and use with little fuss. This is assuming you are using CentOS or a compatible clone. Kudos to the whamcloud crowd for putting those packages online and hopefully more packages will appear for different distros.

I really appreciate the way that whamcloud are handling the developement of the project. The following resources are really useful.

I do admit that we're primarily a GPFS user in work, but there is no harm is keeping tabs on what others are using and doing. From the last time we were using Lustre, I was quite happy with the performance but it lacked some polish and documentation for the High Availability setups compared to GPFS at the time. This is probably a issue for smaller groups who don't have enough resources to maintain complex systems. Hopefully with all this new development Lustre will become a better and more sysadmin friendly (at least for the less experienced sysadmin) product over time.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Astro, Particle, High Energy physics is cool
Posted

Creating a git branch with no ancestry
Posted

Windmill Lane graffiti for 2011-04-24

Two mosaics...

The usual Windmill Lane and Creighton St.


Related posts:

Windmill Lane graffiti for 2011-05-15
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

Distro's for building a HPC machine?

What's a good distro for building a HPC cluster? Well what do you want to do? There are linux distro's out there which are nice as a building block, and then there are turnkey solutions if you are really lazy and then there are the distro's that sit in between.

There's the usual set of distro's like

Then there are turnkey solutions such as

or else you might come across distros that sort of sit in between.

I've helped to setup and maintain a few debian and scientificlinux based clusters over the years.

From experience, as nice as debian was, there was almost no end to pain with commercial software and drivers for new cutting edge hardware, filesystem drivers (for things like lustre or GPFS) and packaging stuff up for debian. I would probably not pick debian again as distro for a HPC machine unless I knew that the requirements did not involve commericial software or brand spanking new hardware which requires binary blobs and custom startup scripts. I would imagine fedora would not be too far off from how debian worked for us.

scientificlinux and it's RHEL cloned cousins are a much better for a production environment. A lot of vendors tend to have support for the upstream RHEL distro, so it's nice to be able to plug in software into a free clone and see it work. In general you get almost no problems with commercial codes and drivers. In fact we use scientificlinux in work.

If someone ever tries to convince me to use a particular distro for building a cluster I would entertain the idea. I'd usually ask a few questions about the distro first (to see if it meets my criteria for sucessful deployment and management), such as:

  • How long is it supported for (the longer the better)?
  • Will things like Lustre/GPFS build cleanly?
  • Is the distro stable and tested, does the upstream vendor maintain real bug fixes and security fixes as opposed to "upgrading the package to a newer version will fix the problem"?
  • Is the kernel version stable? i.e. like the bug fixes, I want a stable API/ABI etc... so I can let users target a known system that will be consistant for a long time so they can be sure their codes will work and they won't need to spend time writing solutions to edge cases on a changing system.
  • How easy is it to create packages and a repository for the distro? So the deployment and management process can be streamlined, automated and repeated.

In general I like things to be supported for a long time, and be compatible with a bunch of commercial applications. Over time I've noticed that it's really only the RHEL based distro's that seem to really shine for HPC sysadmins (who run clusters bigger than say 32nodes). I would lean towards RHEL based distro's myself for building a cluster.

The outliers of RHEL derived distros such as caos and chaos are linux pretty good. These two distro's include a bunch of tools that I would install after I have done a base install, but they do require the sysadmin to know what they are doing, but that is no different from rolling my own with scientificlinux. There is a market for these types of tools, that's why there are things like oscar and rockscluster. There are turnkey solutions for those with no experience or budget in setting up a cluster, but be warned they are "limiting" in the way that you can configure your cluster to your tastes.

In the real world (for HPC sysadmins and users), people want to do work and not debate over who is more free or who has the better package manager or best and newest selection of software. Of course if it is free people will gravitate towards it. If it's easy to use then even more people gravitate to it.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Astro, Particle, High Energy physics is cool
Posted

Creating a git branch with no ancestry
Posted

Windmill Lane graffiti for 2011-04-20

It's just amazing, in a span of 24hrs the wall on Windmill Lane got painted twice!


Related posts:

Windmill Lane graffiti for 2011-05-15
Posted

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

Windmill Lane graffiti for 2011-04-16

My first panorama of the year!, I haven't really been doing many panoramas recently.

Now with the usual Windmill Lane Graffiti


Related posts:

Windmill Lane graffiti for 2011-05-15
Posted

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

I got my new camera -- Panasonic LX5
Posted

A new kitchen utensil -- Universal Slicer
Posted

Linux ready for the masses? I think not.

Ubuntu and Fedora does look great these days, the default install is pretty functional. Of course that assumes that you don't need or want 3d acceleration, the latest flash plugin's, mp3 support and a bunch of other multi-media and webby stuff.

It was entertaining to see a newish user making the move from Ubuntu to a Fedora based distro and seeing the amount trouble that it caused. The user wanted to install an accelerated driver. The choices were - Install from source? Use the packaging system? There's lots of choice which is great for a power user, but for the average joe who is just starting to use linux it can be a recipe for disaster.

Even so, gone are the days when installing the NVIDIA or ATI drivers was a thing that you did once and then never upgraded. Upgrading in the past meant that you would have needed to figure out which kernel versions you needed, which kernel headers you needed, which version of the x11/xorg driver modules you needed then you needed to figure out which NVIDIA or ATI package you needed. Or else you have gotten the classic "BSOD" when x11 would just fail to start up. Which is funny to see a typical Windows or Mac user just goes "WTF?"

These days most distro's package up all the correct and needed drivers to get 3d acceleration. So it's just a case of yum install DRIVER or apt-get install DRIVER or whatever your package manager commands are. I guess if you are in the know, you just install from a known repo and never deviate from it. If you did deviate, then you probably know what you are doing or else you are in lots of trouble.

Printing is also another bundle of fun in linux, but like the graphics side stuff, things have improved lot since I started using linux. Mind you, I use my Mac in work to admin linux servers and workstations. I do at times miss linux as a desktop for all it's worth, but I can live without it.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

I got my new camera -- Panasonic LX5

Finally my LX5 has arrived, I mainly got it because my trusty LX3 is beginning to die. That is the shutter button is dying, it's been through a lot with me in the past two or three years. I hope that the LX5 will do me good for the next few years!

The unpacking of the LX5

As you can see in the above pictures, I didn't exactly take care in taking the pictures as I really wanted to power on the camera to play with. I pretty much left everything in the box, the only things I took out to use is the charger, the camera itself (obviously) and the little cord for the lens cap. I've decided not to use the camera strap, instead I am going to use a small hand strap that is detachable. I'm going to need to get a 45mm male to 46mm female step up ring to use my barrel adapter and 18mm super wide angle lens on the LX5. Now to plan a holiday and wedding so I can take lots of pictures!

Oh here are some sample shots, I'm glad the dynamic b&w mode hasn't changed much its very very nice.

The LX5 isn't too different from the LX3, if it weren't for the fact that my old camera was dying I wouldn't have upgraded. It's still a powerful little point and shoot camera for hobbiests and power users.


Related posts:

Windmill Lane graffiti for 2011-05-15
Posted

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

A new kitchen utensil -- Universal Slicer
Posted

Breaking my archlinux laptops with e4rat

It's Sunday, I'm bored, I've posted a few things already and now it's time to pointlessly play with my archlinux laptops. So, let's begin with trying to make my laptop pointlessly boot faster, it's something that I rarely bother to do on the servers that I run, but on a laptop it might be worth the hassle. Anyway the following only works on Linux and systems with an ext4 root filesystem. So start by installing e4rat

packer -S e4rat

This will install a few small applications, once it is installed I profiled my boot sequence by doing what was suggested on the e4rat homepage. This is done by going into the boot menu and changing the kernel line to include an init script to the one supplied by e4rat for example I added this to grub kernel line

init=/sbin/e4rat-collect

Then boot up as normal, by default e4rat will record the disk activity for 120seconds if this is not long enough it can be extended by editing /etc/e4rat.conf. If it is too long you can just kill the process. Now you need to run the reallocator,

sudo init 1
e4rat-realloc /var/lib/e4rat/startup.log

Then substitute init=/sbin/e4rat-collect with init=/sbin/e4rat-preload. The above steps would probably need to be re-run everytime when you upgrade a package that is related to the boot process or add more things to the boot process.

I didn't notice any appreciable improvment, probably because my machines are pretty minimal to begin with, and I don't bother booting into X automatically. It's probably more worthwhile if I booted into X directly and ran a more heavyweight window manager. This caused me to bother to setup xdm on both archlinux laptops and create a bunch of boot options to run e4rat. To be honest my machines had probably spent most of its time trying to connect to the network on boot.

So what was the point of the above? I guess not much other than the fact I was bored and it looked interesting at the time.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Do people give a crap when they can not give a crap?

It just occured to me, I've been blogging randomish crap for the last few years. Mostly just posting photos of my holidays, food and other random stuff that I might want to comment on. I am exercising my right as an interweb user to spout out nonsense to a deaf audience. On thinking about that, it just occured to me that I haven't really been following up on the local politics and financial mess that Ireland has gotten itself into in the last few years.

With the recent happenings in Japan and their problems with a natural disaster and a potential set of nuclear disasters, it seems all the media has shyed away from Ireland's own problems. I guess why should the average joe give a crap when they can not give a crap. Especially when the previous government decided that it was a good idea to bailout the banks (supposedly the cheapest bailout) is now probably the worlds most expensive bailout.

I guess I careless right now about it all since I don't own any property or anything of significance, I do however feel a little concerned about my savings.


Related posts:

Astro, Particle, High Energy physics is cool
Posted

State of the Irish Banks
Posted

How long does it take to get naturalised in Ireland?
Posted

Otaku and in Ireland?
Posted

Your Country Your Call, if I had a choice...
Posted

org-mode and git as an offline wedding planning tool

So you're getting married and a need to plan your wedding? The first obvious solution was to use google docs, which was okay till I noticed that all sorts of junk had ended up on my google docs account. I couldn't make sense of some of the naming of the documents and which versions where which. It was a classic case of me just either not understanding or me just being plain stupid.

So my solution to the problem? Setup a git repository and startup emacs with org-mode and migrate all my data over to org-mode because it made sense to me at the time. I think my wife to be is still having issues that I decided to do such a thing.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Apparently I'm a luddite

I haven't used wordpress or drupal for my blog. I use vi and emacs for most of my text editing. I like vanilla, my mobile phone doesn't do text messages and has no electronic phone book.

I am missing some cool features that other blogging systems have. So I whipped up a dirty template to show related posts. It kinda slows things down when I add new posts or edit things, but it's a price that is worth paying.

On the note of adding more fancy things to my blog, perhaps I should start thinking about creating posts in a more sensible directory structure instead of dumping everything into the posts directory.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

A new kitchen utensil -- Universal Slicer

What's better than sliced bread? Well apparently slicing your own bread and other stuff.

The unpacking...

Now to slice me some bread and salami...


Related posts:

Making the most of Hugin -- erasing unwanted objects
Posted

Windmill Lane graffiti for 2011-05-15
Posted

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

Windmill Lane graffiti for 2011-04-02

As the weeks go by Windmill Lane is smelling more and more of fresh spray paint...


Related posts:

Windmill Lane graffiti for 2011-05-15
Posted

Windmill Lane graffiti for 2011-04-24
Posted

Windmill Lane graffiti for 2011-04-20
Posted

Windmill Lane graffiti for 2011-04-16
Posted

I got my new camera -- Panasonic LX5
Posted

Test, test, test and more testing

The value of testing, do I really need to stress it? It ensures consistantly working code and a bunch of examples of what the code should or should not do. What I don't like is the large mass of frameworks out there. Almost every language has their own framework that a user must learn. That said, I'm beginning to appreciate wvtest far more now, since it is very very simplistic and almost dumb, but it works well.

I've learnt that I should really run tests randomly on new applications and libraries to ensure that the system that I am working on really works.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Cool trick with gcc - note for myself before I forget

I occassionally need to know some system dependant stuff when messing around with some C code. I keep forgetting this neat little trick

gcc -dM -E - < /dev/null

It will dump out useful things like this...

#define __DBL_MIN_EXP__ (-1021)
#define __UINT_LEAST16_MAX__ 65535
#define __FLT_MIN__ 1.17549435082228750797e-38F
#define __UINT_LEAST8_TYPE__ unsigned char
#define __INTMAX_C(c) c ## L
#define __CHAR_BIT__ 8
#define __UINT8_MAX__ 255
#define __WINT_MAX__ 4294967295U
#define __ORDER_LITTLE_ENDIAN__ 1234
#define __SIZE_MAX__ 18446744073709551615UL
#define __WCHAR_MAX__ 2147483647
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2 1
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4 1
#define __DBL_DENORM_MIN__ ((double)4.94065645841246544177e-324L)
#define __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8 1
#define __FLT_EVAL_METHOD__ 0
#define __unix__ 1
#define __x86_64 1
...and so on...

Actually, have I posted the above already and just forgot?


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Creating a git branch with no ancestry
Posted

Using gource to visualise projects stored in version control
Posted

Git is a bit too clever sometimes

Consider this chain of events

mkdir annex
cd annex
git init
cd ..
mkdir annex.git
cd annex.git
git init --bare
cd ../annex
git annex init stuff
git remote add bare ../annex.git
git push bare master

Now I decide to not use git in my annex directory and remove the .git directory from annex. Then if I were to do a

git clone host:annex annex-clone

git gets a bit too clever and skips over the annex directory and goes straight to annex.git. Although some people might like this behaviour, I would prefer that git should do what it is told and spit out an error to me saying that host:annex doesn't exist.

Trusting my data to git-annex

I've been recently testing out git-annex, and I've just started trusting my data to it. My use case for it has been for archiving and storing old project and research data that I no longer want stored on my workstation and to manage source tarballs from the cports stuff that I have been hacking at.

The main advantage of having git-annex in my case is that I can now have a directory full of stub entries with no actual data. All my data is stored either on a remote server with all the raid5/6 and tape backup available to it and an additional copy is stored on a portable disk which will probably end up sitting on my shelf.

My setup is currently...

  • Workstation - annex'd project directory in ~/Projects which may or may not have all the data that is being tracked.
  • A portable disk - with a bare repo storing a replica of the my workstation
  • A remote server - with effectively a clone of my workstation's ~/Projects directory

I guess the sensible thing to do is actually this...

  • Remote server - with a bare repo (storing all the data as well)
  • Workstation - where I keep a clone of the annex'd files but have only a partial copy of the data.
  • Portable disk - for storing a replica as a secondary backup that can be removed.

The nice thing about using git-annex is now, I can track my files on a big fat server or workstation. Then take a clone of the annex'd repo and get whatever files I need and work offline. It gives me the possiblilty of managing my large archives of files offline and possibly track replicas of data.

I'm not totally sure I would recommend anyone to use it to for industrial grade data archiving or anything like that just yet. It is certainly a tool for system admins, techies and power users who like to work offline and truly manage their data in distributed fashion.

I can see myself using this more for tracking and managing tarballs of user data that I need to backup to protect students and postdocs from user stupidity.

cports snapshot - it's been brewing for a while now

I've finally decided that it's about time to share the big dirty hacks inside of the cports fork that we've been cooking up in work. It's been sitting long enough without me wanting to plug more stuff in the core of cports/mpkg for the past month or two. I've only been adding new packages to the system.

For now there are just snapshots which I put online at my workplace

I've posted to the Caos mailing list, so let's see who bites. If there is enough interest (if at all) I'll put up a public git repo to allow for submissions of packages and fixes. I suspect the right thing to do is that it might be worth handing the whole thing back upstream so I don't need to deal with it.

There's still a bunch of outstanding issues that we haven't resolved. Such as having MKL as a BLAS/LAPACK alternative to ATLAS/LAPACK, dummy packages to provide modules for vendor MPI's (e.g. MPI stacks from QLOGIC or VOLTAIRE).

On other cports related stuff, it seems that Caos-NSA isn't as dead as I thought it was.


Related posts:

Memory debuggers and garbage collectors for C/C++
Posted

Alternatives to using GOLD, Maui for accounting and banking for HPC systems
Posted

The ultimate sandbox game, ever!
Posted

Astro, Particle, High Energy physics is cool
Posted

Creating a git branch with no ancestry
Posted

St. Patrick's day 2011

St. Patrick's day 2011! I wandered around town for the day hoping to see the parade but I was too late, all the good spots were taken. Dublin city centre was a bit crazy, the streets were full of pleasant and unpleasant drunken people.

This was a real WTF >!? moment, are teenagers really like this these days?!

As you can see the streets were full of people and the whole city was partying!

I subsequently ended up going to the pub with some friends and friends of friends who ended up taking lots of pictures with my camera. I had the lensbaby on the camera at the time so it was a nightmare for people to focus and expose the shots.

Was wandering around Templebar after going to the pub, as per usual, it was crazy. I got some nice shots with the lensbaby here.

Adding some colour to this blog.

Traditionally ikiwiki has a pretty plain anti-theme, of no colours and no fancy fonts etc... I decided well, I've started using emacs, I'm just catching up to the eighties. It's time to move on to the 90's by adding some CSS coloured goodness.

It's not where near a piece of art, it's mildly pleasant and should hopefully be friendly to colour blind people. All I'm missing now is the scroll and blink tags!

Enterprise tools in academia

After my last post about Enterprise reporting tools in academia, I have had the (mis)fortunate experience of seeing a presentation on Enterprise Architecture. It seems to be all the rage right now with academic institutions, along with implementing CERIF. Yes, this is going to be another random rant.

Some things that I really appreciate as a systems administrator and user is the simplicity of some solutions over other solutions. Generally anything that lets me be lazy makes me like it even more. All these buzz words for Enterprise Architecture just worries and scares me that people are missing some of the concepts and what to target first. Admittedly I don't understand Enterprise Architecture fully, I can see it being useful, I just hope that it's not a fad.

On a similar note, as a proof of concept I had setup CAS, Liferay and Drupal to use single sign on which I suppose is just a piece of the big puzzle for Enterprise Architecture. As per usual, there is a bunch of tools out there which is free and easy to use and implement to provide enterprise grade solutions.

The depressingly long naturalisation times in Ireland

It's just depressing when I occasionally check back at the INIS site for updates on how long it takes to process applications. It hasn't really changed that much. I was curious how often that page gets updated, but it seems more it's updated less often than I would have liked it to be.

Checking The Wayback Machine just makes me even more depressed about waiting. The average wait time is now 26months!

Just to recap,

Date Average wait time
2009-06-21 23 months
2010-02-28 24 months
2011-03-08 26 months

Average wait time seems to be increasing by 2 months for every year that I am waiting, I guess at this rate I have waited just over 12 months, so I guess I have at least another 18 months to wait in the queue. Go Go Go team bureaucracy!


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Urge to replace my LX3

My LX3 is beginning to die, the shutter button has started to stick and I can't focus sometimes. I also found this page http://www.leica-camera.info/2010/04/leica-d-lux-4-service-mode.html it describes how to get into the service mode. One of the features of the service mode is a stats page with how many times I have used the flash, and how many photos I have taken since I have turned the camera on.

I've apparently taken 20,559 photo's, fired off the flash 323 times, powered on the camera about 8059 times and the camera has gone into power save mode about 1347 times. In short my LX3 has served me well, I'm just annoyed that I didn't used my superwide angle adapter as much as I should have.

I could upgrade the LX3 to a LX5 which would be really nice. I really have a few cameras to pick from at the price range that I am willing to spend. I could get one of the following

What will I get?! They are all nice cameras, but for ~500 euro they better be good! Then again I could be saving up to get a Canon 7d or a Canon 5d.

Enabling the gnu coreutils and findutils with macports on OSX

This has been driving me a bit mental for a while now. "BSD != GNU" tools!!!!! I have macports installed, you can enable variants when you install packages. There used to be a variant called +with_default_names, it used to be possible install coreutils and findutils so that they aren't prepended with g. They recently removed this variant and it confused me a bit. It seems that they now have the normal binaries without the g in front of the gnu tools located in /opt/local/libexec/gnubin/.

So all you need to do is just

sudo port install coreutils findutils
export PATH=$PATH:/opt/local/libexec/gnubin

It depends on if you want the extra gnu utils underneath the BSD layer or the GNU layer on top, so experiment with the order of the PATH's

Getting "resource exhausted (Resource temporarily unavailable)" errors on OSX?

It seem's that OSX has been causing me some more headaches, I've been toying with git-annex which is interesting and useful for managing a bunch of large files with git. I had kept running into errors such as the following when one runs the tests for git-annex

### Error in:   1:blackbox:13:git-annex migrate:1
forkProcess: resource exhausted (Resource temporarily unavailable)
Testing 1:blackbox:14:git-annex unused/dropunused                             
### Error in:   1:blackbox:14:git-annex unused/dropunused
forkProcess: resource exhausted (Resource temporarily unavailable)

This was a result of the low limit set by default for the maximum allowed processes in the system and per user. If you do a ulimit -a you can find out what the limit is, it's usually 266 for a normal user.

As a result I had googled up the problem and found that all I needed to do was something like this...

# this is really for the run time, you can set these settings in /etc/sysctl.conf
sudo sysctl -w kern.maxproc=2048
sudo sysctl -w kern.maxprocperuid=1024

# tell launchd about having higher limits
sudo echo "limit maxfiles 1024 unlimited" >> /etc/launchd.conf
sudo echo "limit maxproc 1024 2048" >> /etc/launchd.conf

Then ideally reboot, since I have no idea on how to restart launchd for myself to get it to inherit the new settings as such shell would have been a child process of launchd. Most users probably won't run into this limit, but I think most heavy and power users coming from a Linux background probably will come across this at some point. If only Apple made this limit higher it would be great, or else make it dynamic.

Debugging tools on OSX

I really missed using things like ldd and strace from my old linux workstation. The OSX equivalent tools are

  • otool, e.g. otool -L mybinary.x to see what libraries are linked
  • dtruss or dapptrace for tracing binaries, e.g. sudo dtruss -f mybinary.x you need admin access to run dtruss which is a little annoying, but it's better than nothing.
Building a time lapse with a DSLR, a mac and some free software

The tools

  • Canon 1000d DSLR camera with a Sigma 10-20mm super wide angle lens
  • A Twin1 ISR cable release/intervalometer
  • One iMac
  • ffmpeg, dcraw, libraw and Raw Photo Processor
  • A bunch of shell scripts (notes really)
  • One Hahnel Tripod

There's probably plenty of online tutorials on how to setup your camera to do interval shooting with a remote, so I won't go into that here. What is more useful and interesting is how to rename and stitch all the images together (either on linux or on a mac) with ffmpeg.

All you really need is just this

#!/bin/bash
x=1
for i in *JPG;
do
    counter=$(printf %06d $x)
    mv "$i" img"$counter".jpg
    x=$(($x+1))
done

and this

ffmpeg -f image2 -i img%06d.jpg -sameq mytimelapse.mp4

Both of the above was pretty much lifted from the ffmpeg documentation. The generated output should be usable in iMovie.

I'll post the time lapse I did recently once I get permission.

Cray XT5, compiling parallel programs

Having helped someone build quantum espresso on a Cray XT5 I'm going to have to remember to make a mental note of reading the documentation before I try and build things. Also mpicc/mpif90/mpicxx aren't there in the default software stack. The stack just uses CC=cc, CXX=CC and FC=ftn. These are just wrapper scripts which will also (at least on the ROSA machine) wrap up linking and compiling MPI based programs. It didn't occur to me till I read the documentation!

Gerrit Code Review

The android project uses gerrit for code reviews and as far as I know, google seems to use gerrit internally for some of their git repository hosting and code reviews (see http://www.youtube.com/watch?v=VzkudNGUepQ). I've been on the look out for some simple issue trackers and or project management software that isn't overly complicated to use and to setup. It was just an accident that I had come across gerrit after I was looking at some of the kiteware released code for things like VTK, ITK and cmake, whamcloud who are supporting lustre also seem to be using gerrit to review code.

Although it's not an issue tracker or a project management piece of software, it seems to be an interesting tool to open up repositories for contributors who are not apart of the core group of developers of a project. It seems to be like a gatekeeper to a repository, anyone can clone and submit code to the system, but the changes aren't applied till the changes are reviewed and verified.

There's already a few blog posts out there such as Code Review with Gerrit, a mostly visual guide and Gerrit Code Review Installation which describe how to install and configure gerrit, and also how to import existing repositories into the system. It's not difficult to install and configure gerrit for evaluating or production for small projects, it's almost pleasant to setup.

One thing to note about newer versions of gerrit is that you will need to configure new projects with the enough temporary permissions to push a branch or tag. Apart from initially playing with the permissions for a project there's isn't really too much need to fiddle with it.

In just testing the system, I have found that doing something like this in my .git/config to be useful

[remote "review"]
        url = ssh://gerrithost:29418/project
        push = HEAD:refs/for/master

and using the commit-msg hook is also very useful

scp -p -P 29418 gerrithost:hooks/commit-msg .git/hooks/

or else if you have a different ssh identity for your gerrithost, you can configure ssh with the appropriate identity

Host gerrithost
User username
IdentityFile ~/.ssh/id_rsa
Port 29418

So far I've been toying with gerrit as a single user, it doesn't seem very useful projects with less than two or three developers. What I have found it useful for, it is to use it as a dumping ground for changes and a place for making notes for the changes. It's acting as a barrier for myself, protecting my projects from me. Other useful scenarios would be to use it to review code submissions from student projects. I think if I replaced org-mode and milli with a simple to install bug tracker like roundup things might work a bit better.

Building up software stacks for HPC environments

The problem

No linux distro ever has exactly the set of software that you want use and install, and when it does it's too old or too new. Everyone thinks that their linux distro has the perfect or the best packaging system for managing packages. There are lots of good if not great packaging systems out there, but for a HPC sysadmin they usually suck when you need 3-4 versions of MPI compiled up with different compilers.

A good example is something like NAMD, if you wanted to build a customised version of NAMD such that it runs optimally on your cluster . The sysadmin usually needs to pick a compiler that works, pick an MPI implementation that either works or is optimised for the network that the sysadmin has, decide on whether to build fftw by hand or install it from the packaging system, charm++ and a few other things, decide whether to use MKL to back the FFTW implementation or not. Once all the decisions are made, the sysadmin configures and compiles things up and discovers there are updated versions of some of the software that he or she has built. Now the sysadmin goes back and changes some versions of software and re-links everything. Wash, rinse and repeat the said process till a usable and desired binary is produced.

Once a usable binary is produced, the sysadmin (me) usually goes back and tries to document what was done, and then repeats the whole build process on another machine for the user who asked for the software to be installed (of which there might be 2-3 versions of NAMD or whatever is installed). Or else the instructions are stored somewhere for another sysadmin to reproduce the said binary at a later point in time.

The possible solutions

Write shell scripts to repeat the build, write lots of documentation. Write you own packaging system and lay it on top of the distro of your choice. The guys at nixos did this, but it doesn't really allow users to sanely pick which versions of software they want to use and it is a little overly complicated to bootstrap and to allow a user to build stuff in the user land. The PETSc project and FEniCS project took the approach of building a build system for their software, since they require the usual MPI's, BLAS's, LAPACK's and FFTW's. It seems to work, but it's not too flexible to plug other stuff into the systems that they have built.

Some conclusions and a possible solution

It's a known and hard problem for package system builders to design systems that are smart and maintainable and has the capability of doing dependency checking, resolution and conflicts. It's often these design requirements that make the packaging system so smart that it makes them a little bit too complicated and useless for HPC sysadmins. That is these fancy systems often do not allow the HPC sysadmin to install multiple versions of the same software compiled up in different ways on a single system.

When you think about it, it's not a totally crazy idea that a HPC sysadmin would want to build their own set of scripts and framework to repeat a build. Most sysadmins think they have it bad when installing software for users, but HPC sysadmins have it much worse! But there's hope!

One particular build system had caught my eye a few months ago,

  • mpkg - http://staff.e.kth.se/mpkg/index.html
  • cports - http://www.caoslinux.org/ and http://caoswiki.infiscale.org/

mpkg kind of worked, but the cports developers took the concept and made it more useful, especially for HPC sysadmins, it generated environment module files and had been extended to allow different compilers to be used. It also had basic dependency checking so that when you installed a package it would also install its dependencies.

Sadly cports seems to be dead, but being free software I've taken a copy of the source code and have effectively forked the project. So far I have fixed a bunch of weird behavior/bugs in the build system. It's beginning to look promising as me and some colleagues of mine have plugged in a bunch of bio applications (for gene sequencing) for some users of our clusters in work.

cports is working out quite well for us in work, it's not really a build system, or a packaging system. It's really a building framework to make a HPC sysadmin's life much easier. We've been using it on our clusters to provide updated software in our module system. It's been refreshing to see our software stack installed in a pretty consistent way, and module files are automatically generated and tagged correctly for the user and that the builds could somewhat be repeated on similar systems. It's still a pretty dumb and semi manual system, but it works for us and it doesn't pretend to be clever.

It's not quite ready for prime time yet. It needs some more testing and probably way more polish before it can be published and advertised to other HPC sysadmins to play with.

New toys and a new phone

I got me some gadgets and stuff from thinkgeek recently. A utility key thing, a sonic grenade (for annoying people), a tv remote thing to switch stuff off, swedish firesteel for starting fires (for my survival kit?) and a book on weapons of mass destruction for the office.

I also got a Johnsphone, which frau-klein got for me. It's neat, it does nothing but make and receive calls. With a 2-3week standby time it's pretty good. I just need a new clock/alarm and a watch now, since I now have no clock or watch which I also used my old phone for.

It will be refreshing to see how much I will like or despise this phone. I really like the idea of this phone, but not being able to receive text messages might annoy some of my friends and family. I've been meaning to get a basic dumbed down phone for a while now, but this is the ultimate minimalistic phone!

Windmill Lane graffiti for 2011-01-08

First post of the year for graffiti on Windmill Lane. Some of the pieces were put up before Christmas while I was away in Hong Kong. Seems my LX3 is beginning to give up the ghost, it's beginning to lose focus when I try and half press the shoot button.

What 2011 holds for me in work, naturalisation and marriage

So it seems that Work permits to foreign nationals dropped 19%, and I'm one of 346 people (Chinese) who hold a work permit (I think stamp4 based permits are counted in the stats). It's a year on into my application for naturalisation and apparently they are processing early 2009 applications now and the average processing time as stated on the INIS site is 26months. There are also budget cuts like everywhere else, and possible changes in rules and so on. It's going to be a bundle of fun waiting for my application to go through. By the way, I'm getting married! I better check to see if marriage before naturalisation has any major issues for my application.


Related posts:

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

2011-05 -- Update on long naturalisation times in Ireland
Posted

macports, re-installing it on my desktop at work

My macports install got pretty messed up recently due to the way that perl5 (5.8, 5.10 and 5.12) is installed. It seem's that the default installation installs 5.8 which is old and conflicts with packages such as ikiwiki etc...

Pro-tip,

sudo port perl5 +perl5_12

Then install the apps that you want to use, but before doing so, do a

port variants PKG

before installing it, and install perl5 early on so that other packages that depend on it won't automatically pick perl 5.8.

State of the Irish Banks

It's the end of 2010 I've just been on vacation to visit family back in Hong Kong and one of the first questions I get asked is "How is Ireland doing?, I hear you guys are in trouble". I'd usually answer, "Yes Ireland is pretty messed up financially. By the way, I'm expecting a paycut and increased taxes next month."

What wasn't a shock to me was that AIB had effectively been nationalised while I was away and there is still a slim chance that Ireland might still default in 2011. Yet there is still more money being poured into the banking system from taxes, levies and raiding state pension funds.

It's terrible that Ireland now owes so much money to the EU and IMF at such a high interest rate (6% as far as I recall). The EU and IMF are like loan sharks with a knife at the Irish government's throat forcing them to take the loan. I'm still of the opinion that the banks should not have been bailed out and that NAMA is still a terrible idea and defaulting would still have been a better option. The motto of "Fail fast and fail early" is definately not a bad idea, Ireland would be on the path of recovery much earlier.

Thanks to the crisis my pension with the university that I am working with is probably wiped out and "invested" into the broken banking system. I guess if the cost of bailing out the banks was so minimal to the taxpayer in the first place then Ireland should have nationalised one of the banks early on or else let one of the banks default.

Hong Kong 2010 Part2

This is the second part of the post for Hong Kong 2010. There's significantly more photos here for this post. We went to China for two days on tour around Guangzhou. A good chunk of the photos were taken on our trip to China.

As with most of my more recent photos that I have been taking (on the 1000d), I've been shooting with raws and a uniwb whitebalance, I just throw all the raws at Rpp in batch mode leaving pretty much all the settings at the defaults. The only settings I customise is the TrueFilm option which I set to the K64 mode. I currently don't spend as much time as I should in fixing up the white balance of things. So far the automatic settings mostly work for me.

Hong Kong 2010 Part1

Another year and another holiday and christmas in Hong Kong, this is a first of a two part post with photos. These photos are from day 1 till day 7. It's mostly pictures of family members and places that we have visited. I might go back and comment on some of these photos at some point.

Pleasantly surprised about "Raw Photo Processor" for OSX

I've been pleasantly surprised with the results of RPP and in particular the TrueFilm simulations which simulate various films to get that classic look.

The first one is the default color mode, and the second one is a k64 film simulated mode.

I also tried the a25 b&w mode which I really liked as well. I think I will be sending the RPP developers a donation soon for such a nice and useful program.

I just wish I had shot more of my earlier photos (from my DSLR) as raws so I can process them again.

Re-installing some image processing software on my macbook

I'm yet again going on another trip away soon (to Hong Kong). I've uninstalled all of my applications from macports and I am now reinstalling a bunch of apps so I can process my photos during my holiday. So far I've re-installed,

  • hugin - Cause I like doing panoramas when I get a chance.
  • libraw - The new 0.12.0 release has a bunch of new demosaic methods which are nice to play with.
  • pfstools - and friends for doing HDR images, but I suspect I will just end up using enblend instead
  • enblend and enfuse - for stacking images to give a poorman's HDR image
  • ImageMagick - so I can resize, blur, process images as I need too.
  • RPP - for those moments where I need to manually go in and tweak an image.

As per usual nearly all the software is either opensource or free.

Windmill Lane graffiti for 2010-12-03

So, I've been away in London and Dresden for parts of the last month or so, plus I have been busy so there hasn't been much content in the recent months. There will be more to come as soon as I sort through all the photos.

Experimenting with darcs for version control

As I have been playing with tahoe-lafs I wanted to checkout the latest version of the trunk after I had reported some bugs on a release candiate. Their choice of VCS was darcs, I have had a look at darcs before in the past and just dismissed it due to the size of the community and the lack of performance when it comes to tracking large repositories of files.

Just to experiment I converted one of my own repositories to darcs to see how it works. I used this tool to do the conversion http://hackage.haskell.org/package/darcs-fastconvert and following the instructions from this post http://lists.osuosl.org/pipermail/darcs-users/2010-September/025251.html.

Just in case if the archives don't work

cabal install darcs-fastconvert

To convert a git repo to darcs

(cd gitproject; git fast-export --all --progress=500) | \
   darcs-fastconvert import gitproject-darcs

To convert a darcs repo into git

mkdir darcsproject-git
cd darcs-project-git
git init
(cd ../darcsproject; dracs-fastconvert export) | git fast-import

First impressions of darcs is, I kinda like the theory behind the patch sets and the interface. But it's slow and it doesn't handle binary blobs too well and the implementation is done with haskell. So I'm not entirely sure of it, it's a bit of a pain to compile but if you just use the binaries it's fine.

I'll keep using git for now, I might give vng more of a try since I like the darcs interface somewhat.

Using tahoe-lafs for doing secure backups

At first I had dismissed tahoe-lafs for any distributed filesystems needs that I had. Recently a project had cropped up in work that needed backups done. The nature of the data is probably considered to be sensitive and I'd rather not know what I am backing up. Not only does the data need to be backed up, but it has to be secure and safely stored for disaster recovery.

Assuming that you have managed to install tahoe-lafs, following the instructions to install it is pretty straightforward. Setting up your own grid or a node isn't as clear.

The basic tahoe grid needs to have 1 introducer node and at least 10 storage nodes, this is all based on the default settings. The first test setup that I created had the introducer node on one of my storage node.

On my first node

$ tahoe create-introducer /data/tahoe-introducer
... edit /data/tahoe-introducer/tahoe.cfg and set a nickname ...
$ tahoe start -d /data/tahoe-introducer
$ tahoe create-node /data/tahoe-node
... edit /data/tahoe-node/tahoe-node/tahoe.cfg and set a nickname ...
$ cp /data/tahoe-introducer/introducer.furl /data/tahoe-node
$ tahoe start -d /data/tahoe-node

Then on my other 9 nodes I just did...

$ tahoe create-node /data/tahoe-node
... edit /data/tahoe-node/tahoe-node/tahoe.cfg and set a nickname ...
$ scp node01:/data/tahoe-introducer/introducer.furl /data/tahoe-node
$ tahoe start -d /data/tahoe-node

Assuming all goes well and everything starts correctly, you can then setup a client node, for example on my desktop

  $ tahoe create-client
  $ scp node01:/data/tahoe-introducer/introducer.furl ~/.tahoe
  $ tahoe start

Once I started the client on my desktop, I was able to view the gateway page on my desktop by going to http://localhost:3456. The webgui is nice to view things, but its not very useful for managing your files in the system. You can do things like

 $ tahoe ls
 $ tahoe cp SOURCE alias:DEST

where alias is a mapping to a URI or hash in the tahoe-lafs system, this took me a while to figure out and understand, so its worth reading up the docs for more info on the concept of CAPs and ROOTCAPs etc... these are pretty much identifiers for retrieving your data.

You will need to protect these hashes and store them somewhere safe outside of the tahoe-lafs system if you with to keep your data private. Without these hashes you cannot access the data.

So far I've just experimented with the backup command and the deep-check command for repairing data.

   $ tahoe backup IMPORTANT alias:backup/IMPORTANT
   $ tahoe deep-check --add-lease --repair alias:

I'm so far impressed with how resilient the system is and how easy it was to setup the basic system. Performance wise, it's a bit slow compared to plain old rsync, tar and scp for uploading data. Downloading data from the system seems just as fast as any other method.

I've been testing on a mac desktop and a bunch of linux servers and a few windows machines. So the system is quite portable and runs on lots of different platforms.

I'm curious to see how many files can I put into the system before it grinds to a halt. I wonder how big files have to be before tahoe-lafs chokes. There's also a bunch of other issues with upgrades and migrating data as new releases of the software comes out, I guess this needs to be experimented with a bit more.

This is also of interest to read, its a thread on client-caching it does what I think I want to do.

autoconf and automake replacements

In the last few months I've been toying with cports, a continuation of the mpkg packaging system. The original version of mpkg had autoconf and automake as a part of the basic configuration of the system. This somewhat made it easier for the basic system to be setup by less advanced users. cports had removed this functionality so it could be packaged up pretty simply as an rpm for caoslinux, while this is well defined and usable for caoslinux it wasn't as useful for me.

After digging around for some autoconf and automake replacements I found these...

I eventually just chose to use Configure even though it had no basic checks for C++ or Fortran. It's pretty self contained and small, there's enough bits and pieces there for small projects to add new checks to the system. It's certainly not as powerful or full featured as some of the above listed tools. One of the main problems of the other tools is the requirement for them to be pre-installed on the machine that you wish to run the configurator on.

On the other hand Configure is just written in sh, it'll pretty much run on any UNIX like system, but it's probably never going to work well on windows. waf looks interesting enough for me to take a look at somepoint, it's almost as light weight as Configure except it requires python to be installed. For now Configure suits cports just fine, it small, simple and pretty hackable.

Dealing with large datasets

There's lots of free software out there that will provide distributed filesystems and grid based storage systems. Some notable ones which I've experimented and used in production before, from the point of plain data storage and management on HPC clusters

Each one of the above systems has there own features, issues and traits (and costs for running). Some are better suited for general purpose computing and some are suited for specialised storage and computing.

The traditional distributed systems like Lustre, GlusterFS and GPFS are pretty useful and offers a pretty good way of scaling out and expanding your system to meet your requirements. The only thing they lack is a complete storage management system, this is where iRODS shines, you can do much more with metadata with the data that you have. SECTOR sort of sits in between, it's interesting to see the p2p design of the system and the capabilities it has, it just lacks a coherent user interface right now, the FUSE client for SECTOR doesn't provide full posix compliance, but it's good enough to do most things.

A quick review of some of these systems (based on my experience)...

  • Lustre - Has great read/write performance, scales out well. Reliability is an issue, it relies on the sysadmin to have lots of experience in setting up LinuxHA. Although the software is free, the running cost and start up cost in learning how the technology works is quite high. There were issues quotas the last time we tried Lustre out on our compute clusters in work. Doing FSCK's on a Lustre is complicated and messy, recovering from failed OST's and OSS's used to be pretty involved. Setting up the MDS's and MDT's such that it is reliable, redundant and highly available can be an involved process. Migrating the OSS's, MDS's, OST's and MDT's between systems can be complicated.

  • GPFS - Like Lustre read/write performance is pretty good and scales quite well. Once you read the basic design document of GPFS and run through the process of setting up and adminning a GPFS you will appreciate how easy it is to run and maintain the system. The sysadmin doesn't need to learn about LinuxHA and all the complexities that go with it. IBM techsupport for GPFS (if you have a support contract) is extremely good, if you have ever needed to log an issue with them you will know, things get fixed and fast! The overall management of the NSD's and nodes in a GPFS cluster is pretty trivial.

  • GlusterFS - Write performance suffers as you add nodes to the system (the last time I tried it), but that depends on how you set it up (if you want raid1/0 with replication etc...). Read performance is as good as the above two filesystems. The setup cost is medium, I have no experience with setting up and monitoring linuxHA with GlusterFS, although GlusterFS does have some redundancy features, it lacked some management tools and documentation at the time when I tested it.

  • iRODS - I've only used this for research projects (developing micro-services and work-flows). It layers itself on top of an existing filesystems (or many machines and filesystems) and it can scale quite well from the looks of it. The main issue with iRODS is that to make the most of the cool features you will need to be using a GSI infrastructure for authentication, read this as grid certs! I'd pretty happy to use this for large scale archiving and storage of data that doesn't have a high rate of change. The start up costs for this is probably as high as Lustre and users will need to learn how to use the system as well since there isn't really a filesystem interface to the storage system. iRODS also lets the admin/user to write rules/work-flows and user defined functions (micro-services) to manipulate the metadata or data in the system. I have yet to try the FUSE client for iRODS. Having a LDAP back end to the user system would be nice.

  • SECTOR - I've only been testing this out on a small scale on a few nodes. The startup costs is quite low. Adding additional slave and master nodes to the system is trivial. Basic management of the system seems trivial. From the minimal amount of testing write performance is not great, but from first glance read performance scales pretty much linearly with the number nodes that you add to the system. The concept of a topology for your storage nodes is also interesting and useful as you can define replication numbers and distance based on the topology. The user interface is a bit weird and requires a little bit of technical knowledge to use. There is a FUSE client, but it isn't fully posix compliant. SECTOR like iRODS lets you write user defined functions to operate on your data, but in SECTOR's case every slave node becomes a compute element. SECTOR's user experience could be better, but it's not the worst. The user management probably needs more work. The system it provides is sufficient but it would be nice to have a bit more such as GSI, LDAP or PAM support. There isn't really metadata in the system, but is there a need for it?

So what was the point of the above? Not much really, except that there are lots of good solutions out there and you don't have to pay much for it if you have knowledgeable techies working for you and know what you want. If I want to run a cluster with a distributed and parallel filesystem I'd probably pick in the order of GPFS, Lustre then GlusterFS. If I want to analyse lots of data in parallel, from what I've played with I'd pick (in this order) iRODS/SECTOR, GPFS, Lustre then GlusterFS. For archiving there is no question that iRODS would be picked, but beyond that, anything else with a commercial support contract would probably be ok.

I spend some of my time in work getting paid to mess with these systems and sometimes I just do it for fun. I've a few ideas in the pipeline for the above tools. Some include

  • Doing analysis with bio-applications and bio-data (MRI, aligning, sequencing) with SECTOR.

  • Plugging in SECTOR as a backend storage system for iRODS to get the WAN capabilities of SECTOR.

  • Update the Python interface to SECTOR (as soon as I learn CPP and swig).

  • Test SECTOR in a WAN environment and maybe build a mini storage cloud for the bio people who have data distribution problems between our sites in work.

  • Test out the FUSE interface for iRODS and update some of the test code I have for iRODS.

Another year another GNIB card

Like every year, I queue up at 8am in the morning behind hundreds of people for hours. Well more like 142 people in front of me, I was number 143. I queued from 8am till about 11:30am before I talked to anyone and by 12:30pm I was done.

This year was different, I qualified for stamp4. This means I no longer need a work permit to work in the sector that I am employed in. All I needed to bring with me this time to the GNIB office was just my past work permits, P60's for the last few years and recent payslips.

It was still 150e to get the same though and they only do it for one year at a time.

But cases like this http://www.irishtimes.com/newspaper/ireland/2010/0911/1224278636072.html just scares me sometimes.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Windmill Lane graffiti for 2010-08-30

I've been playing more with UniWB, it's interesting so far. From the limited messing, when the colours are right, you can get some real nice pop effect. I'm not too confident with adjusting white balance myself.

If the auto white balance from dcraw was a bit better I'd probably use UniWB more.

UniWB and Whitebalance Multipliers for the LX3

I've recently discovered the whole idea of UniWB for maximising the dynamic range of digital cameras that can take raw images. In principle it sounds like a great idea if you are already shooting raws. I'm lazy but I was curious anyway. To be honest it's a bit useless on my LX3 due to the amount of optical distortions I get, plus I'm too lazy to process the images. Anyway, as a result of messing around with UniWB I found setting the white balance of the images to be interesting. I was reading this site http://www.guillermoluijk.com/tutorial/uniwb/index_en.htm. So to make a UniWB you just take a dark frame (lens cap on) at say ISO100, f/22, 1/4000sec, note that these files are usually camera model specific. Take the resulting raw file and check it with dcraw by doing

dcraw -v -w FILENAME.RW2

The multipliers are all supposed to be as close to 1 as possible (say with in a 5% error, this is just a guess), if it is then great, if not try again. You then take this raw file and set it as your custom white balance and then take pictures as normal. The previews will all come out greenish, so you will need to post process the image and correct the white balance. The basic idea is that the custom white balance will cause the histograms on your camera to give you a more realistic view of what's happening, you could probably get 1 or so stops more out of your shots.

So I got these settings by using dcraw -v -w FILENAME.RW2 for my LX3

  • Daylight
    • 2.049430 1.000000 1.661597 1.000000
  • Cloudy
    • 2.281369 1.000000 1.539924 1.000000
  • Shade
    • 2.528517 1.000000 1.418251 1.000000
  • Flash
    • 2.326996 1.000000 1.517110 1.000000
  • Halogen
    • 1.361217 1.000000 2.319392 1.000000

Some sample images....

This is what the camera see's in terms of white balance based on the UniWB custom white balance

This is dcraw's auto white balance

This is applying the mulitplier's got from above

This one was balanced using my whibal keyring using ufraw cause I was too lazy to figure out the patch to use dcraw to calibrate the image

The auto white balance in dcraw was pretty similar in look to the whibal calibrated image. I'm not too sure if I and committed enough to use raws. I do like being able to maximise my exposures for HDR or panoramic stitches though. It's more useful on my DSLR than on the LX3, since the LX3 doesn't have an RGB histogram.

Windmill Lane graffiti for 2010-08-23

Here's this months batch of tags and graffiti and so on... sadly some of the later images a bit blurry cause my LX3's shutter button gets stuck sometimes so I can't get good focus for every shot. I probably just need to get the camera cleaned or serviced.

ikiwiki and cgit

Since we migrated to this new machine I left the gitweb.cgi disabled, but I recently wanted to publish some stuff with git. Of course having a web based interface would be nice. So I thought, what the hey, I should try cgit. Feature wise it's probably on par with gitweb. Installation wise, it's pretty straight forward if you read the docs. The best thing was the configuration is much clearer than gitweb imho.

What was nice was the config for ikiwiki is now...

 historyurl => 'http://www.sgenomics.org/cgi-bin/cgit.cgi/jtang/wiki.git/log/[[file]]',
 diffurl => 'http://www.sgenomics.org/cgi-bin/cgit.cgi/jtang/wiki.git/commit/?id=[[sha1_commit]]',

which wasn't too hard to figure out and setup.

DWM on OSX (revisited)

Having recently completely moved to OSX in work, I really miss DWM and I'm want automatic window management. I previously wrote about using DWM on OSX but that was long ago on OSX 10.5. OSX 10.6 has changed a few things and my previous setup now no longer works.

Firstly, download and install XQuartz from http://xquartz.macosforge.org/trac/wiki/X112.5.3 and then... checkout these instructions from here http://gist.github.com/311377 this has saved me lots of time figuring stuff out, but it's more or less easy to get DWM to work on OSX.

Not much to say about this really, but it isn't too hard to get going.

Checksumming files with openssl

I always seem to forget what tool I use to checksum files when I'm building packages for macports, so here's a note for myself

openssl sha1 fossil-src-20100805100943.tar.gz 
openssl rmd160 fossil-src-20100805100943.tar.gz 
openssl md5 fossil-src-20100805100943.tar.gz 

OpenSSL should be available on most if not all unix/linux/osx like systems.

While I'm at it, here's a macports file for fossil scm

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- 
# vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
# $Id$
PortSystem          1.0
name            fossil
version         20100805100943
categories      devel
maintainers     tchpc.tcd.ie:jtang
description     Fossil: Simple, high-reliability, distributed software configuration management
homepage        http://www.fossil-scm.org/
master_sites        http://www.fossil-scm.org/download
distfiles       ${name}-src-${version}${extract.suffix}
distname        ${name}
worksrcdir          ${name}-src-${version}
depends_lib     port:zlib

use_configure       no

checksums           md5     33c2cf512c72f5b153dbfc26867e16ed \
            sha1    8840609b75106a9d685486a2778d9b1319b37bc8 \
            rmd160  80b92c63fd431c35a793eddcab067e992de0bd3b


destroot.destdir  DESTDIR=${destroot} prefix=${prefix}

build.target        all

configure {}

destroot {
     xinstall -m 755 -d ${destroot}${prefix}/bin
     xinstall -m 755 ${worksrcpath}/fossil ${destroot}${prefix}/bin
}
Checklist for securing a Scientificlinux 5.x based system

I've recently had to migrate some services (relatively quickly) from one machine to another. One of the requirements of the system is that it should be relatively securely setup. So here's a checklist of some of the software and things to watch out for (at least for me when setting things up).

Some, if not all of the notes were taken from the Centos wiki as a reference for myself.

Software

  • shorewall - for firewalling
  • etckeeper - for tracking changes in /etc/

Checklist for setting up the machine

  • Did a stock SL5x install with only the server option ticked. I left selinux turned on in targetted mode.
  • Used the default partition layout, ideally sometime should be spent on partitioning things correctly and enable nosuid,noexec where appropriate.
  • Generate a list of installed apps for reference, you can do this by doing either yum list installed > ~/installed.txt or rpm -qa --qf "%{name}\n" | sort | uniq > ~/installed.txt
  • Make sure regular updates are on, in my case I have a prepared rpm for our local mirrors and repos which I install and then run an upgrade/update to make sure I have all the latest security fixes, then reboot.
  • Install etckeeper
  • Run a chkconfig --list and see what is enabled, then turn off what I don't need.
  • Do the suggested changes to protect the machine at a physical level, I haven't done all these steps yet. I'm not sure if I need all the extra security since the machine is in a secure physical location.
  • Enable a bios password
  • Change boot order of the machine to not allow booting from network or removable physical media.
  • Require a password for the bootloader (grub or lilo)
  • Require a password for single user mode
    echo "Require the root pw when booting into single user mode" >> /etc/inittab
    echo "~~:S:wait:/sbin/sulogin" >> /etc/inittab
    echo "Don't allow any nut to kill the server"
    perl -npe 's/ca::ctrlaltdel:\/sbin\/shutdown/#ca::ctrlaltdel:\/sbin\/shutdown/' -i /etc/inittab
  • Disable USB mass storage devices
    echo "Disabling USB Mass Storage"
    echo "blacklist usb-storage" > /etc/modprobe.d/blacklist-usbstorage
  • Restrict the root account
    echo "tty1" > /etc/securetty
    chmod 700 /root
  • Setup some password policies
    echo "Passwords expire every 180 days"
    perl -npe 's/PASS_MAX_DAYS\s+99999/PASS_MAX_DAYS 180/' -i /etc/login.defs
    echo "Passwords may only be changed once a day"
    perl -npe 's/PASS_MIN_DAYS\s+0/PASS_MIN_DAYS 1/g' -i /etc/login.defs
  • Change the default password hashing from MD5 to sha512
    authconfig --passalgo=sha512 --update
  • Change the default umask restrictions
    perl -npe 's/umask\s+0\d2/umask 077/g' -i /etc/bashrc
    perl -npe 's/umask\s+0\d2/umask 077/g' -i /etc/csh.cshrc
  • Enable pam_tally2 such that if a user fails to login after three attempts, lock them out for 60mins before they can try again. The file /var/log/tallylog is a binary log containing failed login records for pam. You can see the failed attempts by running the pam_tally2 command without any options, and unlock user accounts early by using pam_tally2 --reset -u username
    touch /var/log/tallylog
    cat << 'EOF' > /etc/pam.d/system-auth
    #%PAM-1.0
    # This file is auto-generated.
    # User changes will be destroyed the next time authconfig is run.
    auth        required      pam_env.so
    auth        sufficient    pam_unix.so nullok try_first_pass
    auth        requisite     pam_succeed_if.so uid >= 500 quiet
    auth        required      pam_deny.so
    auth        required      pam_tally2.so deny=3 onerr=fail unlock_time=60

    account     required      pam_unix.so
    account     sufficient    pam_succeed_if.so uid < 500 quiet
    account     required      pam_permit.so
    account     required      pam_tally2.so per_user

    password    requisite     pam_cracklib.so try_first_pass retry=3 minlen=9 lcredit=-2 ucredit=-2 dcredit=-2 ocredit=-2
    password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=10
    password    required      pam_deny.so

    session     optional      pam_keyinit.so revoke
    session     required      pam_limits.so
    session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
    session     required      pam_unix.so
    EOF
  • Logging users out automatically after 15mins
    echo "Idle users will be removed after 15 minutes"
    echo "readonly TMOUT=900" >> /etc/profile.d/os-security.sh
    echo "readonly HISTFILE" >> /etc/profile.d/os-security.sh
    chmod +x /etc/profile.d/os-security.sh
  • Restrict cron and at
    echo "Locking down Cron"
    touch /etc/cron.allow
    chmod 600 /etc/cron.allow
    awk -F: '{print $1}' /etc/passwd | grep -v root > /etc/cron.deny
    echo "Locking down AT"
    touch /etc/at.allow
    chmod 600 /etc/at.allow
    awk -F: '{print $1}' /etc/passwd | grep -v root > /etc/at.deny
  • Turn of wireless if it exists
    for i in $(find /lib/modules/`uname -r`/kernel/drivers/net/wireless -name "*.ko" -type f) ;
        do echo blacklist $i >> /etc/modprobe.d/blacklist-wireless ;
    done
  • Sysctl Security (/etc/sysctl.conf)
    net.ipv4.ip_forward = 0
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0
    net.ipv4.tcp_max_syn_backlog = 1280
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    net.ipv4.conf.all.accept_source_route = 0
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.secure_redirects = 0
    net.ipv4.conf.all.log_martians = 1
    net.ipv4.conf.default.accept_source_route = 0
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.default.secure_redirects = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    net.ipv4.icmp_ignore_bogus_error_responses = 1
    net.ipv4.tcp_syncookies = 1
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.tcp_timestamps = 0
  • Use TCP Wrappers
    echo "ALL:ALL" >> /etc/hosts.deny
    echo "sshd:ALL" >> /etc/hosts.allow
  • Disable remote root logins

    • edit /etc/ssh/sshd_config and make sure that PermitRootLogin=no
  • Create a wheel group if it doesn't exist add a line like this for the sudoers file

    %wheel  ALL=(ALL)       ALL
  • Add some users to the wheel group
    gpasswd -a jtang wheel
  • Check that selinux is enabled, and confine applications/users as needed, but generally speaking the defaults are fine.
    sestatus
  • Configure sendmail so I get system emails. Make sure tcpwrappers is configured to only allow localhost to connect

  • Configure shorewall, edited these files

    • policy
    • routestopped
    • rules
    • shorewall.conf
    • interfaces
    • zones
My first time lapse

I got an intervalometer over the weekend, and here's my first time lapse video. It's not as good as I hoped it would be, but it's not too bad.

Tokyo 2010, and it was fun

I've been back for a week now, the photos are sorted and are online. So while I wait for stuff to compile, I'm posting these pics.

The pics are certainly the highlights of the trip to Tokyo with Frau Klein.

Macports - Portfile - mr - a Multiple Repository management tool

Since I now have a mac for work, I've been moving my old work files and creating some packages for macports. Here's the first of the lot that I use regularly, it's for mr - a Multiple Repository management tool

# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- 
# vim:fenc=utf-8:ft=tcl:et:sw=4:ts=4:sts=4
# $Id$
PortSystem      1.0
name            mr
version         0.48
categories      devel
maintainers     tchpc.tcd.ie:jtang
description     a Multiple Repository management tool
homepage        http://kitenet.net/~joey/code/mr/
master_sites    http://ftp.de.debian.org/debian/pool/main/m/mr/
distfiles       mr_${version}${extract.suffix}
distname        mr
depends_lib     port:perl5.8

use_configure       no

checksums           md5     3d3770f7d4b1bb4f92cdfa9c57090dfe \
                    sha1    bdaac495252c562f708e9ddda47b3f54424eb07a \
                    rmd160  ee55d5c9ec37719c5c56d98ed314d4693dd0e102


build.args    PERL_PATH="${prefix}/bin/perl" DESTDIR=${destroot} prefix=${prefix}
destroot.destdir  DESTDIR=${destroot} prefix=${prefix}

build.target      build

configure {}

destroot {
     xinstall -m 755 -d ${destroot}${prefix}/bin
     xinstall -m 755 ${worksrcpath}/mr ${destroot}${prefix}/bin
     xinstall -m 755 ${worksrcpath}/webcheckout ${destroot}${prefix}/bin

     xinstall -m 755 -d ${destroot}${prefix}/share/man/man1
     xinstall -m 755 ${worksrcpath}/mr.1 ${destroot}${prefix}/share/man/man1
     xinstall -m 755 ${worksrcpath}/webcheckout.1 ${destroot}${prefix}/share/man/man1
}

dwm or pristine-tar is on my hit list next.

Setting up a FreeRADIUS server on Scientificlinux

The following notes are a summary of HOWTO: Incremental Setup of FreeRADIUS Server for EAP Authentications

Prerequisites

  • Scientificlinux 5.x
  • A working LDAP directory - I'm pointing to a LDAP directory that I already have on our network.
  • FreeRADIUS - I just installed this from the SL5.x repositories

Testing

Step 1 - test on the localhost

Debugging the server

/usr/sbin/radiusd -X

Testing the setup, first add this to your /etc/raddb/users file

dummyuser User-Password := "dummypasswd"

Note that I am using version 1.1.x, in newer FreeRADIUS versions you will need to use Cleartext-Password

Then do

radtest dummyuser dummypasswd 127.0.0.1 1812 testing123

The testing123 part is a default shared secret password. The above steps are just used to see if the basic setup works or not.

Step 2 - test from a remote host

Now edit /etc/raddb/clients.conf and add in your host

client 226.134.191.112 {
secret = testing123
shortname = 226.134.191.112
}

Restart the radiusd process then do this from 226.134.191.112

radtest dummyuser dummypasswd 226.134.191.113 1812 testing123

where 226.134.191.113 is the radius server. It should complete sucessfully

Step 3 - EAP (with TLS) and PEAP configuration

Prerequisites

  • SSL keys and certs for the radius server, see the eap.conf for comments.
  • Step 1 and Step 2 works.

at minimum you will want to have this type of configuration in your eap.conf file.

eap {
    default_eap_type = md5
    timer_expire     = 60
    ignore_unknown_eap_types = no
    cisco_accounting_username_bug = no
    md5 {
    }
    leap {
    }
    gtc {
        auth_type = PAP
    }
    tls {
        private_key_password = whatever
        private_key_file = ${raddbdir}/certs/cert-srv.pem
        certificate_file = ${raddbdir}/certs/cert-srv.pem
        CA_file = ${raddbdir}/certs/demoCA/cacert.pem
        dh_file = ${raddbdir}/certs/dh
        random_file = ${raddbdir}/certs/random
    }
    peap {
        default_eap_type = mschapv2
    }
    mschapv2 {
    }
}

For testing you may want to just comment out the group=radiusd and user=radiusd options in radiusd.conf

Step 4 - test with WPA Supplicant or similar tool

I used the eapol_test program from wpa_supplicant with this configuration file

network={
    eap=PEAP
    eapol_flags=0
    key_mgmt=IEEE8021X
    identity="dummyuser"
    password="dummypasswd"
    #ca_cert="/home/jtang/ca.crt"
    phase2="auth=MSCHAPV2"
    anonymous_identity="anonymous"
}

run like this

eapol_test -c radeap.conf  -a226.134.191.113 -p1812 -stesting123 -r1

eapol_test was built with

wget http://hostap.epitest.fi/releases/wpa_supplicant-0.5.10.tar.gz
tar xvf wpa_supplicant-0.5.10.tar.gz
cd wpa_supplicant-0.5.10/
cp defconfig .config
make eapol_test

Making FreeRADIUS look up LDAP

Add this to your /etc/raddb/dictionary file

VALUE           Auth-Type               Local                   0
VALUE           Auth-Type               System                  1
VALUE           Auth-Type               SecurID                 2
VALUE           Auth-Type               Crypt-Local             3
VALUE           Auth-Type               Reject                  4
VALUE           Auth-Type               LDAP                    5

In eap.conf in the PEAP section set

default_eap_type = peap

In radiusd.conf in the LDAP section set at least this

server = "ldap.tchpc.tcd.ie"
identity = "cn=admin,dc=tchpc,dc=tcd,dc=ie"
password = myadminpassword
basedn = "ou=people,dc=tchpc,dc=tcd,dc=ie"
#filter = "(posixAccount)(uid=%u))"
filter = "(uid=%u)"
# access_attr = "dialupAccess"
password_attribute = userPassword
set_auth_type = yes

then in the authorize section uncomment the ldap option and comment out the files module.

Patching 'fsl_sub' for SLURM

I recently patched 'fsl_sub', which is apart of fsl, to allow it to batch submit tasks to our computer clusters. I'd submit the patch to the authors of the software but I couldn't figure out where to send the patch to. I really didn't need to sign up to another mailing list. So here it is.

--- fsl_sub.orig        2010-06-11 13:03:35.279077000 +0100
+++ fsl_sub     2010-06-11 13:04:33.409821000 +0100
@@ -100,6 +100,10 @@
     fi
 fi

+if [ "x$SLURM_JOB_ID" != "x" ] ; then
+       METHOD=SLURM
+fi
+

 ###########################################################################
 # The following auto-decides what cluster queue to use. The calling
@@ -123,6 +127,11 @@
        queue=verylong.q
     fi
     #echo "Estimated time was $1 mins: queue name is $queue"
+
+    # if slurm environment is detected use the compute partition, change this to suit
+    if [ $METHOD = SLURM ] ; then
+           queue=compute
+    fi
 }


@@ -200,7 +209,7 @@
 # change. It also sets up the basic emailing control.
 ###########################################################################

-queue=long.q
+queue=compute
 mailto=`whoami`@fmrib.ox.ac.uk
 MailOpts="n"

@@ -364,6 +373,40 @@
        ;;

 ###########################################################################
+# SLURM method
+# this is a very naive way of doing things, its just to simply fire off all
+# the tasks individually to the resource manager
+###########################################################################
+
+       SLURM)
+               if [ $verbose -eq 1 ] ; then
+                       echo "Starting Slurm submissions..." >&2
+               fi
+               _SRMRAND=$RANDOM
+               _SRMNAME=$JobName$SRMRAND
+               echo "========================" >> sbatch.log
+               echo "= Starting submissions =" >> sbatch.log
+               echo "========================" >> sbatch.log
+               date >> sbatch.log
+while read line
+do
+        if [ "x$line" != "x" ] ; then
+sbatch -J $_SRMNAME -o "slurm-log-$_SRMNAME-%j-%N.out" -t 01:00:00 -p $queue -n 1 <<EOF
+#!/bin/sh
+echo 
+echo $SLURM_JOB_NAME
+echo $SLURM_JOB_ID
+echo $SLURM_JOB_NODELIST
+echo
+date
+echo
+$line
+EOF
+        fi
+done < $taskfile >> sbatch.log 2>&1
+       ;;
+
+###########################################################################
 # Don't change the following - this runs the commands directly if a
 # cluster is not being used.
 ###########################################################################

It's not the prettiest thing around, its quick and dirty and it spams the queue system pretty good. It's cut down a job which did take 5-6days to about 8-9hrs. That is it used to run on 1cpu, now it runs on 24-32cpus at a time.

Migrating my development folder from linux to OSX

One of the things that I don't like too much about OSX is the case-insensitive and type sensitive defaults. It's certainly nice and easy to use for most people, but if you are wanting to write shell scripts that you want to run elsewhere it can be a bit of a pain. I prefer to have case and type sensitive file system for development. "Disk Utility" is pretty handy for creating disk images with the properties that I need.

Planning my trip to Tokyo
  • Thursday - be jetlagged, going to arrive in the afternoon, so I will probably just wander around the local area at night.
  • Friday - Shinjuku and Shibuya - need to kill some time on my own.
  • Saturday - Tsukiji Fishmarket, Ginza, Imperial Gardens then to Shinjuku for dinner (as it suggests in the Lonely planet)
  • Sunday - Harajuku & Aoyama - surrounding areas, which ever day it is when all the dressed up action is going to be.
  • Monday - Ueno - museums around Ueno, Parks and Temples.
  • Tuesday - Odaiba - to see all those crazy buildings.
  • Wednesday - Akasaka & Roppongi - surrounding area, Roppongi Hills.
  • Thursday - Ghibli museum at Mitaka and surrounding areas.
  • Friday - Akihabara, Electric town and surrounding areas.
Backing up a Flickr account

It just occured to me in the last few days that I should probably do a complete backup of my flickr account. I came across these apps which seem useful but not all of them are cross platform apps.

Migratr seems to be the most useful as it tries to grab album/set information, it appears to be a windows only application.

Going to Tokyo (A very unplanned holiday)

frau-klein is going to Tokyo for work and I'm tagging along. It's a badly planned (well unplanned) holiday, considering we were going to go to Dresden originally.

Some useful links for preparing on a short notice for i.e. hotels and plane tickets

Some guides/reading material that I ended up getting

  • Lonely Planet - Toyko Encounter
  • AA Citypack guide and foldout map

Now that I have a plane ticket and a hotel, I need an itinery for the trip.

Secretwars 2010

I completely missed sorting out these photos last month, but here they are, secretwars at Bernard Shaws from last month (April 2010).

An iMac for work - finally!

Finally after a few years of moaning and complaining I finally got a mac for work! Upon installing it, the first thing I do is install the apps that I use all the time. I guess I should compile a list of the things I installed.

  • Xcode
  • Google Chrome
  • Adium X
  • Colloquy
  • VLC
  • Cog
  • Perian
  • Macports (which points to my own local mirror)
  • VirtualBox

From Macports, the first set of apps that I went to install were

  • tsocks
  • git-core
  • ikiwiki
  • emacs / emacs-app
  • org-mode
  • rsync
  • keychain
  • unison
  • xfig
  • transfig
  • tig
  • latex
  • gnuplot
  • dar
  • aria2
  • mercurial
  • darcs
  • subversion

It seems I have quite a few VCS's installed, I guess that comes with the territory when you need to checkout lots of source code and stuff.

I also installed mksh cause I couldn't deal with bash doing weird stuff with its configs when I use screen.

How I would build a High Performance Computer

So I've talked about "How not to build a High Performance Computer". I'd now like to rant on how I would build a "Cheap High Performance Computer". If one does an online search for "How to build a cluster", you will more than likely get hundreds if not thousands of hits. You're also likely to get performance analysis reports and cost analysis reports etc...

Clusters are systems, not just individual components. It's the sum of all the components which makes the system useful and or usable for the user who wants to do their work. Clusters usually contain at least the following.

  • Controller Nodes
  • Compute Nodes
  • Storage Nodes
  • Management Network
  • Compute and or IO Network
  • Data center / Machine room with the necessary power and cooling
  • Software stack to manage and provide the service

Ideally I would group the Controller, Compute and Storage nodes all into the same purchasing pool but only vary them slightly to allow them to function with their designated services. It makes purchasing easier.

Step 1

Decide on what application you want to run. Will be bio-informatics, quantum chemistry, high energy particle physics.

Step 2

Find out what the code does. Does the code do lots of file IO? Does it do lots of communications. Who develops the code and if its a local developer, can you influence how the code is designed.

Step 3

Once Step 2 is done, optimise your budget for the required hardware to complete your task. Procuring a capacity machine is different from a capability machine. Of course there are also different types of capacity machines and capability machines.

You could for instance go for high clock rate machines with 1gb ram per core and cheapish infiniband/gig-e/10gig-e if your jobs are BLAS/LAPACK operations intensive. Or else if you are latency/communications sensitive you might really care about a fast scalable comms network, you might care less about clock rate, so cheaper processors will be fine. You may even not care about comms or high clock speeds, but rather more memory is useful.

It all boils down to an optimisation problem.

Step 4

Once you've decided on the type of machine/equipment you think you need, you might want to start and build a list of the applications you want to run with some realistic sample inputs that the user wants to run, use this as benchmarks for evaluating whether the suggested hardware that vendors suggest is good or not. Synthetic benchmarks like HPL are pretty useless in most cases for evaluating and are only good for PR purposes.

Step 5

Try and build a list of the required hardware and see if you can estimate roughly how much you can get it for.

Step 6

Armed with a list of benchmarks and a manifest of what you think you might need. Write a request for procuring this machine and issue it to the various vendors you are interested in.

Step 7

Collect the replies if any and evaluate the performance of the suggested machines based on the benchmarks that the vendors have run for you. Evaluate whether the proposals meet your budget or not.

Step 8

Procure the machine, install it.

Step 9

Benchmark the machine that you have procured to make sure it does what you and the think the cluster should do. Make sure everything is working, if there are failover components, try breaking the system in controlled ways to make sure you have the necessary redundancy required.

Step 10

Signoff, and profit!

Setting up a KVM based virtual machine on Scientific Linux

This set of notes is based on the wiki page on the CentOS Wiki Page on KVM

Required Hardware (Ideally)

Intel or AMD based machine that supports virtualisation. On intel based machines you can check by

grep vmx /proc/cpuinfo

On and AMD based machine you can check by

grep svm /proc/cpuinfo

Your host machine should probably be multi-core if possible, and have at least 2gigs of ram, but more is preferred. Have the appropriate kernel module loaded.

If its Intel do

modprobe kvm-intel

If it's AMD do

modproble kvm-amd

Required Software (What I used)

  • The ScientificLinux 5.4 distro installed on my host system.
    • The kmod-kvm package - it's apart of the SL repos as far as I know.
    • The kvm package - it's apart of the SL repos as far as I know.
    • The QEMU stack, which came with the kvm package.
    • The kvm-qemu-img - it's apart of the SL repos as far as I know.
    • The bridge-utils package to setup a bridged network.
    • The tunctl package which is available from the epel repository.

KVM should be apart of recent linux kernel releases (since 2.6.20 according to the releasenotes).

Specifically these were the packages I used...

[root@duo ~]# rpm -qa | egrep 'kvm|bridge-utils|tunctl'
kvm-83-105.el5_4.13.x86_64
kmod-kvm-83-105.el5_4.13.x86_64
kvm-tools-83-105.el5_4.13.x86_64
etherboot-zroms-kvm-5.4.4-10.el5.0.sl.x86_64
bridge-utils-1.1-2.x86_64
kvm-qemu-img-83-105.el5_4.13.x86_64
tunctl-1.5-2.el5.x86_64

Creating a Virtual Machine

First make sure the permissions for the KVM device node is correct

chown root:kvm /dev/kvm
chmod 0660 /dev/kvm

and add yourself to the kvm group

usermod -G kvm -a jtang

The process is

  1. Create a disk image with qemu-img or use dd
  2. Boot the disk image with qemu-kvm with a valid install media

There are some choices to be made when creating the images, you can either use the raw format which will result in pretty big full sized disk images. Or else you can use the qcow2 format which is a sparse format where the disk image will grow to the maximum size that you have defined.

Create the disk image

qemu-img create -f qcow2 disk.img 5G

or

dd if=/dev/zero of=disk.img bs=1G count=5

Boot the image for an install

Note in Scientific linux the qemu-kvm binary is located in /usr/libexec/qemu-kvm, you could create a symlink for it if you wish.

/usr/libexec/qemu-kvm -hda disk.img -cdrom archlinux-2009.08-core-x86_64.iso -m 512 -boot d

Once it is booted up you can just follow the instructions on installing your guest OS.

Bridged Tap Networking and KVM

By default KVM supports NAT for its guests, it's not ideal for production environments. I would prefer to have a bridged network so my guests can appear on my network with a real IP address . Edit /etc/udev/rules.d/90-kvm-rules such that

KERNEL=="kvm",          NAME="%k", GROUP="kvm", MODE="0660"
KERNEL=="tun",          NAME="net/%k", GROUP="kvm", MODE="0660"

or else do this in /etc/rc.local

chown root:kvm /dev/net/tun
chmod 0660 /dev/net/tun

Now run these commands

brctl addbr br0
ifconfig eth0 0.0.0.0
brctl addif br0 eth0
dhclient br0
tunctl -b -u jtang
ifconfig tap0 up
brctl addif br0 tap0
iptables -I RH-Firewall-1-INPUT -i br0 -j ACCEPT
qemu-kvm ~/win2k.img -m 512 -net nic -net tap,ifname=tap0,script=no

If you do not have a DHCP server to assign your NIC an address you could do this in place of the dhclient command assuming you are on a 192.168.1.0/24 network.

ifconfig br0 192.168.1.120 netmask 255.255.255.0 up
route add -net 192.168.1.0 netmask 255.255.255.0 br0
route add default gw 192.168.1.1 br0

Attempting to run the Virtual Machine

There are some issues that I haven't addressed, such as monitoring the kvm guest.

/usr/libexec/qemu-kvm -hda disk.img  -m 512 -net nic -net tap,ifname=tap0,script=no -nographic  -daemonize

I should probably configure the guest to be accessible via a serial console as well.

Using virtio for the networking

/usr/libexec/qemu-kvm -hda disk.img  -m 512 -net nic,model=virtio -net tap,ifname=tap0,script=no

Using virtio for the storage

/usr/libexec/qemu-kvm -drive file=disk.img,if=virtio,boot=on  -m 512 -net nic,model=virtio -net tap,ifname=tap0,script=no

You will need to correct references to sda/hda to vda in grub and in your fstab, and also make sure your initrd has the virtio, virtio_blk and virtio_pci loaded up at boot. The best thing to do is to use disk labels or blkid's for fstab instead of device names.

Installing ScientificLinux 5.4 as a guest

Some notes on installing SL5.4 as a guest. There's probably more efficient ways of doing this.

First create a disk image, I chose to use qcow2 32gb is size.

qemu-img create -f qcow2 sl5x.img 32G

Then install the OS, I used the SL5.4 install dvd. I suppose you could do a network install if you wanted.

/usr/libexec/qemu-kvm -drive file=sl5x.img,if=virtio,boot=on,cache=writethrough  -m 768 -cdrom SL.54.110309.DVD.x86_64.disc1.iso

Running the virtual machine

/usr/libexec/qemu-kvm -drive file=sl54-kvm.img,if=virtio,boot=on,cache=writethrough \
-net nic,macaddr=DE:AD:BE:EF:25:12,model=virtio \
-net tap,ifname=tap0,script=no  -m 768  -nographic

I also enabled a serial console and made grub redirect its output to the serial console.

My grub.conf file is now something like this

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
serial --unit=0 --speed=9600 --word=8 --parity =no --stop=1
terminal --timeout=10 --dumb serial console
title Scientific Linux (2.6.18-164.2.1.el5)
    root (hd0,0)
    kernel /vmlinuz-2.6.18-164.2.1.el5 ro root=/dev/VolGroup00/LogVol00 console=tty0 console=ttyS0,9600n8
    initrd /initrd-2.6.18-164.2.1.el5.img

And in the inittab near the bottom,

# Run gettys in standard runlevels
co:2345:respawn:/sbin/agetty ttyS0 57600 vt100-nav
1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2
3:2345:respawn:/sbin/mingetty tty3
4:2345:respawn:/sbin/mingetty tty4
5:2345:respawn:/sbin/mingetty tty5
6:2345:respawn:/sbin/mingetty tty6

The above changes lets me connect to the virtual machine without needing a graphical interface. This however fails to work if you daemonize the process. According to the man pages it is possible to redirect the virtual machine serial port to the hosts serial port in /dev/xxx somewhere, I didn't get this to work. However I did get the redirection to a pty to work and I was able to connect to it via minicom.

Enlarging a disk image

I found this online, its the quickest way to enlarge your disk image, note it must be a raw image (assuming you started with a 10gb image you will end up with a 20gb image with the following commands)

dd if=/dev/zero of=foo.img bs=1 count=1 seek=$(($(stat -c%s foo.img)+10*1024**3-1))

or

dd if=/dev/null of=foo.img bs=1b seek=20G count=0

or you could cat a file full of zeros and append it to your diskimage, its up to you.

Apparently the best method and the safest method is to do this...

qemu-img create -f raw morespace.raw <size>
qemu-img convert -f qcow2 existing.img -O raw existing.raw
cat morespace.raw >> existing.raw
qemu-img convert -f raw existing.raw -O qcow2 existing-larger.img

I haven't tried any of the above yet to confirm which is the best method.

Notes

  1. If you find that bridged networking doesn't work, make sure the tap interface has been added to the bridge.
First release of Hugin for 2010

I'm a bit of a fan of the software Hugin, the recently did the release. As per usual, the windows release is a bit behind compared to the linux and mac binaries. I'll need to fire up my mac or my linux vm to try it out for now.

ufraw 0.17 is out

ufraw 0.17 got released on april fools day. I must say it's neat, I might stop using rawtherapee in favour of ufraw. I quickly tested it by taking an old raw that I had (of Frau Klein) and just running the raw file through a few raw converters

Picasa 3.6.0, with the default settings

ufraw 0.17, with the default settings

Rawtherapee 2.4.1, with the default neutral setting

I kinda like the Rawtherapee colours more than the ufraw ones overall, but Frau Klein looks much better in the ufraw processed image, the colours of the sky is best in the Rawtherapee image. The lens correction in ufraw certainly is useful, it's pretty automatic, I think I might shoot more raws, if only Rawtherapee had lensfun support it would be great.

Forcing a reboot on linux (CentOS 5.x and friends)

I recently had some issues with a linux box that had a few processes in a disk wait state. I couldn't do a "reboot" or even a "reboot -f" the system would just stop waiting for the processes to be killed. But this worked...

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

It was on a CentOS 5.x based system.

My first batch of madeleine cakes

I kinda got bored and I decided to learn how to bake me some madeleine cakes. After googling around for some receipes I decided they all were pretty easy to make so I adapted one from a few others that I saw.

List of ingredients

  • 100g flour
  • 100g granulated sugar
  • 100g unsalted butter
  • 1/2 tea spoon of baking powder
  • 1/2 tea spoon of vanilla extract
  • 3 large eggs
  • zest of a lemon (I guess as much or as little as you want)
  • pinch of salt

Procedure

  1. Melt the butter
  2. Sift the flour, salt and baking powder together
  3. While butter is melting beat, the eggs and sugar for about 3-5mins in a bowl, stop when the mixture has gain about 2-3x in volume, add lemon zest and vanilla extract.
  4. Add some flour to the egg mixture bit by bit with a spatula or else you will lose the air in the egg mixture.
  5. Spoon some of the egg, sugar and flour mixture into the melted butter, mix up the butter and the small amount of egg, sugar and flour mixture together. Add this back into the main bowl of egg, sugar and flour mixture. Fold it all together.
  6. Leave the mixture to sit in the fridge for about 30-60mins
  7. Preheat oven to ~190-200c.
  8. Spoon the mixture into a greased up and floured baking tin, or else use those little paper cup things so you don't need to grease and flour the tin. Make sure you fill each unit up to about 90%.
  9. Bake for about 12mins or so.
  10. Remove from tin and let it cool

Conclusion

I made a mess of my first batch in terms of filling up my tins too much so they spilt out and I also baked them a little too long so they were more like cookies than cakes. The cakes tasted the way they are supposed to, so that is a good thing. Photo's to follow in my next attempt to make them.

Linking against SCALAPACK

Having compiled scalapack, I now want to link against it to test out some examples. Here's a copy of my Makefile that I butchered from the example and testing directory from the scalapack distribution.

#
# scalapack example LU
# 


#
# where the libraries are
#
HPCE2LIB = /misc/shared/apps/hpce2/libs
SCALAPACKLIB = $(HPCE2LIB)/libscalapack.a

BLASLIB = $(HPCE2LIB)/blas_LINUX.a
LAPACKLIB = $(HPCE2LIB)/lapack_LINUX.a

BLACSLIB = $(HPCE2LIB)/blacs_MPI-LINUX-0.a
BLACSCINIT = $(HPCE2LIB)/blacsCinit_MPI-LINUX-0.a
BLACSFINIT = $(HPCE2LIB)/blacsF77init_MPI-LINUX-0.a

FBLACSLIB = $(BLACSFINIT) $(BLACSLIB) $(BLACSFINIT)
CBLACSLIB = $(BLACSCINIT) $(BLACSLIB) $(BLACSCINIT)

LIBS = $(SCALAPACKLIB) $(FBLACSLIB) $(LAPACKLIB) $(BLASLIB)

#
# compilers and the compiler options 
#
F77 = mpif90
CC = mpicc
NOOPT =
F77FLAGS = -O3 $(NOOPT)
CCFLAGS = -O4
SRCFLAG =
F77LOADER = $(F77)
CCLOADER = $(CC)
F77LOADFLAGS =
CCLOADFLAGS =
CDEFS = -DAdd_ -DNO_IEEE -DUsingMpiBlacs


#
# build the program
# 
dmatgen = pdmatgen.o pmatgeninc.o
dlinchk = pdlaschk.o pdlafchk.o
dlu = pdludriver.o pdluinfo.o pdgetrrv.o $(dmatgen) $(dlinchk)
dluexe = xdlu

$(dluexe): $(SCALAPACKLIB) $(dlu)
    $(F77LOADER) $(F77LOADFLAGS) -o $(dluexe) $(dlu) $(LIBS)
    $(MAKE) LU.dat

default: $(dluexe)

$(dluexe): $(FRC)

FRC:
    @FRC=$(FRC)
clean: 
    rm -f *.o

check: $(dluexe)
    mpirun -np 4 ./$(dluexe) > LU.out

.f.o : ; $(F77) -c $(F77FLAGS) $*.f

pdludriver.o: pdludriver.f
    $(F77) $(DRVOPTS) -c $<

All the code referenced in the above can be found in the scalapack distribution. Some things to note, please use the same fortran compiler that was used to build scalapack or you will get weird compiler errors.

Running the tests for any numerical libraries that you choose to build yourself

I had previously posted some notes on Compiling scalapack on SL5x, but I forgot to check whether the library works or not and now I need to use it. So I compiled up and ran the test code that shipped with scalapack.

[jtang@duo TESTING]$ mpirun -np 2 ./xdgsep
[duo:26526] *** An error occurred in MPI_Comm_group
[duo:26526] *** on communicator MPI_COMM_WORLD
[duo:26526] *** MPI_ERR_COMM: invalid communicator
[duo:26526] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 26525 on
node duo exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[duo:26524] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[duo:26524] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

The error didn't make much sense, except it was blindingly obvious that I had not configured BLACS (hence MPIBLACS) correctly.

Going back to my BLACS build directory I run the tests

[jtang@duo EXE]$ mpirun -np 2 ./xFbtest_MPI-LINUX-0 
BLACS WARNING 'No need to set message ID range due to MPI communicator.'
from {-1,-1}, pnum=0, Contxt=-1, on line 18 of file 'blacs_set_.c'.

BLACS WARNING 'No need to set message ID range due to MPI communicator.'
from {-1,-1}, pnum=1, Contxt=-1, on line 18 of file 'blacs_set_.c'.

 Sample BLACS tester run                                                         
==============================================
==============================================
BEGINNING BLACS TESTING, BLACS DEBUG LEVEL = 0
==============================================
==============================================
BLACS ERROR 'Illegal grid (2 x 2), #procs=2'
from {-1,-1}, pnum=0, Contxt=-1, on line -1 of file 'BLACS_GRIDINIT/BLACS_GRIDMAP'.

BLACS ERROR 'Illegal grid (2 x 2), #procs=2'
from {-1,-1}, pnum=1, Contxt=-1, on line -1 of file 'BLACS_GRIDINIT/BLACS_GRIDMAP'.

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 1473 on
node duo exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[duo:01472] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[duo:01472] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

It had turned out that MPIBLACS required a different TRANSCOMM setting, according to http://www.open-mpi.org/faq/?category=mpi-apps#blacs, after correcting this mistake I recompiled MPIBLACS and re-ran the tests and followed up by recompiling scalapack. The tests all pass after I corrected this issue.

On a side note, OpenMPI seemed to want default to the verbs backend for communications. To run OpenMPI compiled programs with TCP/SMP only communications you can do this

mpirun --mca btl tcp,self -np 4 --hostfile hostfile ./myapp

you could also edit /etc/openmpi-mca-params.conf and set this option

btl = tcp,self

this will make OpenMPI use only TCP/SMP communications by default.

I guess the moral of the story is to run the tests to make sure things are working as expected.

Swapping the Caps Lock with Control L

I've been meaning to make a note of this for a while now. I've been a happy hacker keyboard user for a while now, I really like it. I really like the lack of a caps lock key. To get the same effect on a normal keyboard under linux you could use the following .Xmodmap

 !
 ! Swap Caps_Lock and Control_L
 !
 remove Lock = Caps_Lock
 remove Control = Control_L
 keysym Control_L = Caps_Lock
 keysym Caps_Lock = Control_L
 add Lock = Caps_Lock
 add Control = Control_L

Or if you don't want the caps lock any more

 !
 ! Swap Caps_Lock and Control_L
 !
 remove Lock = Caps_Lock
 remove Control = Control_L
 !keysym Control_L = Caps_Lock
 keysym Caps_Lock = Control_L
 !add Lock = Caps_Lock
 add Control = Control_L

or else do setxkbmap -option “ctrl:nocaps” for something less permanent.

Weak references are not implemented in the version of perl

Gah, I'm getting errors like this

Error: Weak references are not implemented in the version of perl at
/usr/lib/perl5/site_perl/5.8.8/CGI/FormBuilder.pm line 15 BEGIN
failed--compilation aborted at
/usr/lib/perl5/site_perl/5.8.8/CGI/FormBuilder.pm line 15. Compilation
failed in require at (eval 77) line 2. BEGIN failed--compilation
aborted at (eval 77) line 2.

This pretty much means that the packages Scalar::Util and Compress::Zlib are outdated and need to be updated. First and obvious thing to do is to use cpan to re-install the broken modules, but unfortunately if you try and re-install Compress::Zlib from cpan it will get into a circular dependancy. The best thing to do is probably manually update/reinstall the following packages.

Go to cpan and grab a copy of

and then install them by hand.

How long does it take to get naturalised in Ireland?

This one slipped through the cracks recently, I haven't been keeping an eye on my post much recently. Two weeks after I submitted my Birth Affidavit, I have received a confirmation that the Irish Naturalisation and Immigration Services (INIS) have received my application. It is waiting to be reviewed and has been placed in chronological order with other recently received applications. It's now the end of February 2010 and they are currently processing applications from early 2008, and the estimated turn around time is 24 months. It's going to be a long wait assuming it all goes well.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Astro, Particle, High Energy physics is cool
Posted

assert.h is where it's at for debugging C programs

Having spent a few hours refreshing myself with C programming and being influenced by the java programmers in work, I've decided to be a bit more sensible about how I write C code. Armed with the that in mind, I looked around at how people did unit testing to do test driven software development for C.

It turns out that a lot of the frame works are just over engineered packages that rely on automake and autoconf hell to get working properly. I resorted to looking up and refering to The C Programming Language (book), it had reminded me of just using assert.h.

Using assert() is easier than you would expect and people should probably use it a bit more than not using it!

#include <assert.h>
void assert(scalar expression)

If the assertion is true the program will continue, otherwise it will abort. If you wish to turn of the assertions you could just add #define NDEBUG ahead of the #include and the assertions won't be compiled.

Here's an example usage of assert()

#include <stdio.h>
#include <assert.h>

int main(int argc, char **argv) { 
  int i; 
  for (i=5; i>0; i--) {
    fprintf(stdout, "i => %d\n", i); 
  } 
  assert(i > 0); 
  return 0; 
}

The assertion is that i cannot be zero, so the code will fail. This is probably a bad example of how to use assert() but it does demonstrate what it does. I find that I usually just end up using assertions to check that things aren't NULL or to make sure that things are greater than or less than expected values for variables in my code.

Writing and debugging iRODS micro-services

Before I forget, here's list of useful resources for learning how to write a micro-service.

Some micro-services that I've been thinking about cooking up are resolving an irods path to a physical path, a plotting service (either with gnuplot or plplot). So far my gnuplot micro-service attempt hasn't been too sucessful, doing gnuplot_init() seems to cause my micro-service to exit for some reason.

Flattening out a XML containing metadata for my iCAT metadata (iRODS)

I've been thinking again about flattening some metadata that is stored in a hierarchical structure in an XML file to a form that the XML micro service understands. The microservice effectively only takes in XML files with this type of data

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
<AVU>
<Attribute>Alice</Attribute>
<Value>1</Value>
<Unit>Years Old</Unit>
</AVU>

Of course my metadata.xml doesn't fit this model, it is of the form.

<?xml version="1.0" encoding="UTF-8"?>
<Metadata xsi:schemaLocation="http://www.hpc-europaII.org/jra3 schema.xsd" 
xmlns="http://www.hpc-europaII.org/jra3" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <Source>
        <Experiment>
            <expUID>SOMEVALUE</expUID>
            <expURI></expURI>
        </Experiment>
        <Tools>
            <ShortName>
            UG
            </ShortName>
            <Version>0bcfb63f7c6c67cd954609c3a90a27b2</Version>
        </Tools>
        <Authors>
            <Name>
                <First>
Alice
</First>
                <Middle></Middle>
                <Last>
Bob
</Last>
            </Name>
            <Role>
            Group Leader
            </Role>
            <Address>
                <Street>Some Street</Street>
                <PostalCode>Some PostalCode</PostalCode>
                <City>Some City</City>
                <Country>Some Country</Country>
                <Phone>Some Number</Phone>
                <E-mail>Some E-mail Address</E-mail>
                <Web>Some Web Address</Web>
            </Address>
        </Authors>
        <Institutes>
            <ShortName>Some ShortName for a Lab</ShortName>
            <Address>
                <Street>Some Street</Street>
                <PostalCode>Some PostalCode</PostalCode>
                <City>Some City</City>
                <Country>Some Country</Country>
                <Phone>Some Number</Phone>
                <E-mail>Some E-mail Address</E-mail>
                <Web>Some Web Address</Web>
            </Address>
        </Institutes>
    </Source>
    <Science>
    </Science>
    <Results>
    </Results>
</Metadata>

I had a look at http://nwalsh.com/docs/tutorials/extreme04/, http://www.w3.org/TR/xslt20/ and the example posted at http://groups.google.com/group/irod-chat/browse_thread/thread/1a1669bf1e36c205/46a904d3e5d651fd?lnk=gst&q=irods+%26+ddi%3F#46a904d3e5d651fd

In the end I came up with this simple xsl file

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" 
                xmlns:jra3="http://www.hpc-europaII.org/jra3"
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
                exclude-result-prefixes="xsi jra3">

  <!-- output will be in XML with indents for ease of reading -->
  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>

  <!-- skip over first tag -->
  <!-- xsl:template match="jra3:Metadata" priority="2"/ -->

  <!-- strip out the white space at the top of the file -->
  <xsl:strip-space elements="*"/>

  <!-- puts <avu></avu> around the attribute-value-unit triplets -->
  <xsl:template match="/">
    <metadata>
      <xsl:text>&#xA;</xsl:text>
      <xsl:text>&#xA;</xsl:text>
      <xsl:apply-templates/>
    </metadata>
  </xsl:template>

  <!-- call the templates -->
  <xsl:template match="jra3:Source">
    <xsl:apply-templates select="jra3:Experiment//jra3:expUID"/>
    <xsl:apply-templates select="jra3:Experiment//jra3:expURI"/>
    <xsl:apply-templates select="jra3:Tools//jra3:ShortName"/>
    <xsl:apply-templates select="jra3:Tools//jra3:Version"/>
    <xsl:apply-templates select="jra3:Authors//jra3:Name//jra3:First"/>
    <xsl:apply-templates select="jra3:Authors//jra3:Name//jra3:Middle"/>
    <xsl:apply-templates select="jra3:Authors//jra3:Name//jra3:Last"/>
    <xsl:apply-templates select="jra3:Authors//jra3:Role"/>
    <xsl:apply-templates select="jra3:Authors//jra3:Address"/>
    <xsl:apply-templates select="jra3:Institutes//jra3:ShortName"/>
    <xsl:apply-templates select="jra3:Institutes//jra3:Address"/>
  </xsl:template>

  <!-- begin output -->
  <xsl:template match="jra3:Experiment//jra3:expUID">
    <AVU>
      <Attribute>Source/Experiment/expUID</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit>
        <!-- xsl:value-of select="normalize=space(.)"/ -->
      </Unit>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Experiment//jra3:expURI">
    <AVU>
      <Attribute>Source/Experiment/expURI</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Tools//jra3:Version">
    <AVU>
      <Attribute>Source/Tools/Version</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Tools//jra3:ShortName">
    <AVU>
      <Attribute>Source/Tools/ShortName</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Name//jra3:First">
    <AVU>
      <Attribute>Authors/Name/First</Attribute>
    </AVU>
    <xsl:apply-templates select="jra3:Authors//jra3:Name//jra3:First"/>
  </xsl:template>

  <!-- begin output -->
  <xsl:template match="jra3:Experiment//jra3:expUID">
    <AVU>
      <Attribute>Source/Experiment/expUID</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit>
        <!-- xsl:value-of select="normalize=space(.)"/ -->
      </Unit>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Experiment//jra3:expURI">
    <AVU>
      <Attribute>Source/Experiment/expURI</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Tools//jra3:Version">
    <AVU>
      <Attribute>Source/Tools/Version</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Tools//jra3:ShortName">
    <AVU>
      <Attribute>Source/Tools/ShortName</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Name//jra3:First">
    <AVU>
      <Attribute>Authors/Name/First</Attribute>
    </AVU>
    <xsl:apply-templates select="jra3:Authors//jra3:Name//jra3:First"/>
  </xsl:template>

  <!-- begin output -->
  <xsl:template match="jra3:Experiment//jra3:expUID">
    <AVU>
      <Attribute>Source/Experiment/expUID</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit>
        <!-- xsl:value-of select="normalize=space(.)"/ -->
      </Unit>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Experiment//jra3:expURI">
    <AVU>
      <Attribute>Source/Experiment/expURI</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Tools//jra3:Version">
    <AVU>
      <Attribute>Source/Tools/Version</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Tools//jra3:ShortName">
    <AVU>
      <Attribute>Source/Tools/ShortName</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Name//jra3:First">
    <AVU>
      <Attribute>Authors/Name/First</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Name//jra3:Middle">
    <AVU>
      <Attribute>Authors/Name/Middle</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Name//jra3:Last">
    <AVU>
      <Attribute>Authors/Name/Last</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Role">
    <AVU>
      <Attribute>Authors/Role</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

  <xsl:template match="jra3:Authors//jra3:Address">
    <AVU>
      <Attribute>Authors/Address</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(jra3:Street)"/>,
        <xsl:value-of select="normalize-space(jra3:PostalCode)"/>,
        <xsl:value-of select="normalize-space(jra3:City)"/>,
    <xsl:value-of select="normalize-space(jra3:Country)"/>,
        <xsl:value-of select="normalize-space(jra3:Phone)"/>,
        <xsl:value-of select="normalize-space(jra3:E-mail)"/>,
        <xsl:value-of select="normalize-space(jra3:Web)"/>.
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>



  <xsl:template match="jra3:Institutes//jra3:ShortName">
    <AVU>
      <Attribute>Institutes/ShortName</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>,
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>


  <xsl:template match="jra3:Institutes//jra3:Address">
    <AVU>
      <Attribute>Institutes/Address</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(jra3:Street)"/>,
        <xsl:value-of select="normalize-space(jra3:PostalCode)"/>,
        <xsl:value-of select="normalize-space(jra3:City)"/>,
        <xsl:value-of select="normalize-space(jra3:Country)"/>,
        <xsl:value-of select="normalize-space(jra3:Phone)"/>,
        <xsl:value-of select="normalize-space(jra3:E-mail)"/>,
        <xsl:value-of select="normalize-space(jra3:Web)"/>.
      </Value>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>

<!--      
  <xsl:template match="jra3:Tools//jra3:">
    <AVU>
      <Attribute>Source/Tools</Attribute>
      <Value>
        <xsl:value-of select="normalize-space(.)"/>
      </Value>
      <Unit/>
    </AVU>
    <xsl:text>&#xA;</xsl:text>
    <xsl:text>&#xA;</xsl:text>
  </xsl:template>
-->

<!-- delete the rest -->
  <xsl:template match="jra3:Results"/>
  <xsl:template match="jra3:Science"/>

</xsl:stylesheet>

Running this xsl file on my metadata.xml file I get this type of output

<?xml version="1.0" encoding="UTF-8"?>
<metadata>

<AVU><Attribute>Source/Experiment/expUID</Attribute><Value>SOMEVALUE</Value><Unit/></AVU>

<AVU><Attribute>Source/Experiment/expURI</Attribute><Value></Value><Unit/></AVU>

<AVU><Attribute>Source/Tools/ShortName</Attribute><Value>UG</Value><Unit/></AVU>

<AVU><Attribute>Source/Tools/Version</Attribute><Value>0bcfb63f7c6c67cd954609c3a90a27b2</Value><Unit/></AVU>

<AVU><Attribute>Authors/Name/First</Attribute><Value>Alice</Value></AVU>

<AVU><Attribute>Authors/Name/Middle</Attribute><Value></Value></AVU>

<AVU><Attribute>Authors/Name/Last</Attribute><Value>Bob</Value></AVU>

<AVU><Attribute>Authors/Role</Attribute><Value>Group Leader</Value></AVU>

<AVU><Attribute>Authors/Address</Attribute><Value>Some Street,
    Some PostalCode,
    Some City,
    Some Number,
    Some E-mail Address,
    Some Web Address.
      </Value></AVU>

<AVU><Attribute>Institutes/ShortName</Attribute><Value>Some ShortName for a Lab,
      </Value></AVU>

<AVU><Attribute>Institutes/Address</Attribute><Value>Some Street,
    Some PostalCode,
    Some City,
    Some Country,
    Some Number,
    Some E-mail Address,
    Some Web Address.
      </Value></AVU>

</metadata>

This flattened output would allow the XML micro service to load it up and associate it as metadata to my collection of files. I think using the XSLT microservice to transform the data on the server side would be ideal.

It seems this is the most effective strategy for a number of iRODS users out there in the world.

Libtool was broken on my macbook

I recently got errors similar to this

libtool: Version mismatch error. This is libtool 2.2.6, but the
libtool: definition of this LT_INIT comes from libtool 2.2.4.
libtool: You should recreate aclocal.m4 with macros from libtool 2.2.6
libtool: and run autoconf again.

This was happening when I wanted to do a "port upgrade gimp2", basically almost anything that needed libtool and automake would fail miserably. Libtool had probably been upgraded at some point and since I don't blanket upgrade everything I didn't notice that it wasn't working. I tried a number of things to upgrade/fix the brokeness.

So I tried doing

port -v -nR upgrade --force libtool

This would rebuild a handful of packages that depended on libtool. Doing this still failed for me, it appeared that autoconf was picking the system versions of libtool, autoconf and automake. This had baffled me for a while since when I echo'd my $PATH it was showing the correct path.

Then it had occured to me to check if macports was resetting/setting my environment before it built a package. After looking at http://guide.macports.org/chunked/reference.variables.html I checked my macports.conf, it turned out that my binpath was just wrong, I had the system paths defined before my macport install. I just commented out the binpath variable and ran a rebuild of gimp2, everything was working again.

I must have changed the binpath for some reason or another at somepoint to fix something else, that something else makes me think it was openssl that was boned.

Compiling scalapack on SL5x

Just some notes for myself nothing to see here...

List of needed software

Steps

To install scalapack from source you will need to above list of dependancies installed. We will assume that your system has gcc, g++ and gfortran installed.

Configure, Compile and Install openmpi

tar zxvf src/openmpi-1.4.1.tar.gz 
cd openmpi-1.4.1/
./configure --prefix=$HOME/local
export PATH=$HOME/local/bin:$PATH
export LD_RUN_PATH=$HOME/local/lib
export LD_LIBRARY_PATH=$HOME/local/lib

Configure, Compile BLAS

tar zxvf blas.tgz
cd BLAS
make all FORTRAN=gfortran LOADER=gfortran
cp blas_LINUX.a $HOME/local/lib/libblas.a

you may or may not want to edit make.inc to customise the build more.

Configure, Compile LAPACK

tar zxvf lapack.tar.gz
cd lapack-3.2.1/
cp make.inc.example make.inc

The default settings are pretty sensible and should work fine with gfortran. Copy the BLAS library that you compiled earlier on to the lapack build directory. (or edit make.inc to point to the correct blas library)

cp ../BLAS/blas_LINUX.a .

Then just type make in the lapack directory. It will compile the lapack library and also run some tests. You should go into the TESTING directory and look for failures in the *.out files and investigate any significant failures. You should be left with a lapack_LINUX.a in the build directory

Configure, Compile mpiblacs

tar zxvf mpiblacs.tgz
cd BLACS
cp BMAKES/Bmake.MPI-LINUX ./Bmake.inc

Before you can compile BLACS/MPIBLACS you should go into the INSTALL directory and type make help. You will need to run some of the tests to correctly set the options needed to make BLACS work correctly.

You will need to know the location of the build directory in my case it was /home/jtang/develop/hpce2-scalapack/BLACS

So I did

cd INSTALL
make xintface F77=gfortran BTOPdir=$HOME/develop/hpce2-scalapack/BLACS
make xsize F77=gfortran BTOPdir=$HOME/develop/hpce2-scalapack/BLACS
make xsyserrors F77=gfortran BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so"

Run the above programs

./EXE/xintface
For this platform, set INTFACE = -DAdd_

./EXE/xsize 
ISIZE=4
SSIZE=4
DSIZE=8
CSIZE=8
ZSIZE=16


mpirun -np 4 ./EXE/xsyserrors 
libibverbs: Fatal: couldn't read uverbs ABI version.
--------------------------------------------------------------------------
[[34681,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: duo

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
If this routine does not complete, you should set SYSERRORS = -DZeroByteTypeBug.
Leave SYSERRORS blank for this system.

Take note of the output as you will need it to build BLACS. You should also build and run the following targets to make sure you get the right settings.

make xtc_CsameF77 F77=mpif90 BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so"

mpirun -np 4 ./EXE/xtc_CsameF77
--------------------------------------------------------------------------

[[42923,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: duo

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
libibverbs: Fatal: couldn't read uverbs ABI version.
libibverbs: Fatal: couldn't read uverbs ABI version.
libibverbs: Fatal: couldn't read uverbs ABI version.
libibverbs: Fatal: couldn't read uverbs ABI version.
 If this routine does not complete successfully,
 Do _NOT_ set TRANSCOMM = -DCSameF77


[duo:27485] *** An error occurred in MPI_Comm_free
[duo:27485] *** on communicator MPI_COMM_WORLD
[duo:27485] *** MPI_ERR_COMM: invalid communicator
[duo:27485] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
 Do _NOT_ set TRANSCOMM = -DCSameF77
--------------------------------------------------------------------------
mpirun has exited due to process rank 3 with PID 27486 on
node duo exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[duo:27482] 3 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[duo:27482] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[duo:27482] 1 more process has sent help message help-mpi-errors.txt / mpi_errors_are_fatal

The next two are just sanity checks

make xcmpi_sane F77=gfortran BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so"
make xfmpi_sane F77=mpif90 BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so"

mpirun -np 4 ./EXE/xcmpi_sane
mpirun -np 4 ./EXE/xfmpi_sane

With the above information I go back into the build directory. (see also http://www.open-mpi.org/faq/?category=mpi-apps#blacs, the TRANSCOMM option needs to be set to -DUseMpi2)

make F77=mpif90 BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so" TRANSCOMM=-DUseMpi2 INTFACE=-DAdd_  mpi what=clean
make F77=mpif90 BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so" TRANSCOMM=-DUseMpi2 INTFACE=-DAdd_  mpi

Then cd into the TESTING Directory

make F77=mpif90 BTOPdir=$HOME/develop/hpce2-scalapack/BLACS MPIdir=$HOME/local MPILIBDir=${MPIdir}/lib MPILIB="$HOME/local/lib/libmpi.so" TRANSCOMM=-DUseMpi2 INTFACE=-DAdd_

Then if you wish and have time, run the tests.

Going back to the root of the BLACS directory you should see that there is now a LIB directory containing the libraries that you have built.

Configure, Compile SCALAPACK

You will need to know the locations of the above compiled BLAS, LAPACK and BLACS libraries and some of the settings from the BLACS compilation.

tar zxvf scalapack.tgz
cd scalapack-1.8.0
cp  SLmake.inc.example SLmake.in
make home=$HOME/develop/hpce2-scalapack/scalapack-1.8.0 BLASLIB=$HOME/develop/hpce2-scalapack/BLAS/blas_LINUX.a LAPACKLIB=$HOME/develop/hpce2-scalapack/lapack-3.2.1/lapack_LINUX.a BLACSFINIT=$HOME/develop/hpce2-scalapack/BLACS/LIB/blacsF77init_MPI-LINUX-0.a BLACSCINIT=$HOME/develop/hpce2-scalapack/BLACS/LIB/blacsCinit_MPI-LINUX-0.a BLACSLIB=$HOME/develop/hpce2-scalapack/BLACS/LIB/blacs_MPI-LINUX-0.a CDEFS="-DAdd_ -DNO_IEEE -DUsingMpiBlacs" SMPLIB=""
Otaku and in Ireland?

It just occured to me that I am a bit of an anime junkie, or well I used to be. I feel too old to watch anime, yet I still watch my weekly fix of Bleach and Naruto. I still look forward to all the good stuff that Studio Ghibli has to offer and of course, I'm quite partial to the Gundam and Macross franchise. There's so little time and so much anime that needs my attention.

So back to being an Otaku in Ireland, you really need to be a part of eirtakon.com. There's also an irc channel that the members frequent. The irc channel can be found at irc.synirc.net on channel #eirtakon and irc.synirc.net on channel #eirtaku

By the way did I mention I'm a narutard?

DWM and Java based programs

Every so often I need to run java based programs or programs that use a java based front end. For example I occasionally need to fire up openoffice, matlab, mathematica, ireports and almost everytime I forget what environment variables to try and set to get the interfaces working properly under DWM

So try

export AWT_TOOKIT=MToolkit
./app

or

wmname LG3D
unset AWT_TOOLKIT
./app

Usually one or the other will work. If not I usually just start up Xnest or a secondary Xserver running a different windowmanger to run that one application. Note, wmname is a suckless.org tool.

Your Country Your Call, if I had a choice...

It seems that the smart economy is just yet more glarg..%&^"?!'... it's a clear sign of desperation. I'm not too sure on how to react or respond to this. According to the website that the prize fund is provided fully by Irish and non-Irish corporations, as well as a small number of Irish firms and individuals. It also has the backing of the Irish government. I wonder what the break down is in terms of funding between the various bodies. Still 1.2 million euro as prize money plus development money is no small amount. Still, I question what the government did to convince the various corporations to put forward the funding.

One Step Closer to OAI Compliance

After Setting up a handle.net account for DSPACE I needed to start and look at making my DSPACE install OAI compliant. I was directed to Open Archives, after a bit of googling I ended up at the wiki page Use the OAI-PMH interface. It turns out that to enable the OAI-PMH interface all I had to do was just drop the dspace-oai.war file into the same webapps directory as my dspace.war deployment, restart my app server and bob's you're uncle.

To quote the dspace wiki page on OAI-PMH

  • Identify . used to retrieve information about a repository: http://web-address-to-my-dspace/dspace-oai/request?verb=Identify
  • ListMetadataFormats . lists metadata formats supported for OAI-PMH: http://web-address-to-my-dspace/dspace-oai/request?verb=ListMetadataFormats
  • ListRecords . permits metadata harvesting of all or a selection of items: http://web-address-to-my-dspace/dspace-oai/request?verb=ListRecords&metadataPrefix=oai_dc
  • ListSets . lists the .sets. (i.e. DSpace Collections) available in a repository: http://web-address-to-my-dspace/dspace-oai/request?verb=ListSets

Using the above knowledge I tested my OAI-PMH interface with a web browser. I was quite pleased at yet again how easy it was to get this functionality working. That's another two thumbs up to DSPACE and friends.

What's my name?

I've recently sent in an application to get naturalised, but it was returned to me due to some mistakes in the application form. For the love of god, if you are applying for naturalisation in Ireland, read the forms and double check simple things like names and make sure they match up to what you have on your birth certificate. If not you will require a Birth Affidavit.

The process so far has been obscure, painful and full of legal mine fields.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

State of the Irish Economy

I had previously posted this rant elsewhere, but here it goes again here.

It's cracked. There is a load of media hype targeted at the public sector in Ireland right now. The public sector is also getting (or has got) yet another pay cut. To top it all off, the Irish government has been pushing for a smart knowledge based economy, but guess what, all those contract staff that aren't seeing any benefit of being a public civil servant is getting a pay cut too!

Other sucky things so far in the system is NAMA and the raiding of state pensions, the government is just driving the state into a huge hole. It's forcing people to leave, and thus creates a feedback loop in creating even more debt or rather the lack of paying it back.

If the state had any sense then they should let the system sort itself out and not intervene. Let the strong prevail and let the weak fail. I mean it is meant to be a capitalistic state/society.

Using dar to archive directories of files

I've been migrating user home directories from one machine to another machine recently. In doing so I've come across the age old problem of users having too much junk, old user directories that need to be backed up for future reference and so on.

Instead of using tar like I normally would for most things, I chose to use dar. dar gives me the possibility of being able extract a single file without needing to stream through the entire archive.

To archive a user's home directory this is what I run

dar -c user -R / -g home/user -v

I guess rtfm and check man pages to see what the above options mean. Note I also have this in my .darrc

    #########################
    #           #
    #   Shared options pro  #
    #    DAR archiver   #
    #           #
    #########################

# Jakub Holy, 25.4.2005
# This file: /etc/darrc
# Note: If the user has her $HOME/.darrc, this file is ignored => it must be included with the option -B /etc/darrc to dar

###
### Files that shall not be compressed (because they're already)
###
# archives (Note: .tar is archive, but not compressed => do compress it).
-Z "*.bz2" -Z "*.deb" -Z "*.gz" -Z "*.Z" -Z "*.zip" -Z "*.rar" -Z "*.tbz2" -Z "*.tgz" -Z "*.jar" -Z "*.ear" -Z "*.war"
-Z "*.BZ2" -Z "*.DEB" -Z "*.GZ" -Z "*.Z" -Z "*.ZIP" -Z "*.RAR" -Z "*.TBZ2" -Z "*.TGZ" -Z "*.JAR" -Z "*.EAR" -Z "*.WAR"
# media - images
-Z "*.gif" -Z "*.jpeg" -Z "*.jpg" -Z "*.png" 
-Z "*.GIF" -Z "*.PNG" -Z "*.JPEG" -Z "*.JPG"
# media - audio
-Z "*.ogg" -Z "*.mp3"
-Z "*.OGG" -Z "*.MP3"
# media - video
-Z "*.avi" -Z "*.mov" -Z "*.mp4" -Z "*.mpg"
-Z "*.AVI" -Z "*.MOV" -Z "*.MP4" -Z "*.MPG"
# documents - compressed formats
-Z "*.pdf" -Z "*.swf" -Z "*.sxw"
-Z "*.PDF" -Z "*.SWF" -Z "*.SXW"
# strange formats, binaries and other hard to compress (empirical)
-Z "*.gpg" -Z "*.rnd" -Z "*.scm" -Z "*.svgz" -Z "*.vlt" -Z "*.zargo" -Z "*.wings" -Z "*.xpi" -Z "*.chm"
-Z "*.GPG" -Z "*.RND" -Z "*.SCM" -Z "*.SVGZ" -Z "*.VLT" -Z "*.ZARGO" -Z "*.WINGS" -Z "*.XPI" -Z "*.CHM"
-Z "*.obj" -Z "*.tif" -Z "*.tiff"
-Z "*.OBJ" -Z "*.TIF" -Z "*.TIFF"

###
### Ignored files
###
-X "*~" -X "*.o"    
# *~ are backups, *.o are compiled unlinked files

### jt's custom options
# -m dont compress files less than this size in bytes
# -v verbose
# -s slice up in this size
# -y compress with bzip2
-m 256 
-v
-s 2000M
-y

Once the archive has been created, it's probably not a bad idea to test it, you can do so by doing

dar -t user --noconf
Made ikiwiki go faster

I eventually made ikiwiki go much faster than what I was experiencing before. It had turned out that the aggregate plugin left some trash in my tree of files, I some how ended up accumulating about 10k number of small files.

After I had removed the loose files I ran a rebuild/refresh, and now its down to ~3seconds for a refresh of my site. Not bad I guess.

Archlinux on my eeepc 701

Been playing with archlinux recently, its everything that gentoo should have been. I downloaded the USB image then proceeded to install it on to my eeepc 701. I then found this http://wiki.archlinux.org/index.php/Asus_Eee_PC_701, which I so rightly read and took some hints from it.

I did all the usual things of turn off the swap disk, mounting /tmp and /var/log into a tmpfs and I made sure that noatime was enabled for the filesystems that I do have on the SSD.

Another thing I did was put nilfs2 on my SD card and mounted /home on it. So most of my writes are now either in memory or on my SD card which I can replace relatively cheaply and easily.

Setting up a handle.net account for DSPACE

This assumes you have DSPACE 1.5.x already running and working and that you had previously only used a dummy handle. It's as simple as doing

/dspace/bin/dsrun net.handle.server.SimpleSetup /dspace/handle-server

I found this buried in the DSPACE installation manual http://www.dspace.org/1_5_2Documentation/ch03.html#N10AA6 this link was forwarded to me from the handle.net crowd, as far as I know its the CNRI who run the show.

Just follow the instructions, have a credit card and paypal account ready to pay and make sure you have at least these details prepared...

  • Description of server site
  • Organisation name
  • Contact person's name
  • Telephone number
  • Email address for a contact
  • Two passwords for securing the system, I guess make sure you have them in a safe place and don't lose them.

Once the simple setup is completed and you have followed all the instructions, you will be directed to a page to pay the fee for the setup. Then you will be asked to upload sitebndl.zip to the handle.net site. Overall I found the experience with setting a handle.net account and server with DSPACE to be generally pleasant. Now all I need to do is put the system into production. Overall is two thumbs up for CNRI and the DSPACE devs for making it easy to setup this stuff and being helpful in the process.

Making Ikiwiki go faster

I've been using ikiwiki for the past year or two for my blog, but it's gotten progressively slower. So I've turned off some of the RSS and ATOM feeds that are on my blog in hopes that things will run faster. Maybe I should turn off more stuff? The question is, how much do I turn off? Anyway, step 1. upgrade ikiwiki, step 2. upgrade all its dependancies!

My first archlinux package - ttyrec

Having been experimenting with archlinux on eeepc for a few months now. I've been itching to package something up for fun to see how easy/hard it would be. I've finally got a chance make a package. Here's the resulting PKGBUILD file for ttyrec

# Maintainer: Jimmy Tang <jtang@tchpc.tcd.ie>

pkgname=ttyrec-git
pkgver=20100201
pkgrel=1
pkgdesc="ttyrec is a tty recorder - this is not the official upstream version"
arch=('i686' 'x86_64')
url="http://git.kitenet.net/?p=ttyrec.git"
license=('BSD')
makedepends=('git')

_gitroot="git://git.kitenet.net/ttyrec.git"
_gitname="ttyrec"

build() {
  cd "$srcdir"
  msg "Connecting to GIT server...."

  if [ -d $_gitname ] ; then
    cd $_gitname && git pull origin
    msg "The local files are updated."
  else
    git clone $_gitroot
  fi

  msg "GIT checkout done or server timeout"
  msg "Starting make..."

  rm -rf "$srcdir/$_gitname-build"
  git clone "$srcdir/$_gitname" "$srcdir/$_gitname-build"
  cd "$srcdir/$_gitname-build"

  #
  # BUILD HERE
  #
  make CFLAGS="-O2 -Wall -DSVR4 -D_XOPEN_SOURCE=500" || return 1
  install -d $pkgdir/usr/share/doc/$pkgname $pkgdir/usr/share/man/man1 $pkgdir/usr/bin
  install -D ttyrec.1 ttyplay.1 ttytime.1 $pkgdir/usr/share/man/man1
  install -D README $pkgdir/usr/share/doc/$pkgname
  install -Dm755 ttyrec ttyplay ttytime $pkgdir/usr/bin
}

It's actually a butchered version of what I found inside the PKGBUILD file for ikiwiki-git. The above isn't for the official version of ttyrec, but rather a patched up verion from Joey Hess. I had stumbled on his repo for it when I was reading his blog post on termcasting and got interested in ttyrec. I can think up a few useful things to do with ttyrec already, such as replacing a user's shell with ttyrec to record their sessions to playback :)

Hugin 2009.4.0 and RawTherapee 3.0a1

Having been away for a few weeks in the last few months and just taking snaps of things, I failed to notice that RawTherapee changed its license to a GPL one! So we should see RawTherapee getting compiled and released on different systems. I downloaded it and tried out the new 3.0 alpha 1 release and it's abit unstable for now. It has a tendancy of crashing randomly on my machine, I will be sticking with the 2.4.x stable for now.

Another one of my favourite pieces of software for photography has got to be Hugin, they too seemed to have done a new release and its rather spiffy, there is even some GPGPU support which works quite well on my cheapo nvidia g210 based graphics card. Here's a rather boring panorama stitched from 4 images.

Using ikiwiki for documentation

I've previously tried to use ikiwiki do project management and issue tracking. I consistantly failed at using it with none technical people. So I'm revisiting it for using it for documentation. I've successfully use it for my own things for documentation, but the trick is to make it accessible for people who don't know it too well to do documentation.

Using the stock blog/software template with tagging I've setup a wiki that we're using for documenation. I guess the nice thing is when I am done I can dump the output as static html for archiving. Another big win for using ikiwiki for documentation is the ability to fork and work offline.

Whibal - my new toy for my camera

My new toy arrived today. It's a Whibal keyring which I got from http://www.rawworkflow.com/. I've heard about it before, I never really took much notice to it till now. I wasn't so sure about it till I bought one and did a fairly simple test. I waited till it was dark, and went for a walk and set my camera to take raw images and took some snapshots of my Whibal keyring.

Went and fired up RawTherapee and then converted my raw into jpgs. One was set to use my cameras AWB and the other was calibrated using the Whibal keyring

Image using camera AWB setting

Image using Whibal to calibrate whitebalance

Here's the original raw file P1180274.RW2, I took it with my LX3.

Overall, I think I'm impressed with it enough to recommend the Whibal keyring. I'm considering in picking up a second one and or with the pocket sized card as well. It's a worthwhile investment since it is an easy way of making sure the colours look somewhat correct without much effort.

I'm almost convinced to start using the RAW mode on my LX3 if it weren't for the fact that I am lazy in doing the lens corrections (since I don't use silkypix or any commercial apps to process RAWs).

Panoramas from videos and images

I was recently asked by my work colleagues to take some images of a computer cluster that we had recently procured. Of course I said "Ok" that sounds like a great idea, only the space that I had to take the photos was pretty tight and I had a super wide angle lens to take the images with. I wasn't totally satisfied with the results, lots of distortion, no clear shot of the machine, noise, bad perspectives etc...

I had previously found this post http://www.flickr.com/photos/36383814@N00/4207656698/ which was quite interesting. It seems on of the Hugin devs (I think) wrote some perl scripts called Panotools-Script

So my tools used were ffmpeg, Panotools-Script and Hugin including its dependancies for enblend and enfuse. I tried to install the latest stable version of these tools.

First I obviously took lots of photos, some with a Canon 1000d with a sigma 10-20mm lens, then some more photos with my Panasonic TZ3, I also took a video as perpendicular as I could get of the subject that I wanted to stitch. The video was taken at 30fps and it was about 1m40secs long.

Following the instructions posted at the flickr post http://www.flickr.com/photos/36383814@N00/4207656698/

ffmpeg -i p1050357.mov -r 10 -f image2 foo-%04d.png

The above generated ~1200 image and took about ~5mins. I only extracted 10 frames from each second of video.

match-n-shift -o p1050357.pto -n -v 65 foo-????.png
ptochain -m output.pto.mk  -o output.pto p1050357.pto
make -f output.pto.mk -j 3

The make process took quite some time on my desktop machine so I just worked while it processed the data.

autooptimiser -a -l -s -o output-optimised.pto output.pto
pto2mk -o output-optimised.pto.mk -p output_stitched output-optimised.pto
make -f output-optimised.pto.mk ENBLEND=enfuse

It gave me an aweful lot of errors, I guess it doesn't work too well when I do panoramas up too close! So instead here's a sample I did from a short video clip, the clip was taken with a Panasonic TZ3

Here's the source video P1050396.MOV and the original output in as a tif file output stitched.tif. The final output was a little blurry, it's not the best, but not the worst either.

Experimenting with DRBD for data replication for backup purposes

Software used,

My test configuration

#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd83/drbd.conf
#

global {
    usage-count no;
}

common {
    syncer { 
        rate 33M;
        verify-alg sha1;
    }
}

resource r0 {
    protocol C;
    startup {
        wfc-timeout 0;
        degr-wfc-timeout 120;
    }
    net {  
        cram-hmac-alg "sha1"; 
        shared-secret "mysharedkey"; 
    }
    on myhost.mydomain1 {
        device /dev/drbd0;
        disk /dev/sdb2;
        address 10.0.0.1:7788;
        meta-disk internal;
    }
}

Where myhost.mydomain1 is just my first host, I wish to just create a one node drbd cluster for now so I can add additional hosts to the pool to replicate my data at a later point. In short I want run the cluster in a degraded mode till I get more hosts to add redundancy for backup purposes.

/dev/sdb2 is just a small 50gb partition for testing purposes and the 10.0.0.1 address is just a private ip address that I intend on binding to the second network interface on my server such that I can use a cross over cable to connect to my backup host later on.

Once I had the above config file, I needed to run the following commands.

/etc/init.d/drbd restart
drbdadm create-md r0
/etc/init.d/drbd restart
drbdadm -- --overwrite-data-of-peer primary r0
/etc/init.d/drbd restart
mkfs.ext3 /dev/drbd0
mount /dev/drbd0 /mnt/r0

The about should give you a single node drbd setup, from reading the documentation It should be possible to add a second host to the setup, by adding this to the config file.

on myhost.mydomain2 {
    device /dev/drbd0;
    disk /dev/sdb2;
    address 10.0.0.2:7788;
    meta-disk internal;
}

then copy the updated drbd.conf file to myhost.mydomain2, then issue these commands on myhost.mydomain1, assuming you want to make this the primary node

drbdadm disconnect all
drbdadm invalidate-remote all
drbdadm primary all
drbdadm connect all

On the secondary node myhost.mydomain2 issue these commands

drbdadm disconnect all
drbdadm invalidate all
drbdadm connect all

Please be careful on which nodes you issue these commands or else you might end up blowing away data. Once the above commands have been issued, have a look at /proc/drbd there will be some statistics on whats happening.

I'm not really using drbd for high availability, but rather for an easy way of providing a backup copy of my data. Note you would probably not want to automount the drbd partition on boot so I have set my system not to automount and to do no fsck's

So I now have this in my fstab for /home

/dev/drbd0      /home           ext3    defaults,noauto,usrquota,grpquota 0 0

and to mount it on my primary machine, I do

drbdadm primary all
mount /home

Notes: I found that if both my disks are invalid (after some messing to test the backups) you might get something like this

[root@myhost.mydomain1 ~]# drbdadm primary r0
0: State change failed: (-2) Refusing to be Primary without at least one UpToDate disk
Command 'drbdsetup 0 primary' terminated with exit code 17

you could force one of the hosts' disks to be valid by doing

drbdadm -- --overwrite-data-of-peer primary all

make sure you do it on the primary node!

Today is pay day 2010-01-20

Today is pay day if you work in TCD. Apparently if you work in a University in Ireland and if you are on a contract (not permanent or tenured) you are still liable for the recent round of pay cuts as suggested by the latest budget. On average I'm currently earning ~5% less from this day forward, and my pension levy has increased by ~4% (since last year it was based of 10months pay and this year is the full 12months). It's not too bad for me as I don't have a mortgage, but it must be hellish for those that have a mortgage and a family, especially those that have bought a place in the last 5yrs.

As the TRSA have pointed out, this is certainly not encouraging for people working in the so called knowledge based smart economy.

Holiday in Hong Kong 2009 - Some highlights

Went to Hong Kong and Beijing for my vacation this year (again) ended up taking about ~5500 photos in the 2 weeks that I was away. Here's some highlights for the Hong Kong part of my trip (that aren't family related).

I've only quickly skimmed through the photos and half of them were taken with a new 10-20mm lens that I bought while I was in Hong Kong (sadly because it was new and I didn't know what I was doing with the lens, some of the pics came out blurred) Here's a quick selection of what I thought were nice photos.

Try two at applying for naturalisation

Since I was about 6months too early to apply for naturalisation, I had to wait a few months. Now to download the forms from http://www.inis.gov.ie/en/INIS/Pages/Form_8 fill them in, photo copy them and then get some sponsors.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Christmas is cancelled in Ireland

It seem's that the public sector has been under quite a lot of fire recently, especially with the recent union sanctioned strikes. BOI and AIB cancel Christmas staff parties and now the public sector is following this move. The media is absolutely grilling the public sector right now. It's only going to get more interesting now, as these social events are being cancelled, the restaurant/catering sector is going to see a sharp decline in their takings this Christmas. It looks like there will be a collapse of yet another sector. The world as we know it is coming to an end, I wonder what will happen when the public sector strikes again this week (they strike on 2009-12-03). Will society fall apart? I guess I better start stocking up on tuna.

SC09 Day 5

Al Gore gives a keynote speech at SC09, he certainly is an interesting speaker

Scinet, there's 400gbit/s of bandwidth for the exhibitors and visitors, the network here is a bit crazy

Just some more random shots of the exhibitor hall

Since all of the meetings and discussions were out of the way and there was nothing left of value to see at the exhibitors hall, and I had attended all the BOF's and paper presentations, a car was hired to do some sight seeing and shopping

SC09 Day 4

Hmmm waffles

Had some time to wander around the exhibitors hall before and after some meetings

WTF?! what's theregister doing here?

Cool video wall

Some random shots of the exhibitions

SGI UV, only about 5yrs too late

SC09 Day 3

Another busy day so not many pics.

Some pics of the sushi that I had for dinner, I was pleasantly surprised and pleased with it :)

Slurm!!! was at the Slurm user meeting and the developers had a few cans of slurm to give out to the attendees, pretty cool :)

SC09 Day 2

Didn't have much time to take snaps all day till the opening of the exhibition hall, most of the shots are from the hip so some of them are blurry.

SC09 Day 1

Funnily because I registered on time and I read the instructions I was registered for everything correctly, as for Geoff, well, he forgot to register for the Tutorials. He ended up registering and paying the late fee!.

Some random pics while walking around looking for a restaurant.

Dinner and more local beer!, the lemon in my Widmer Hefe Weizen odd, it looked and smelt the part of a hefeweissen, but the taste was meh, the following beer that I tried was the Deschutes Mirror Pond Pale Ale. Only had a main course of beer battered oysters which were pretty tasty.

SC09 Day 0

I hadn't even left Ireland yet and I think I'm going to miss the connecting flight in Atlanta. After a few hours on the flight and some wine, checking the inflight display between movies I see that we are 40 minutes ahead of schedule. In other words I should hope that I will not need to stay overnight in Atlanta.

So plans for SC09? Attend tutorials and paper presentations. Talk to vendors about our upcoming purchases and what we are buying (or at least what we think we will be buying). Shopping for Frau Klein for a notebook and an Apple Magic Mouse (though I do prefer my Logitech G9 mouse).

Time is 16:40 in Atlanta, we managed to reach the gate and not miss our connecting flight!.

mmm, double espresso to keep me awake

American airports are funny and quiet

A quick bite of food and some local beer, tried the Sinister dark ale which was quite good, and the Fat Tire this was less impressive

Bugger, jet lag is setting in, I can't sleep, its 5am

High end computers and HPC

I filled out this http://www.theregister.co.uk/2009/11/05/high_end_servers_hpc_barometer_survey/ yesterday while I was waiting for something to finish compiling. I must remember to check the results when I am back from SC09. How co-incidental is this survey, seeing that SC09 is on in a week's time. I will be talking to vendors about both storage and compute seeing that we are probably going to be procuring yet another HPC machine for work, I certainly hope more vendors are more clued in on the Irish/EU market than in the past. It's also a shame that Si-Cortex went bust they made some pretty cool kit. Anyway I've previously blogged about How not to build a High Performance Computer, perhaps I should also put up a post on How I would build a High Performance Computer. I would imagine that every HPC sysadmin type person would have their own ideas on how to do it, which is probably why I made a list of things not to do instead of things to do.

L4D2 Demo

Pre-ordered the L4d2 game today and for pre-ordering I got access to the L4D2 demo as well. Played one game of it today on advanced and it was fun the new melee weapons are hugely entertaining, being able to wack a zombie with a guitar or a machete is very satisfying. The new specials are certainly interesting. I can see playing with random people online can lead to all sorts of interesting and fun games.

Go to http://www.l4d.com/ for more info ;) just in case if you're wondering why I'm blogging about this, it's because the game is coming out on my birthday. I still bought a copy for myself though.

From VI to EMACS

I've been a vi and vim user for many many many years now ever since I tried Free-BSD 2.x and then moved onto using Red-hat Linux, followed by Debian Linux. In short I had always used vi or vim for almost every form of text editing that I needed to do.

I recently got introduced to the orgmode and I was pretty impressed with the planner and to do lists features of this package. So I started using emacs 23.1 as a result. It's not as bad as I remember it to be. I guess its probably because computers are now fast enough to make emacs nicer to use than it was in the past.

Given the choice between emacs and eclipse, I think I would prefer emacs as an IDE for writing code.

I've effectively replaced vimoutliner with orgmode for planning and todo lists. vimoutliner still has a place for doing quick checklists for projects. I found that I liked orgmode enough to learn more about emacs and as a result I have started using emacs on the console as my text editor for mutt.

Getting my GNIB card renewed at the Immigrations office

Armed with my work permit, my old GNIB card and a credit card to pay the processing fee and a hang over from Sunday night I decided that I needed to go to the Garda National Immigration Bureau to get myself updated with the system.

According to http://www.citizensinformation.ie/categories/moving-country/moving-to-ireland/rights-of-residence-in-ireland/registration-of-non-eea-nationals-in-ireland I needed to be sure I was legal :) Like every year for the last 5 years that I have had to do this (now on my 6th year) at getting a new GNIB card etc...

I queued up at 8:40am to get a ticket to queue up even more to see a representative, There were exactly 122 people ahead of me, I was number 123. I proceeded to queue till about 10:30am (approximately) before the officials decided that they have met their quota for the day. There must have been about 20-30 people still queuing for tickets. Around 10:45am I get to see a representative, I handed over my documentation and after a 2min wait and a few exchanges of words it's all ok. Now to wait for my documentation and new GNIB to be processed. What I expected to get in 10-15mins to completion, I ended up waiting about 30-40mins. I hadn't really kept a note of how long I waited.

During the wait I over hear people discuss about the GNIB being awkward, sending people around on wild goose chases to gather documentation. The whole system is wonderfully inconsistant, badly documented, badly orchestrated and under funded. I guess they just want to make it difficult for foreigners to stay legally in Ireland. The conversations and people that were around the GNIB office at that time of the day were mostly non-students, whom did not get tickets till 9:30am. God knows how long they must wait and queue to get processed. It's clear that the GNIB offices are working at near 100% capacity (I say this because not all the booths were manned).

It was like this 4yrs ago when it started getting real busy with the boom, and it still is pretty busy now. I reckon they must process about 700-800 people a day at peak periods of the year. In the end I got my documentation, I checked it and left. In a few more weeks time I think I qualify for naturalisation and long term residency, I will need to apply to make my life just that bit easier or crazier given I will need to deal with Irish Bureaucrats that love to change laws.

Windmill Lane graffiti for 2009-10-11

It seems they want to do work on the old Windmill Lane Studios'. Perhaps I should check what they want to do? They've been moving stuff out of the studios recently. I wonder if they want to knock down the old building.

LX3 - 2.0 firmware

Go here http://panasonic.jp/support/global/cs/dsc/download/LX3/index.html for the 2.0 update, here's a copy of the updates copied verbatim from Panasonic's site.

  1. Improved auto white balance performance.
  2. Auto Focus speed-up.
  3. Recording function with an aspect ratio of 1:1 has been added.
  4. A white balance (WB) bracket function has been added.
  5. [HIGH DYNAMIC] has been added to scene mode.
  6. [LENS RESUME] has been added to the [SETUP] menu.
  7. [MENU RESUME] has been added to the [SETUP] menu.
  8. The exposure compensation and the auto bracket compensation can now be set up to ±3EV.
  9. Position of the guide line can now be set.
  10. It is now possible to display the highlights in playback mode.
  11. It is now possible to record the user’s name in the picture.
  12. Digital red-eye removal has been modified.
  13. Items saved in the custom set have changed.

Overall the new features look like it's going to make the LX3 even more fun to play with.

links for 2009-09-20

Been quite busy this week so the list of links that I have looked at is quite small compared to previous weeks

Enterprise reporting tools in academia

I've recently been working with an internal home brew system for research support systems at work at TCD. The system aggregates data from different sources, provides an interface to the members of staff to publish metadata about themselves and most importantly automate some processes to publish information into a digital repository called TARA - Trinity's Access to Research Archive. It's actually not too bad of a system, it works and the users and university get what they need out of the system. Another aspect of the system is the ability to extract data and export some of the data as an XML file, I was the unlucky soul charged with loading up this exported XML and rendering it on a different system.

Talend Open Studio, jasper server and jasper ireport came to my rescue. With much help from http://www.frau-klein.de it was possible to extract and load the data that we were interested in to a useful destination, in our case a MySQL database. After the data was imported into a MySQL DB, it was possible to generate reports with ireport using the MySQL DB as a data source.

As Frau Klein suggested

  1. Find out what data you need to present.
  2. Find out where to get your data from your data source.
  3. Decide how your data should look like in your target DB.
  4. Get Talend Open Studio or JasperETL and design your ETL/ELT process, and make sure you have a small and valid dataset to test with. Note, I had to use TOS because JETL had some bug with the multiple schema xml component.
  5. Use ireport to generate your needed reports.
  6. Profit! - well not really.

What did learn from the process of doing this

  1. Working in academia is different from the enterprise, there are lots of tools out there to do things, so don't reinvent the wheel or do a half assed job. I was going to hack up a perl script to do the above task. I am glad I didn't.
  2. Make people tell you what they want and try to meet those targets and write down these targets so people can agree on them, such that it cannot be changed without good reason.
  3. If there is a standard (in my case, the input XML data file) make sure your source is validated according to the schema, and make people provide data in the proposed format. Deviating from this will cause pain, misery and lots of work. I haven't quite experienced this yet but I can see it coming my way soon.
  4. Need to learn how to say no to feature requests half way through the project.
  5. Make sure there is plenty of time to do the project!

I would highly recommend anyone who needs to do an ELT process to take a look at Talend Open Studio and for anyone wanting to generate reports to look at the Jasper range of opensource tools.

Copying files with tar and ssh

I've recently had the luxury of being handed 6x1tb external HDD's filled with scientific data that needed to be "uploaded" to a storage array for analysis. At first I thought to myself I should just use rsync to upload the files. Obviously it was going to take a while to scan the files then upload the files, this would typically be the command that I would use.

$ rsync -arxvHP /SOURCE/ host:/DEST/

After a bit of thinking, I thought, may be using gnu tar with ssh might have been a better choice to do the initial transfer, then just use rsync to update the dataset if there are any changes. I used something similar to the following command.

$ tar czf - SOURCEFILES | ssh host 'tar xzf - -C DESTDIR/'

The above is just a note to myself as I keep forgetting I can just pipe stuff over ssh.

links for 2009-09-13

Was at the No to NAMA demonstration this weekend, photos can be seen at http://www.flickr.com/photos/jcftang/sets/72157622348784478/

links for 2009-09-06
Another evening out and about with my point and shoot camera

It was a nice August evening so me and Frau Klein went for a walk...

These two pics camera from the camera.

These pics were going to be HDR pics, but it seems the LX3 really isn't suited to do HDR, its just too noisy. I think having a DSLR would be better for HDR. Instead of doing HDR I ended up using enfuse and enblend to merge the pictures.

Here's two classic HDR images. They look pretty nasty and noisy.

links for 2009-08-30
GOVERNMENT PLANS to charge immigrants who receive long-term residency in the State €500 each

Immigrant groups criticise 500 euro residency application fee - sigh I can see more legislation coming down the road for migrant workers as well as more fees. The real kicker is that it takes almost two years for them to process long term residency.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Open64 compilers

I recently came across the Open64 compiler chain,

I was a bit surprised that I had never came across this compiler before, I had been more GCC and commercial compiler focused to have noticed this compiler. So far I have only installed it and compiled OpenMPI 1.3.3 for testing purposes. This may be a nice alternative to the Pathscale compilers which are pretty much dead in the water in terms of developement etc... till the company that used to maintain it gets sorted out.

Panasonic LX3 Settings for 2009-08-25

I've been messing with my LX3 settings and found that setting the noise reduction to -2 gives me the best image that I like.

The current settings for my LX3

C1:
Aperture Priority
Intelligent ISO (with MAX ISO set to 3200)
RAW + High Quality JPG
Dynamic B&W mode
Image Stabilistaion Mode 2
AWB

C2-1:
Aperture Priority
Intelligent ISO (with MAX ISO set to 3200)
RAW + High Quality JPG
Dynamic Color mode
Image Stabilistaion Mode 2
AWB

C2-2:
Aperture Priority
ISO 80
RAW + High Quality JPG
Dynamic B&W mode
Image Stabilistaion Mode 2
AWB

C2-3:
Aperture Priority
ISO 80
RAW + High Quality JPG
Dynamic Color mode
Image Stabilistaion Mode 2
AWB

Film settings

STANDARD COLOR:
Contrast: -1
Sharpness: 0
Saturation: 0
Noise Reduction: -2

DYNAMIC COLOR:
Contrast: -1
Sharpness: 0
Saturation: 0
Noise Reduction: -2

VIBRANT COLOR:
Contrast: +1
Sharpness: 0
Saturation: +1
Noise Reduction: -2

NOSTALGIC COLOR:
Contrast: 0
Sharpness: 0
Saturation: +1
Noise Reduction: -2

STANDARD B&W:
Contrast: +1
Sharpness: 0
Noise Reduction: -2

DYNAMIC B&W:
Contrast: 0
Sharpness: 0
Noise Reduction: -2

The RAWTherapee profiles that I'm using

The above settings and setup mostly works for me, I'm mostly satisfied with the results that I get.

links for 2009-08-23
links for 2009-08-16

Been pretty busy this week so the updates to my blog have been pretty sparse...

General News, pending doom and disaster

Youtube clips

Ireland isn't much better off...

Tech links

links for 2009-08-09

Photography Related

links for 2009-07-26
Windmill Lane graffiti for 2009-07-25

U2 recently had a few performances in Dublin. This of course attracted a lot fans to Windmill Lane to take pictures of the walls of graffiti. The local graffiti artists have been busy as well.

I've been trying to take photos of the graffiti on a regular basis and then blog/post them online, below are a selection of images from the past few weeks.

Using rsync to backup a physical machine

Every so often I need to backup an entire machine, I'm usually lazy too and I prefer to rsync the data between machines so I can restart the process if it fails to complete for whatever reason. The following is what I use to backup or migrate machines

PROMPT> rsync -arvpz --numeric-ids --exclude dev --exclude proc --exclude tmp --exclude sys \
-e "ssh -l root@source" root@source:/. /data/backup_target/.

I could have used tar and piped the data over ssh to another host as well, but I didn't.

links for 2009-08-02

Haven't had much time to blog about stuff in the last week, but instead here's a selection of links that were interesting to me in the past week.

News in general

Photography related

  • Rawdrop - for converting raws to a more usable format
  • dcraw - windows builds of updated versions of dcraw
  • Rawtherapee 2.4 - released recently
links for 2009-07-22

Just a round of links that I've come across that are remotely related to the recent An Bord Snip Nua report and other related mishaps of the government.

What does an economist know about Science and Technology

To follow up on my previous rant Current state of affairs for Ireland in light of "An Bord Snip Nua", http://www.siliconrepublic.com were nice enough to publish a few brief overviews of the recent An Bord Snip Nua report. Though they do lack any real commentary on the issues. It seems they are waiting for a popular opinion to rise so they can summarise the feelings and opinions of the general public who do care.

These three articles caught my eye

I guess the first article's commentary makes sense, but the second article clearly points at another slap in the face to basic research, science and technology. Without basic research there can be no innovation, how does one expect to commercialise on science and technology if the academic is not allow to experiment and try out new ideas and so on.

All I can do is still sigh at some of the findings and recommendations published for third level institutions. Colm McCarthy is probably thinking "What could possibly go wrong with cutting off funding to science and tech?!". The view is a quick intermediate saving for the next 1-2yrs to keep the deficit down. In the long run, there will be even less innovation and commercialisation of ideas. This would make it even less appealing for a phd graduate to want to stay in Ireland.

Right now, Irelands smart economy seems to me to be nothing more than highly educated grunts for the Intel's, IBM's, Google's and Nokia's etc... As Frau Klein pointed out and the report as well, the ICT sector will move to where ever it's cheapest, not being an expert at running an economy, I'm not sure what I would do in the governments shoes. Cutting salaries and raising the retirement age is certainly a start. Perhaps making science and technology sexy again for young people is another thing that can be done by the third level institutions as well.

Panasonic LX3 resources

I've been messing with my LX3 settings and playing with different tools to develop raws etc... here's a collection of useful blogs, pages and websites that I found useful

The current settings for my LX3 (based on the above information)

C1:
Aperture Priority
Intelligent ISO (with MAX ISO set to 3200)
RAW + High Quality JPG
Dynamic B&W mode
Image Stabilistaion Mode 2
AWB

C2-1:
Aperture Priority
Intelligent ISO (with MAX ISO set to 3200)
RAW + High Quality JPG
Dynamic Color mode
Image Stabilistaion Mode 2
AWB

C2-2:
Aperture Priority
ISO 80
RAW + High Quality JPG
Dynamic B&W mode
Image Stabilistaion Mode 2
AWB

C2-3:
Aperture Priority
ISO 80
RAW + High Quality JPG
Dynamic Color mode
Image Stabilistaion Mode 2
AWB

Film settings

STANDARD COLOR:
Contrast: -1
Sharpness: 0
Saturation: 0
Noise Reduction: -1

DYNAMIC COLOR:
Contrast: -1
Sharpness: 0
Saturation: 0
Noise Reduction: -1

VIBRANT COLOR:
Contrast: +1
Sharpness: 0
Saturation: +1
Noise Reduction: -1

NOSTALGIC COLOR:
Contrast: 0
Sharpness: 0
Saturation: +1
Noise Reduction: -1

STANDARD B&W:
Contrast: +1
Sharpness: 0
Noise Reduction: -1

DYNAMIC B&W:
Contrast: 0
Sharpness: 0
Noise Reduction: -1

The RAWTherapee profiles that I'm using

The above settings and setup mostly works for me, I'm mostly satisfied with the results that I get.

Current state of affairs for Ireland in light of "An Bord Snip Nua"

Last week An Bord Snip Nua published a report recommending various ways of saving money for the government. I have in the past wondered what all these experts have been smoking. I'd like some too! Here are the recently published reports

A few months ago, I posted an article Why Ireland's knowledge industries will lead the nation to economic recovery. In light of what's recommended in this recent report, it is yet another slap in the face to the knowledge based economy. I agree with cutting back on costs, but some of the approaches I don't particularly agree with. Cutting back on academia is probably a bad idea for building up a good smart economy, what is likely to happen is with the suggested amalgamation of some of the funding agencies. Irish academics will be advised and instructed to focus on specific areas that this new super agency will identify as a cash cow. In short what will probably happen is that basic and fundamental research will just suffer in general. With the focus on generating income from start ups to form the Nokia's of Ireland other areas such as arts, humanities and other less popular areas (in relation to generating income) will suffer hugely.

What I did find interesting (funny) is starting on page 65 some of the suggestions that they have published. The academics in DIAS must be worried that they might be merged in with another university, which I don't particularly agree with. Although amalgamating the IT's does make sense. Other areas which might be worth noting is that agencies setup to provide a service should stick to providing a service and not doing/applying for research grants. In particular is the suggestion that phd student output should be lowered as industry can't absorb the output so phd graduates simply leave Ireland which is a loss of return for the investment that the Irish government. It's almost comical to suggest the reduction of output of phd students, cut funding, and increase the teaching load of senior research staff. Senior research staff tend to spend most of their time writing grant proposals, reports and just running their research group. I guess if they have no students and no funding its not a problem that they teach more then.

An economist wrote this An Bord Snip Nua report, they are academics too. One should wonder, wtf?! are these guys thinking. Hopefully they have committed to some money to some projects before they decide to commit to pulling all the funding. It would be a disaster to stop all the momentum gained from previous years of investment.

Compiling a 32bit verison of openmpi on a 64bit system

Someone recently asked me to help them to test out a computational code called gamess, unfortunately the code is distributed in binary form only. Not only that, there were many different versions for different architectures and mpi implementations. The kicker was we spent half an hour trying to get it to start up in parallel only to find that it kept spawning off single processes on different nodes. The reason for this was that it was a 32bit blob, our mpi libraries were all 64bit!

I ended up having to compile up a 32bit mpi implementation on the 64bit machines. The following command was used to configure openmpi 1.2.8

./configure --prefix=/misc/shared/apps/openmpi/gcc/32/1.2.8 \
CFLAGS=-m32 CXXFLAGS=-m32 FFLAGS=-m32 FCFLAGS=-m32 \
--with-wrapper-cflags=-m32 --with-wrapper-cxxflags=-m32 \
--with-wrapper-fflags=-m32 --with-wrapper-fcflags=-m32

This tip was found at http://www.open-mpi.org/community/lists/users/2008/08/6337.php and http://www.open-mpi.org/community/lists/users/2006/04/0992.php

Wines - Via Romana

Rating - 8/10, I certainly prefer this over the Elias Mora wine. Via Romana is I think similar to a pinot, it's peppery! I need to describe this more or Frau Klein will comment again that I don't describe it enough.

WTF!? The Minister for Justice in Ireland just makes stupid decisions

WTF?! this new legislation is just stupid, see this Irish Times article. I wonder how this will be implemented, will all people that are Irish be monitored? will you be monitored if you are not Irish but live in Ireland? Will all accounts hosted in Ireland be monitored? Will it be monitored at the SMTP level? What about companies like google who have data centres in Ireland hosting accounts for people across Europe? will these accounts and systems be monitored?

This whole business of keeping this type of information and monitoring emails is abit worrying. One should ask the question, WTF?! is the Minister for Justice in Ireland (Dermot Ahern) doing or thinking.

This is all in the wake of the Libel and blasphemy bill that was passed recently.

Howth and some random pics

Wandered out to Howth today, didn't take too many photos. Here they are with a few other random snaps from the last few weeks that I thought were nice.

My wine from Spain

Graffiti around where I live. I live near Windmill Lane Studio where U2 used to record their music

Howth, just some random pics that I thought were nice

Wines - Elias Mora

I opened up a bottle of wine from my shipment from Spain, it was a bottle of Elias Mora. Since I'm not a fine wine connoisseur I'm simply going to rate it on a basis of do I like it and out of a scale of 1 to 10. So for this bottle, I like it, and I would rate it a 7.5/10. I would definitely recommend it with red meat or duck. Now on to the rest of the red wines!

My shipment of wine from Spain

I recently had a friend ship me 12 bottles of wine. I had no preferences for anything in particular except that I wanted to try out different types of wines. Below is a list of the wines that I got, note that the first bottle Bestue got damaged in the post, so I ended up with 11 bottles of wine in the end.

  • Bestue (€12)
  • 2xAzpiricueta (2x€8.30=€16.60)
  • Elias Mora (€9)
  • Via Romana (€7)
  • Porto Dulce (That is the kind, not the make) (€9.90)
  • Carramimbre (€12)
  • Paramo de guzman (€11)
  • Don Bernardino de Barrica ("De barrica" means aged in a wooden barrel) (€13)
  • Terras Gauda (€11.50)
  • Vinha Somoza (Which is a Godello) (€6.90)
  • Vinha de Martin (€9.70)

Total cost of the wine: €118.50 Shipping: €77.20 Insurance (for €88, which was the maximum allowed for that weight): €4.30 Absolute cost: €200

I will probably setup a table in a wiki page at some point just listing all the wines I've tried and ones that I like/dislike.

meson do forno

I just got back from Spain and I'm starting to sift through my photos that I took on my trip. Although my Spanish is non-existant, I had much fun at meson do forno. It's an Arabic Galician restaurant (as Jose puts it) in Lugo, Spain.

The food was good, the wine was good, but most of all, the owner is quite funny, entertaining and cool ;), it was a bit of a shame that we had a late and big lunch. It's a must visit if you pass by Lugo, note that this is my second time visiting this restaurant. I won't mention too much about the entertainment otherwise it will spoil it. The owner Suso also has a website http://www.mesondoforno.com/, unfortunately I don't read Spanish :P

Here comes the photos.

Free and opensource First Person Shooter games

There's a few other opensource FPS's that I keep an eye out on, Alien Arena recently did a new release. This prompted me to have a look at the other free FPS's that I occassionally play and or download to look at to see if its gotten any better.

The games all vary in community size and quality, I find that Urban Terror is probably the best to play, it's very CounterStrike like and people tend to pick it up quicker than the other games which tend to be pretty wacked out looking. I am a bit partial to the look and feel of Warsow, its a nice alternative to Open Arena or even Quake Arena for deathmatch games. World Of Padman and Alien Arena might be suitable for the younger players or more casual gamers.

One important note about the listed games, they all have windows, linux and osx distributions so everyone can play!

Japan 2006 - some old photos from a trip with some friends

I recently dug out an old hard drive full of photos from a trip to Japan from a few years with some friends. They were taken with fairly good point and shoot cameras at the time. But the resolution and naming of it all were mixed. So I needed to batch resize all the images and sanitise the filenames.

First I used detox to santise all the file and directory names

$ detox -r Japan/

Then i used find and imagemagick's mogrify command to resize the images.

$ find Japan/ -type f -exec mogrify -quality 90% -resize  800x600">" {} \;

There are a few options you can give to mogrify but the above suited me the most, I didn't have to think too much about it on what dimenions to use, whether to up or downscale etc... run the above command and then wait a while. This dropped the size of the original archive from 2gbytes to about 210mbytes.

Next, I used a fairly old python script called curator, which can be found at http://furius.ca/curator/, to generate a static gallery. I think with a few hundred photos ikiwiki just falls apart.

$ cd Japan
$ curator

Then again, wait a while... this added about 40mbytes of data. Here's the gallery of my Japan trip. This might disappear cause its not in the repo since its a bit big!.

How not to build a High Performance Computer

Having helped build a largish infiniband cluster a few years ago (it was big at the time), and admining it for a few years. I think I've learnt what not to do when building, setting up or admining the machine. This is just a rant, perhaps I should compile a list of software/hardware that I like for building and running a High Performance Computer. There are probably many other things that I have probably missed or you may not agree with my opinions.

Some of the do not do's when building, procuring and or admining a cluster

  • Try and avoid tier two or tier three sized vendors for medium to large machines
  • Try and avoid none standard linux distros that deviate too far from RHEL, SLES or Debian based distros. If you are on a small budget centos or scientificlinux is a good RHEL substitute. Pick a distro that will have decent support for the hardware and or software you intend on running.
  • Try and pick tools and helper applications for running the infrastructure and the cluster that are standard and opensource when possible. Don't overly rely on vendor tools (that the company has written). E.g. why use a commercial job scheduler when you can use a free one that you have access to the source code and can install it on different clusters to provide a uniform interface for your users.
  • Try not to edit files and then copy the files out to your compute nodes. It's a better idea to script it or package up the changes so you can systematically and automatically roll out these changes.
  • Survey the type of applications your intend on running, get sample datasets and job runs for benchmarking. You should try and benchmark your codes on the machine that the vendor is trying to sell to you. It's usually a bad idea to listen to vendors raving on about how great performance is going to be. When requesting benchmarks, be reasonable about the job sizes that will get tested otherwise the vendors will probably ignore you.
  • Sometimes its also good to ignore your own users requests, e.g. your users might request more ram in a node than they actually need, sometimes its better to force the users to scale out to more nodes to force them to develop better scalable codes. Or to force them to pick better codes that scale better.
  • Use a centralised user management system such as ldap and not NIS which doesn't scale.
  • Make sure your boot and install process for your compute nodes are scalable and repeatable. In theory network booting and doing an auto install or check on every boot is a good idea, but in practice it doesn't scale unless you have a few boot/install servers running on your network. If you have more than 100nodes on your system and you can only boot 10 nodes at a time it can get quite annoying.
  • When procuring a new machine, try to avoid enabling licenses for software that you have purchased until the last possible moment when you are sure that the hardware is functioning within expected parameters.
  • Vendors like simulating hardware failures, network failures and power outages to test redundancy (if you have requested that certain parts of your machine be redundant). Let the vendor do their thing, but don't trust them, physically go in and unplug network cables, or pull out power cables on machines that have a UPS attached and make sure things failover or shutdown correctly in case of a failure.
  • Make sure you have access to some sort of a serial login or virtual console to your compute nodes, that is make sure IPMI or something similar is available. Otherwise you will be making lots of trips to the machine room to turn things on and off.
  • If your cluster is going to be sufficiently big, try and get a second login node and replicate some of the system services there. This makes management of the cluster much nicer if you need to test things out or reboot the head nodes for security updates etc... or to get users to only compile things on one node and not the one.
  • Try and budget for support for the storage or file system that you are going to get (especially if you plan on running a parallel file system). Or else budget for tape backup for disaster recovery. The last thing you want is loosing user data that might have taken months to generate in the event of a file system / storage problem.
  • Try and get a commerical compiler or two for your hardware. GCC is good, but it doesn't always generate the best possible code for the hardware you are running on.
  • Sometimes vendors write software for monitoring compute nodes, and sometimes this software is gets too big in terms of memory and cpu usage, and sometimes its just a good idea to ignore it and use more lightweight apps to do the same job.
  • Don't listen to the sales guy and buy the biggest and fastest machine they have, it's just not worth paying 10-20% more for a less than 10% performance increase. Look at the price vs. performance curve and decide.
  • Whilst GPU's and offload cards are great for certain applications, sometimes vendors will try and sell you some. Don't believe in some of the marketing or sales hype. If you can test it out then do so, and if your applications take advantage of it then great. If not, don't spend too much time at thinking about or discussing it with the vendor.
  • Don't upgrade the software stack for the sake of upgrading, when your cluster works, just put a freeze on things. You don't want to break a complex and working machine, assuming it is working.
  • Use modules, or at least adopt the methodology of installing multiple versions of software into shared and versioned directories (see gnu stow for an example of what I mean)
  • No matter how much documentation you do for your cluster/queuing system, users will almost never customise the example scripts your give them, so make sure you give them scripts that output lots of useful debugging info from the queuing system and make sure you give them good examples.
  • Make sure the racks you get have wheels!!! or else make sure your vendor will deal with all the hardware installation at your site etc...
  • Don't assume that your machine room has sufficient cooling and power. Check and make sure your site meets the minimum requirements.
Friday evening fun with scammers

It's friday, and I saw an opportunity to have some fun with a scammer...

From: Chuck Norris <jtang@tchpc.tcd.ie>
Date: Fri, 12 Jun 2009 15:30:58 +0100
To: mr.fredpeters1@msn.com, 529229@bugs.debian.org
Subject: Re: Bug#529229: Agent Name: Mr. Fred Peters
X-OSBF-Lua-Score: 17.90/0.00 [H] (v2.0.4, Spamfilter v2.0.3)

Dear Mr. Freed Peters,

On Fri, Jun 12, 2009 at 02:28:22PM +0300, BNL wrote:
>
>
> Agent Name: Mr. Fred Peters
> Tel: +44-702-409-3921
> Email: ( mr.fredpeter01@msn.com )
>
> This is to inform you that you have been
> selected for a cash prize of
> #1,000,000 (British Pounds) held on the
> 10th of June 2009 in London Uk.

Thank you for your notification, but would it be possible to receive the
cash prize in either european euro's or chinese RMB's ? i've been
keeping up to date on the current market trends and british pounds are
just a bad idea, please see this link

http://www.bloomberg.com/apps/news?pid=20601103&sid=avgZDYM6mTFA&refer=news


 1. Name: Carlitos Ray Norris
 2. Address: Rm 208, Long Room Hub, 87 E Santa Clara St, San Jose, CA, USA
 3. Country of Residence: USA
 4. Telephone Number: (209) 966-3110
 5. Age: 49


Best REgards,
Carlitos.

In the midst of all the fun I forgot to remove the to flag to the bugs.debian.org list, sorry guys :P hopefully you got a smile from it though...

Dresden, Germany - Night 3

Went for dinner at the Ontario, it's a really touristy place apparently. I had a pretty good Moose steak, Alex had a steak too, as did Frau Klein (she had the kobe steak) and Andrea had a cheese platter type thing...

Walked around the city center at night, it was pretty nice, there were no tourists around :)

Dresden, Germany - Day 1

Day 1 of my trip to Dresden, and took these photos

It's early in the morning at Dublin airport,


At Berlin, going to get the ICE highspeed train to Leipzig


Hmmm, not even midday yet. I got a kebab for breakfast


Waiting for the ICE train on the platform, there were delays due to some accident with another train on the opposite platform from where our train was supposed to arrive.

This was the offending carriage, there was a fire on the carriage

Still waiting...

Here comes the ICE train!!!

We travelled in first class for the fun of it


Leipzig train station, spot the goths


At a friend of Frau Klein's place, we had beer, the dark beer Koestriker was great

Exporting a tree of files from a git repo

I recently updated the topgit rpm package from opensuse such that it works with centos/scientificlinux. To do this, I needed to get the version numbers and export a tree of files for my tarball and spec file.

I cloned the topgit git repo, and inside the repo i did the following

[jtang@duo topgit]$ git archive --prefix topgit/ HEAD --output  $(git describe).tar

or alternatively

[jtang@duo topgit]$  git archive --prefix topgit/ HEAD  | bzip2 >  $(git describe).tar.bz2

which should give you either

topgit-0.7-17-g78e14da.tar

or

topgit-0.7-17-g78e14da.tar.bz2

These are really just notes for myself since I keep forgetting these little magic commands. In the process of doing this, I also found that the default behaviour of git tag doesn't do what I expect, especially after reading this post http://www.rockstarprogrammer.org/post/2008/oct/16/git-tag-does-wrong-thing-default/

This should do the right thing

git tag -a TAGNAME

or

git tag -s TAGNAME

The above two should make git describe and its friends happy, but the following doesn't quite do what you would expect

git tag TAGNAME

It's probably also useful to find out the changes made between the tags and or HEAD of the repo, doing the following will give you some nicely formatted change log

[jtang@duo topgit]$ git rev-list --pretty=short "topgit-0.7..HEAD"  | git shortlog

It's all useful stuff if want to use tags in your own git repo.


Here's the spec file

#
# spec file for package topgit (Version 0.5)
#
# Copyright (c) 2008 SUSE LINUX Products GmbH, Nuernberg, Germany.
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.

# Please submit bugfixes or comments via http://bugs.opensuse.org/
#

# norootforbuild

%define checkout_version 0.7-17-g78e14da
Name:           topgit
BuildRequires:  sed
Version:        0.7
Release:        2
Summary:        TopGit - A different patch queue manager
License:        GPL v2
Group:          Development/Tools/Version Control
Url:            http://git.or.cz
Source0:        topgit-%{checkout_version}.tar.bz2
BuildRoot:      %{_tmppath}/%{name}-%{version}-build
Requires:       git-core
Requires:       coreutils
Requires:       /bin/sed /bin/grep /bin/awk
BuildArch:      noarch

%description
TopGit aims to make handling of large amount of interdependent topic
branches easier. In fact, it is designed especially for the case
when you maintain a queue of third-party patches on top of another
(perhaps Git-controlled) project and want to easily organize, maintain
and submit them - TopGit achieves that by keeping a separate topic
branch for each patch and providing few tools to maintain the branches.


%prep
%setup -q -n topgit

%build
make prefix=%{_prefix} \
    mandir=%{_mandir} \
    cmddir=%{_prefix}/lib/topgit \
    sharedir=%{_datadir}/topgit

%install
make install \
    prefix=%{_prefix} \
    mandir=%{_mandir} \
    cmddir=%{_prefix}/lib/topgit \
    sharedir=%{_datadir}/topgit \
    DESTDIR=$RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT/etc/bash_completion.d
install -c -m 0644 contrib/tg-completion.bash $RPM_BUILD_ROOT/etc/bash_completion.d/tg-completion.sh

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root)
%doc README
%{_bindir}/*
%{_prefix}/lib/topgit
%{_datadir}/topgit
/etc/bash_completion.d/*

%changelog
* Thu May 28 2009 jtang@tchpc.tcd.ie 0.7-2
- update to 0.7-17-g78e14da
* Mon Mar  9 2009 jtang@tchpc.tcd.ie 0.7-1
- update to 0.7-3-gd145a7b8
- made spec file work on centos5 based systems
* Thu Dec 18 2008 tiwai@suse.de
- updated to HEAD a062d4f8ef8e7d3174b5a572e1002b6c4b2273e7
  * Change tg help exit code to 0
  * Check for cmddir earlier
  * Print help output when no command is given
  * Require an argument to tg -r
  * Print help message when command is not proper
  * Note that do_help is used when short messages might be wanted
  * Add Vim modelines for consistent spacing
  * Pass -- to rev-list for branch/filename disambiguation
  * tg-export: implement skipping empty patches for quilt mode
  * tg-summary: -t and --graphviz are mutual exclusive
  * README: spelling fixes
* Wed Nov 19 2008 tiwai@suse.de
- updated to version 0.5
  as of HEAD 9f5917da92ab8c32ab765a331ebcaf227b3e2f85
* Mon Sep 29 2008 tiwai@suse.de
- updated to version 0.4
* Wed Sep 10 2008 tiwai@suse.de
- updated to version 0.3
* Tue Sep  9 2008 tiwai@suse.de
- initial version
Scammer baiting material

Ever want to bait scammers? with links like these really it can be fun...

http://www.bloomberg.com/apps/news?pid=20601103&sid=avgZDYM6mTFA&refer=news

and

http://formaementis.files.wordpress.com/2008/07/250m-zwd.jpg

demand that you don't want to be paid in USD, and ask for Japanese yen or Euro's. I should really stop replying back to scammers, maybe I would get less spam if I did. I'm always inspired by http://www.419eater.com, but realistically I just spam the scammers back more than baiting them. Maybe I need to re-adjust my sense of humor

Backing up files on a *NIX system

I've been doing my routine backups (read this as snapshots) of my Maildir folder in work recently. Doing a du -s Maildir

[jtang@marduk ~]$ du -s Maildir
3840080 Maildir

Which is approximately 3.7gb of text files. I naively did what I usually do to snapshot/archive my mails by just tar'ing up and bzip2'ing the the directory.

[jtang@marduk ~]$ tar -cf - Maildir | bzip2 > Maildir-$(date -I).tar.bz2

The above resulted in a compressed tarball that was about 690mbytes. So I looked around to see what else was good for compressing data. There's p7zip, paq, dar and lzma. lzma seemed to be the most popular for now. So I gave that a try

[jtang@marduk ~]$ tar -cf - Maildir | lzma -c > Maildir-$(date -I).tar.lzma

Here's the results of compressed tarballs

[jtang@marduk ~]$ ls -l Maildir-*
-rw-r--r-- 1 jtang users 3034542080 May 23 22:37 Maildir-2009-05-24.tar
-rw-r--r-- 1 jtang users  722575723 May 23 22:37 Maildir-2009-05-24.tar.bz2
-rw-r--r-- 1 jtang users  586280476 May 24 00:11 Maildir-2009-05-24.tar.lzma

lzma was easily better in this case than bzip2 for compressing my Maildir directory of files. But lzma is certainly less convenient than bzip2 when it comes to moving my files around to machines that might not have lzma. dar was the other interesting backup program, I'd probably use it more if it was tar like :) and everywhere! I've used dar a few times already to backup entire machines. It's nice being able to extract individual files from an archive without the need to process the entire archive.

Another Saturday, another day of food and being lazy

For our late lunch we had some fillet steaks which I bought for Friday evening (which ended up being a night of nasty food and beer!?)

Nothing too fancy, just some steaks with a mushroom and red wine sauce and some crusty baguettes.

For dinner, Frau Klein cooks white asparagus and a white asparagus soup.

Nerd porn - some high performance machines

I took the opportunity to take a few snaps of a few of our machines in work after a power outage.

The new production machine sitting next to the new (S2A-9900) and old (S2A-9550) DDN storage, there is about 100tb of storage excluding the old IBM DS4400 storage. Only ~1200 compute cores, its of a modest sized for a university I guess.

The old production machine, there is only ~700 cores, but its over three years old now, it was an early infiniband cluster, this is just the front of the machine, its 11racks in size unlike the the new machine which is only 4racks, this is all excluding storage racks.

The rest are just service racks for other things..., notice the cabling mess, we took the opportunity to clean up three years worth of uncontrolled cabling and installing of new machines.

Beer and loud music

I kinda didn't expect to go out for beer and some metal type gig type thing...

sigh they were doing the egyptian sigh and sms bombing Geoff, "not nerd!" the MBA will take away Geoff's Nerdness...

Dermot left his bag at the pub, notice that frau-klein has Dermot's bag

Grabbing some nasty junk food from supermacs, frau-klein wasn't impressed at all

The rest of these were just from the gig at some pub (I think it was Murrays)

I took more photos, but I only had my t200, it was rather disappointing most of the pics came out blurred. I think I am getting spoilt by my Panasonic LX3

Fast Food - Stir fried udon noodles

Was hungry last night and didn't really know what to eat, so I picked up some quick fry steaks, mushrooms and eggs. I was going to make a steak sandwich cause I had some bread left. Instead I found a pack of udon noodles

In the meal there was,

  • chilli
  • ginger
  • soyasauce
  • oyster sauce
  • one egg
  • sliced steak
  • udon noodles

it was either this or a hot dog, chips and onion rings from the local chipper.

Problems with long term storage and archiving data forever

With more and more libraries seeing a drop in student numbers using the physical facilities, libraries are increasingly moving more towards digital repositories that are meant to last forever. There are clear advantages of having a digital repository. Such as being able to search for titles easily, if there is a copy in the archive then there is no worry that you won't be able to check it out if you have the appropriate authorisation and many people can check out the same copy of work if needed. There are probably a whole host of other reasons why its a good idea.

On the flip side, setting up a long term storage system with archival qualities is a nightmare. There seems to be a range of issues with

  • data preservation
  • tagging data with metadata
  • authentication and or authorisation for copyrighted works
  • disaster recovery issues
  • backups
  • possible audit trails if metadata is edited
  • vendor problems, you will need to commit to a system for a long time (software and or hardware vendors), so who and what do you commit to?
  • preserving data formats and tools to manipulate and convert these formats
  • need for HSM? are spinning disks better than tapes?

The list of problems probably goes on. So how do people deal with this problem? The only game in town right now seems to be dspace or fedora commons for the software side. For the hardware it seems to be a bit more vague, I get the impression people are just buying robust mid ranged kit that mostly fits their needs for archival, then upgrade a few years down the road with the equivalent hardware and then just migrate the data or replicate it to the new system.

How do people tackle these issues? Do people build on top of dspace or fedora commons with mid ranged hardware solutions from a tier 1 vendor like IBM, HP, sun or whatever tier 1 vendor? Do people outsource it to a tier 1 company such as the previously listed ones?

Strict rules on Naturalisation

Today I got a letter back from my initial posting of my application for naturalisation. It appears that I don't fully qualify to apply, I'm at 54 of 60 months of the required time. It only took two weeks for them to process and reject my application. I guess this will be put on the back burner for another 6months before I try again, now that I know that I need to meet the minimum requirements for applying in terms of time.


Related posts:

I'm now naturalised and Irish!
Posted

24 months and two weeks
Posted

2012-02 -- Update on long naturalisation times in Ireland
Posted

GNIB 2011 (till 2012) - Here I go again
Posted

New Irish citizens will have to swear oath of fidelity
Posted

Using ikiwiki for project management and issue tracking

We've been having arguments and discussion on project management in work recently. Personally I like the idea of using an issue tracker for tracking todo's, bug's and generic issues that a project might have. Most of the software I've tried are often over engineered for what I like and want to use in practice. Things like bugzilla, mantisbt, request tracker, redmine and a few others that I can't remember do almost what I want and need but tend to miss features that another system has etc...

I simply took the softwaresite ikiwiki example and renamed it to my project name, and added a "todo" page for tracking issues. With the web interface, its easy enough to use if one doesn't know how to use git and the command line etc... I think the softwaresite example needs to be refined a bit more for general project management usage. I plan on using tags for milestones and marking severities for issues that arise in my projects.

Montys of Kathmandu and a stroll around of Dublin city centre at night

The dinner at Montys of Kathmandu

Went out for dinner at a Nepalese restaurant near Templebar.

The starters

The main course

Overall the dinner was pretty tasty, we were highly satisfied.

A pint at a pub near the cinema

This was followed by a pint and the new Star Trek film!

Dublin at night

Dublin at night, apart from the drunktards screaming at each other, the girls in short skimpy dresses and dodgy characters Dublin city centre can be nice in places, though some places in Dublin at night is like a warzone.

clearly from these pictures, I need to play around with my camera more for night time photos. Some of them came out pretty good. I suspect I would need a cheap tripod to get some nicer ones for the long exposure photos. I'm still pretty impressed with the iA mode on the Panasonic LX3

Lifestream and blogging at home

Frau Klein wanted a server at home for her lifestream and me being lazy and the such didn't do much for a while. Till I got the great idea of turning my eeepc 701 4g into a server. I blew away the install with eeebuntu. Pro tip, use a decent external usb cd/dvd rom drive to install it. Or else go read the docs on how to make a bootable usb memory stick.

I chose to install eeebuntu netbook remix, I was pleasantly surprised at how well it worked. I probably should have just installed the base version to save space, but I have a spare 8gb SDHC card for the eeepc anyway. It's configured so that my /home is on the SDHC card. The pages can be found at...

I'm using the dyndns.com dynamic dns service, which gets automagically updated as my WAN addresses changes every 24hrs. The Linksys WRTG router that I have, which I have flashed with the tomato firmware handles all this automatically. As a side effect of all this, I have a nice little fairly power efficient machine that I can ssh to from anywhere in the world and I get a mirror of my blog locally.

Compiling GCC 4.x on older systems

Recently I was asked to install the latest GCC tool chain and mvapich for someone, I've done it a few times but keep forgetting what I need to install. So here are some quick notes on what I usually do. GCC 4.x, well the latest stables require that you have a few dependancies installed, such as a working compiler, your system compiler will do and two libraries.

The following was carried out on scientificlinux 4.5 based cluster.

First download the following packages

and of course gcc itself

The GCC documentation on Installing GCC was pretty useful, it can be found at http://gcc.gnu.org/install/

We first install libmpfr

tar xvjf mpfr-2.4.1.tar.bz2
cd mpfr-2.4.1
./configure --prefix=/usr/support/arch/x86_64/GCC/4.4.0 \
     --exec-prefix=/usr/support/arch/x86_64/GCC/4.4.0 \
     --libdir=/usr/support/arch/x86_64/GCC/4.4.0/lib64
make 
make check
sudo make install

Now install libgmp

tar xvjf gmp-4.3.0.tar.bz2
cd gmp-4.3.0
./configure --prefix=/usr/support/arch/x86_64/GCC/4.4.0 \
     --exec-prefix=/usr/support/arch/x86_64/GCC/4.4.0 \
     --libdir=/usr/support/arch/x86_64/GCC/4.4.0/lib64
make
make check
sudo make install

Note that it is important to run a make check to be sure that these numerical libraries will give the correct numbers.

To configure and install GCC you need to set your LD_RUN_PATH and LD_LIBRARY_PATH such that configure will find the needed libraries.

export LD_LIBRARY_PATH=/usr/support/arch/x86_64/GCC/4.4.0/lib64:$LD_LIBRARY_PATH
export LD_RUN_PATH=/usr/support/arch/x86_64/GCC/4.4.0/lib64:$LD_RUN_PATH
export LDFLAGS='-L/usr/support/arch/x86_64/GCC/4.4.0/lib64  -Wl,--rpath -Wl,/usr/support/arch/x86_64/GCC/4.4.0/lib64'
tar gcc-4.4.0.tar.bz2
mkdir gcc-4.4.0-build
cd gcc-4.4.0-build
../gcc-4.4.0/configure --prefix=/usr/support/arch/x86_64/GCC/4.4.0 \
    --with-gmp=/usr/support/arch/x86_64/GCC/4.4.0 \
    --with-mpfr=/usr/support/arch/x86_64/GCC/4.4.0
make
sudo make install

By default these language are built,

 c,c++,fortran,java,objc

You could probably replace the path /usr/support/arch/x86_64/GCC/4.4.0 with something like /usr/local or even $HOME/local

To use the compiler you will need to set a few environment variables, I prefer to prepend my environment to make sure I get the right libraries and executables.

export LD_LIBRARY_PATH=/usr/support/arch/x86_64/GCC/4.4.0/lib64:$LD_LIBRARY_PATH
export LD_RUN_PATH=/usr/support/arch/x86_64/GCC/4.4.0/lib64:$LD_RUN_PATH
export LDFLAGS='-L/usr/support/arch/x86_64/GCC/4.4.0/lib64  -Wl,--rpath -Wl,/usr/support/arch/x86_64/GCC/4.4.0/lib64'
export PATH=/usr/support/arch/x86_64/GCC/4.4.0/bin:$PATH

Your paths may differ, so set them correctly. The LDFLAGS line I guess is optional, since it just overrides the LD_RUN_PATH variable with your own custom one.

Once its all compiled and installed and my environment set, I was pretty much able to compile mvapich with the updated compilers.

Ireland and other tax havens for US companies getting their advantage limited

http://www.irishtimes.com/newspaper/frontpage/2009/0505/1224245943467.html - this was going to happen eventually, it was a question of when some of these loop holes were going to get closed. As I have said before the recent article in Silicon Republic on Why Ireland's knowledge industries will lead the nation to economic recovery is a bit retarded. Everything is not fine and we haven't hit rock bottom yet with the economy. I can almost see a lot of US companies either downsizing their Irish operations or leaving the country, the act of not increasing corporation tax on these foreign companies didn't really encourage them to stay.

I wonder if this is an idle threat?

Cool tools for panoramic and HDR photos

Been playing with Hugin and qtpfsgui both are free, though I'm not 100% sure of the licensing of the tools. With a small bit of work I think you can get some pretty cool photos without having to pay for commercial software. They may or may not be as powerful as the commercial alternatives depending on what you like, but at least these two tools are cross platform.

Rawtherapee is another free tool for anyone who takes raws from their DSLR or high end point and shoot cameras. Now all I need to do is find some nice free and cross platform photo management software. 'ls' comes to mind.

A Panorama of Grand Canal Docks

Here's another panorama, it took ages to generate from the 7x10megapixel images. The final jpg that was generated was ~125megs in size at a resolution of ~24kx15k. I had to crop it to bits afterwards. I need to figure out how to get hugin to crop the image before the final output.

The pieces of the image were taken with my Panasonic LX3 camera.

A Panorama of Howth - the lighthouse

While at Howth, I took a few photos to see if I could make a decent looking panorama picture, my motif kinda failed, but the panorama image came out ok. The shots were taken by hand with no tripod on my Pansonic LX3 point and shoot camera.

Slowly and surely I'm forgetting about my old Canon AE1 and my Sony T200. The image was created using http://hugin.sourceforge.net/

Click on the image above to view the fullsized jpg, be warned its about 10000x1700 in size.

Why Ireland's knowledge industries will lead the nation to economic recovery

Why Ireland’s knowledge industries will lead the nation to economic recovery is the title of Silicon Republic's article. I'm scratching my head a bit, I always thought Ireland's main industry was exporting value added products and pharmaceuticals etc..., and it is not as knowledge based as it wants to be. Partly due to low numbers of students going into third level education science and engineering. This may be changing now as a result of the economic down turn as doing "Business course" might not be the best idea in the world. The Tougher criteria for work permits doesn't really bode well in creating a high end knowledge based economy especially when they could now be snapping up quality American/Asian/Australian migrant workers. To top that off, the increase in taxes, the high cost of living and government levies also doesn't make it too attractive for prospective workers.

The recent slap in the face of an emergency budget and the collapsing (or collapsed) property market has really screwed with the 30 somethings who have recently bought a house and have started a family. It wouldn't be inconceivable that some of these families might leave Ireland for a better place. Ireland will once more be a great exporter of people :P I'm not sure how Ireland will build a knowledge based industry.

Some times I wonder what these guys are smoking, cause I want some to smoke as well. But it's mostly WTF?

Seafood dinner after going to Howth

After going to Howth, we also bought some seafood. Namely some Tiger Prawns, Razor Clams and a Lobster...

The starter, some big prawns. Cooked with garlic, scallions, salt and pepper, light soyasauce and some oil. Baked it in a hot oven for about 10mins.

The second course. Drunken prawns, well it's my version of it. It's just some scallions, chilli, garlic, soyasauce and about a cup of Shao Xing rice wine, we let it soak for a few hours before cooking.

Razor clams with ginger, onions and spring onions and a small bit of garlic, just stir fried after lightly poaching the clams. The clams took ages to clean properly. It was worth the effort to clean :)

The Lobster for the last dish. We just cooked it in a vegetable soup. (the lobster was made by Frau Klein, the Lobster was killed by jtang by putting it into boiling wate