Sigmonsay

Musings and Banter

Archive for the ‘Uncategorized’ Category

Interesting links

without comments

Kinder egg banned in the US? Really, wtf USA.

http://www.eatliver.com/i.php?n=7410

http://news.blogs.cnn.com/2011/04/21/easter-reminder-kinder-eggs-banned-in-the-united-states/

Vancouver riots at loss

http://www.eatliver.com/i.php?n=7413

Making Love, Not War http://yfrog.com/z/h03elanp

I’m sure someone already is working on a “Will it Fry?” blog

http://www.wimp.com/friedkoolaid/

Now that’s fast, even though i’m not sure it hit 500km/h in this video..

http://www.wimp.com/maglevtrain/

I would do this in a heart beat. Although I’d be scared, the lion would sense that and likely bite
my head off.

http://www.wimp.com/lionnap/

Written by sig

June 21st, 2011 at 9:04 pm

Posted in Uncategorized

What Should I Do? Choosing SQL, NoSQL or Both

without comments

Written by sig

June 16th, 2011 at 7:29 pm

Posted in Uncategorized

Tagged with ,

how to use a phone

without comments

Written by sig

June 13th, 2011 at 6:44 am

Posted in Uncategorized

cholula, honey and rice

without comments

I was too lazy to make any real food and had some left over rice. Give it a try, it’s quite amazing.

Written by sig

June 4th, 2011 at 3:54 pm

Posted in Uncategorized

Excavator lifting itself to the top

without comments

Written by sig

June 3rd, 2011 at 10:33 pm

Posted in Uncategorized

drop dead simple HOME synchronization

without comments

Something that is always annoying to me is the setup, scripting and management of keeping a hosts files and scripts in a safe place. Well i’ve made various reincarnations of various scripts that attempt to do this. I’ll ommit the history here but wanted to drop a quick note of a drop dead simple solution that has worked pretty well for me.

drop the contents below into Makefile and git init it.

.PHONY: backup

all:
 @echo try 'make backup' or 'make restore'

backup:
 @echo backup file from system started
 rsync -arv --files-from manifest.txt / backup/
 python filemeta.py backup -P backup/
restore:
 @echo restore file to system started
 rsync -arv --files-from manifest.txt backup/ /
 python filemeta.py restore

Since git does not preserve file mode and permission I hacked up a quick script to handle backup and restore of the key file meta data (mode and ownership). Save the contents here as filemeta.py

import sys, os, stat
from optparse import OptionParser

_, cmd = sys.argv.pop(0), sys.argv.pop(0)

parser = OptionParser()
parser.add_option("-c", "--config", dest="config", help="config file", metavar="FILE", default=None)
parser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False)

parser.add_option("-f", "--file-meta", dest="file_meta", default="./file.meta", help="store file meta data path [%default]")
parser.add_option("-P", "--path", dest="path", default=".", help="path to scan [%default]")

if cmd == 'backup':
   (options, args) = parser.parse_args(sys.argv)
   path = os.path.realpath(options.path)
   file_meta = os.path.realpath(options.file_meta)
   file_meta_tmp = file_meta + ".tmp"

   fh = open(file_meta_tmp, "w+")
   for dirpath, dirnames, filenames in os.walk(path):

      st = os.stat(dirpath)
      fh.write("%d %d %d %s\n" % (st.st_mode, st.st_uid, st.st_gid, dirpath))

      for filename in filenames:
         fullpath = os.path.join(dirpath, filename)
         st = os.stat(fullpath)
         abspath = fullpath.replace(path, '')
         fh.write("%d %d %d %s\n" % (st.st_mode, st.st_uid, st.st_gid, abspath))

   fh.close()
   os.rename(file_meta_tmp, file_meta)
   print "Created", file_meta

elif cmd == 'restore':
   (options, args) = parser.parse_args(sys.argv)
   path = os.path.realpath(options.path)
   file_meta = os.path.realpath(options.file_meta)

   for line in file(file_meta):
      st_mode, st_uid, st_gid, abspath = line.split(None, 3)
      st_mode, st_uid, st_gid = int(st_mode), int(st_uid), int(st_gid)
      abspath = abspath.strip()

      if stat.S_ISDIR(st_mode) or stat.S_ISREG(st_mode):
         os.chmod(abspath, st_mode)
         os.chown(abspath, st_uid, st_gid)
      else:
         continue

else:
   raise Exception("invalid command: %s" % (cmd))

Now all you have to do is place the filenames to backup (files or directories) in manifest.txt and run ‘sudo make backup’ or ‘sudo make restore’. git commit or add backup/* and push it off to a origin of your choice. This is nothing that complex but the simple approach to this has made box recovery and migration a breeze. Lots of times you want to track something but don’t have a good place to put it. This is that place. all of those pesky $HOME/.bash_profile files are now in one place and can easily be fetched from a single location. I’ll leave branching and cherry picking to the reader.

Written by sig

May 23rd, 2011 at 1:26 am

Posted in Uncategorized

Acp

without comments

Acp.

– SNIP –

Acp is a rough implementation of an optimized filesystem walker. In general, when doing a full read of a directory tree you touch three different groups of objects.

 

  • Directory data
  • Inode data (things returned by stat(2))
  • File bodies

Acp creates queues corresponding to each of these groups, and tries to do work in bulk in each one. As it finds files and directories the are sorted by either inode number (the default) or by the first block in the file (acp -b). Run acp -h to see the other command line options.

In general, the sorting makes the IO started by acp much closer to the order the filesystems store things on disk. It is especially effective on ext3 when htree is used, because htree causes readdir to return files in random order. This graph shows the performance difference while reading a single kernel tree with tar and with acp.

 

– SNIP –

Written by sig

March 31st, 2011 at 7:58 pm

Posted in Uncategorized

brick

without comments

 

 

Written by sig

March 22nd, 2011 at 6:52 pm

Posted in Uncategorized

Varnish, Couch and S3

without comments

Sending screenshots around is something I do quite often. Rather than rely on external screenshot services I like to build my own using available cloud services. Because doing everything on the cheap is better, more fun and challenging.

Configuring varnish to fetch assets from CouchDB and then S3 as a fail safe if couch is MIA was a fun use of available technologies. Couch lets you save attachments in documents and it was quite a simple addition to the screenshot program I use.

python screenshot program

# add metadata record and image to couchdb
 couchone_uri = "http://XXXXXX.couchone.com/screenshot"
 scheme, netloc, path, query, fragment = urlparse.urlsplit(couchone_uri)
 ss = httplib.HTTPConnection(netloc)

 # save meta data
 rec = {
 "year" : int(time.strftime("%Y")),
 "month" : int(time.strftime("%m")),
 "day" : int(time.strftime("%d")),
 "doctype" : "metadata",
 "s3_url" : s3_url,
 "s3_key" : s3_key,
 }
 headers = {
 "Content-Type" : "application/json",
 }
 couch_key = s3_key.replace("/", "-")
 ss.request("PUT", os.path.join(path, couch_key), json.dumps(rec), headers)
 res = ss.getresponse()
 ret = json.loads(res.read())
 print res.status, res.reason, ret

 # Upload the image as an attachment
 aurl = os.path.join(path, ret['id'], "image")
 aurl_params = "?rev=" + ret['rev']
 ss.request("PUT", aurl + aurl_params, file(filename).read(), { "Content-Type" : "image/jpg" })
 res = ss.getresponse()
 print res.status, res.reason, res.read()
 attach_url = scheme + "://" + netloc + os.path.join(path , ret['id'], "image")
 print "URL", repr(attach_url)

Written by sig

March 20th, 2011 at 9:17 pm

Posted in Uncategorized

Japan: One week later

without comments

From The Big Picture, Boston.com:

 

 

 

 

From Japan: One week later – The Big Picture – Boston.com.

Written by sig

March 18th, 2011 at 10:31 pm

Posted in Uncategorized