Sigmonsay

Musings and Banter

Acp

without comments

Acp.

– SNIP –

Acp is a rough implementation of an optimized filesystem walker. In general, when doing a full read of a directory tree you touch three different groups of objects.

 

  • Directory data
  • Inode data (things returned by stat(2))
  • File bodies

Acp creates queues corresponding to each of these groups, and tries to do work in bulk in each one. As it finds files and directories the are sorted by either inode number (the default) or by the first block in the file (acp -b). Run acp -h to see the other command line options.

In general, the sorting makes the IO started by acp much closer to the order the filesystems store things on disk. It is especially effective on ext3 when htree is used, because htree causes readdir to return files in random order. This graph shows the performance difference while reading a single kernel tree with tar and with acp.

 

– SNIP –

Written by sig

March 31st, 2011 at 7:58 pm

Posted in Uncategorized

bash path cache

without comments

All UNIX shells cache the command paths based on the contents of PATH enviromental variable. This can cause a problem if a cached path no longer exists. For example, you have a command “foo” installed in /usr/bin and /usr/local/bin. Your PATH variable is set to “/usr/local/bin:/usr/bin“.When you run “foo“, it is searched under each directory listed in PATH and the results are cached. In this case, the path “/usr/local/bin/foo” will be cached for “foo“.Now suppose you delete the command file “/usr/local/bin/foo“. You still have another copy in “/usr/bin/foo“. However, the next time you type “foo“, the shell will return an error such as this:

-bash: /usr/local/bin/foo: No such file or directory

To clear the cached path of foo command, you can run

$ PATH=$PATH

This basically resets the PATH variable, thereby clearing the cache. For bash shell, you might be able to do the same thing using

$ hash -r

Although, the previous method should also work for bash.

 

Written by sig

March 28th, 2011 at 5:27 pm

Posted in bash

brick

without comments

 

 

Written by sig

March 22nd, 2011 at 6:52 pm

Posted in Uncategorized

Varnish, Couch and S3

without comments

Sending screenshots around is something I do quite often. Rather than rely on external screenshot services I like to build my own using available cloud services. Because doing everything on the cheap is better, more fun and challenging.

Configuring varnish to fetch assets from CouchDB and then S3 as a fail safe if couch is MIA was a fun use of available technologies. Couch lets you save attachments in documents and it was quite a simple addition to the screenshot program I use.

python screenshot program

# add metadata record and image to couchdb
 couchone_uri = "http://XXXXXX.couchone.com/screenshot"
 scheme, netloc, path, query, fragment = urlparse.urlsplit(couchone_uri)
 ss = httplib.HTTPConnection(netloc)

 # save meta data
 rec = {
 "year" : int(time.strftime("%Y")),
 "month" : int(time.strftime("%m")),
 "day" : int(time.strftime("%d")),
 "doctype" : "metadata",
 "s3_url" : s3_url,
 "s3_key" : s3_key,
 }
 headers = {
 "Content-Type" : "application/json",
 }
 couch_key = s3_key.replace("/", "-")
 ss.request("PUT", os.path.join(path, couch_key), json.dumps(rec), headers)
 res = ss.getresponse()
 ret = json.loads(res.read())
 print res.status, res.reason, ret

 # Upload the image as an attachment
 aurl = os.path.join(path, ret['id'], "image")
 aurl_params = "?rev=" + ret['rev']
 ss.request("PUT", aurl + aurl_params, file(filename).read(), { "Content-Type" : "image/jpg" })
 res = ss.getresponse()
 print res.status, res.reason, res.read()
 attach_url = scheme + "://" + netloc + os.path.join(path , ret['id'], "image")
 print "URL", repr(attach_url)

Written by sig

March 20th, 2011 at 9:17 pm

Posted in Uncategorized

Brace yourself

without comments

Brace yourself

 




Written by sig

March 19th, 2011 at 5:38 pm

Posted in Videos

Japan: One week later

without comments

From The Big Picture, Boston.com:

 

 

 

 

From Japan: One week later – The Big Picture – Boston.com.

Written by sig

March 18th, 2011 at 10:31 pm

Posted in Uncategorized

Genesis from garito on Vimeo

without comments

Written by sig

March 18th, 2011 at 10:10 pm

Posted in Uncategorized

First Person Video of Tsunami Flooding Japanese City

without comments

I too am without words. AMAZED at the damage that unfolds, by the cars at first, then floored by entire buildings getting washed away.

 

Written by sig

March 16th, 2011 at 9:59 pm

Posted in Uncategorized

Kick out the blogs

with one comment

Sittin here on a rainy day in SF skipping the swim swum getting wordpress less broken

 

And i’m having some troubles getting posts working with direct links…

Written by sig

March 15th, 2011 at 10:59 pm

Posted in Uncategorized