Wensheng

Creating ooxml word docx file with python and XSLT

2010-05-16T01:52:00.011-05:00

The idea is taking an existing Word document, making it a template, and using it with external data to create a new Word document.

If you're dealing with 'doc' file (word2003 and before), you can use Pywin32. There are good information in the Chapter 10 of "Python Programming on Win32" book, in a section called "Automating Word".
But 'doc' file is on the way out. Staring from version 2007, Word is using the new 'docx' format. The good news is that 'docx' use OOXML (Office Open XML), an open industry standard. This means we can, in theory anyway, create Word DOCX file without using Windows. In practice, there's currently no good library that facilitate creating a docx file from xml data from scratch. It's tedious to do so without a good library. Hopefully someone will create one soon.

A short-cut exists though, i.e. create a template docx, create XSLT from it, then use XSLT to transform xml data into a new docx file.

Here's a post that describes this process using CSharp. My code is based on ideas and codes in this post.

Here's how to do it in Python (using the excellent lxml module):

1, Create a template docx file
Just create a regular docx file with Word (example.docx)

2, Create xslt file from docx file.
A docx file is a zip file. You can get "word/document.xml" file by unzip the docx file. But the resulting file has no line breaks. It's very hard to edit it using regular text editor like VIM. (you can use xml editor, but there ain't good one that's free). I have a python program that does this (getxslt.py). All it does is getting 'word/document.xml', prettifying it(adding linebreaks), and adding XSLT header and footer.
We need to use the docx file later, but without 'word/document.xml' file (python's zipfile module can't do file replacement in zip). You can achieve this by "open archive" the docx file with '7-zip' on windows, then delete 'word/document.xml'.

3, Modify xslt file.
Determine where you need to change the data, add things like:
<xsl:value-of select="..."/>
and
<xsl:for-each select="...">
...
</xsl:for-each>
You need to study a little bit xslt to do this, it's not too hard.

4, Do XSLT transform to convert xml data to 'word/document.xml' and add it to new docx file.

import shutil
import zipfile
from lxml import etree

xsl = etree.XSLT(etree.parse("xslt1.xml"))
xml = etree.parse("cddata.xml")
newxml=xsl(xml)

#nodocxml.docx is original docx file without word\document.xml
shutil.copyfile("nodocxml.docx","mycds.docx")

mycds = zipfile.ZipFile("mycds.docx",'a',zipfile.ZIP_DEFLATED)
mycds.writestr('word/document.xml',str(newxml))
mycds.close()

explanation:
'xslt1.xml' is the modified xslt file from the step 3;
'cddata.xml' is xml data file (source: w3schools xslt tutorial );
newxml is the result xml tree of xslt transformation;
'nodocxml.docx' is the same as example.docx, the original docx file, except it doesn't have 'word/document.xml';
'mycds.docx' is our final product.

The example files can be downloaded from here: http://bitbucket.org/wensheng/pydocx/downloads/pydocxml.zip

The example does very few transformation. If you want to more advanced stuff like adding images, you have to be more familiar with OOXML. (Add image to media directory, change relationships xml file, add relationship anchor to document.xml etc.)

Getting back data from lvm on raid1 on a single disk

2010-04-13T00:40:00.005-05:00

One of my servers died. The only thing I have left is a hard-drive from that server. But I have important data I need to recover from this disk.

The disk was part of RAID1 (software-raid) disks. It's not boot-able, and only contains LVM volumes.

Here's how I get my data back:

1, Connect the hard-drive to my PC. I used a USB docking station.

2, My OS (Fedora) assigned it to /dev/sdd. It can not mount it since it's raid.

3, make raid node, and attach the partition
mknod /dev/md0 b 9 3
mdadm --create /dev/md0 --level=raid1 --name=0 --auto=md --raid-disks=1 -f /dev/sdd1
"-f" switch is needed to force creating raid1 with only 1 disk.

4, now if I do pvscan, vgscan, and lvscan, it shows my vg (vg1) and lvm's:
pvscan #it shows PV /dev/md0 VG vg1
but I still can not mount them, because no volume group actually existed. Do this:
vgchange -a y vg1
Now /dev/vg1 is activated.

5, Do the mount as usual
mount /dev/vg1/wensheng /mnt

After I got back all my data, I disassemble the hard-drive to get the magnets, and throw the rest to trash. Why? because of this:

from my experience, this drive will die soon anyway.

IGot24 - a javascript game for calculate 24, hosted on AppEngine

2009-11-26T00:38:00.003-06:00

I wrote a small game in the last few days. The game is called IGot24.
The idea is to use 4 basic arithmetic operations to get result 24 from 4 randomly drawn cards. I talked about the algorithm and python implementation in an old post. In fact, it's currently the first result on Google for "calculate 24".

The game is at:
http://calculate24.appspot.com/

I first wrote the game using pure JavaScript. Later I thought whoever play it may want to get solutions. So I created the backend with Google AppEngine that supply cards and solutions. Then I added a few more options, like getting only solvable cards, disabling timer, calculate 42 instead of 24, etc.

The heart of the JavaScript code is a finite state machine. At first I just coded away thinking it'll be simple, then gradually realized there're more states. I then drew a state diagram and found out there were 21 states! They are all branches, there's no loopback, so I don't think they can be optimized. (Maybe they can, but I don't want to re-read my circuit design book from ages ago to find out)

I think the game can be easily re-done in flash using Flex, since JavaScript can be ported to AS3. It can be ported to IPhone too but it will be waste of time and money (to buy a mac to do iphone dev) because I found a "calculating 24" iphone app got downloaded a whopping 12 times.

Chrome OS on Xen HVM

2009-11-21T23:55:00.000-06:00

Google announced open-source Chromium OS 2 days ago. today I tried to build it on a Ubuntu virtual machine. But I failed, had numerous problems I won't elaborate. I will try again sometime later.

I searched web and found a VMWare virtual disk of ChromeOS here. So I downloaded it.

Not wanting to install VMWare, I converted it to a raw disk image and loaded it to XEN with HVM, it worked. But it basically unusable for me mainly because of mouse movement was too slow. I will try with a different vnc client later to see if it improves.

Here's how to convert vmdk image to raw:
$qemu-img convert -f vmdk chrome-os-0.4.22.8-gdgt.vmdk -O raw chrome-os.img

Then just specify the image file in 'disk' line of domain config file, like:
disk = [ 'file:/root/chrome-os.img,hda,w' ]

Here's screenshot of chrome OS:

Here's chrome OS login screen, I have a Fedora 12 hvm domain and a Ubuntu 9.04 32bit domainU also running on this machine. The domain0 itself is Fedora 12 64bit with pvops kernel 2.6.31.6.

I had to use standard VGA driver(stdvga=1), the default Cirrus Logic driver give me a screen like this after I login:

POC: online barcode qrcode scan with webcam

2009-10-27T01:24:00.004-05:00

This is proof of concept(POC). The idea is to scan rebates, coupons, tickets etc. in the forms of barcode and qr-code using your web-cam.

Such an idea is really nothing new. With cellphone, you can now scan the barcode when you pick up an item in store, and get price information on whether it is cheaper at nearby stores or online. Also barcodes on cellphone can now be used for mobile ticketing.

But I have not heard a lot about scanning rebates/coupons/tickets with webcam online. So I created this POC project.
Screenshot:

It somewhat works. QR-code works consistently but most of times the program can not read UPC and code 128. I am sure with some added backend photo processing, I can improve the result, but I doubt it will help a lot. Two of my 3 webcams have a focal length that make it impossible to snap a clear code image, unless of course the image is real large like 400x400 or 500x500px. So I guesstimate most webcams are not suited code scanning?

Here's actual page in a iframe:

The frontend is build with Flex SDK. The webcam code come straight from here.
The backend is using Zxing.
The QRcode test sample is generated from here with google chart api.

JW media player with lyrics scroller

2009-09-02T23:52:00.012-05:00

Yesterday, I made a lyrics scroller. The thing works, but it's missing quite a few things, like, progress bar, seek, time display, volume control, videos.

So today I made a new one, this time using jw media player. JW media player is a full featured web media player. It has almost everything. Now it has a lyrics scroller;)

Below is a demo, it's a iframe of this page.

Usage:
Include 4 javascript files in the html header:
jquery.js, jquery.scrollTo-1.4.2-min.js, swfobject.js, jwplrc.js

Then put these javascript code in html:

var flashvars = {
file:"somesong.mp3",
lrc:"somelrc.lrc"
};
create_jwplrc("player_divid","320","80",flashvars,{},"some_uniq_id");

The "file" flashvar can be used to specify a media file such as mp3 or a playlist xml file.
If it's a single media file, you need to specify "lrc" flashvar to tell the player where lrc file is. (update on 1/18/10)It can be in your web directory or anywhere on the web.
if it's a playlist, no "lrc" flashvar is needed. You imply that lrc and media are at
same location/directory, and lrc and media have the same filename. If it's a playlist, you need to specify lrc file in "<info> </info>" inside playlist xml file. The media files can be anywhere on the web, but lrc files has to be on your own site.

"create_jwplrc" is just a wrapper function that wraps "swfobject.embedSWF". The first argument for create_jwplrc is the id of a "<div>" that will be the player, 2nd/3rd are width and height, 4th is flashvars, 5th is parameters, it can be empty {}. The 6th argument is a unique name for the player.

File download wensheng.com/code/jwlyrics.tgz (updated 1/18/10)

A javascript mp3 player with scrolling lyrics (lrc) display

2009-09-02T03:00:00.008-05:00

When listening to music on PC, I use a Chinese mp3 player called TTPlayer. The main feature of TTPlayer is it display lyrics that synced with the music.
Searching for a web equivalent, the closest thing i found is this. But it has issues, for me anyway, i.e. play only one song, no play/pause control, can't scroll back, can't replay, use an unfamiliar animation framework. So I look at the code and come up with my own. The player is based on soundmanager2 demo code. It use jquery and jquery scrollto plugin.

The demo is at: http://wensheng.com/code/sm_lyrics/

The code can be downloaded here.

Happy Girls

2009-07-31T00:12:00.009-05:00

I will be watching Tonight's Super Girls (or Happy Girls, as it's called now). I don't know why I watch it. I never watched American Idols or any other pop contest shows. And I absolutely hate the most populor one of previous super-girls winner - Li Yuchun.

But since I watched this year's happy-girls one night with wife, I got hooked. Maybe it's just the pretty girls I'd like to watch. I told a co-worker the other day I like Happy-girls, she said I'd have become dirty middle-aged man (龌龊中年男人). Is that the reason? Ouch..

I am rooting for Liu Xijun 刘惜君. (big pic)

I also like Tan Lina 谈莉娜, she's very pretty. (pic)
I like Li Xiaoyun 李宵云 too （video), but she's not good-looking. Huang ying's good too, but her voice style gets old after a while. I really don't care for other girls.

Here's a video of Liu from youtube singing "伤痕". In my opinion, it's better than Lin yilian's original.

set up geo dns (geodns) on Fedora using geoipdns pt 2

2009-07-30T23:38:00.000-05:00

Here I describe the steps of set up Geoipdns (a Tinydns fork at does geo dns). First following these steps here to set up Tinydns. Even though Geoipdns is a fork, it doesn't provide the configuration programs such as tinydns-conf that's in Tinydns. So it's much easier to set up Geoipdns after we already have Tinydns set up.

You can read Geoipdns document on http://pub.mud.ro/wiki/Geoipdns.
Here are quick steps:
$yum install inotify-tools-devel
#geoipdns use libinotifytools.

$mkdir vdns
$cd vdns/
$wget http://pub.mud.ro/~cia/files/vdns-src.tgz
$tar xfz vdns-src.tgz
$vi conf-cc
#Add " -include /usr/include/errno.h" to the first line of conf-cc

$LOCAL_CFLAGS="-DUSE_LOCMAPS -DUSE_SFHASH -DUSE_TOUCH_RELOADS -DDEBUG_MODE -DHAVE_MMAP_READAHEAD"
$make
$mkdir /usr/local/apps
$./install
$cp -rp /usr/local/apps/vdns/bin/* /usr/local/bin/
The last step is to allow us access vdnsd vdnsdb without add to PATH.

Now geoipdns is installed. Next we need to configure it so it does geo dns.
$cd /etc/tinydns
$cp run run.tinydns #backup run
$vi run
#inside run, we change /usr/local/bin/tinydns to/usr/local/bin/vdnsd

$cd root
$vi Makefile
#inside Makefile we change tinydns-data to vdnsdb

$make
$svc -t /service/tinydns

Now vdnsd should be running, use "ps -ef" to see that vdnsd is running tinydns is not, but you still should see "supervise tinydns".
Do some dig to make sure everything still works just like Tinydns is running.

Now you need to query the server from at least 2 different locations, otherwise you don't know if geodns works or not.
If you set up dns on local network, such as 192.168.1.0, you can put IP's of local machines in "data" and test geodns from these local machine.
If you have a world facing DNS server and you want to test geodns, you can put the IP's of your home, work, colocated server, or VPS in data, then test dns from those locations.

However if you want to test it fast at where you are, you are also in luck, because there're several online dig sites you can use. I will use 2 such sites as examples:
http://dig.menandmice.com/knowledgehub/tools/dig ip is 207.57.2.84
http://www.subnetonline.com/pages/network-tools/online-dig.php ip is 85.17.250.238
Note these Ip's are as of this writing. If they have changed at the time of your testing, you need to change them too in your data file.

Now the data. Suppose your DNS server name is ns1.myserver.com (replace it with your real dns server name), add these to your /etc/tinydns/root/data file:

%onlinedig1:207.57.2.84:32
%onlinedig2:85.17.250.238:32
.example.com::ns1.myserver.com:259200
+www.example.com:1.1.1.1:3600::onlinedig1
+www.example.com:2.2.2.2:3600::onlinedig2
+www.example.com:3.3.3.3:3600::nomatch

Do a "make" to update the data hash.

Now you can start testing. First do a "dig @ns1.myserver.com www.example.com" locally. It should say 3.3.3.3. Then go to http://dig.menandmice.com/knowledgehub/tools/dig , enter name server "ns1.myserver.com", enter domain name "www.example.com", click "perform query", it should say 1.1.1.1. Now do the same on http://www.subnetonline.com/pages/network-tools/online-dig.php, it should say 2.2.2.2.
If they display correct fake information, your geo dns works.

Next you need to add real IP location data and setup geo dns for your real domains.
I will talk about this in another post.

Why geo dns - setting up geodns on fedora part 1

2009-07-29T23:50:00.001-05:00

First of all, what is geodns? Let's look at an example. The following show the outputs of dig and ping on www.google.com from 3 different geographical locations.
First from Dallas:

270 06:13 PM wang@ns1)dig www.google.com +short
www.l.google.com.
74.125.47.99
74.125.47.103
(truncated)
271 06:13 PM wang@ns1)ping -c 2 www.google.com
PING www.l.google.com (74.125.47.99) 56(84) bytes of data.
64 bytes from yw-in-f99.google.com (74.125.47.99): icmp_seq=1 ttl=54 time=21.2 ms
64 bytes from yw-in-f99.google.com (74.125.47.99): icmp_seq=2 ttl=54 time=21.0 ms

Next from Xi'an, China:

[wang@www ~]$ dig www.google.com +short
www.l.google.com.
64.233.189.104
64.233.189.147
64.233.189.99
[wang@www ~]$ ping -c 2 www.google.com
PING www.l.google.com (64.233.189.147) 56(84) bytes of data.
64 bytes from hk-in-f147.google.com (64.233.189.147): icmp_seq=1 ttl=242 time=45.7 ms
64 bytes from hk-in-f147.google.com (64.233.189.147): icmp_seq=2 ttl=242 time=45.8 ms

Finally from Zhejiang China:

137 03:06 PM wang@cn)dig www.google.com +short
www.l.google.com.
66.249.89.99
66.249.89.104
66.249.89.147
138 03:06 PM wang@cn)ping -c 2 www.google.com
PING www.l.google.com (66.249.89.99) 56(84) bytes of data.
64 bytes from jp-in-f99.google.com (66.249.89.99): icmp_seq=1 ttl=243 time=47.4 ms
64 bytes from jp-in-f99.google.com (66.249.89.99): icmp_seq=2 ttl=243 time=47.4 ms

You can see the IP's for www.google.com from these locations are different. The reason for this is that Google want to send www.google.com visitors to their nearest web servers. Why? you might ask. Because of network latency. Here's output from pinging US google server from China:

[wang@www ~]$ ping 74.125.47.99 -c 2
PING 74.125.47.99 (74.125.47.99) 56(84) bytes of data.
64 bytes from 74.125.47.99: icmp_seq=1 ttl=44 time=258 ms
64 bytes from 74.125.47.99: icmp_seq=2 ttl=44 time=261 ms

So the ping time is >10 times as long as ping time from within US. If Google doesn't have servers in China (or hk, jp, or whatever closer to China), the experience of Chinese www.google.com visitors will be really bad (long response time, slow page load).

The 3 popular DNS software (bind, powerdns, tinydns) all have geo capability, either with patch, backend, or in tinydns case, a fork called geoipdns. I have been using Tinydns for several years and very satisfied with its ease of use and performance. So I stick with Tinydns for my geodns.
The geodns fork of Tinydns is called geoipdns, it's written by Adrian Ilarion Ciobanu.
I will talk about how to set up Geoipdns in the next post.

Set up Tinydns on Fedora

2009-07-29T20:23:00.000-05:00

Installing Tinydns (djbdns) on Fedora consists of 4 steps:
1, Install daemontools
2, Install ucspi-tcp
3, Install djbdns
4, configure tinydns

Step 1: Daemontools
First of all, make sure we have all the packages that's needed:

yum install gcc gcc-c++ make flex bison # etc.

This will install all the compilers, kernel-headers, tools and stuff.
Then

mkdir daemontools
cd daemontools
wget http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
tar xvfz daemontools-0.76.tar.gz
cd admin/daemontools-0.76/
vi compile/conf-cc #add " -include /usr/include/errno.h" at the end of 1st line
./package/install
cd /command
\rm * #get rid of links, we will use actual excutables here
cp -rp /root/daemontools/admin/daemontools/command/* .
vi /etc/inittab #get rid of that svscanboot line because it's not used from fc9
vi /etc/event.d/svscan #create this new file

The content of this file /etc/event.d/svscan is:

start on runlevel [2345]
stop on runlevel [016]
respawn
exec /command/svscanboot

Now daemontools setup is complete. Do a "ps -ef" to see if svscan is running. If not, manually start it, or reboot.

Step 2: ucspi-tcp
This is simple:

mkdir ucspi-tcp
cd ucspi-tcp/
wget http://cr.yp.to/ucspi-tcp/ucspi-tcp-0.88.tar.gz
tar xvfz ucspi-tcp-0.88.tar.gz
cd ucspi-tcp-0.88
vi conf-cc # put " -include /usr/include/errno.h" at the end of 1st line
make
./install

Step 3: djbdns
This is quick too:

mkdir djbdns
cd djbdns/
wget http://cr.yp.to/djbdns/djbdns-1.05.tar.gz
tar xvfz djbdns-1.05.tar.gz
cd djbdns-1.05
vi conf-cc # add " -include /usr/include/errno.h" to the 1st line
make
./install

Step 4: configuration
I will only tinydns here, setting up dnscache is very similiar to setting up tinydns.

useradd –s /bin/false tinydns
useradd –s /bin/false dnslog
tinydns-conf tinydns dnslog /etc/tinydns your_external_ip
ln -s /etc/tinydns /service/
svstat /service/tinydns

That's it, do "svstat /service/tinydns" again, it should show the seconds it's up keep increasing. If not, like for example it stuck at 0 or 1 second, then your setup is not correct.
To debug the problem, do some of these:
Look at /etc/tinydns/log/main/current to see if there's error.
Make sure there's a supervise directory in /etc/tinydns/
Make sure there's No directory under /etc/tinydns/env
remove /service/tinydns link and re-link
remove /etc/tinydns directory and re-create them using tinydns-conf
stop and start the service using "svc" ("svc -d" then "svc_u" or "svc -t" etc.)

Next, just add records to "/etc/tinydns/root/data", and then "make", tinydns should now happily serve any requests from anywhere. If not, use "dig" to debug the problem.

Creating a Fedora 11 image

2009-06-12T11:42:00.004-05:00

Fedora 11 is out. I made a fedora 11 image as my domU. The steps for making a Fedora 11 image is little different than those for Fedora 10 that I have described here. But there are some caveats, all related to RPM.

First you have must rpm version > 4.6, otherwise you have have this MD5 errors (fedora 11 doesn't use md5 checksum anymore).

Second, the new rpm in fedora 11 use LUA, but liblua.so is not installed by yum groupinstall. So inside fedora 11, we have this chicken and egg problem. To install liblua.so from liblua.rpm (and anything else from rpm) we need rpm; but to run rpm, we need liblua.so. To solve this, just copy liblua.so from fedora 10 to fedora 11's /usr/lib when fedora 11's image is mounted in fedora 10.

Other than these, everything's fine with making Fedora 11 image.

Vim takes 9 G memory

2009-05-29T15:12:00.003-05:00

Today I used Vim to edit a big file, about 1.5Gig in size. At the end of editing, Vim occupied 9.1G memory. Fortunately the machine has 64G. I was using visual mode to do column delete. I can't use cut and paste (Unix command line, not windows cut and paste) because I didn't delete whole columns. The whole editing session took more than 1 hour with most time spent waiting for commands to complete.

Chinese language test

2009-05-16T13:41:00.004-05:00

My colleague passed me a Chinese language test. It's an Excel spreadsheet. There are 100 2-choice character questions. I was kinda surprised I only scored 79. After all I'm Chinese and I thought my Chinese was good. But anyway it's a good test. So I made a web version:

http://wensheng.com/code/cntest/

I shuffle the questions so it doesn't come out the same every time. You can also score yourself anytime. (In Excel you have to answer all 100 to get your score.) The test works in Firefox, Opera, Chrome, and Internet Explorer 8. It doesn't work in IE7 and IE6 because they don't' allow setting the NAME attribute of radio button dynamically. There are ways around it but I don't want to waste my time fixing it.

msvc2005 release crash debug OK

2009-04-28T12:35:00.004-05:00

I had a strange problem with Visual C++ 2005 today. I needed to add some feature to a program I developed several years ago. I used VC2005.

During debugging, everything's fine. But when I built a release and tested it, it crashed. So I set a breakpoint before the location where I thought it crashed, and run the debug, this is what I got:

How is that even possible? "0<0" is true?! Apparently the memory is messed up. I don't have time to get to the bottom of this so I just released the debug version.

(update 5/15/09: In case I forget. If I set Property->Configuration->C/C++->Code Generation->Basic Runtime Checks to "Stack Frames (/RTCs)", the program will works fine. From MS (http://msdn.microsoft.com/en-us/library/aa289171.aspx): /RTCs - Stack Frame Run-Time Error Checking
This option does several things that help to protect the stack from corruption.
* Initialize all local variables to non-zero values each time the function is called. This prevents inadvertent use of values on the stack from previous calls.
* Verify the stack pointer to check for corruption, such as that caused by defining a function as __stdcall in one place and __cdecl in another.
* Detect overruns and underruns of local variables. This differs from /GS because it is available only in debug builds, and it detects corruption on either end of a buffer and for all buffers.
)

The adventure of (re)installing linux on Tyan s5150 with rocketraid 2220 and Hitachi Deskstar T7K250

2009-01-20T00:46:00.012-06:00

More than 3 years ago, I built a backup server for others to use. It's set up with a Tyan S5150 motherboard with Pentium 630 (3.0Ghz), Highpoint RocketRaid 2220 (rr2220), 7 SATA drives in a 3U rackmount Antec case. I don't remember how much it costed me, but it's sure more than $2000 (hard disks alone was $1000).

Recently the server was freed up. Even though the hard disks are real cheap now (the cheapest 1T drive is $94 now at newegg, Seagate 7200.11 1.5T drive is selling for $129) I hate to see those hard-drives idle, so I brought it home and intended to build a file server.
5 of the 7 SATA drives are 250GB Hitachi Deskstar T7k250. These drives are from the same batch, all produced in July 2005. The problem with these drives is that they can not be used by any other computers. Either bios doesn't read these disks or the it read as 0MB and can not use them. I also have 3 USB SATA hardware enclosures, none of those disks work with any of the enclosures. These are the only SATA drives I have that have this problem (I have a lot of drives). What's strage is that the same 250G T7K250 drives that produced a month later (August 2005) have no such problem.
So I have to use rr2200 (which is a pci-x card) for these drives, which means I have to use Tyan S5150 board because it has a PCI-X slot. As I try to install the board, I found I have to use a EPS12V power-supply. But all I have is ATX. So i pulled EPS12V PS from the Antec case (I can't use the case, it's like a helicopter when it's turned on). Basically the whole server got re-used except the metal shell.

The server used to have Fedora Core 4 on it. I remembered it took a lot of time for me to install and configure it. But unfortunately I didn't document what I did. Huge mistake!! Now the problem only begins.

First of all, With the raid card rr2220 inserted, and a raid array configured. The server can't even boot to the point it trying to load OS (any OS on any media, hard disk or CD or USB).
It only boot to an OS after I deleted the raid array, leaving the 5 disks unconfigured. Still, I can not boot Centos, Ubuntu installation CD. I can only boot with Fedora 10 CD, and only after adding "edd=off" to boot line.
It take a long time to get to welcome screen, at times it appeared stalled. But tty4 shows it's some disk error, and still trying.
During the configuring the installation, this show up.
You bet it's a BUG! The installation aborted.
This bug is caused by conflicting drivers sata_mv and tg3. You may wonder why a sata driver (sata_mv) conflict with a ethernet card driver (tg3), beats me.

Only after I pulled out the rr2220 can I successfully install fedora 10. So I did the installation. Boot the new OS, blacklisted sata_mv, compiled and installed so-called "open-source" driver from Highpoint. Did a modprobe hptmv6, it shows the driver loaded succesfuly and it saw 5 drives.
But I can not use them. Because somehow they can not be used a individual disks, They can only be used as raid arrays. So I went ahead and configured RAID arrays. But lo and behold, I can NOT boot again. I tried a GRUB CD, it doesn't boot either. I do have spare floppy drive, but I remember I tried that 3 years ago it didn't work so I didn't bother.

So I did some research (i.e. google) and it seems the problem is that rr2220 was trying to be first and only boot device no matter what you told motherboard bios. The problem can only be corrected by change in rr2220 bios. The change requires disabling INT13 and/or EBDA reallocation.
But the Linux bios update utility from Highpoint have no such options, only with their windows version can you make this change. What could I do?

Running out of options, I installed windows xp (after un-insert rr2220 of course), download flash utility and new bios, re-insert rr2220, reboot, run the utility, check "disable INT13" and "disable Allocating EBDA " and "disable stop or error", then flashed the bios.
Yes, I wasted an hour just to install Windows and run Highpoint bios flashing utility.

Now it's Fedora 10 installation all over again. Pulled out raid card, install fedora 10, boot to fedora 10, blacklist "sata_mv", download and built and install "open-source" rr2220 driver, install driver. Shutdown, re-insert Raid card, boot, create raid arrays, continue boot, fdisk, mkfs, mount, install and configure Samba.

Finally my file server is set up. It has 1.2 Tera Byte of storage. Not much, but I got to use the 5 disks that otherwise can't be used. Besides, it has room to expand. RocketRaid 2220 is an 8 port SATA2 card. Tyan motherboard itself has 4 SATA ports. So in theory, at today's 1.5T per drive, I can have 1.5*12 = 18 Tera Byte of storage (at 5M/song, it's 3 million MP3 songs, or at 6G/dvd it's 3000 DVD movies).
When the price on 1.5T drive drops to sub-$50 range, I will get some of those (assuming per GB SSD price is still much higher then).

on GFW of China

2008-12-24T01:58:00.003-06:00

GFW=great fire wall.
On my vacation to China this last summer I tried to access this blog from different locations in China: Xi'an, Beijing, Dalian, Shanghai. Never once did I succeed.

I can read Wikipedia, but I heard from other people it's hit & miss. I had no problem with porn sites though.
I always thought this censoring by Chinese government is really stupid. Today I read a excellent column on NY Times by World-is-flat author Tom Friedman, towards the end of the article, it struck a chord with me:

America still has the right stuff to thrive. We still have the most creative, diverse, innovative culture and open society — in a world where the ability to imagine and generate new ideas with speed and to implement them through global collaboration is the most important competitive advantage. China may have great airports, but last week it went back to censoring The New York Times and other Western news sites. Censorship restricts your people’s imaginations. That’s really, really dumb. And that’s why for all our missteps, the 21st century is still up for grabs.

For the benefit of the China and Chinese people's future, Mr. Hu, Tear down this great fire-wall. (tear down this wall). 胡哥, 这墙拆了得了.

Emacs utf8 and Chinese on Windows

2008-12-21T12:52:00.005-06:00

Download EmacsW32 latest binary (emacs 23 + emacsw32). Then add following to your .emacs

(setq locale-coding-system 'utf-8)
(set-terminal-coding-system 'utf-8)
(set-keyboard-coding-system 'utf-8)
(set-selection-coding-system 'utf-8)
(prefer-coding-system 'utf-8)

All your problems of viewing/editing Chinese on Windows Emacs will go away.

Adding Emacs to right click context menu

2008-12-19T01:49:00.005-06:00

To add Emacs (or any program) to right click context menu for any type of files, open regedit, add a key under HKCR/*/shell, name it whatever you want, I name it "run emacs".

Then under it, create a new key called "command", change it data to your path to emacs, plus a "%1" at the end.

If you want to add Emacs to just a specific type of file, for example, ".org" file for Orgmode, it's a little more work.
Create a new key ".org" under HKCR, give it data value "Orgmode.File". Then create a new key "Orgmode.File" under HKCR. Under it, create key "Shell", under which create key "Emacs", under which create key "command", change its data to emacs path plus "%1".

If you want to add Emacs to directory context menu, create the same keys under HKCR/Folder.

Of course, all these apply to any programs, not just emacs.

how to make fedora 10 image from old version

2008-11-26T12:53:00.009-06:00

Fedora 10 just came out. If you have an old redhat OS (fedora or CentOS) and you want a new fedora 10 image, to be used as a Xen domainU or a customized Amazon EC2 AMI or whatever, here's how to do it:

Step 1: Create a 2Gig sparse file or lvm and mount it to /mnt.
examples:
for sparse file:
# dd if=/dev/zero of=f10.img bs=1024k count=1 seek=2000
# mkfs.ext3 f10.img
# mount -o loop f10.img /mnt
for lvm:
# lvcreate -L2G -n f10 vg0 (assume you have a volume group called vg0)
# mkfs.xfs /dev/vg0/f10
# mount /dev/vg0/f10 /mnt

Step 2: edit /etc/yum.repo.d/fedora.repo
go to /etc/yum.repo.d, create a new directory, then move everything to the new directory. (you will move them back up to /etc/yum.repo.d after creating fedora 10). Create a new file call fedora.repo with content:
-----------------

[fedora]
name=Fedora 10 - i386
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/releases/10/Everything/i386/os/
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-10&arch=i386
enabled=1
gpgcheck=0

-----------------

Step 3: do a "yum groupinstall"
#yum clean all
#yum makecache
# yum --installroot=/mnt -y groupinstall Base
Go have a cup of tea, or read some blogs, or downmod some reddits.
After the installation is complete, to make it actually useable, you need to create these devices:
# MAKEDEV -d /mnt/dev -x console
# MAKEDEV -d /mnt/dev -x null
# MAKEDEV -d /mnt/dev -x zero
And create a root passwd:
# chroot /mnt
# pvconv
# passwd
# exit
Create some files:
# vi /mnt/etc/fstab
# vi /mnt/etc/sysconfig/network
# vi /mnt/etc/sysconfig/network-scripts/ifcfg-eth0
# vi /mnt/etc/resolv.conf
The contents of these files depends on your settings, use the files on your current OS as references.

That's it, you're done. Unmount the image. Now you should be able to boot it. After you boot it(e.g. as a Xen domU), do some configurations. These are what I did:
# chkconfig NetworkManager off
# chkconfig network on
# service NetworkManager stop
# service network start #use network instead of NetworkManager
# chkconfig atd off
# chkconfig bluetooth off
# chkconfig cups off
# chkconfig haldaemon off
# chkconfig ip6tables off
# chkconfig mdmonitor off
# chkconfig netfs off
# chkconfig netfs off
# chkconfig nfslock off
# chkconfig pcscd off
# chkconfig portreserve off
# chkconfig rpcbind off
# chkconfig rpcidmapd off
# chkconfig rpcgssd off #all these services are useless for me
# yum install xfsprogs #my filesystem is xfs
# vi /etc/rc.d/rc.sysinit #comment out a bunch of lines to get rid of false err/warning device-mapper messages during boot-up

Don't forget to delete (in your current OS) fedora.repo in /etc/yum.repo.d and move everthing back in there.

Super Easy Python powered Christmas light controller for $40

2008-11-07T23:57:00.014-06:00

There're lots of ways to control Christmas lights. The easiest (but expensive) is just getting a lightorama. I'm not spending $400 on a controller. So I decided to build my own.

Computer Christmas has many articles showing you how to do this. There's a how-to for a 320 channel controller. All I need is 8 channel on/off (no dimming), so a simple parallel port relay box is perfect for me. This how-to tells you how to building one. But actually you don't need to build a relay box yourself, there are parallel port relay kits sold online for cheap. You just buy the kit and connect to electric box.

So Here's how to build a Python powered Xmas light controllers for $40(more or less).

1, get a parallel port relay kit for about $32, for example: here or here. You can get it assembled or in module form for a few more bucks. I didn't want to spend much time assembling so I got a module.

2, get electric box and outlets from Homedepot or Loews for total $8. Again I'm lazy so I just got a power strip from Walmart for $4, but it only has 7 outlets, I couldn't find one with 8.

3, Wire the kit to your electric box.

4, Connect to computer and fire up Python (see my previous post).
Video of testing session.

Here's source code for the GUI program (in wxpython), most of code were generated by wxglade, I just created the event handlers for the mouse clicks. The speed control (slider) is not implemented yet.

python parallel port on Windows

2008-10-18T01:44:00.003-05:00

Tried different ways of controlling parallel port with Python on Windows. The easiest way is with Inpout32.dll.

Just download the dll, put it in system32 folder. Then from python:

from ctypes import windll
p = windll.inpout32
p.Inp32(0x378) #default 255(all high) on my pc
p.Out32(0x378, 0) #put all low on port 2-9

The address 0x378 might be different on your machine, open System->Hardware->Device Manager->Ports->ECP Printer Port->Properties->Resources, use the first number as your address.

Create QRcode with google chart API

2008-07-18T19:02:00.006-05:00

Google recently add QRCode to its Chart API. This make QRCode generation a breeze. All you need to do is sending a GET request to google.
QR Code is the most popular 2d barcode. I put a page up that make it easy to generate QR Code with Google Chart API:

http://wensheng.com/gqrcode.html

To create a URL, you need to put 'http://' at the beginning. Otherwise code reader will see it as just text. Likewise to create a telephone number, you need to put 'tel:' at the beginning, otherwise it's just code and you can't dial it from your phone.

There're a lot of code readers. I have Kaywa Reader on my cellphone, it works really well.

Can't believe my brain is only 20

2008-06-30T23:33:00.004-05:00

So I took this japanese flash test for brain age, it show my brain age (脳年齢) was 20. I don't know if it helped that I was drinking a beer at the time of testing.

Add: Hmm, Chimps do much better.

Grid/Cloud/Utility computing is expensive!

2008-06-30T12:06:00.008-05:00

Utility computing is hot now. Two major players are Amazon EC2 and Google AppEngine. There are many minor players too. It's also evident that some ISP's are jumping onto the bandwagon.
I did some quick research on several vendors and sumarizd by findings here:

click to enlarge

For me, Google AppEngine makes most sense. You pay nothing for up-to 5 million page-views per month. The only problem, albeit a major one, is that it's only python. It's not a problem for me though since I know python.

Other providers are way too expensive. Look at engineyard.com, $400 per month get you a meager 760Mb ram, only 250GB transfer. Someone must be out of his mind to purchase it.

For those who don't go with AppEngine, I would say it's best to just get a VPS. You can do whatever you want with VPS just like a dedicated server. And it's cheap, you can easily find a 512M ram VPS with plenty of diskspace and bandwidth for less than $50, that's all you pay. (Just make sure your VPS is Xen based, not Virtuozzo/OpenVZ based, as they allow ram bursting, which means you will NOT get your ram when you really need it, as other users on the same host machine as you stole it.)