Checked on BlueHost support site and was informed that they have upgraded PostgreSQL and have recompiled Apache. I suppose I couldn’t ask BlueHost tech support regarding the issue since Subversion is not something included in the hosting package.

Found this thread on Google suggesting to ‘rebuild apt-util’ but without giving any solution, so I decided to roll my sleeves and solve the problem on my own.

Here was my installation hosting environment:
1. Bluehost 64bit box
2. subversion 1.5.5

And here is how I installed subversion in the first place, you may want to check this post (referring as post #1 form now on) first as the following steps are partially in the same context with it.

Failed trying to reconfigure

Tried to install all subversion dependencies downloaded in post #1, it ignores the installation and says the libraries have been installed.
Tried to reconfigure and link, got lots of ‘cannot read symbols‘ error. One of the errors seems to be version related problem which I encountered before while doing apache upgrading.

I guess I may need higher version of apr and apr-util and hope they will still work with subversion 1.5.5, otherwise I’ll have to upgrade my svn on the server as well.

Download apr and apr-util

cd src
wget http://www.apache.org/dist/apr/apr-1.3.5.tar.gz
wget http://www.apache.org/dist/apr/apr-util-1.3.7.tar.gz
tar -xzvf apr-1.3.5.tar.gz
tar -xzvf apr-util-1.3.7.tar.gz

Install apr

cd apr-1.3.5
./configure --enable-shared --prefix=$HOME LDFLAGS="-L/lib64"
make && make install

Install apr-util

cd ../apr-util-1.3.7
./configure --enable-shared --prefix=$HOME --with-apr=$HOME --without-berkeley-db LDFLAGS="-L/lib64"
make && make install

Install neon

Since neon depends on apr and apr-util, I’d better reinstall it as well, tried to install the neon comes with subversion-dep-1.5.5.tar.gz and this time it succeeded, great!

cd ../subversion-1.5.5/neon
EXTRA_CFLAGS="-L/lib64 -fPIC"
CFLAGS="-L/lib64 -fPIC"
./configure --prefix=$HOME --enable-shared LDFLAGS="-L/lib64" --with-libs=$HOME
make && make install

Reinstall Subversion

cd ..
./configure --prefix=$HOME --without-berkeley-db --with-apr=$HOME --with-apr-util=$HOME --with-neon=$HOME -without-apxs --without-apache
make && make install

Something you might want to know if you are going to follow my steps to save your svn server on BlueHost. As the information in this weblog is provided “AS IS” with no warranties : )

1.Check your BlueHost hosting environment (32 bit or 64 bit)
2.Check your subversion version (a new repository schema has been introduced since version 1.4.?, you may want to download all repositories and upgrade them locally)
3.Backup all repositories.
4.If you are not sure about what those steps are all about please do read post #1, don’t replicate them blindly – for your server’s safety, although BlueHost backs up your home directory in daily and weekly intervals.
5.There might be a better solution beyond my knowledge and you can wait for someone to find out – I am just a brave armature :)

1. Install Subversion 1.5.5 on Bluehost Linux 64bit Box
2. Configure svn+ssh on Windows OS

After relentless pounding of my fingers on the keyboard just to feed the same password again and again and again to subversion repository in Mac terminal, I realised that typing password is not as fun as developing finger habits in my beloved Emacs, although they are both repetitive.

I was reluctant to set up password-less svn+ssh access on my new Mac machine. The reason is that the key pair I am using were both generated by Putty, one deployed on Linux server and one for Windows clients. I would rather play Street Fight 4 online for half hour other than reconfigure SSH access across three different operating systems – if it is not absolutely necessary. They both involve the phrase “pain in the ass”, if you know what I mean :)

Good news is that Mac OS comes with a SSH agent:

ssh-agent is a program to hold private keys used for public key authentication (RSA, DSA). The idea is that ssh-agent is started in the beginning of an X-session or a login session, and all other windows or programs are started as clients to the ssh-agent program.

With the ssh agent, we can add the identity once per login. It means ‘add once, use everywhere’! It will be really helpful if you use more than one clients to connect to the same repository server.

Step1:
Use mac os build in ssh-keygen to generate a pair of openssh format keys. Deploy the public key on your server (assume your server uses openssh) and set up the appropriate permissions (see the related post1 to lean more). If you already have a .ppk key, then a conversion from Putty format to openssh would probably be the easiest way to go. It can be done either using PuttyGen in Windows OS or using a Putty Mac port (MacPorts or DawinPorts).

Step2:
Grant the key file minimum permissions. It needs to be:
chmod u=r,ga-rwx

If you are curious about ssh agent, read step3 and step4, otherwise, just skip to step 5.

Step3:
Find out the ssh-agent environment variable values: SSH_AUTH_SOCK and SSH_AGENT_PID. For more info about ssh agent in Mac OS, please read here.

Step4:
Set SSH_AUTH_SOCK and SSH_AGENT_PID environment variables in your subversion client of choice. If you only use command line clients, this step can be skipped.
Update: It turned out the SSH_AGENT_PID value changes form login to login.

/tmp/ssh-XXXXXXXXXX/agent.<ppid>
Unix-domain sockets used to contain the connection to the authentication agent. These sockets should only be readable by the owner. The sockets should get automatically removed when the agent exits.

In this case, the clients need to talk to the OS to find out the value. Since ssh-agent servers as a lower level service in the OS, it seems to be impossible to set up those values for clients manually and I assume it is not necessary.

Setp5:
Use ssh-add command to add the key to ssh authentication agent.
ssh-add yourkey

Find the following three files in the ~/.ssh/ directory (create if not exist).

 ~/.ssh/identity
    Contains the protocol version 1 RSA authentication identity of the user.
 ~/.ssh/id_dsa
    Contains the protocol version 2 DSA authentication identity of the user.
 ~/.ssh/id_rsa
    Contains the protocol version 2 RSA authentication identity of the user.

Append your private key in the right file (depends on the key type) and change the permission to owner read only, the ssh-agent will automatically read those files and add identities to the agent every time a ssh request occurs.

Now it’s really password-less!

Look how happy they are! This must be enjoyable!

However, one probably should get himself fully covered when it comes to serious business – even it means sacrifice of agility and strength.

This also applies to coding. When one’s code needs to interact with other people’s code or intends to serve as a service, the less exposure, the better. As one of the three pillars of OOP, encapsulation is defined as below:

In computer science, Encapsulation is the hiding of the internal mechanisms and data structures of a software component behind a defined interface, in such a way that users of the component (other pieces of software) only need to know what the component does, and cannot make themselves dependent on the details of how it does it. The purpose is to achieve potential for change: the internal mechanisms of the component can be improved without impact on other components, or the component can be replaced with a different one that supports the same public interface.

Encapsulation also protects the integrity of the component, by preventing users from setting the internal data of the component into an invalid or inconsistent state.

Another benefit of encapsulation is that it reduces system complexity and thus increases robustness, by limiting the interdependencies between software components.

Two weeks ago, I was asked to integrate a third-party online game into our system at work, here is the API I received.

For registration:

http://domain/service/commandHandler.php?Command=doCommand&
RequestID=b5e555ec-7788-486e-8426-37d82a97a287&Code=234ab13&Name=XXXXXXXXX&
Username= XXXXXXXXX&FirstName=NTVA&LastName=NTVA&Password=XXXXXXXX&
Email=XXXXXXXXX&CountryID=2&State=NTVA&PostCode=NTVA&City=NTVA&Address=NTVA&
PhoneNumber=NTVA&GenderID=1&DateOfBirth=20081111&IsAffiliate=FalseDiscountCode=&
BannerID=XXXXXXXXX&IBAN=&BankAccountOwnerName=&SwiftCode=&BankName=

In the manual, it says only parameters marked as XXXXXXX are mandatory, others are all optional. sounds simple!

So here is what I did:

1. Send request like this: http://domain/service/commandHandler.php?Command=doCommand&RequestID=[Guid]&
RequestCode=RandamCode&Name=XXXXXXXXX&Username=XXXXXXXXX&
Password=XXXXXXXX&Email=XXXXXXXXX&
BannerID=XXXXXXXXX.

Response: Error in FirstName, LastName.... blar blar blar.

2. OK I guess it expects every parameter to be passed in even they are optional. So sent request like the below – I would not say it’s dumb:

http://domain/service/commandHandler.php?Command=doCommand&
RequestID=[GUID]&Code=RandomCode&Name=XXXXXXXXX&
Username= XXXXXXXXX&FirstName= &LastName=&Password=XXXXXXXX&Email=XXXXXXXXX&
CountryID=&State=&PostCode=&City=&Address=&PhoneNumber=&GenderID=1&
DateOfBirth=20081111&IsAffiliate=False&DiscountCode=&BannerID=XXXXXXXXX&IBAN=&
BankAccountOwnerName=&SwiftCode=&BankName=

Response: Error in FirstName, LastName... blar blar blar. This time all ‘parameter=’ didn’t complain, but all ‘parameter=NTVA’ failed.

3. After 10 more times try, I finally figured out:

• The RequestCode need to be exact the same value as printed in the API example although there isn’t a single word of explanation about the mythical code.
• All Parameters which have value NTVA in the printed API need to be set to something, if the values are available, you still need to set them to NTVA, NTVA means NULL or empty string.
• The countryCode and GenderID must be numbers and again there is not a single sentence about those being mentioned in the manual. Who cares, let’s pass in the countryCode as 1 (I assure there will be at least one country) and GenderID=1
• The DateOfBirth, you need to pass in a string in ‘yyyymmdd’ format, let’s pass 19800101 to make sure every user I registered can access whatever porn you are going to show…
• Hooray, registration succeeded! But I got no userID?
• Let me give those guys a call to check if I did it right. Then I was astounded by what I heard: the RequestID is not suppose to be unique, just as the magical RequestCode, you need to send the same value (exact as printed in the API) every tim. UserID is not returned with registration command because they can’t see why we need it, but all other commands would return userID so you can get it later if you want – So my assumption is that they don’t know how to return an ID for newly inserted row.

After all these steps, my first thought was to send the API to The Daily WTF.

How hard it could be to check whether a parameter is present in a HTTP request? Is it really impossible to determine if the value of the parameter is empty so I need to pass some stupid string like NTVA?

If things have been done down to such a level, I would not expect them to know encapsulation. (Notice all bold emphasised parts have been violated). But then how could you expect me to send user details into your database? Who knows what the data would end up to? Did you notice there are bank details to be expected?

The contact from the API provider is very enthusiastic about the project. Out of courtesy, I didn’t tell him that the API sucks.

From a tech perspective, I wouldn’t had gave it a go anyway, although the integration was cancelled due to other reasons.

Sometimes, actually most times, you need to get yourself well equipped before going out hunting for business. No one would choose vulnerable partner or hire naked soldiers. In a similar vain, skilled coders would not use overly exposed APIs, which will set everyone in trouble.

Show your robustness, not vulnerabilities.

Refactor early, refactor often, refactor thoroughly.
Test early, test often, test thoroughly.

For 9-5 programmers (The programmers who come to work at 9am, shut down their computers at 5pm and go home with coding-proof bubbles around them.), my points would seem to be over killing. The overwhelming effort needed to practice the principles is enormous, after all, not every coder would like to put these things as his/her epitaph.

For well versed desktop application developers, these three words, early, often and thorough, might not be innovative, as we all have read The Pragmatic programmer and Steve McConnell’s Code Complete. However, from a web developer’s point of view, they are quite interesting.

A typical web application development circle includes requirement, specification (optional), design, coding, testing and delivery. We apply different three word formulas to each stage.

Requirements and Specification

Gather early, gather often, gather thoroughly.

Clients, especially shareholders, usually are non tech savvy type of people, they come to you: ‘We want a website, we want it next month and we like Facebook pop-up bubbles!’. I am so glad you said ‘pop-up bubbles’!

Gathering requirements should be absolutely essential and it is the first deference of disappointment. Some small agencies, including the ones I worked with, tend to pitch their ‘free designs’ in the first meeting, do the hard sells, tell the client nothing could not be done – even if the client want the web to trap real fish.

Instead of trapping clients with free design and ocean deep low price and then rip them off when it comes to tiny changes and maintenance, a well composed contract that is based on enough requirements would be more appropriate. For web applications, personally I don’t think specifications are usually necessary and somehow they produce more disappointment than satisfactions. Reason? it’s not easy to get the specific level right. How specific is specific enough? ‘Membership management’ equals nothing when it comes to design and coding, but ‘The site needs three roles, respectively gusts, registered users and vip users’ will guarantee future changes to be incurred. For a requirements-change-everyday application, you might want to spend more time to gather the right requirements to ensure the solidity of design – both visual and coding.

Design and Coding

Refactor early, refactor often, refactor thoroughly

Not to repeat many great talks (PDF) and discussions on this topic, I will only say one point that I found very interesting yet paradoxical. DRY is the first thing I learned from my C++ class and I believed in it for many years. But for web applications, it becomes quite disadvantageous sometimes: If the page only needs to be alive for three days, code generator and wizard are your best bets. Duplication? don’t worry about that, you would not have a chance to modify them before they die out from Google.

Testing and Delivery

Test early, test often, test thoroughly.

User involve early, user involve often, user involve thoroughly.

Test is a big topic and will not be discussed here.

User involving might be a pleasant way to work with, it is definitely the most efficient way of avoiding disappointment and disagreement. Let the end users involve from the beginning to the end. Give them a prototype of interface to play with, ask for feedback after each stage/component. There is a thing called one-mind, but you and your client don’t usually have it. Sometimes user doesn’t know how much effect would be involved for a small change of his mind; sometimes a big change the client is afraid of telling you may only requires one line code change. Communicating with user can synchronize user’s expectation and developers’ decision, reducing the unhappiness caused by the parts which developers put a huge amount of effort in but lives out of user’s expectations and prevent client changing mind like a kid – which is far more efficient than specification and user would appreciate it.

Ok It’s a lie but seriously, I am not a big fan of mice and GUI, they usually slow me down and distract me from concentrating. Besides, some really bad designed HCI, such as Windows Vista UAC, can easily push my buttons in all circumstances. People from Microsoft said

“Disabling UAC risks your computer for allowing some worms to execute commands secretly without a prompt confirmation from end user.”

and I up voted this as the correct answer:

“Enabling UAC risks me throwing my computer out the window from frustration with a poorly designed and horribly executed security model.”

Forgive me being loquacious, the point here is I don’t want to use TortoiseSVN – I 100% agree with you that it’s a fantastic software, I do! I just need to get svn+ssh work with my Emacs shell mode.

Let’s get started.

1. Download Putty, if you are like me always afraid of installing untrusted software from the Internet because you believe that they are mostly badly written piece of crap that they would mess your already rubbished Windows OS completely then you can be greatly relieved here, I promise.

2. Start Puttygen.exe and generate a pair of keys, save the private key WITHOUT passcode,

3. Copy the public key, yes COPY not SAVE, paste/append it to $HOME/.ssh/authorized_keys, create the file if it doesn’t exist.

4. After modified the authorized_keys file, remember to:

chmod 700 authorized_keys
chmod 600 .ssh

This is important as you may get connection errors if the file is group writable.

5. Next, test the connection with putty.exe or plink.exe, (side note:TortoiseSVN has a plink.exe windows implementation which would not pop up messages.)
plink.exe -i theprivatekey.ppk username@hostname

6. Modify your subversion configuration file. It’s in
c:\Users\yourname\AppData\Roaming\Subversion in Windows Vista
add the following line to your [tunnels] section:
ssh=x:/path/plink.exe -i x:/path/privatekey.ppk

Now you should be able to access your svn+ssh repository without typing password, in Microsoft Windows! Vista!

Some extra info for folks who love GUI – for TortoiseSVN users, you need to generate the OpenSSH private key from server, then load it with puttygen.exe and get the public key – it’s due to the fact that there are differences between private key implementations in putty and OpenSSH, otherwise you will get the lovely “server refused our key” message!

There are already several tutorials in the Internet, why I am writing this again? Well, I just wanted to prove that I too, can write blog! Among all those tutorials, none of them worked for me, I guess either I was trying to install the newest version of SVN or the BlueHost hosting environment has changed, or they just hate me. So I am constructing this and hope to contribute something I learned along the frustrating process of setting up SVN on 64bit shared hosting so you don’t need to smash your keyboard.

Once you enabled SSH on BlueHost control panel with a photo ID, it’s time to set up Subversion.

1. Open a terminal, type in:
ssh username@yourdomain.com
and hit enter, then input your hosting password.

2. It’s better to create a separate folder for all the operations, in case our home directories got messed up by any mistakes.

mkdir src
cd src

3. Download Subversion and dependencies:

wget http://subversion.tigris.org/downloads/subversion-1.5.5.tar.gz
wget http://subversion.tigris.org/downloads/subversion-deps-1.5.5.tar.gz
tar -xzvf subversion-1.5.5.tar.gz
tar -xzvf subversion-deps-1.5.5.tar.gz
cd subversion-1.5.5

4. Install apr and apr-util, notice the var LDFLAGS , it’s 64 bit, it’s m-A-g-I-c.

cd apr
./configure --enable-shared --prefix=$HOME LDFLAGS="-L/lib64"
make && make install

cd ../apr-util
./configure --enable-shared --prefix=$HOME
--with-apr=$HOME --without-berkeley-db LDFLAGS="-L/lib64"
make && make install

5. Install neon, I had been stuck here for an hour because of the 64 bit mode problem, remember to explicitly enable compilation of shared libraries and prefix the files into home directory.

cd ../neon
EXTRA_CFLAGS="-L/lib64 -fPIC"
CFLAGS="-L/lib64 -fPIC"
./configure --prefix=$HOME --enable-shared LDFLAGS="-L/lib64" --with-libs=$HOME
make && make install

6. Install Subversion, pass --without-apxs and --without-apache to prevent svn installing Apach modules, also you need to explicitly specify the dependencies.

cd ../
./configure --prefix=$HOME --without-berkeley-db --with-apr=$HOME --with-apr-util=$HOME --with-neon=$HOME
-without-apxs --without-apache
make && make install

7. Modify .bash_profile and .bashrc file

nano -w .bash_profile (pico is always my favourite.)
add $HOME/system/bin to PATH variable, it should look like this:
PATH=$PATH:$HOME/bin:$HOME/system/bin
Also add this line to your .bashrc file, after the ‘fi‘
PATH=$PATH:$HOME/bin

8. Logout the current session and log on again, let’s try:
mkdir ~/repos
cd repos
svnadmin create topsecrets007plus

now you can access your repository like this:
svn+ssh://username@host/home/username/repos/topsecrets007plus/
just replace username with your 8-character user name given by BlueHost.