yahoo2mbox.pl error: Unexpected title page format

The problem

c:\emails\yahoo2mbox-0.24>perl yahoo2mbox.pl --user=pooja13pandey --pass=<password>
1 --delay=5 desimasala
Logging in as pooja13pandey... ok.
Getting number of messages in group desimasala...
Unexpected title page format (DesiMasala).

The solution

This simply means that the user with which we are trying to archive the yahoo group's messages locally, is not part of the yahoo group yet. So now I make pooja13pandey user a part of the desimasala yahoo group.

c:\emails\yahoo2mbox-0.24>perl yahoo2mbox.pl --user=pooja13pandey --pass=<password>
1 --delay=5 desimasala
Logging in as pooja13pandey... ok.
Getting number of messages in group desimasala...
Retrieving messages 1..12389:
Endless redirect loop detected while retrieving message 1.
This error is often due to using incorrect case in the group name.
Saved 0 message(s) in desimasala.

Hmm, so there is still some problem. Although, there is a yahoo group called Desimasala, Im not able to login to it through yahoo2mbox.pl script for some reason. On closer look, one can see that the group name is case-sensitive. It needs to be DesiMasala, instead of Desimasala or desiMasala or desimasala.

So, lets make that correction and try again.

c:\emails\yahoo2mbox-0.24>perl yahoo2mbox.pl --user=pooja13pandey --pass=<password>
1 --delay=5 DesiMasala
Will resume at message 1
Logging in as pooja13pandey... ok.
Getting number of messages in group DesiMasala...
Retrieving messages 1..12389:

.. And, Bingo! It worked.

quick script to extract email ids from a detailed mail header

Since my creative juices are flowing so much today, I thought of posting the simple, yet effective awk script to extract sender's email ids from a detailed mail message (even though if the mail id be ecrypted).

A word of caution is that the file being processed should have a line with "EOF" value (without strings) ONLY at the end of file, otherwise the awk script will hang.

BEGIN { print FILENAME | "wc -l |cut -f1 -d' '" ; }
/^X-Sender:/ || /^From / { #print NR, $0;
           #
           # get the real userid from where the email came
           #
           m=split($0,b,"@");
           from=b[1];
           #print from;
           x=split(from,c," ");
           realfrom=c[x];
           gsub("<","",realfrom);
           #print realfrom;
           #
           # get the domain name of the smtp server now
           #
           while ($0 !~ /HELO/ && $0 != "EOF") getline;
           if ($0 == "EOF") exit;
           domain=$5; gsub(")","",domain); n=split(domain,a,".")
           #print n, domain;
           realdomain=a[n-1]"."a[n];
           if (n>1) print realfrom"@"realdomain;
           next;
         }

How to uninstall a CPAN module ?

CPAN module does not have the option of un-installing a perl module as of now. This can be especially frustrating for newbies (as it was for me). I struggled quite a bit on google and found this solution. For this, you need the CPANPLUS module, which needs to be first installed through CPAN.

C:\Users\Pooja Verma>perl -MCPAN -e shell

cpan> install CPANPLUS

cpan> exit

Unlike CPAN, you cannot invoke CPANPLUS by typing cpanplus on command prompt, even if its installed.

C:\Users\Pooja Verma>cpanplus
'cpanplus' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\Pooja Verma>perl -MCPANPLUS -e shell
CPANPLUS::Shell::Default -- CPAN exploration and module installation (v0.84)
*** Please report bugs to <bug-cpanplus@rt.cpan.org>.
*** Using CPANPLUS::Backend v0.84.  ReadLine support disabled.

*** Type 'p' now to show start up log

Did you know...
    The documentation in CPANPLUS::Module and CPANPLUS::Backend is very useful
CPAN Terminal> help

[General]
    h | ?                  # display help
    q                      # exit
    v                      # version information
[Search]
    a AUTHOR ...           # search by author(s)
    m MODULE ...           # search by module(s)
    f MODULE ...           # list all releases of a module
    o [ MODULE ... ]       # list installed module(s) that aren't up to date
    w                      # display the result of your last search again
[Operations]
    i MODULE | NUMBER ...  # install module(s), by name or by search number
    i URI | ...            # install module(s), by URI (ie http://foo.com/X.tgz)
    t MODULE | NUMBER ...  # test module(s), by name or by search number
    u MODULE | NUMBER ...  # uninstall module(s), by name or by search number
    d MODULE | NUMBER ...  # download module(s)
    l MODULE | NUMBER ...  # display detailed information about module(s)
    r MODULE | NUMBER ...  # display README files of module(s)
    c MODULE | NUMBER ...  # check for module report(s) from cpan-testers
    z MODULE | NUMBER ...  # extract module(s) and open command prompt in it
[Local Administration]
    b                      # write a bundle file for your configuration
    s program [OPT VALUE]  # set program locations for this session
    s conf    [OPT VALUE]  # set config options for this session
    s mirrors              # show currently selected mirrors

Now you can uninstall the module that you want to. If needed, you can use the --verbose and --force option with it.

CPAN Terminal> u MIME::Head --force --verbose

How to scrape emails for online marketing? -- for free

Prologue - Is harmless marketing always Spam?

Ok, so I had started a website and wanted to promote it. At some point, you might be in the same boat too.

Unfortunately, I did not have a while lot of contacts for sending emails to. Well, the website was not really spammy (it was actually a community related website for open source audio resources), and I was not going to solicit for money, so I did not have a lot of compunctions in telling people about it. If they wanted to listen to some audios, they could stick around, otherwise they could do anything else they pleased to.

I knew some owners of yahoo groups that had heavy audience. I thought maybe it was worth a shot explaining about the website and asking if they might be willing to share emails for targeted promotion. The results were quite disappointing, and somewhat expected.

After talking to a few of them, my general observation seemed that most group owners think that as long as they dont give out member emails to others, those emails would always be protected from spam. They tend to think of other groups/email distributions as spam! (as if those people WONT get any other SPAM in the future at all, huh?)

Whatever, I thought. As long as my intentions were pure, I did not need to have second thoughts.

So what are my options?

Hmph! What a Bummer. How long could I depend on google to make the website more popular? It was basically a deadlock.

So one fine day, I started thinking about how ELSE I could approach this problem. For example, could I not use a unix utility like wget or perl to simulate web clicks or use some robot utility to do clicks, web navigation and download information?

Then another thought was -- could I not get emailids from specific yahoo groups or gmail groups that I was subscribed to, and whose target audience might be interested in knowing about a website? After further digging on google, I found that there IS a perl module called WWW::Yahoo::Groups that can programmatically fetch group messages to local PC drive.

WWW::Yahoo::Groups -- Easier said than done..

I tried installing WWW::Yahoo::Groups on a unix box I had access to, but it seems that its easier said than done. It had a ton of dependent modules that wouln't install properly (even with the force option) -- I specifically had a lot of trouble with the Crypt::SSLeay module.

So what next?

This failure forced me to look for other options. And boy, there are quite a few of them available. fetchyahoo is one, then you have yahoo2mbox.

Now, An interesting point about yahoo2mbox is that there are a couple places where you can find it. Apparently, TT solutions has their own version at http://www.tt-solutions.com/en/Products/yahoo2mbox, while there is a debian version available at http://packages.debian.org/lenny/yahoo2mbox. I had problems with the debian version on Ubuntu and tried using the one from tt-solutions, which worked better.

So while fetchyahoo can fetch the contents of a single yahoo user, yahoo2mbox can do the same for a yahoogroup.

Fine, I thought, lets go for the jugular and get yahoo2mbox to work. The results were inconsistent: if you are a moderator of the group or if the group has really loose security setting for not masking email ids, only then can you see the email ids in the downloaded messages. Also, you get weird messages/errors if the group home page has photo and text (got some tag related errors with a yahoo group FHRS_USA).

Email masking on Yahoogroups and Google groups..

Anyway, I realized that both yahoogroups and google groups had elaborate email masking on. True, without it, the online world would NOT be a safe place to be in. Email harvesters would retrieve emails left, right and center and make big bucks.

This is an example from yahoo groups content fetched through yahoo2mbox:

...

...

From devendra999@8_ZAVFsOGj8FjcpG8_oA372HgMSEAl6ol8NKeUSx7VKOtZW8QZCRKJGqiqzqMU-GwjIpYZBzGXe2JmZrjCeniA.yahoo.invalid Sat Aug 05 14:41:35 2000
Return-Path: <devendra999@8_ZAVFsOGj8FjcpG8_oA372HgMSEAl6ol8NKeUSx7VKOtZW8QZCRKJGqiqzqMU-GwjIpYZBzGXe2JmZrjCeniA.yahoo.invalid>
Received: (qmail 6062 invoked from network); 5 Aug 2000 21:41:35 -0000

...

...

This is an example of detailed header fetched using a similar utility called gmail2mbox.pl:

Received: by 10.90.81.11 with SMTP id e11mr6036070agb.27.1237961013457;
        Tue, 24 Mar 2009 23:03:33 -0700 (PDT)
Return-Path: <raviraj.pe...@gmail.com>
Received: from mail-gx0-f164.google.com (mail-gx0-f164.google.com [209.85.217.164])
        by gmr-mx.google.com with ESMTP id 15si1362316gxk.4.2009.03.24.23.03.32;
        Tue, 24 Mar 2009 23:03:32 -0700 (PDT)

Again, what are my options?

So a couple days into the quest and no clear solution yet. Water, water everywhere and not a drop to drink. What an irony.

With these thoughts in mind, I was looking at individual emails (of the yahoo group that I was interested in) in my Yahoo! inbox. Quite absent mindedly, I opened one of the messages a view source and realized that it did not have email masking turned on. In fact, I realized, it could be on, because what would the reply-to address be then? In other words, the From Email id HAD to be available in the mail header of yahoo group email in my inbox.

Aha!

So then, the solution was plain and simple. I would extract the individual email messages for a yahoo group and then extract the email ids from there. Theoritically, it should work. In practice too, it did. Here is an example:

From v_ramesh00001@yahoo.com Thu Feb 10 16:59:16 2005
Return-Path: <v_ramesh00001@yahoo.com>
X-Sender: v_ramesh00001@yahoo.com
X-Apparently-To: sewausa_atl@yahoogroups.com
Received: (qmail 45652 invoked from network); 11 Feb 2005 00:59:15 -0000

So this meant that one would just need to be part of email distribution of a particular yahoo group that one is interested in. It is easier to extract emails if all yahoo group related emails are filtered into a specific folder (then you can utilize --folder option of fetchyahoo).

The quirks of making 'fetchyahoo' work..

For using perl, there are a couple options available, but the best option, in my humble opinion, seems to be Active Perl on windows, available at http://www.activestate.com/activeperl/ . The "other" options on windows are Cygwin or Virtualbox to simulate a unix environment in windows.

If you want to use Virtualbox, the easiest option is to install Ubuntu. You can find a lot of Virtualbox How to articles here. Another option is http://wubi-installer.org. Look at http://www.technobuzz.net/how-to-install-ubuntu-in-windows-with-wubi/

While Cygwin mostly works for simple unix stuff, I found that most of the perl package dependencies do not work out and you end up getting frustrated (I found that Crypt::SSLeay module had problems).

Installing ActivePerl and required modules for fetchyahoo

So activeperl .msi is installed, the perl executable is automatically aded to the windows PATH:

c:\emails\fetchyahoo-2.13.3>perl -version

This is perl, v5.10.0 built for MSWin32-x86-multi-thread
(with 5 registered patches, see perl -V for more detail)
..
..

Also, we will assume that the latest version of fetchyahoo is downloaded and extracted to c:\emails\fetchyahoo-2.13.3 folder:

c:\emails\fetchyahoo-2.13.3>dir
 Volume in drive C has no label.
 Volume Serial Number is DA07-A231

 Directory of c:\emails\fetchyahoo-2.13.3

03/27/2009  09:48 PM    <DIR>          .
03/27/2009  09:48 PM    <DIR>          ..
03/09/2009  12:09 PM            15,182 ChangeLog
03/09/2009  12:09 PM            17,992 COPYING
03/09/2009  12:09 PM             2,747 Credits
03/09/2009  12:09 PM           107,289 fetchyahoo
03/09/2009  12:09 PM             5,359 fetchyahoo.1
03/09/2009  12:09 PM             2,287 fetchyahoo.spec
03/09/2009  12:09 PM             4,907 fetchyahoorc
03/09/2009  12:09 PM             6,314 index.html
03/09/2009  12:09 PM            19,380 INSTALL
03/09/2009  12:09 PM               966 TODO
              10 File(s)        182,423 bytes
               2 Dir(s)  147,889,139,712 bytes free

You should now try to run it. it gave me this error initially:

c:\emails\fetchyahoo-2.13.3>perl fetchyahoo
Can't locate MIME/Head.pm in @INC (@INC contains: C:/Perl/site/lib C:/Perl/lib .
) at fetchyahoo line 59.
BEGIN failed--compilation aborted at fetchyahoo line 59.

Basically, it needs the MIME::Head module installed. For this, we will use CPAN module. You can also use the CPANPLUS module, which is more advanced and has the option of un-installing PERL modules too. CPAN module cannot un-install modules.

This is how you invoke CPAN (you can also just type c:/> cpan). Note that the overwriting the lockfile message might come if a previous session did not terminate properly:

c:\emails\fetchyahoo-2.13.3>perl -MCPAN -e shell

There seems to be running another CPAN process (pid 5220).  Contacting...
Other job not responding. Shall I overwrite the lockfile 'C:\Perl\cpan\.lock'? (
Y/n) [y]

cpan shell -- CPAN exploration and modules installation (v1.9205)
ReadLine support enabled

cpan> install MIME::Head
Going to read C:\Perl\cpan\Metadata
  Database was generated on Thu, 26 Mar 2009 10:26:54 GMT
Running install for module 'MIME::Head'
Running make for D/DO/DONEILL/MIME-tools-5.427.tar.gz
Fetching with LWP:
  http://ppm.activestate.com/CPAN/authors/id/D/DO/DONEILL/MIME-tools-5.427.tar.g
z
Fetching with LWP:
  http://ppm.activestate.com/CPAN/authors/id/D/DO/DONEILL/CHECKSUMS
Checksum for C:\Perl\cpan\sources\authors\id\D\DO\DONEILL\MIME-tools-5.427.tar.g
z ok
Scanning cache C:\Perl/cpan/build for sizes
DONE
MIME-tools-5.427/
MIME-tools-5.427/testin/
MIME-tools-5.427/testin/multi-simple.msg
MIME-tools-5.427/testin/andreas-1296.uu
MIME-tools-5.427/testin/ak-0696.msg
MIME-tools-5.427/testin/short.txt
MIME-tools-5.427/testin/words.txt
..
..
..

  CPAN.pm: Going to build D/DO/DONEILL/MIME-tools-5.427.tar.gz

*** Module::AutoInstall version 1.03
*** Checking for Perl dependencies...
[Core Features]
- Test::More     ...loaded. (0.72)
- Mail::Header   ...missing. (would need 1.01)
- Mail::Internet ...missing. (would need 1.0203)
- Mail::Field    ...missing. (would need 1.05)
- MIME::Base64   ...loaded. (3.07_01 >= 2.2)
- IO::File       ...loaded. (1.14 >= 1.13)
- IO::Handle     ...loaded. (1.27)
- IO::Stringy    ...missing. (would need 2.11)
- File::Spec     ...loaded. (3.2501 >= 0.6)
- File::Path     ...loaded. (2.04 >= 1)
- File::Temp     ...loaded. (0.18 >= 0.18)
==> Auto-install the 4 mandatory module(s) from CPAN? [y]
...
...
Appending installation info to C:\Perl\lib/perllocal.pod
  GBARR/TimeDate-1.16.tar.gz
  nmake install  -- OK
Running install for module 'Test::Pod'
Running make for P/PE/PETDANCE/Test-Pod-1.26.tar.gz
Fetching with LWP:
  http://ppm.activestate.com/CPAN/authors/id/P/PE/PETDANCE/Test-Pod-1.26.tar.gz
Fetching with LWP:
  http://ppm.activestate.com/CPAN/authors/id/P/PE/PETDANCE/CHECKSUMS
Checksum for C:\Perl\cpan\sources\authors\id\P\PE\PETDANCE\Test-Pod-1.26.tar.gz
ok
..
..
t/require.....ok
t/send........ok
All tests successful.
Files=8, Tests=127,  2 wallclock secs ( 0.00 cusr +  0.00 csys =  0.00 CPU)
  MARKOV/MailTools-2.04.tar.gz
  nmake test -- OK
Running make install
Prepending C:\Perl\cpan\build\MailTools-2.04-fCc9N7/blib/arch C:\Perl\cpan\build
\MailTools-2.04-fCc9N7/blib/lib to PERL5LIB for 'install'
...
...
Module 'MIME::Head' installed successfully
No errors installing all modules

Interestingly, you can also check this using Perl Package Manager GUI Utility (invoked by typing ppm on the windows command prompt). It just takes lot of time to load the GUI's data:



Now, let us check if it works:

c:\emails\fetchyahoo-2.13.3>perl fetchyahoo --nodownload
No username specified.
Please enter your Yahoo! username: pooja33pandey
Please enter your Yahoo! password:
No mailbox or mailspool specified.
Please enter the path to and name of your mail spool or mailbox (eg /var/spool/m
ail/username): pooja33pandey.mbox
Logging in securely via SSL as poojagverma on Fri Mar 27 22:29:50 2009
Failed: Invalid ID or password entered (username: pooja33pandey )

If you are running Vista, you might see this infamous pop-up window, which you will need to unblock:

All right! So it works. Now, lets try a more comprehensive example:

c:\emails\fetchyahoo-2.13.3>perl fetchyahoo --onlylistmessages --username=pooja1
3pandey --password=<pwd> --spoolfile=pooja.mbox --logout
Use of uninitialized value $ENV{"HOME"} in concatenation (.) or string at fetchy
ahoo line 1992.
Logging in securely via SSL as pooja13pandey on Fri Mar 27 22:45:49 2009
Country Code 'in' not found. We will try the translation for 'us'.
Country code : in       FetchYahoo! Version: 2.13.3
Successfully logged in as pooja13pandey.
Marking messages read on the server

Fetching mail from folder: Inbox
Getting Message ID(s) for message(s) 1 - 25.
1. new "Public Records " - Locate anyone. Search public records. 7:46 AM 6KB
2. new "Pooja Pandey <p" - online skype number (678) 534-2725 2:47 AM 3KB
3. old "Nimesh Bhuva <b" - Re: [GHPCSB_MCA_2k] Re: Happy Holi 27/3/09 35KB
4. old "Nimesh Bhuva <b" - Re: [GHPCSB_MCA_2k] Happy Holi 27/3/09 32KB
5. old "Birthday Remind" - First Reminder for Vibha Deshmukh's Birthday 26/3/09
4KB
6. old "Sharma, Ashish " - RE: [LIKELY JUNK]RE: [LIKELY JUNK]Re: So it 25/3/09 6
0KB
....
....
Got 90 Message IDs
Not downloading messages
Messages have not been deleted.
Logged out.

Note that fetchyahoo limits the messages fetched to 90 by default, because there is a download limit of 65mb per hour per user per IP address that is set by yahoo. You can use --safedownload option to give a gap of 5-10 seconds between each message fetch. This way, you can run a single command for a long time, without hitting the yahoo imposed download limit (per user, per IP).

Note that once you download the messages locally, they will be marked as read. If you want to terminate the download in between, you can do so and resume it later with the --newonly and --msgidarchivefile option. By defeault, the messages are appended to the archive/spool file:

D:\emails\fetchyahoo-2.13.3>perl fetchyahoo --folder=<foldername>  \
--username=<username> --password=<password>                        \
--safedownload  --spoolfile=<foldername>.mbox                      \
--msgidarchivefile=<foldername>_msgids  --newonly

Conclusion: The strategy in a nutshell

So there you have it. A simple mechanism to get targeted email ids for making your online marketing campaign successful:

1) Identify the Yahoo Group that you are interested in. This is a strategic decision. You want to limit your focus to people who would be interested in your idea. The demographics are important for high return on interest.

2) Become a member of the group and subscribe to individual emails.

3) Setup a filter to direct all Group emails to a specific folder. Free Yahoo account allows for 100 such filters now. Make sure the traffic is flowing in.

4) Sit on your ass for 6 months to 1 year to allow of significant volume of emails. If it is high activity/volume group, then your wait time would be lesser.

5) Fetch the Yahoo folder contents to your local PC. Now you are sitting on the goldmine.

6) Filter out the email ids using simple shell script provided here. Feel free to extent it to your needs. Always manually check the email ids retrieved.

7) Last, but not the least, input the contacts gathered to your mail broadcast software and reap the benefits by inviting them to your newsletter/broadcast.

Remember, the golden rule of thumb to retain the interest of your audience -- Do not send too many similar mails in too short a period of time. Start very moderately and hope that most of them would join your newsletter.

Conclusion - With great power comes great responsbility..

Well, I hope that this article was helpful to you, if your intentions are true and pure. I DO NOT support mis-use of this method for spamming people's inboxes and for immoral or lucrative purposes (that is NOT the intention with which this article has been written).

If you find this article useful or would like to discuss it further, please leave a comment here. Have a great day and Good luck.

Quick script for extracting emails from unformatted text

Often, we face a need of extracting emails from some un-formatted text like or html tags etc. For this, the following script can come handy for extracting emails into simple text file, which can be uploaded to mailman or other mailing software contact lists:

$ more xtract_emails.sh
#/bin/ksh
sed -e 's/\,/\n/g' -e 's/ /\n/g' $1 |   \
                             grep '@' | \
sed -e "s/[<>();]//g" -e 's/mailto://g' \
                              | sort -u > ${1}.extracted.txt

wc -l ${1}.extracted.txt

Example:

$ ./xtract.sh emails_unformatted.txt

131 emails_formatted.extracted.txt

Hope it is useful for someone else for extracting emails in a single shot. Otherwise, it takes a lot of time for doing several passes by examining the post-processed output. Even with the above heuristic rules, the output may not have 100% proper email, so some proof reading would be needed.

If this is useful to you, please leave a comment here.

Failed opening required 'wp-blog-header.php' while using cformsII wordpress plugin

Problem

All right, so I was trying to insert a cforms II contact form in Wordpress 2.7.0 Installation and was facing this error in a pop-up dialog box:

Warning: require_once(wp-blog-header.php) [function.require-once]: failed to open stream: No such file or directory in /home/wisdom/public_html/prashna/wp-content/plugins/cforms/js/insertdialog25.php on line 9

Fatal error: require_once() [function.require]: Failed opening required 'wp-blog-header.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/wisdom/public_html/prashna/wp-content/plugins/cforms/js/insertdialog25.php on line 9

I ignored it for a little while, but a point came when I had to really get it going. As everyone else, I googled it. The first 3-5 pages of it are sheer nonsensical links of websites facing the same error, but without solution. Then finally, I hit upon a support forum of cforms II, which suggested many things.

Solution

This is what worked for me. Apparently, the abspath.php was MISSING from the cforms plugin directory:

wisdom@wisdomspeak.org [~]# find . -name abspath.php
./public_html/mantra/wp-content/plugins/cforms/abspath.php

wisdom@wisdomspeak.org [~]# find . -name cforms
./public_html/mantra/wp-content/plugins/cforms
./public_html/gita/wp-content/plugins/cforms
./public_html/prashna/wp-content/plugins/cforms
./public_html/vedvaani/wp-content/plugins/cforms

# cp ./public_html/mantra/wp-content/plugins/cforms/abspath.php \
./public_html/prashna/wp-content/plugins/cforms

And everything was golden..

Thanks to that, I was able to have a contact form like this in the wordpress page:

Setting up a common login across multiple wordpress installations

A pre-requisite for promoting interaction across sub-domains..

After setting up wisdomspeak.org/mantra, wisdomspeak.org/vedictalks and wisdomspeak.org/gita, i realized that the only way the parent website, wisdomspeak.org, was going to get popular were through inviting user interaction on discussion boards. But for this, users would need to be logged into different sub-domains through a single login (like a global single sign-on feature of Oracle applications).

Assumptions:

First and foremost, the version of wordpress installed on the multiple sub-domains was version 2.7. Also, the wordpress installations were done in the same database, with different table prefixes.

The table_prefix of the first wordpress installation (wisdomspeak.org/mantra) was "wp_", whereas the table prefix of the other wordpress installation with which I wanted to share the userbase of wisdomspeak.org/mantra, was "gita_".

Now I had to make wisdomspeak.org/gita use the same users/roles as wisdomspeak.org/mantra.

What worked finally..

Once you try googling it yourself, this is probably the most straightforward and common response that you would see:

In the wp-config.php of the 2nd wordpress installation, after this line:

/* That's all, stop editing! Happy blogging. */

..add the following:

define('COOKIE_DOMAIN', '.wisdomspeak.org');
define('COOKIEPATH', '/');
define('CUSTOM_USER_TABLE', 'wp_users');
define('CUSTOM_USER_META_TABLE', 'wp_usermeta');

But when I was going to wisdomspeak.org/gita/wp-login.php and trying to login as admin user, I was getting you do not have " You do not have sufficient permissions to access this page." error message while using firefox browser, while IE just gave up with a page not found error.

Interestingly, this error is also encountered a log while people upgrade wordpress installations, so there is a lot of scope of mixing up resolutions while trying to solve this. Better be wary of mixing different recipes.

And so, in desperation, I searched for more options/resolutions and came across a post that suggested changing the prefix values of meta_key column in the <table_prefix>_usermeta table of the 2nd word press installation. Using this hint, I did some more tweaking by opening the phpMyAdmin interface of the gita_metadata, but the permissions error was still coming.

So, i searched a bit more and came across a posting in which a user was not able to make it work, in spite of manually the prefix of <table_prefix>_capabilities table referred in $WORDPRESS_HOME/wp-includes/capabilities.php to the table_prefix of the earlier wordpress installation (which was wp_ in my case). The change is actually quite simple: you are fixing the name of the capabilities table for a user to be wp_capabilities.

function _init_caps() {
global $wpdb;
/* $this->cap_key = $wpdb->prefix . 'capabilities';  -- the earlier entry*/
$this->cap_key = 'wp_capabilities';  /* the new entry */
$this->caps = &$this->{$this->cap_key};

And when I tried with this, not only was I able to login with the admin user's password for wordpress installation 1 in wordpress installation 2, but when I changed the URL from wisdomspeak.org/gita/wp-admin/ to wisdomspeak.org/mantra/wp-admin/, i was not redicted to a login page again, which meant that the same user was shared across different wordpress installation. (It did ask me the login once for wisdomspeak.org/mantra/wp-admin/, so perhaps the cookie was not shared really, but that is something I need to doublecheck).

Extending this concept to roles?

While going across capabilities.php, I came across a similar looking entry for user_roles and thought that logically speaking, a similar fix would be needed in capabilities.php to re-use the user roles :

function _init () {
global $wpdb;
global $wp_user_roles;
 /* $this->role_key = $wpdb->prefix . 'user_roles';
$this->role_key = 'wp_user_roles';

But when I tried this, i started getting the "You do not have sufficient permissions to access this page." error again. On reverting it back like this, the login started working again:

function _init () {
global $wpdb;
global $wp_user_roles;
 $this->role_key = $wpdb->prefix . 'user_roles';
/* $this->role_key = 'wp_user_roles'; */

So was it really needed..

Looking back, changing the meta_key column's entries in gita_usermeta table did not make any sense, since I was never going to use that table. To prove this point, i changed the prefix values for meta_key column back to gita_ and the admin user was still shared across wisdomspeak.org/mantra nad wisdomspeak.org/gita.

How to share cookies across wp installs in a common domain?

Thanks to Ben (whose comment can be seen below) for sharing the secret for making cookies work across wordpress installs in a common domain:

1) Use the same auth_salt, logged_in_salt, secret key in /wp-admin/options.php page for each sub-domain blog

2) Install Root-cookie plugin ( by LINICKX )

* http://www.linickx.com/archives/831/root-cookie-path-14-an-update-for-wordpress-27

1. Upload root-cookie.php to the /wp-content/plugins/ directory
2. Activate the plugin through the 'Plugins' menu in WordPress
3. Log out
4. Log in
5. Done :o)

3) And in: wp-settings.php

Change this source code like this:

351 - define('COOKIEHASH', md5(get_option('siteurl')));

to:

351 -  define('COOKIEHASH', md5(get_option('http://www.mysite.com')));

Yippee!

All right, so that was a good learning on an early Sunday morning; All in 1 hour of googling and trying out options. Dont they say that early mornings are the most productive of times.

My first Aesthetic wordpress theme: Modifed Lotus Flower Theme for 3 columns

A big Leap..

The past few days have been a big leap for me. Thanks to Google's search engine, tons of prolific writers across the world, and Tanvi Chanchalani, I learnt something about wordpress and css, which enabled me to modify a beautifully aesthetic 2-column wordpress theme called Lotus Flower into a 3-column wordpress theme. Not only this, using it, I was able to quickly make two websites on my newly purchased domain, wisdomspeak.org.

The purpose of this post is to recount my learning experience in slow motion for my own benefit and also for the benefit of those thousands of budding wordpress theme creators. Honestly, there is so much creativity out there. People get great ideas all the time. It would be a pity not to be of any service to them, so that they could help humanity. Well, seriously, I just want to help.

I Love you!.. but...

Ok, back to the mission.

When I initially saw the Lotus Flower theme on http://www.freewpthemes.net, I was so impressed (bowled over might be the right term) that it immediately struck me "the" one. And I used it very well, feeling happy.

But soon, I realized that to encourage the community to adopt the audio resources website that I was planning to build, Im going to need people to donate as well, which meant that they needed to see the Donate through Paypal plugin on the sidebar WITHOUT scrolling down.

Necessity is the mother of Invention..

The Support us plugin simply had to catch the attention of the person browsing the website, or else the idea would be buried in the annals of .. uhm.. the sidebar. So I had this really urging need of bringing it on the right hand side, along with some more visible links to related websites. Cmon now, I had to make it easy on the users, right, or else I wouldn't be doing my marketing homework right.

And so, i started looking around for means to add another sidebar. But there was this ONE problem. I didn't know squat about PHP or wordpress themes or CSS !

Hmm, but I had a couple of big factor on my side. I had the fire in the belly to make it work and I had TIME. Did anyone tell you that time is also a big resource? (The Capitalist says: Time is Money. In Hindi they say, समय बड़ा बलवान - time is very powerful).

And so, the hunt begins..

Ok, so the first thing to understand for customizing a wordpress theme is that the heart of the Look/Feel/User Interface is the style.css (cascading style sheet) file, which is usually in the home directory of the Wordpress theme ($WORDPRESS_HOME/wp-content/themes/<themename>). The directives given in style.css regulate the placement and cosmetics of the elements that make up the website.

The contents or elements themselves come from Wordpress's infrastructure (content entered through .../wp-admin.php console) and have got nothing to do with a style.css file. You could simply modify the sytle.css and achieve visually very different results. For example, check out wisdomspeak.org/vedictalks and wisdomspeak.org/mantra. They are both using the SAME theme, but just slightly different style.css  etc.

A different path of hit and trial..

But thats not how it played out for me. The paragraph above me is just me talking in retrospect.

First, I tried adding a second sidebar to the Lotus Flower theme. After googling a lot of pages, I could finally see sidebar2 in the Configure Widgets Page (through the Dashboard link):

1) Prefixed an argument "2," as the first argument to the register_sidebars() function in functions.php. Earlier, the function call was for register_sidebar() -- notice the singular case.

register_sidebars(2, array(
'before_widget' => '<li id="%1$s" class="widget %2$s">',
'after_widget' =>  '</li>',
'before_title' => '<h2 class="widgettitle">',
'after_title' => '</h2>',
));

2) Copied $WORDPRESS_HOME/wp-content/themes/<themename>/sidebar.php to $WORDPRESS_HOME/wp-content/themes/<themename>/sidebar2.php and made the following changes in sidebar.php and sidebar2.php:

# diff sidebar.php sidebar2.php
1c1
< <div id="sidebar-left">
---
> <div id="sidebar-right">
3c3
<               <?php if (!function_exists('dynamic_sidebar')
|| !dynamic_sidebar(1)): ?>
---
>               <?php if (!function_exists('dynamic_sidebar')
|| !dynamic_sidebar(2)): ?>

Earlier, the function call inside sidebar.php was only for dynamic_sidebar(). It wasn't working without it. Also, the <div id..> tag value was made more specific to -left or -right.

Later on, I realized that the advantage of this was that I could influence the look and feel of each sidebar individually.

3) Included <?php include( "sidebar2.php"); ?> in page.php and index.php like this:

<?php get_header(); ?>
<?php include( "sidebar2.php"); ?>

Then did the same for $WORDPRESS_HOME/wp-content/themes/<themename>/404.php, archive.php, search.php, single.php. The really dumb rule of thumb that I followed was: if there is a <?php get_sidebar(); ?> in the php file, it needs the 2nd sidebar too.  Common sense, eh?
4) At this point, lo and behold, I COULD see the 2nd sidebar in the Configured widgets page on Wordpress admin console and happily assigned the PayPal widget to the right sidebar. At least, I was getting somewhere.

5)  When I tried previewing the website live, it WAS showing the right sidebar, but the left sidebar was pushed way down on the left. So the content was very much there, but the UI/cosmetics was not displaying it right. This is where we need to brush up our CSS skills.

learning basic CSS..

Now, it was down to manipulating the style.css for accommodating the extra sidebar.This is where these tutorials from html.net really helped me understand the overall scheme of css. I still don't understand that much about how floating works. My approach to making the theme work was .. change one property, preview it and if it wasn't working, revert it back. Pretty much hit and trial based.

From common sense, I could reason that there was a need to play with a lot of width values width/padding/marging values for various elements in style.css, because ultimately that's what was causing the mis-adjustment.

Changed the style.css a bit to accomodate the 2nd sidebar to the right. I had to play with a lot of width values. I just guessed that some of those width/padding/marging values were infuencing the display. But I didn't understand the overall structure properly yet.

It was actually at http://www.html.net/tutorials/css/lesson13.asp that a lightbulb lit up in my mind. The example had used width: 33%;.

Ok, I thought, if this directive can be used for columns in a list, then why cant it be used for any other element? Or in other words, it should be possible to use a similar % directive for other elements in css too.

First blood..

This is where Tanvi helped me a lot in advising which sections and their properties to be changed. I wanted to cover more width on the screen (800x600 resolution is history). Keeping this in mind, she not only widened some settings, but also sent me widened images to go along with the new theme.

When I initially replaced the style.css for theme, its working on firefox, but not on IE: The left sidebar was not able to adjust on the top somehow, and was pushed down below. Also, the header and top navigation pane were not centered.  Thats when I realized that the new widened images had not been overlaid. After copying them, the screen is much wider, finally! However, there seems to be some sort of an adjustment issue with the radio widget's size or floating adjustment for IE. From googling, it seems that there are some CSS bugs for IE, especially around wrapping or floating.

After playing with width: <number>% directives for content, sidebar-left, sidebar-right sections of style.css (I identified them with the background image tag -- compared the filenames to the theme's images visually.), I was able to bring all the contents under a very uniformly formatted pattern.

I was experimenting with % values for width on top of the css and  this seems to be setting up the alignment very well for sidebars and the top! It almost like what is needed. But there was still a gap between the left sidebar and the main content box. It seems to be related to some margin setting for the content element. This is what worked out for me:

#sidebar-left {
        FLOAT: left; WIDTH: 20%
}
#sidebar-right {
        FLOAT: right; WIDTH: 20%
}
#content {
        BACKGROUND: url(images/img10.jpg) no-repeat left top;
 FLOAT: right ; WIDTH: 58%
}

In an effort to center and widen the header & logo and remove extra padding between header/logo, I kept playing with the properties of their sections:

#header {
        BACKGROUND: url(images/img02.jpg) no-repeat center center;
MARGIN: 0px auto; WIDTH: 1200px; HEIGHT: 60px
}
#logo-wrap
 {
       PADDING-RIGHT: 0px; PADDING-LEFT: 0px;
BACKGROUND: url(images/img03.jpg) no-repeat left top;
PADDING-BOTTOM: 20px; MARGIN: 0px auto;
WIDTH: 1200px; PADDING-TOP: 10px
}

Now the top alignment was perfect, but the menu links at the top (links to the Wordpress "Pages") were aligned to the very left. Once again, Tanvi came to the rescue and said that the answer was to change the left padding for menu UL section:

#menu UL {
    PADDING-RIGHT: 0px; PADDING-LEFT: 145px;
PADDING-BOTTOM: 0px; MARGIN:0px; LINE-HEIGHT: normal;
PADDING-TOP: 0px; LIST-STYLE-TYPE: none
}

This is when It was very much like what I wanted it to be. I was elated and thankful to have been able to achieve my goal.

Later that night, I saw that the padding for the elements in the right sidebar was off, not uniform. It was related to the left-padding pixel value being somewhat different for the list and element. I wouldn't say that even now all the left/right padding values for left/right sidebar's units/lists elements are uniform at 0px or 5px or 10px, but its just that the combination of hits and trial without a particular scheme worked out. You should take a more methodical approach and keep a uniform pixel side to make things easier.

The Modified theme ..

The theme is available here for your download and further modification. The original theme's zip file is available here.

Later on, I modified the following two things to make a different theme for wisdomspeak.org/vedictalks (the new one is style.css):

# diff style.css.gold1 style.css
2c2
<       PADDING-RIGHT: 0px; MARGIN-TOP: 10px; PADDING-LEFT: 0px; FONT-SIZE: 13px;
BACKGROUND: url(images/img01.jpg) repeat-x left top; PADDING-BOTTOM: 0px; COLOR: #333333;
PADDING-TOP: 0px; FONT-FAMILY: "Trebuchet MS", Arial, Helvetica, sans-serif
---
>       PADDING-RIGHT: 0px; MARGIN-TOP: 10px; PADDING-LEFT: 0px; FONT-SIZE: 13px; 
BACKGROUND-COLOR: #FFF380; PADDING-BOTTOM: 0px; COLOR: #333333;
PADDING-TOP: 0px; FONT-FAMILY: "Trebuchet MS", Arial, Helvetica, sans-serif
98c98
<       BACKGROUND: url(images/img05.jpg) no-repeat left top;MARGIN-LEFT: 4px;
MARGIN: 0px auto; WIDTH: 950px; HEIGHT: 200px;
---
>       BACKGROUND: url(images/img05.jpg) no-repeat left top;MARGIN-LEFT: 4px;
MARGIN: 0px auto; WIDTH: 950px; HEIGHT: 20.3em;

The beauty of  HEIGHT: <number>em property is that it can scale up the image very well.

The new theme looks like this:

Feedback..

I know this is not the best of the tutorials out there on the internet, but I would appreciate feedback/validation from you, whatever your level of expertise. I am still learning CSS and would love to try new things like scalable flash images in a wordpress blog.  So please leave a comment here. Thanks for your consideration.

Preface - Joining the Android revolution

Last year, I joined the Android open source revolution as a user. I took a difficult and controversial decision. I bought the G1 phone. Many people call it the T-mobile G1 phone, but I like to call it the HTC G1 phone, to give credit to the actual phone manufacturer. Agreed, it couldn't have happened without Google sponsoring the Android platform and creating the Open Handset Alliance. T-mobile got exclusive rights to the marketing of the very First device to run the Open source android platform and got a sweet deal for itself. It even convinced many people to buy the data plan.

I am sure a lot of people thought that they had to get the data plan to make the phone even work. Well, that was true, but only for the very first power up of the phone. Once it had registered the google account for the phone, it didn't really care if it had the data plan or not.

I bought the G1 through Ebay (unlocked). After utilizing Microsoft Cashback Livesearch feature with Ebay, I got a sweet deal for $375, including some accessories. Back in Nov 2008, I got a 25% cashback! A week later, Microsoft announced that they would give instant rebates, instead of delaying the cashback for 2 months via Paypal, as originally announced.

Later, I learnt that Tmobile also sells the unlocked handset to customers for $399 without contract, although its not openly said so. So if you are thinking of getting it from ebay/amazon, try hammering a Tmobile representative at a store. It might be worth getting it from Tmobile and make them liable (just in case the handset develops some issues).

After playing with the G1 Phone for a while, I thought it was a worthwhile investment for me, as I was able to get on the internet anywhere with the data plan. K-9 Email (just released a more stable version yesterday) and AIM IM (continuously improving) work well on this RC30 revision of the platform now, and honestly, that was all I needed to work efficiently; effectively, I had bought my independence.

The Android Market/Application Store

One of the most fascinating selling points of the android platform is the Market, which is in Beta stage till 15th Jan 2009, which effectively means that all applications are free on it. (The Pacman game is the lone exception, I guess). There are some very creative people out there who are developing applications left and right. I remember that when I got hooked to the market, I would check out the market almost every night and try out new programs.

This was also the time when I logged a lot of bugs/enhancements through http://code.google.com/p/android/issues/entry website. It was quite concerning to see that this website was not very much publicized, though. I guess Google wanted to keep it quiet till the Beta stage was over.

Streamfurious - Great potential

When I first played the StreamFurious application, I was fascinated at the various radio stations that I could play. It also set me thinking as to what was shoutcast/icecast in the first place.

On doing a bit of Googling, I came across some posts and youtube videos of how to build a shoutcast server, which seemed lucid enough.

http://www.frihost.com/forums/vt-15354.html - This is the one I used for my shoutcast server
http://wizardskeep.org/mainhall/tutor/shbroad.html -- this one has screenshots

http://www.youtube.com/watch?v=c1leou1vqE4 - This is a good video for Linux installation (recommended for stability)

This morning, I had some free time on my hands and decided to look into creating my radio station. I only used the first link, http://www.frihost.com/forums/vt-15354.html, and I was up and running with my station within 1+ hour (includes discovery and trial/error time). Now thats really OPEN technology.

Apparently, there are a lot of FREE shoutcast servers on the internet, and the most popular of them seem to be WINAMP's shoutcast plugin (from winamp.com) and the shoutcast server from shoutcast.com.

How does it work..

Come to think of it, the technology itself is quite simple, after all the abstractions and implementations.

The actual audio track is played in Winamp. Winamp, in turn, has a plugin, which talks to another Shoutcast streaming service (installed through the Shoutcast server). The Shoutcast service talks to the shoutcast.com servers for registration/status etc. and makes the audio available to internet users.

The entire flow could be depicted as:


Winamp (play audio here) --> Winamp shoutcast Plugin --> Your Shoutcast Server <--> Shoutcast.com

/\

|

Internet users (Shoutcast clients)

In addition, I had to enable port forwarding in my home router for port 8001 for 192.168.0.2 IP, so that external users could talk to my shoutcast streaming server.

Some Setup considerations..

One thing to note here is that I had to wait for an hour or so before my station actually got registered with shoutcast.com and was showing up in the search.

It seems that when the shoutcast server log shows this message, the registration can be assumed to done:

I think this is the right log message to look for the station registration..

<01/07/09@12:09:28> [yp_tch] yp.shoutcast.com touched!
<01/07/09@12:19:37> [yp_tch] yp.shoutcast.com touched!

Another thing to mention here is that the upload speed of the internet connection on your shoutcast server matters and would be key to decide what encoder setting you would use. If you expect N concurrent users and you arebroadcasting at 24kbps encoder setting, then your overall upload requirement is (N x 24) kbps at any time.



You got to make sure you can sustain that upload bandwidth. Even with 24 or 48 kbps, you get pretty good results. The default number of maximum users is 32, which is not bad for experimentation over cable modem internet speeds.



Also, remember that the lower your encoder settings, the more people can connect to your radio station. Know that BBC UK streams audio at 24kbps, so its not bad to use smaller settings. (but then they must be having a cluster of shoutcast servers and multiple relay server too, I'm guessing).

I started off with 24kbps first, but later switched to 64kbps to experiment with downloading speeds for StreamFurious. Also, I was using Comcast High speed Internet with download speed of 3.5mbps and upload speed of 1.5mpbs (speedtest.net would tell you).

Mantra Pushpam @ Atlanta - My Radio Station

I always wanted to listen to Vedic Mantras/Chants whenever I wanted to. Thanks to Open source technology (shoutcast), Google, HTC, Tmobile and ebay, now I am able to play Vedic chants/Mantras on the G1 phone through StreamFurious client.



All I had to do was click on http://71.204.14.23:8000, click on Listen tab and the browser redirected the request to Streamfurious, which then added the station through the listen.pls file and buffered media upto 2mb before playing the streaming audio. Now, I was on an EDGE connection, so it took a while for the buffering to happen. On 3g Networks, it should be even faster.

Note that Streamfurious does not have the capability to add a radio station manually as of now, but if you go to the source URL of a streamcast server and click on Listen, the default browser is smart enough to redirect the request to Streamfurious. That's how I'm getting around it for now.

For the iPhone users..

For the iPhone users, there are FlyCast (FREE! Very cool article here: http://cybernetnews.com/2008/09/16/free-iphone-app-streams-music-in-the-background/) , iRadio and Shoutcast radio applications that came out in the market. I am still having my iPhone friends try this out.

Here is a very good article on how to listen to Shoutcast radio stations using iTunes.. http://cybernetnews.com/2008/10/23/cybernotes-add-shoutcast-radio-streams-to-itunes/

Another good article with Video review is at http://www.appvee.com/t/shoutcast-radio

For them BlackBerry users..

For the Blackberry power users, FlyCast works well for them. Information on how to download and install FlyCast is available at http://www.blackberryforums.com/aftermarket-software/158658-flycast-download-links.html

When the installation dialog comes up, click "Set Application Permissions" and then click "Download". On the configuration screen, change the "Connections" and "Interactions" settings from "Custom" to "Allow". Press "Back" and then "Save". The application will download and install. If you have a Bold - Run "FlyCast" from the Downloads folder

If you have a Curve - Run "FlyCast" from the general "Ribbon"

You have have to register here - FlyCast - Now your smartphone is complete, then  log in from your BB.

FlyCast has set up a support site for iPhone and BlackBerry users. Please visit FlyCast Forums Index page for tips on getting the most from FlyCast on your BlackBerry.

Additionally, You can also run audio streaming through the Slacker application, as seen here : http://blog.laptopmag.com/video-hands-on-with-slacker-application-for-blackberry

StreamFurious Vs AntPlayer

When I tried running the station again later today with StreamFurious (version 0.0.6), it started deleting the buffer first (which it was not recycling -- seemed like a bug for sure. I wrote to feedback@streamfurious.com about this and did not get any response from them). The deletion of the buffer resulted in timeouts and not a very good user experience. This forced me to look for more alternatives in the android market for playing shoutcast streams.

Through http://www.howardforums.com/showthread.php?t=1468651, I found AntPlayer, which seems to be more mature in terms of finding stations (the search feature actually works) and also a good buffer recycling mechanism.

On day 1, I had better success with AntPlayer for some reason. However, on the subsequent days, at random times, StreamFurious seemed to work better than AntPlayer. Actually, AntPlayer seemed to hang while buffering totally, while StreamFurious did not seem to repeat the buffer deletion syndrome anymore. (It COULD have to do with the fact that I had restarted the shoutcast server a couple times on the first day itself).

A caveat with StreamFurious seems to be that it uses up the battery very quickly. Also, it seems StreamFurious is getting more publicity in the market due to the keyword Stream in its application name.

http://mantra.podzone.org:8000/listen.pls - Making the radio station URL  user friendly and Pnemonic to memory

Since I am running this station from my home on a regular Dell latitude D630 laptop on Windows XP and Comcast Cable internet (behind a Netgear router), the actual external IP would change from time to time. I wondered: How could I make the radio station's URL independent of the IP?

Then I remembered that a long while back, I had used free DNS services provided by dyndns.com. DynDNS has many paid services, but you can associate an IP to some predetermined domains (you get to choose the sub-domain though) for FREE.

So I created a free account on dyndns.com and associated my shoutcast server's external IP 71.204.14.23 (got through http://www.Whatismyip.com) with mantra.podzone.org.



Whenever I would have to reset the internet connection or restart Windows, the external IP would change. To re-associate the latest external IP, I would simply go to https://www.dyndns.com/account/services/hosts/ and assign the latest IP (it detects the external IP auto-matically) to mantra.podzone.org. This way, the users do not have to remember anything, but  http://mantra.podzone.org:8000/index.html.

If you click on the Listen subtab, the browser should be able to open the listen.pls (playlist) file with a suitable application on the mobile OS.

You've got to utilize the power of free stuff on the internet.

Network utilization Stats..

Now, I was also interested in the network utilization stats on my wireless network due to shoutcast broadcast per user. When I played the station through streamfurious on my G1 phone, i monitored the local wireless network interface's bandwidth usage (11mpbs maximum bandwidth) and got this graph:



As you can see, it was using less than 1% of 11mbps bandwidth per mobile user.

Feedback welcome

If you found this article helpful and would like to discuss/express your views, please do leave a comment and we can discuss over web/email.

Thanks a lot for your time to read this article!