Tuesday, January 15, 2013

Multi-homed MythTV backend

This is the first of my non OpenWRT related posts.

I have a peculiar situation.  I have two houses that I split my life between.  I don't want to pay for cable in both houses, but I also want access to some of the TV shows that you only get newer episodes on cable and not via Netflix/Hulu/Amazon.

So to solve this, I have set up MythTV in both houses.  Both setups run Mythbuntu 12.04.1 (www.mythbuntu.org).  In the house without cable I have a Silicondust HD Homerun.


The HD Homerun is set to record OTA content exclusively.  I get all the basic TV networks this way in that house.

In the second house I have a Silicondust HD Homerun Prime.  I'm fortunate that in that house I have Comcast, which is one of the more friendly companies with regard to copy protection.  I'm able to record pretty much everything except the premium networks like HBO and Showtime.
 

In the second house I duplicate all the OTA recording schedules but also schedule things on the other stations I care about.

Secondary House

Now to get the recordings from the second house to the first house, I had to setup a collection of scripts that run in staggered cron jobs.  The first one is on the secondary house.  I first determined which channel ID's I needed to export from.  This can be found by looking through mythweb or by examining the database with a tool like phpmyadmin.  Look for 'chanid' in the 'channels' table.

Once I had those channels figured out I created this script.  On each daily run it will find all the recordings who's starttime happened to be "today".  It exports all relevant SQL data about that recording into a flat SQL file that can be imported on the primary house.

Next it creates a list of recordings that will need to be synced over to the primary house.  This list is based upon the last successful sync date from the primary house.  It only syncs all recordings between the lastrun date and "tomorrow".  This sets it up so that if for some reason the primary backend doesn't sync a day it will still work.  Also it's important to sync only a handful of days because the autoexpire will be happening at different rates on the two backends.

I store all this content in the directory /var/lib/mythtv/exported-sql

sudo mkdir -p /var/lib/mythtv/exported-sql 
sudo chown mythtv:mythtv /var/lib/mythtv/exported-sql

/home/mythtv/export.sh 

#!/bin/sh

#export SQL
chanid="2657 2659 2622"
DIRECTORY=/var/lib/mythtv/exported-sql/
BIGGEST_DATE=`find $DIRECTORY -maxdepth 1 -name '*.sql' | sort -r | head -1 | sed 's,.*exported-,,; s,.sql,,'`
for chan in $chanid;
do
[ -n "$where" ] && where="$where or"
[ -n "$BIGGEST_DATE" ] && date=" and starttime > '$BIGGEST_DATE 00:00:00'"
where="$where (chanid='$chan'$date)"
done
CONFIG=$HOME/.mythtv/config.xml
DATE=`date '+%F'`
if [ "$DATE" = "$BIGGEST_DATE" ]; then
echo "Already ran today, not running SQL generation again"
else
db=`xpath  -q -e 'string(//DatabaseName)' $CONFIG 2>/dev/null`
user=`xpath  -q -e 'string(//UserName)' $CONFIG 2>/dev/null`
pass=`xpath  -q -e 'string(//Password)' $CONFIG 2>/dev/null`
host=`xpath  -q -e 'string(//Host)' $CONFIG 2>/dev/null`
fname=/var/lib/mythtv/exported-sql/exported-$DATE.sql
mysqldump -h$host -u$user -p$pass $db recorded recordedseek recordedrating recordedprogram recordedmarkup recordedcredits --where="$where" --no-create-db --no-create-info > $fname
fi


#generate a recordings list
lastrun=/home/mythtv/lastrun
tomorrow=$(date --date="tomorrow" '+%Y%m%d')
if [ -f $lastrun ]; then
        tmp=$(cat $lastrun)
else
        tmp=$tomorrow
fi
while [ "$tmp" != "$tomorrow" ]
do
        test_dates="$test_dates $tmp"
        tmp=$(date --date="1 day $tmp" '+%Y%m%d')
done
test_dates="$test_dates $tomorrow"
for date in $test_dates;
do
        for chan in $chanid;
        do
                from="$from /var/lib/mythtv/recordings/${chan}_${date}*"
        done
done
ls $from 2>/dev/null | tee /var/lib/mythtv/exported-sql/recordings-list

Next I set up rsync to export my /var/lib/mythtv directory (read only) and my lastrun successful (write only).

/etc/rsyncd.conf

max connections = 2
log file = /var/log/rsync.log
timeout = 300

[mythtv]
comment = mythtv
path = /var/lib/mythtv
read only = yes
list = yes
uid = nobody
gid = nogroup
auth users = rsync
secrets file = /etc/rsyncd.secrets

[lastrun]
comment = last rsync run
path = /home/mythtv/
write only = yes
read only = no
list = no
uid = mythtv
gid = mythtv
auth users = rsync
secrets file = /etc/rsyncd.secrets

Last thing to do on the secondary house is to set up the crontab to run at night.  I set it for 11:05 PM every day.  It should only take a 10-15 seconds to run.

5 23 * * * /home/mythtv/export.sh

Because of the way this all works, I decide to leave my secondary house backend on all the time.  

Primary House

Now in my primary house I need to sync recordings, SQL data, and then update the last successful run at the secondary house.

/home/mythtv/import.sh

#!/bin/sh

domain=rsync@address
sql_directory=/home/mythtv/sql
recordings_directory=/var/lib/mythtv/recordings
password_file=/home/mythtv/password-file
lastrun=/home/mythtv/lastrun

sync_recordings()
{
today=$(date '+%Y%m%d')
from=$(cat $sql_directory/recordings-list | sed "s,/var/lib/mythtv,$domain::mythtv,")
RET=30
while [ $RET -eq 30 ]; do
#rsync -avz --partial --timeout 120 --progress $from --password-file=password-file $recordings_directory
rsync -av --partial --timeout 120 --progress $from --password-file=$password_file $recordings_directory
RET=$?
done
echo "rsync return code: $?"
[ $RET -ne 0 ] && exit 1
echo "$today" > lastrun
rsync -avz --password-file=password-file lastrun $domain::lastrun/
}

sync_sql()
{
rsync -avz $domain::mythtv/exported-sql/* --password-file=$password_file $sql_directory
}

insert_sql()
{
CONFIG=$HOME/.mythtv/config.xml
db=`xpath  -q -e 'string(//DatabaseName)' $CONFIG 2>/dev/null`
user=`xpath  -q -e 'string(//UserName)' $CONFIG 2>/dev/null`
pass=`xpath  -q -e 'string(//Password)' $CONFIG 2>/dev/null`
host=`xpath  -q -e 'string(//Host)' $CONFIG 2>/dev/null`
old_host=supermario
new_host=kingkoopa
for fname in $(find $sql_directory -maxdepth 1 -name '*.sql');
do
if [ ! -f ${fname}.imported ]; then
cat $fname | sed "s,${old_host},${new_host},g; s,INSERT INTO, INSERT IGNORE INTO,g" > ${fname}.imported
mysql --host=$host --user=$user -p$pass $db < ${fname}.imported
fi
done
}

suspend()
{
sudo /usr/local/bin/setwakeup.sh $(date --date='18:00' '+%s')
sudo pm-suspend
}

mythfilldatabase
sync_sql
sync_recordings
insert_sql
#suspend

I'm careful about the order I do things.  The SQL has to get inserted last in case for some reason the recordings fail to sync or don't all sync while i'm watching.

I currently don't suspend afterwards due to some instability on my system, but I have been experimenting with that too.  If S3 is reliable you can configure the setup to suspend after the sync is done and wake up next time you need to use it or record from it.

I set the cronjob to run at midnight every day on the primary backend.

0 0 * * * /home/mythtv/import.sh

Thursday, December 1, 2011

Garmin Forerunner 405 and Ubuntu 11.10

Recently I purchased my first Garmin running watch from Amazon.  I decided to go with one from the Forerunner 405 family as I preferred the style, size and features compared to the others I came across.
    

At home I only have Ubuntu machines, but I still want access to all of my running data from the watch.  The older Garmin watches connected over USB to transfer data, but these newer ones use something called ANT+ to communicate with other wireless peripherals and to transfer data to computers.

The watch came with a USB dongle.  Unfortunately, Ubuntu 11.10's kernel doesn't have any drivers that automatically load when plugging it in.  After searching a little bit, I discovered that the usbserial module can be loaded using custom vendor and product ID's.

To do this, you can check lsusb to find the dongle.  Dynastream is the subsidiary of Garmin that owns the proprietary ANT+ technology.

# lsusb | grep Dynastream

Bus 001 Device 015: ID 0fcf:1008 Dynastream Innovations, Inc. 

Now that you have the vendor and product ID, you can manually load the usbserial module. 

# sudo modprobe usbserial vendor=0x0fcf product=0x1008

This will create a new /dev/ttyUSBx device on the system.  I only have one /dev/ttyUSB0, so t he rest of this assumes that's the same for you.  Next I found out that there is actually an app in the archive for fetching this data, it's just very poorly documented.  You can install it from the Ubuntu Software Center or manually install it:

# sudo apt-get install garmin-ant-downloader

Once installed, you have to wake up the watch, and get it in pairing mode.  For me this meant messing around with the bezzle until I could find the pairing menu and enable pairing.  Once pairing is on, it's just a matter of running the app manually once

# garmin-ant-downloader

The watch should say pairing worked properly.  Now go back to the training menu, and reset your run using the quit button so that the timer is back at zero.  This tells the watch that it's ready to send the data that it recorded before.  Put it in date/time mode.  It's OK if it falls asleep, ANT+ works either way.  Rerun the tool and it should place a tcx file for your run in the current working directory.

# garmin-ant-downloader

The tcx file that it spits out is complete, but in it's current form won't upload to http://connect.garmin.com.  Because of an empty name field.  This can be fixed with a simple sed command however.


So for me this is enough to at least get my runs recorded and data somewhere I can manage with Ubuntu 11.10 without too much pain.  On the TODO:
  1. Find a way to automatically load usbserial when this vid/pid shows up.  It doesn't seem to have any modaliases defined, so this might be troublesome and just require a udev rule.  Any suggestions here would be helpful.
  2. Figure out why the Name field isn't populated properly when downloading data

Tuesday, February 22, 2011

Updated packages for backfire

Recently an evolution of the Asterisk 1.8.x build that I submitted a ticket for at OpenWRT SVN has been accepted to OpenWRT trunk.

I've since then decided to recover my build environment and rebuild 1.8.2.3 using what landed upstream in trunk for backfire on ar71xx and brcm2.4.  I've also included the WIP patch for invisible support from http://www.personal.psu.edu/wcs131/blogs/psuvoip/2011/01/asterisk_hack_make_your_google.html

Note, invisible support is shared across clients because it's a proprietary google extension to Jabber.  If you set it on Asterisk, you still won't be able to be shown as visible from a regular client like GTalk for Android or Chat in GMail.

I'll try to keep an eye on https://issues.asterisk.org/view.php?id=18727 to see when it's properly included so that I'll switch to the patch upstream instead though.

Thursday, December 9, 2010

Outbound Google Voice dialing "proper" fix

As I mentioned in http://supermario-world.blogspot.com/2010/12/nov-30th-break-of-asterisk-18-w-gv.html, proper Google Voice outbound dialing got busted at the start of the month because of a protocol change on Google's servers.

While upstream worked on the patch (as detailed in https://issues.asterisk.org/view.php?id=18412), I had a workaround that used an AGI call back for the dialer.

Well a final patch has been developed now that accounts for the protocol changes and accepted into 1.8.x svn (https://issues.asterisk.org/file_download.php?file_id=27904&type=bug).  I expect there will eventually be a 1.8.1 maintenance release that includes this patch.

In the interim, i've rev'ed my packages to 1.8.0-2 and include this patch now.  You just need to opkg upgrade and you should receive the updates.

# opkg update
# opkg upgrade

You should undo any AGI dialer hacks that were added and revert to the original outbound google talk dialer.  If you have been using my dialplan, this should just be a two line change to comment out the AGI line and uncomment the Talk line.

so the outbound section of your extensions.conf should look like this:


[outbound]
include => seven-digit
include => local-devices
include => tollfree
include => talk-gmail-outbound
include => talk-numeric-outbound
include => dial-uri


As always, you can view my dialplan and settings at http://www.arctangent.net/~superm1/gv_configs/ in case you are missing something or joining in late.

And thanks to everyone who helped to make this patch happen!

Thursday, December 2, 2010

The Nov 30th break of Asterisk 1.8 w/ GV

On November 30th, Asterisk GV outbound dialing started breaking for people.  The cause was unknown, and it became more prevalent the next two days.

The symptoms are a continuous ringing for the outbound call with the following type of stuff in the logs:


[Dec  3 00:54:00] NOTICE[18023]: chan_gtalk.c:1942 gtalk_parser: Remote peer reported an error, trying to establish the call anyway

Inbound calls (including those initiated from the website) are still functional.  It's not clear what the cause is, but an issue has been filed with Asterisk upstream at https://issues.asterisk.org/view.php?id=18412

In the interim, I've come up with a way to fix the problem by reviving the old AGI dialer originally used with Asterisk 1.6, but reworking it to use Talk instead of Gizmo.

This involves some modifications to the dialplan to support using AGI instead.  If you don't already have AGI setup, here's the basics for that:

1) Install AGI support for Asterisk:

opkg install asterisk18-res-agi

2) Create the AGI directory

mkdir -p /usr/lib/asterisk/agi-bin/

3) Fetch the updated AGI script from http://arctangent.net/~superm1/agi/google-voice-dialout.agi and save it in /usr/lib/asterisk/agi-bin.  Modify it to include your login information for google.

cd /usr/lib/asterisk/agi-bin
chmod +x google-voice-dialout.agi

4) Modify your dialplan.

outbound
Instead of 
exten => _1NXXNXXXXXX,1,Dial(Gtalk/superm1/${EXTEN}@voice.google.com)
Make the line
exten => _+1NXXNXXXXXX,1,AGI(google-voice-dialout.agi)

inbound
Add the following two rules:
exten => superm1@gmail.com, 1, GotoIf(${DB_EXISTS(gv_dialout/channel)}?bridged)
exten => superm1@gmail.com, n(bridged),Bridge(${DB_DELETE(gv_dialout/channel)}, p)

This will bridge outgoing calls with an incoming dialback handler.

As previously, i'll post my updated configs to http://www.arctangent.net/~superm1/gv_configs/.  The way that i've implemented it is actually just a different AGI handler that's included.  So when regular talk support is restored (as mentioned in https://issues.asterisk.org/view.php?id=18412) then it's a one line change to re-enable the standard outbound talk handler.

Thursday, November 18, 2010

Better support for sipdroid on Android

For a while I've wanted to use my fancy Asterisk setup with my HTC Evo 4G, but i've always been getting horrible performance from sipdroid as well as any other clients I was trying on the phone.

It seems that i've identified the problem as actually being a codec problem.


I looked a little closer at the verbose logs in asterisk and saw messages like this:
[Nov 19 02:57:44] WARNING[9373] chan_gtalk.c: Asked to transmit frame type alaw, while native formats is 0x4 (ulaw) (read/write = ulaw/ulaw)


It seems that whenever I was making calls, the formats kept on flipping from ulaw to alaw and vice versa.  When it did work, only one end could hear the other.  To make matters worse, I couldn't get G722 to work whenever I tried to force it.

Comparing the OpenWRT Asterisk 1.8.x install to a standard install on Ubuntu, it looks like the codec files for G722 weren't even getting installed.  I modified the packaging and republished additional packages at my repository for Asterisk on arctangent.net (http://arctangent.net/~superm1/openwrt)

Now there is two more packages available, one for alaw and the other for g722.  You should be able to update your opkg list and install them:

# opkg install asterisk18-codec-alaw asterisk18-codec-g722

After they're installed, modify sip.conf to make sure you explicitly mention that you now support G722:

[101]
username=101
secret=101
type=friend
callerid="Mario"
host=dynamic
context=outgoing
outgoinglimit=1
incominglimit=1
canreinvite=no
disallow=all
allow=g722
allow=alaw
allow=ulaw
allow=gsm
Now restart Asterisk and Sipdroid should be able to use G722 for some much better audio.

Sunday, November 14, 2010

Caller ID with Google Voice and Asterisk 1.8

Previously it was possible to do a lookup of your contact's name against the numbers in your GMail/Google Voice contact list and apply that to your CallerID via John Baab's method outlined on his blog.

I've since then discovered that since updating to Asterisk 1.8 with native Google Voice this no longer functions.  This is because the format of the CallerID string that is sent from incoming calls is now different.

The entire string is sent in the "Name" field of the caller ID.
For Voice Calls, this is what it looks like:
+18475551212@voice.google.com/srvres-MTAuMTIuMTM4LjI2OjD4NdM=
For calls coming in from Talk or GMail, this is what it looks like:
user@gmail.com/gmail.424BF12F

Fortunately, this entire field can be processed using built in asterisk functions to extract the data we need, restoring the functionality of John Baab's script.

Here's all the steps that i've taken to add Caller ID support:

1) Install python, python-expat, python-openssl
opkg install python python-expat python-openssl

2) Fetch John Baab's script and place it somewhere useful.
wget http://www.baablogic.net/googlecontacts.py


3) Download gdata-python from http://code.google.com/p/gdata-python-client/
wget http://gdata-python-client.googlecode.com/files/gdata-2.0.12.tar.gz


4) Manually install it.  The setup.py script doesn't work on OpenWRT, so this method can be used:

tar xzzf gdata-2.0.12.tar.gz
cp -r gdata-2.0.12/src/* /usr/lib/python2.6/site-packages/
rm -rf gdata-2.0.12*

5) Modify John's script to include your gmail login information.  Run it once manually to populate your database
./google-contacts.py

6) Modify your [google-in] context to process the number properly and do a database lookup:

[google-in]
exten => superm1@gmail.com, 1, NoOp(Callerid  ${CALLERID(name)})
exten => superm1@gmail.com, n, Set(CALLERID(num)=${SHIFT(CALLERID(name),@)})
exten => superm1@gmail.com, n, Set(CALLERID(name)=${DB(cidname/${CALLERID(num)})})
exten => superm1@gmail.com, n, Dial(SIP/101, 180, D(:1))

7) (optionally) Add this to your crontab so that it will sync new numbers on a regular basis.  John's script will actually redo the database on every run, so it will also get updates then for every removal too.
crontab -e

This will only take effect for calls routed in through google voice or talk because we only modify the google-in context.  If there is no data available, the name field will just remain empty and the number will be shown on Caller ID instead.