Media Players Documentation

Media Players DocumentationRelease 2.5

Isuma

June 10, 2015

Contents

1 Hardware platforms 31.1 Shuttle XPC small desktops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Logic Supply desktops (v2.5 series) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Advantech rugged servers (1.0 series) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Installation 52.1 Install Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Install local server software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Configure Local Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 SSH upload account configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 SSH tunnels configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.6 First file sync . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Maintenance 93.1 External synchronisation drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Syncing a media player . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 User guides 114.1 How to setup a playlist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 Troubleshooting 195.1 Test procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.2 Troubleshooting stuck queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.3 Queue is full but media player sees it empty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.4 Logging in to media players on the console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.5 Password resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.6 Inspecting status from the commandline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.7 Remote login to media players . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

6 Development 236.1 XMLRPC API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236.2 Debian packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

7 Specifications 257.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.2 Hardware possibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287.3 Auto-mounting Hotplugged Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297.4 Synchronization scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297.5 Test scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

i

7.6 How to set up kiosk mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

8 Changing this document 33

ii

Media Players Documentation, Release 2.5

The Isuma Media Players project is a fully free and open source software project to create a two-way, distributedcontent distribution network for communities with poor connexions to the internet.

Caution: This document covers teh 2.x generation. A new set of media players was designed and is covered inthe 3.x documentation. This is kept for historical reference.

Contents 1


2 Contents

CHAPTER 1

Hardware platforms

The media players project doesn’t require any special hardware to run, although we do expect the usual:

• network interface

• display and sound card (to play videos)

• storage

However, since we are storing videos, we are aiming for large storage specifications. As of July 2014, the Isuma.tvstore is around 1 TB (terabyte, or 0.9 TiB, tebibyte), so we currently deploy 2TB hard drives.

The rest of this section is dedicated to current and previous platform specifications.

1.1 Shuttle XPC small desktops

We have one Shuttle machine in the office, in the XPC series, there isn’t much to say about it other than its peculiarform factor is not very convenient.

1.2 Logic Supply desktops (v2.5 series)

Around 10 machines were built with some Logic Supply Mini-ITX cases, although the original product link is nowdead. We also had trouble with shipping and delivery to Canada. Finally, some hard drive sockets were damagedduring travel, which makes us doubt of the viability of this platform on the long term.

1.3 Advantech rugged servers (1.0 series)

We have deployed some Advantech UNO-3282 servers on the field.

Two of those servers were provisionned for Isuma and are still running after years of service. They have sealed casesthat are very solid.

Advantages:

• very sturdy

• sealed, so it won’t collect dust

• power button protected from accidental tripping

3

http://global.shuttle.com/products/productsList?categoryId=19

http://www.logicsupply.com/components/cases/mini-itx/

http://www.logicsupply.com/products/ci57_2766

http://buy.advantech.com/UNO+3282+S/UNO-3282-S/system-2926.htm


Disadvantages:

• heavy

• power supply is external

4 Chapter 1. Hardware platforms

CHAPTER 2

Installation

This describes how to set up a local server from scratch. You may be able to install the components on an existingsystem, but this is currently not supported.

2.1 Install Debian

To begin, boot into the Debian “stable” installer, either from USB, CD-ROM or from netboot. Note that the bootpriority in the BIOS of the UNO-3272 has options for both USB CDROM and normal IDE CDROM so make sure youpick whichever one you are using.

Set up keyboard, timezone, and other settings as normal.

When you get to the partitioning screen, set up partitions as follows: First hard drive (sda):

1. Primary, filesystem ext3, size 200MB, bootable flag, mount as /boot

2. Primary, LVM, size 1TB

Second hard drive (sdb):

1. Primary, LVM, size 1TB

Now choose “Write changes to disk and configure LVM.” Set up LVM as follows:

Physical volumes: sda2 and sdb1

Volume group: call it main, and set it to use all of both physical volumes.

Logical volumes:

1. Name: root, size 80GB

2. Name: swap, size 3GB

3. Name: media, size 2TB

Save the changes, and go back to the partitioning screen. Set up the logical volumes to be mounted as follows:

1. root: filesystem ext3, mount point /

2. swap: use as swap

3. media: filesystem ext3, mount point /var/isuma

Now choose to continue and the volumes will be formatted. Formatting the media partition takes about a half hour andlooks like it’s stuck at 33% completed for that entire time, so go have a coffee.

Set up root and normal users.

5


Next, choose a local Debian mirror. On the software selection screen, select the “standard system” and “Web server”software collections.

2.2 Install local server software

In /etc/apt/sources.list add:

deb http://debian.koumbit.net/debian stable main

Then run:

apt-get updateapt-get install koumbit-archive-keyring

You will see this warning:

W: GPG error: http://debian.koumbit.net stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0x6DC388D8B7C0A70A

It is normal, you need to bootstrap the trust to the repository. Optionnally, you can also download the key directlyfrom the web of trust and verify the signatures to check its authenticity.

Then run:

apt-get install isuma-local-servers isuma-autossh

Optionally, if you want to setup kiosk mode, run:

apt-get install isuma-kiosk

2.3 Configure Local Server

Create the config.inc file in the /usr/share/isuma-local-servers/sync/. Configure the unique username/password toaccess cs.isuma.tv in the form: username:[email protected].

Make sure file is properly formed with opening <?php and no closing ?>

The username refers to a user that is created within the drupal at cs.isuma.tv.

2.4 SSH upload account configuration

There is configuration required to setup the rsync between the media player and the main isuma.tv website.

Copy the [email protected] private key to /var/isuma/.id_rsa on the media player. The key canbe found on the isuma.tv server at /home/cachingserver/.ssh/id_rsa. Be sure that it is readeable by andonly by www-data (chown www-data /var/isuma/.id_rsa; chmod 600 /var/isuma/.id_rsa).

Then run:

su www-data -c "ssh [email protected] -i /var/isuma/.id_rsa"

NB: I didn’t do that for v25n11 as there is no caching server and couldn’t find the specified files.

Important: Save the fingerprint by entering “yes”

6 Chapter 2. Installation


2.5 SSH tunnels configuration

The SSH tunnels allow Isuma administrators to login remotely on the media players, from the central server. Theywork with a package called “autossh” that tries its best to keep an SSH connection to the central server open. Thatconnection is then used to create a tunnel back to the SSH server running on the media player.

The isuma-autossh package should automatically prompt you for the relevant information. You need to provide it withthe public key of the isuma central server, which is:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdr5vyuwKikrW0Vg+mlsRTkG1Jp3wUt4JmLopspln5sWsbAsMDtnUe7T0NW6s6W2tYvlav9m987HrNvAXCWbBeOwIuabbtcUK39QRdEUrAcd1yConPxmnsk7+sN2p7NC76lYIXYQM7hT2pWa3rXpdZQT2leSrrWFW0j/PEvwsEvb+4vmm2POX2L9GJDeb/czdT6/3NNRVlDiSh4rORbtXgDq7cj2qKZ0xnzDOcOpSjXig2eLl6759qLV3p76iNmMoHjWwo3bRypkwtiZ3zmwBPtGlctezgHqukaUKwcbDRNoZgv99QYLzAYqjx7ZqwWBCaNixaXgZhNkKhmlVSRXZB root@ip-10-122-35-248

(!) If that file is changed, you can use the dpkg-reconfigure isuma-autossh command to copy and pastethe proper key back into place.

Then the isuma-autossh package generates a public key in /root/.ssh/id_rsa.pub, which needs to be manuallycopied at the end of the /home/tunnel/.ssh/authorized_keys file on the central server.

2.6 First file sync

(From #7744.)

Here’s the resyncing process for the file queue. Note that we actually inject the files in the download queue to makesure we haven’t missed anything, but the files could also be injected straight into the File queue.

On the central server, list the queues for the media player:

mysql> SELECT csc.uid, csc.caching_server_queue, ns.title, ns.sqid FROM caching_server_config csc JOIN nodequeue_subqueue ns ON ns.qid = csc.caching_server_queue WHERE csc.uid=91;+-----+----------------------+-----------+------+| uid | caching_server_queue | title | sqid |+-----+----------------------+-----------+------+| 91 | 86 | download | 307 || 91 | 86 | upload | 308 || 91 | 86 | file | 309 || 91 | 86 | transcode | 310 |+-----+----------------------+-----------+------+4 rows in set (0.00 sec)

This is an easy way to list the subqueue identifiers for the given media player. In this example, this is media player #6(uid 91). From this we get the queue id (86) and the subqueue id (307) for the download queue.

Then we need to do the same for a reference MP, we take the isuma office one even though we know that it is missingsome files:

mysql> SELECT csc.uid, csc.caching_server_queue, ns.title, ns.sqid FROM caching_server_config csc JOIN nodequeue_subqueue ns ON ns.qid = csc.caching_server_queue WHERE csc.uid=63;+-----+----------------------+-----------+------+| uid | caching_server_queue | title | sqid |+-----+----------------------+-----------+------+| 63 | 62 | download | 214 || 63 | 62 | upload | 215 || 63 | 62 | file | 216 || 63 | 62 | transcode | 217 |+-----+----------------------+-----------+------+4 rows in set (1.75 sec)

Then we make the crazy request to inject all files from the reference player into the poor little media player:

2.5. SSH tunnels configuration 7


mysql> INSERT INTO nodequeue_nodes (SELECT 86,307,nid,position,NOW() FROM nodequeue_nodes WHERE sqid = 216);Query OK, 69566 rows affected, 65535 warnings (3.47 sec)Records: 69566 Duplicates: 0 Warnings: 69566

This may unfortunately introduce duplicates in the file table, but that’s better than an empty table!

Then do clear up the download queue, you need to hit the central server for every single file in there:

su www-data -c "while true; do /usr/share/isuma-local-servers/sync/download.php; done"

8 Chapter 2. Installation

CHAPTER 3

Maintenance

3.1 External synchronisation drives

(See #181 for background.)

3.1.1 External drive format

A drive is identified as carrying Isuma content if it has a folder in its root called ISUMA_CONTENT (all caps).

This folder is further subdivided by content type and content options. Type is, for example, video or picture, andoptions are things like low, high and ipod for various resolutions or bitrates. So, one might find a file structure like thison the drive:

/ISUMA_CONTENTvideohighlowipodpicturehighlow

The synchronization script recurses through this directory structure, copying all files from the external drive to theinternal drive (with the exception of any files named “raw”, because raw files are not needed by any machine exceptthe local server they were originally uploaded to, and the site server). It then does the reverse, copying all files onthe local drive to the external. In this direction, raw files are copied, because if the upload queue is large, it may bepossible for a file to arrive at the site server via hard drive before it arrives via rsync.

3.1.2 Creating a new synchronisation drive

This must be done from an existing media player.

1. connect the drive

2. find the drive identifier (dmesg | tail or cat /proc/partitions)

3. format it with an ext3 or ideally ext4 filesystem: mkfs -t ext4 -j /dev/sdb1

4. mount the drive: mkdir /mnt/sdb1 ; mount /dev/sdb1 /mnt/sdb1

5. create the magic directory: mkdir /mnt/sdb1/ISUMA_CONTENT

9

https://redmine.koumbit.net/issues/181


6. launch the sync script (or reconnect the drive): /usr/share/isuma-local-servers/automount/driveadded.shsdb1

This should take around 10 hours.

See also #9275.

3.2 Syncing a media player

1. connect the drive

2. observe the led start flashing

3. wait for the led to stop flashing

4. disconnect the drive

The media player should now be synced with the drive, and the drive should have the latest content from the mediaplayer.

10 Chapter 3. Maintenance


CHAPTER 4

User guides

4.1 How to setup a playlist

(once the cabletv package is completed this will all be possible to do remotely through a web interface, and parts of itwill be automated)

1. Open www.isuma.tv on your browser.

2. Make sure that you are connected to your IsumaTV Media Player:

3. Sign in with your username and password.

11


4. Go to your playlist by entering this URL: www.isuma.tv/DID/tv/name of your community and click on yourplaylist at the bottom of the page to open it.

12 Chapter 4. User guides


5. Click on “EDIT THIS” under the DASHBOARD

6. Edit your playlist by adding and removing videos as well as changing their order.

Note: Make sure to only select videos that are currently available on the MP and not videos still in the process of beingdownloaded to the Media Player. Note: Don’t forget to save your playlist regularly, even if you have not yet finished(the save button is at the bottom of the page).

Make a note of the node number for your playlist in the address bar (in this example www.isumatv/node/56706)

4.1. How to setup a playlist 13


7. You can now save this new playlist to the Media Player for local broadcast.

Note: The playlist can be modified from any computer but then needs to be saved on the Media Player in order to bebroadcasted.

In the address bar type www.isuma.tv/playlist/playlist/node number for your playlist In this examplehttp://www.isuma.tv/playlist/playlist/56706

8. Make sure you see the IP address of your local Media Player. In this example: 192.168.0.110

If instead of reading numbers like this one, you see isuma.tv or amazon this means you are not connected to the MediaPlayer. In this case, return to www.isuma.tv and make sure you have connected to the Media Player and can see themessage “You are connected to an IsumaTV Media Player”.

9. Right-click in the middle of the page. Select “Save Page as”.


http://www.isuma.tv/playlist/playlist/56706


10. Change the file name to playlist.xspf (Your file can have any name but it is essential to save it as an .xspf file).Select “WebPage, HTML only” option at the bottom right of the window. Then click “Save”.

11. You will then return to the www.isuma.tv/playlist/playlist page. Press the “Ctrl” and “Q” keys on the keyboard toquit this page. The page will close and a VLC Player window will automatically pop up. Click the Play button, thenselect “Add. . . ”



12. Select your playlist (playlist.xspf ). Click “Open”.

13. Click the Loop button and the Full Screen button on the VLC Player. Then Click the Play button to begin playingthe playlist.



You are now done!

To stop the playlist:

1. Press the “Esc” (escape) key on the keyboard.

2. Under the “Media” menu at the top left of the Player, select “Quit”.

3. The VLC Player will close and the web browser will reopen automatically on www.isuma.tv after a short delay.




CHAPTER 5

Troubleshooting

5.1 Test procedure

When a new media player is installed, it needs to be thoroughly tested. This procedure can also be used on existingmedia players to diagnose problems.

1. ping/pong test

2. URL rewriting for recent and old videos

3. upload videos larger than 8MB without errors

4. upload queue immediately updated

5. upload syncs to website

6. upload audio to website

5.2 Troubleshooting stuck queues

Taken from #10232, to be expanded:

1. login to the server (ssh [email protected] ; mp_ssh_into n7)

2. become the proper user (su www-data -s /bin/bash)

3. look at the logfiles (/var/isuma/log/*.php)

4. if nothing comes to mind, run the script by hand, then look at logfiles again (e.g. /usr/share/isuma-local-servers/sync/download.php)

5. look at /etc/cron.d/isuma-local-servers for the location of scripts to tests

To run the download script by hand:

su www-data -c /usr/share/isuma-local-servers/sync/download.php

It can be useful to bump up the debugging by setting the DEBUG variable in/usr/share/isuma-local-servers/sync/conf.php to 2.

19


mailto:[email protected]


5.3 Queue is full but media player sees it empty

If the queue is full of good stuff to download but the media player sees it as empty, it could be that the schedule is toorestrictive. Try to disable the schedule in the central server and try again.

5.4 Logging in to media players on the console

2.5 media players are in “Kiosk” mode by default, which makes it difficult to diagnose or see what is going on. ALinux console should be available if you type control-alt-F2. Then login with the usual password on the rootaccount.

5.5 Password resets

If the password for the media player is lost, it can be recovered by rebooting in a special Linux mode. See Koumbit’sdocumentation for that purpose.

This technique is complicated and should be considered last resort, if other techniques do not work or are unavailable,as it is difficult and prone to errors.

This technique is known as “booting into /bin/sh as init(8)”.

1. reboot the machine (by doing control-alt-delete or by holding the power button for 5 seconds and thenclicking it again)

2. you will then see the BIOS screen flash by quickly, then the GRUB menu, which should be shown for a fewseconds, quickly hit the shift key to disable the timeout.

3. hit the e key to edit the menu item

4. you are now in an editor, with the arrow keys, navigate to the end of the line starting with linux

5. append init=/bin/sh to the line

6. hit control-x to boot the resulting configuration

7. you should end up at a commandline prompt, enter the following command in order (do not type what is afterthe # symbol):

mount -w -n -o remount / # -w read/write, -n don't write /etc/mtabmount /usr # in case passwd is on /usr/bin/usr/bin/passwd #syncumount /usrsyncsync # it cannot hurtumount -a # will mount / read onlyreboot -nf # -n don't sync or write anything, -f don't call shutdown

8. the machine should reboot with the new root password

5.6 Inspecting status from the commandline

The mp_ssh_config script lists all the media players matching the given pattern (or all MPs if no pattern is pro-vided). Example:

20 Chapter 5. Troubleshooting

https://wiki.koumbit.net/PasswordReset#Linux_init.3D.2BAC8-bin.2BAC8-sh

https://wiki.koumbit.net/PasswordReset#Linux_init.3D.2BAC8-bin.2BAC8-sh


$ mp_ssh_config mediaplayerv25status wan_address lan_address ssh_port nameOffline 208.114.191.198 192.168.1.117 14944 mediaplayerv25n1Offline 65.181.41.57 192.168.1.116 12868 mediaplayerv25n2Online 184.144.193.175 192.168.1.112 17900 mediaplayerv25n3Online 70.25.31.106 192.168.2.16 27838 mediaplayerv25n4Online 70.25.31.106 192.168.2.14 1597 mediaplayerv25n5Online 68.67.33.253 192.168.0.77 15959 mediaplayerv25n6Online 70.25.31.106 192.168.2.15 26620 mediaplayerv25n7Online 173.177.189.83 192.168.20.233 31658 mediaplayerv25n8

This provides a quick overview of matching media players.

5.7 Remote login to media players

To login to the media players remotely through SSH, one need first to be logged into to the central server, then use oneof the scripts deployed there (see also #7201 for history).

The mp_ssh_into script allows you to log into one or multiple media players directly. You can specify a pattern orjust call it directly, in which case you will log into working MPs one after the other.

Example of running a command on multiple servers:

antoine@ip-10-122-35-248:~$ mp_ssh_into v25 ls -l /var/isuma/.id_rsafound media player mediaplayerv25n1 on port 14944, sshing...ssh: connect to host localhost port 14944: Connection refusedfound media player mediaplayerv25n2 on port 12868, sshing...root@localhost's password:-rw------- 1 www-data root 1671 oct 16 17:22 /var/isuma/.id_rsaConnection to localhost closed.found media player mediaplayerv25n3 on port 17900, sshing...-rw------- 1 www-data root 1672 oct 16 16:04 /var/isuma/.id_rsaConnection to localhost closed.found media player mediaplayerv25n4 on port 27838, sshing...ssh: connect to host localhost port 27838: Connection refusedfound media player mediaplayerv25n5 on port 1597, sshing...ssh: connect to host localhost port 1597: Connection refusedfound media player mediaplayerv25n6 on port 15959, sshing...-rw------- 1 www-data root 1672 oct 18 13:56 /var/isuma/.id_rsaConnection to localhost closed.found media player mediaplayerv25n7 on port 26620, sshing...-rw------- 1 www-data root 1671 oct 31 18:31 /var/isuma/.id_rsaConnection to localhost closed.found media player mediaplayerv25n8 on port 31658, sshing...root@localhost's password:-rw------- 1 www-data www-data 1671 nov 12 16:13 /var/isuma/.id_rsaConnection to localhost closed.

This logs into all “v2.5” media players and perform one command. If the command isn’t specified, it just logs you inand gives you a shell on the given servers.

I think this completes this request, if there’s anything else, please detail more specifically the problems.

5.7. Remote login to media players 21


22 Chapter 5. Troubleshooting

CHAPTER 6

Development

6.1 XMLRPC API

(to be documented, see the archive for now.)

6.2 Debian packages

The Isuma Media Players make an extensive use of Debian packaging to deploy software but also configuration andpolicy. This section describes how the packages are built and maintained.

6.2.1 Automated package build system

Isuma Debian packages are automatically built by Koumbit’s Jenkins server. The complete documentation about thisserver is available in the Koumbit wiki, this is only a summary applicable to Isuma packages.

When a change is pushed to one of the Debian packages git repository, they are automatically rebuilt within an intervallof around 15 minutes. The package is built within a Debian Wheezy environment and then uploaded into the KoumbitDebian archive, which is automatically signed.

Packages are uploaded to unstable by default. To migrate them to testing or stable, a manual operation mustbe performed on the Debian archive, a server only Koumbit personnel currently has the access to.

6.2.2 Automated package upgrades

Since isuma-local-servers 2.5.0, upgrades are automatically performed on all Media Players. This is donethrough the use of the unattend-upgrades package. Packages from the Koumbit archive and the main Debian archiveare automatically updated. To update more packages automatically, create a new file in /etc/apt/apt.conf.dthe specify a new Origins-Pattern that is appended to the existing list.

See /etc/apt/apt.conf.d/50unattended-upgrades or /usr/share/doc/unattended-upgrades/READMEfor more information about this software.

6.2.3 Manually building a package

To build the current Debian packages by hand:

23

https://redmine.koumbit.net/projects/local-servers/wiki/Wiki

http://koumbit.org

https://jenkins.koumbit.net

https://wiki.koumbit.net/JenkinsService

http://debian.koumbit.net/

http://debian.koumbit.net/

https://wiki.koumbit.net/JenkinsGuide#Migrating_a_package_down_into_stable

https://help.ubuntu.com/community/AutomaticSecurityUpdates#Using_the_.22unattended-upgrades.22_package


git clone [email protected]:isuma-local-servers.gitcd isuma-local-serversgit-buildpackage

To issue a new version, edit files, commit them, then bump the package version and rebuild:

edit file/foo.txtgit commit -m"update foo" file/foo.txtdch -r -i "updating foo" # increments the version number and inserts a commit in debian/changeloggit-buildpackage # or debuild

Make sure you use -D stable, if you want to make a hotfix for stable. Package is now in .. or ../build-area.

To upload the package:

scp isuma-local-servers_* [email protected]:/var/www/debian/incoming

then on the central server:

sudo -u www-data reprepro -b /var/www/debian/ processincoming incoming

kind of klunky but works.

6.2.4 Manually installing a package

Copy the package to the local server and run:

dpkg -i isuma-local-servers_<version>_all.deb

If it complains about some dependencies not being installed, run:

apt-get install

to install them.

After installing the package, you will need to perform a few additional steps:

# get the ssh private key for the site server and place it in ISUMA_ROOT with the name .id_dsa.scp [email protected]:/home/cachingserver/.ssh/id_rsa /var/isuma/.id_rsa# (password in issue #187)

At this point you can check the logs in /var/isuma/log and make sure things are running properly.

6.2.5 Manually upgrading Media Players

Mass upgrades or installs can be performed with our scripts:

mp_ssh_config | grep Onlinefor s in mediaplayerv25n3 mediaplayerv25n4 mediaplayerv25n5; do mp_ssh_into $s apt-get update; donefor s in mediaplayerv25n3 mediaplayerv25n4 mediaplayerv25n5; do mp_ssh_into $s apt-get install isuma-local-servers; done

This should normally not be necessary as the Media Players are automatically upgraded.

24 Chapter 6. Development

CHAPTER 7

Specifications

7.1 Overview

7.1.1 Requirements

1. local caching of media files being downloaded by users on LAN (gets downloaded from main site in back-ground).

2. local caching of media being uploaded by users on the LAN (gets sent to main site in background).

3. local transcoding of uploaded media, so local users can view locally uploaded media before it is uploaded tomain site.

4. remote control and configuration of local servers (perhaps using drupal?), allowing:

• stats

• troubleshooting

• control over what videos are uploaded/downloaded and in what order,

• patches, upgrades, etc.

• bandwidth controls

5. server would be installable from a CD

6. standardized solid hardware: maybe we should store video files on a large internal drive instead of an external,might be safer.

7. local cached version of site – slave db sync from master

8. cached content can be moved via hard drive.

7.1.2 General specs

• pluggable CDN as the model for file delivery, with URL re-writing at the server

• pluggable back-end to add new file types and new file delivery methods

• local server machines are as simple, as plug-and-play as possible. configuration happens on internet-availableservers

• communication between the components using XML-RPC

• use drupal/php for all coding

25


• code to drupal standards for formatting, documentation, etc.

7.1.3 Components

1. Central Server (CS) - keeps track of all the files and their locations and is admin interface.

• Drupal for authentication, XML-RPC and user-interface, minimal footprint

• “Caching Server” module with custom code.

• Nodes on Central Server uniquely identity every file that can be potentially cached (by a combination of filepath,type and option) and an array of delivery methods:

– Local Servers, Qiniq QFile, default amazon CloudFront delivery, etc...

– array (‘backend’ => ‘url’)) where backend can be ‘local_server’ or ‘amazon’ (for now)

• Users as back-end clients (individual local servers) and Roles as different types of back-ends (local server, Qfile).Also roles/users for human admins.

• NodeQueue to track files (as nodes) as they are synchronized with various back-ends (except amazon default,which is uploaded from Site Server and is the base for all other back-ends) and base for user interface to controlsync, etc.

• provide url information for files to SS for url re-writing. Determines which back-end to use, based on requestingcomputer IP (maybe other factors? leave open for future).

2. “Caching Client” module to be installed on Site Server (SS) – communicates with Central Server.

• requests url information for files

• updates CS with latest uploaded files (as MM action)

• implements two-stage file uploading (for video, maybe other filetypes in the future)

3. Local Servers (LS) – simple black box, serve files locally and communicates with Central Server

• transcodes locally uploaded files

• creates static version of site

• solid build, small boxes

• large as possible internal hard drive (2TB? 4TB?)

• multiple ethernet?

• e-sata or other high-speed drive connection

• Debian stable

• Apache, PHP5

• scripts for receiving files, communicating with CS, transcoding, etc.

• scripts for static cache of site

• automount attached e-sata drive

• easily deployable to local servers (could be CD, other)

26 Chapter 7. Specifications


7.1.4 Functional outline – local servers example

1. When a Local Server is plugged in and has internet access, contacts the Central Server with a token generated forlocal servers (all such contact is made using XML-RPC): cs.newcache($token, ‘local_server’) The CS creates a useraccount for the LS, assigns it to the LS role, and returns to the LS its auth credentials (username and password combo).The CS creates a Download node queue, an Upload node queue, and a Files node queue for the LS. The Downloadnode queue tracks all the files to be downloaded to the LS and is filled with all existing active File nodes. The Uploadnode queue tracks the files being uploaded to the Site Server from the LS. The Files node queue tracks the files thatcan be streamed from the LS. These queues can all be manipulated by logged-in admins.

2. On a regular cron cycle (ex. 5 minutes) the LS contacts the CS and sends its auth and LAN and WAN IP:cs.location($auth, LAN_IP, WAN_IP) The CS stores (or updates) these with the LS username, a unique ID and thecurrent timestamp in a “Local Servers” table. The CS removes any stale combos. (When a regular user contacts theSite Server, their WAN IP is used to find the LS in their LAN and URLs are rewritten to this LS.)

3. When a file of a particular type – video SD, video HD, images at various sizes, audio – is uploaded to the SiteServer, a node of type “File” is created on the CS which stores the unique files table filepath from the SS and its type(eg. video) and option (eg. high or low). This node is added to the bottom of all Download node queues (unlessalready present in the Files queue, due to LS conversion). cs.announce($auth, $filename, $type, $option)

5. A LS contacts the CS and is given a url for the file at the top of its Download queue. cs.download($auth, ‘get’)When it has downloaded the file it contacts the CS, the node is added to its Files queue. cs.download($auth, ‘put’,$filename, $type, $option)

6. When a computer contacts the SS its WAN IP is searched for in the “Local Servers” table on the CS.cs.setBackend($auth, $WAN_ip, $session) If found, the LAN IP or IPs are returned to the browser with a JavaScriptwhich looks for the Local Server at the IPs. If found, the CS is alerted and the LS IP and unique ID is stored in thebrowser’s Session. cs.setBackend($auth, $WAN_ip, $session, $local_server) It will be removed from the Session if itis stale-dated in step 2.

7. Download behaviour – when computers on LS networks request files: If requested files (video, audio, images, etc.)are present in the Files node queue for the local server, then the URL to the file is rewritten using the local IP andsent to the SS for page generation. If there is no local server or the file is not in the Files queue, then the URL maybe re-written in other ways (as an RTMP stream from CloudFront, for example.) cs.getURL($auth, $filename, $type,$option, $session)

8. Upload behaviour – when computers on LS networks upload (video) files: Uploading (creating content) will be atwo-stage process. A node creation form creates the content node on the SS, and then a second step is returned to theuser for uploading the file. The file will be uploaded to the LS, and these steps take place:

1. LS contacts SS (or upload script) with name of file and NID (perhaps other form data, like token)and file is created:

• dummy file in files directory, to avoid name collisions

• entry in content field and files table

• somehow Media mover harvesting is inhibited.

2. LS adds file to Upload node queue for LS cs.upload($auth, ‘put’, $filename, ‘video’, ‘high’)

3. when file is uploaded it replaces dummy file and media mover harvesting is enabled. File conversionproceeds as normal.

4. LS converts video file to hi-res formats. LS contacts CS which adds video to Files queue for the LS(will not be downloaded to this LS when converted on SS). cs.download($auth, ’put’,$filename, ’video’, ’high’) This upload process will be implemented for video but maybe adapted to other content (audio, images) if necessary.

9. LS periodically crawls SS to create a static version of the site – urls to media files are the locally cached versions.

7.1. Overview 27


10. Admin accounts can be created on CS allowing admins to alter queue order and remove files from queues. Alsocontrol bandwidth allocation for upload and download.

11. Specially configured hard drives can be connected to LS through high-speed (e-sata) connections. LS will auto-matically mount hard drive and process can be monitored from admin interface. New download files can be copied toLS and will be removed from Download queue and added to Files queue. New upload files can be copied from LS tohard drive and be removed Upload queue (and added centrally when hard drive returns to main office).

7.2 Hardware possibilities

7.2.1 Servers

There are many different classes of system to choose from, depending on exactly how bad an environment these thingshave to withstand. Predictably, there is a direct relationship between price and reliability. Systems listed below arefrom least to most reliable. This is not an exhaustive list of systems or candidates, but will be altered based on feedbackfrom the client until we know exactly what kind of system they need.

7.2.2 Consumer

The Acer Aspire Easystore H340 is a home storage server that normally comes with Windows Home Server. Linuxalso runs on it. If we add 2x2TB hard drives, we have a 5TB system for about $710.

Pros: Cheap Simple Low power Front panel lights

Cons: Requires adapter for keyboard/mouse/video No linux drivers for front panel lights M$ tax Fans may inhaletoo much dust Processor may not be powerful enough for transcoding Swapping out the server for a HP MediaServerwould give us a much more powerful (dual-core pentium) system for $1010 total. This removes the problems withtranscoding, but all other pros/cons remain the same.

7.2.3 Server

We could build a custom system for about $1000 around a case like this:http://ncix.com/products/?sku=37520&vpn=CSE-733TQ-645B&manufacture=SuperMicro

Pros: Cheap Commodity parts means spares are easy to find

Cons: Time-consuming to put together systems from scratch rather than buy pre-made Fans may inhale too much dust

We could also custom-order a server from a big-name manufacturer: A Dell PowerEdge 110 with 4TB would cost$2,769. An HP ProLiant 310 g5 with 4TB would cost $2,953.00

Pros: No assembly required Excellent support

Cons: More expensive Non-commodity parts Fans may inhale too much dust

7.2.4 Industrial

This ARK-3440 system by Advantech is fanless and extremely rugged, and so would hold up very well. No price islisted, but judging by similar systems, it probably costs around $2000.

The slightly older ARK-3420-S and ARK-3420-S1 have Celeron and Core 2 Duo processors and are and are $1080and $1550, respectively.

Pros: Fanless - no danger of dust or smoke damage Robust


http://ncix.com/products/?sku=37520&vpn=CSE-733TQ-645B&manufacture=SuperMicro


Cons: Expensive Limited storage space (2x2.5” laptop drives). This gives us a max of 2TB of HDD or 1TB of flash.Non-commodity parts Edit Milspec God knows how much these cost. Manufacturers don’t publish prices.

They also don’t generally offer dust-proof systems with a lot of storage. Most sealed systems have 1 or 2 2.5” bays,like the above industrial system.

7.2.5 Interface

For machines that don’t have a front panel display, or have one that’s not accessible from Linux, we can use somethinglike The CW1602 This is a 16x2 LCD with 6 buttons and is supported on Linux by LCDproc or lcd4linux software.

7.3 Auto-mounting Hotplugged Devices

7.3.1 udev

udev is a user-mode system that triggers on device insertion or removal, and can be used to change device names in/dev or perform other, more complicated tasks via shell scripts.

Scripts are triggered by matching rules .

The following rule should match any USB drive plugged into the system and run a script to automount it: KER-NEL==”sd*”, DRIVERS==”usb-storage”

This rule matches any SCSI drive: KERNEL==”sd*”, DRIVERS==”sd”

This matches an IDE drive: KERNEL==”hd*”, DRIVERS==”ide-disk”

Annoyingly, these don’t match on remove, only on insertion. Removing the DRIVERS== clause will make themmatch on both.

So, we use these rules to match on insertion and removal of any USB or SCSI disk:

ACTION=="remove", KERNEL=="sd*" RUN+="/home/debian/driveremoved.sh $kernel"ACTION=="add", KERNEL=="sd*" RUN+="/home/debian/driveadded.sh $kernel"

This is the driveadded.sh script that automounts the drives and copies files off them: source:automount/driveadded.sh

And this is the driveremoved.sh script that unmounts the drive when unplugged, then deletes the mountpoint:source:automount/driveremoved.sh

7.4 Synchronization scripts

This is an outline of the scripts that will synchronize files between local servers and the site server (in pseudocode).They run periodically on each local server, launched by cron, at intervals of 5 minutes.

If this causes too much load on the server, it would be a good idea to stagger the cron jobs so they start at differenttimes, or put a random delay at the start of the script.

Stale lockfiles will be deleted from within the same script (rather than through a cron job as has been consideredearlier).

7.3. Auto-mounting Hotplugged Devices 29


7.4.1 Upload script

if there is a lockfile in /var/run:if it is stale, delete itelse exitcreate lockfile in /var/run to indicate upload is runningGet name of file at head of upload queue from cs.upload xmlrpcif there is no file to download:delete lockfileexitcall rsync to copy the file to the site serverif transfer succeeded:do nothingelse if transfer failed:re-add file to end of upload queue using cs.upload

7.4.2 Download script

if there is a lockfile in /var/run:if it is stale, delete itelse exitcreate a lockfile in /var/run to indicate download is runningGet name of next file to download from cs.download xmlrpcif there is no file to download:delete lockfileexitcall rsync to copy the file from the site serverif transfer succeeded:do nothingif transfer failed:re-add file to end of download queue using cs.download

7.5 Test scripts

7.5.1 Introduction

The Local Server software is bundled with a series of test scripts which wrap the XML-RPC communication functionsbetween the local server and the central server. A comprehensive runthrough of all test scripts is outlined below. Thisseries of tests can be run using the testall script located in the local_servers/sync directory.

7.5.2 Testing process

The individual test scripts may be called individually to test a single function if desired. They are:

• addfile.php - Creates a new file node on the Central Server and adds it to all local server download queues.

• addtoqueue.php - Adds an existing file node to a queue. It is an error if the node does not exist.

• createfilenode.php - Create a new file node, suitable for using in addtoqueue or similar.

• getamazonurl.php - Given the filename, type and options of a file, get the URL for its location on Amazon S3.

• getconfig.php - Retrieve the server’s recordedconfig info from the server, including LAN/WAN IPs and band-width limits.



• getfullqueue.php - Get all published and unpublished files in the named queue.

• getqueue.php - Get all published files in the named queue.

• getunpublished.php - Get only unpublished files in the named queue.

• isinqueue.php - Tests whether or not a named file is in a queue.

• movetoqueue.php - Move a file from one queue to another.

• removefromqueue.php - Remove a file from a queue and delete its file node.

• reservefilename.php - Obtain a name for a new file upload that does not conflict with other files already presenton the server.

• testreplace.php - Given a filename and an extension, create a new filename with the given extension on the end.

In addition the following production scripts are in the same directory:

• deleteunpublished.php

• download.php

• setconfig.php

• transcode.php

• upload.php

7.6 How to set up kiosk mode

1. Configure firefox and init scripts according to these instructions: http://jadoba.net/kiosks/firefox/

2. Add further config changes suggested in description and comments here: https://addons.mozilla.org/en-US/firefox/addon/1659/

3. Disable screen blanking and power saving by adding the following to ~/.xinitrc:

xset s 0 0xset -dpms

4. Comment out the startkiosk.sh line in /etc/rc.local

5. Add the following lines to /usr/local/bin/startkiosk.sh right above the “su” line:

echo "Press enter to start kiosk mode."read

6. Add this at the end of /etc/inittab:

7:23:respawn:/sbin/getty -n -l /usr/local/bin/startkiosk.sh 38400 tty1

I highly recommend not using these firefox “kiosk mode” instructions without more research and testing. In practicewe found this setup very unstable. The much simpler setup currently described in the Setup Guide (plain xorg andiceweasel with flash player) works much better. Even better might be the standalone flash player set to the requiredplaylist, with no browser at all... John

7.6. How to set up kiosk mode 31

http://jadoba.net/kiosks/firefox/

https://addons.mozilla.org/en-US/firefox/addon/1659/

https://addons.mozilla.org/en-US/firefox/addon/1659/



CHAPTER 8

Changing this document

This documentation is formatted with ReST and managed by the Sphinx documentation engine.

You can generate the documentation locally with:

make html

You will need to have GNU Make and Sphinx installed for this to work.

Changes should be pushed to the central git repository at:

[email protected]:isuma-doc.git

This requires write access to the repository, which can be requested at [email protected].

Alternatively, a read-only access to the repository is available at:

git://git.koumbit.net/isuma-doc.git

The documentation is also automatically generated on Readthedocs.org when new content is pushed to the sourcerepository.

33

http://sphinx-doc.org/rest.html

http://sphinx-doc.org/

https://redmine.koumbit.net/projects/isuma-doc/repository

http://isuma-media-players.readthedocs.org/en/latest/installation.html

Documents

Media Players Documentation