SETTING UP A NEW EXTERNAL GRABBER
------------------------
The internet epg software can only use xmltv-data for which
a "content" record has been defined in dbChannelMap.dat.
You can do this as follows (see rest of this readme for more details)
-If you run the software on relook, first turn on swap:
swapon /dev/ide/host0/bus0/target0/lun0/part1
-configure the grabber of your choice. For instance:
/opt/bin/tv_grab_se_swedb --configure
-create a channel list
/opt/bin/tv_grab_se_swedb --list-channels --output /tmp/channels.xml
-import the channel list into dbChannelMap.dat
/var/media/epg/software/channelMap.php --import-xmltv-channels=/tmp/channels.xml --db=/var/media/epg/software/data/dbChannelMap.dat --grabber=tv_grab_se_swedb
-If you haven't done so already, import satellite channels from your db.dat
to dbChannelMap.dat. Currently, dbChannelMap.dat contains data for
Hotbird, Astra, Astra 2 and Eurobird. If you receive other satellites
import their channels as follows
/var/media/epg/software/channelMap.php --import-relook=/var/data/db/dat --db=/var/media/epg/software/data/dbChannelMap.dat
-You may need some of the other modes (--gues-by-name, --map-channels etc..)
to associate the satellite channels correctly with the xmltv content.
-finally run:
epg.php --setup
OVERVIEW
----------
The channelMap database is meant to define globally unique
mapping of sattelite channels to specific "content". it also
provides a similar mapping of EPG-data sources to specific content.
With satellite channel we refer to a specific video-stream (with one
or more related audio streams) broadcast on a specific frequency on a
specific transponder of a satellite (or on a specific terrestrial
frequency).
With "content", we refer to what ordinary people would typically
call a "tv-channel", i.e., a set of programs broadcast under a certain
name. However, we avoid the word "tv-channel" in this document.
For example, the content "julie" is broadcast on three different
channels on hotbird at the time this document was written.
In the channelMap database, content is identified by a 32-character
string that looks like a random string. The table SatChannelInfo links
this identifier to one or (usually) more channels. This allows an epg
program to discover that the 3 julie channels broadcast the same
content and can share the same EPG-information.
The table ProgramSourceInfo contains similar data, but for epg-data
sources. For example, the content called "ZDF" by tv-viewers (more
accurately, the content identified using a very specific 32-character
string, let us say 01234567890ABCDEF01234567890ABCDEF) epg-data can be
obtained in several different ways. For instance, through the xml-tv
grabber probrams tv_grab_de_tvtoday under the name 'zdf.de' and using
the grabber tv_grab_fr under the name
'C44.telepoche.com'. Furthermore, grabbers can be built that grab
EPG-information for this content from teletext. Using
ProgramSourceInfo an EPG-system can look up where it can get
information about the content 01234567890ABCDEF01234567890ABCDEF.
Of course, defining a database is one thing, but getting all the
information for it is another. We assume that in practice many people
will work together to insert correct information into the database
and keep it uptodate. The database structure and the program to update
the database was designed to facilitate this cooperation. It is
essential that the following conventions are respected, both by people
that insert information in the database using SQL-statement, or by
programs that update the database
-the same content should always be indentified by the same
32-character Content_ID. Never use a new Content_ID except when
you discover new content
-Never use different Content_ID's for different content. It may be
tempting to associate 'BBC1 England' and 'BBC1 Scotland' with the same
Content_ID, because they broadcast the same programs most of the time
and you prefer to use the same EPG-data for both contents. However,
other people might not agree with this, so you cannot pass your
"corrections" on to other people, who may actually consider them
"errors". If you wish to use the same EPG-info for 'BBC1 England' and
'BBC1 Scotland' then you should handle this at the level of the
EPG-program (through a config file), not at the level of the database.
-for new content, a new Content_ID must be generated. It is very
important that you choose a RANDOM Content_ID for new
content. Otherwise you run the risk that someone else in another
country used the same Content_ID for different content. Again, this
would prevent the exchange of channelMap data and render the database
useless. The preferred way to generate ramdon ID's is using the
md5sum() algorithm, applied to a random input.
-In practice, there will be more channels for which no epg-data can
be found than channels for which grabbers exist. For this reason,
Content_ID's should only be defined for content for which at least
one epg-source exists. In the database, this means that in
SatChannelInfo only Content_ID's are allowed which also occur in
ProgramSourceInfo. All other Channels should have Content_ID=NULL.
Database update programs can overwrite NULL values when they discover
new epg-sources, even if they are not 100% sure that the source
matches the channel. On the other hand, they should be very careful to
overwrite any other value.
-Since multiple epg-sources may exist for the same content, but not
all of them are equally good, the programSourceInfo table has a
Priority column for each content/epg-source combination. EPG-programs
should normally use the source with the highest
priority. Nevertheless, it is very useful to have more than one source
per content, just in case the best epg-source fails (it happens a
lot!).
-SatChannels and ProgramSources also have a human-readable "Name". On
the one hand, this is handy when using SQL-commands to view or update
the database; on the other hand the names can also be used by programs
to "guess" the correct Content_IDs for channels. It is safe to change
a name into a more meaningful one (e.g., remove rubbish characters).
The channelMap program is a program that updates the database. It
recognizes the following commands
*importing channels from an xmltv-grabber's output
--import-xmltv=<xml-file> --grabber=<grabbername> --priority=<priority>
xml-file is a file produces by, e.g. tv_grab_fr. It is
essential to specify the grabbername (in the example: tv_grab_fr),
because this will allow you to share your database with other people.
Setting a priority is only recommended when you have some reason to
believe that the grabber is better or worse than others.
Note that channelMap.php makes some assumptions, e.g., that
programsources with the same xmltv "channel-id" or with the same
display-name are for the same content. Usually these assumptions are
correct, but you can use other commands to correct any errors.
*Importing channels from the relook db.dat database
--import-relook=<db.dat> [--update-names]
This searches db.dat for new channels, identified by Satellite
position, transponder frequency and polarisation, Network_ID and
Program_ID (the information is somewhat redundant; in principle
Network_ID and Program_ID should be sufficient, but in practice they
are not). New channels are inserted into the channelMap database. Old
channels are NEVER overwritten.
NOT IMPLEMENTED: With the option "update-names" the
"Name" of the channels are overwritten with the ones in db.dat but all
other information is kept. This may be useful when channels initially
are named "test 123" and later get a real name. However,
"update-names" is probably to dangerous as it overwrites ALL names of
all channels
*guessing associations between channels and content.
--guess-by-name
This command examines all channels with Content_ID=NULL. It then looks
for programsources with similar names and if it finds them, it uses
their Content_ID. For instance, if a satellite channel named "zdf"
exists and tv_grab_de_tv provides an xmltv-channel named "ZDF" then it
associates "zdf" with the same content as "ZDF". While this works in
many cases, it will also introduce quite a few errors (e.g., may
channels are called "tv1", because the broadcasters lack imagination
when choosing a content name).
It is recommended to use --guess after adding programsources or
satellite channels. You can then correct the errors using other
commands.
--guess-by-name is safe because it will never overwrite existing information.
The "guessed" column in SatChannelInfo indicates whether the
Content_ID field was updated by guessing or by a more clever method.
Important note: currently "guessing" is implemented as follows:
names of channels and programsources are normalised by converting them
to lower case and eliminating spaces and certain characters. Two names
match when they have the same normalised name.
*correcting channel mappings
--map-channels=<file> [--dbdat=db.dat] [--guess] [--overwrite]
file contains a number of lines in one of the following formats:
'zdfd' => 'xmltv:tv_grab_de_tvtoday:zdf.de'
Map all channels with name 'zdfd' in SatChannelInfo to the Content_ID
of xmltv:tv_grab_de_tvtoday:zdf.de
507 => 'xmltv:tv_grab_de_tvtoday:zdf.de"
Map the channel with number 507 on your relook (column ch_order in
ChannelInfo of db.dat) to the Content_ID of xmltv:tv_grab_de_tvtoday:zdf.de
Of course, you must provide --dbdat=... for this to work.
Note the absence of quotes around the number 507. If you use
quotes, the mapping is for the channels named 'to7' rather than
for the channel with number 507.
507 => 'ecccd44dbc666232e60467698642db43'
Same as above, but the right-hand side is interpreted as a Content_ID.
'zdfd' => NULL
507 => NULL
This can be used to delete existing mappings so that they can be
guessed afterwards.
'xmltv:tv_grab_uk_rt:uk.scifi.com' != 'xmltv:tv_grab_de_tvtoday:premscifi.tvtoday.de'
This can be used when distinct contents have been mapped to the
to the same Content_ID by mistake. uk.scifi.com is assigned a
new Content_ID and all existing SatChannels are now linked to
premscifi.tvtoday.de
Note: if filename="-", then data is read from standard input instead
of from a file
The --guess option, when present, not only updates the channels with
the exact name "zdfd" but also channels with similar names, such as
"zdf-d" or "zdf d".
The --overwrite option, when present, will update ALL matching
channels. Without it, only channels with Channel_ID=NULL are updated
*NOT IMPLEMENTED correcting programsource information
--map-sources=file
file contains lines in the following formats:
xmltv:tv_grab_fr:telepoche.com=xmltv:tv_grad_detvtoday:zdf.de
This indicates that the two program sources refer to the same
content. The Content_ID of the left source is replaced with the
Content_ID of the right source in all tables. The information
is also recorded in a table "ContentAliases"
xmltv:tv_grab_fr:telepoche.com!=xmltv:tv_grad_detvtoday:zdf.de
This indicates that the two program sources refer to DIFFERENT
content. A new Content_ID is generated for the left source.
The information is also recorded in a table "ContentAliases".
SatChannelInfo is NOT modified
xmltv:tv_grab_fr:telepoche.com=NULL
This indicates that the record xmltv:tv_grab_fr:telepoche.com
should be removed from the database. If there is no other record
in ProgramSourceInfo with the same Content_ID, then not only
the program source is removed, but also the corresponding content.
This is done by setting Content_ID=NULL for all sattelite channels
that have the same Content_ID as the deleted
xmltv:tv_grab_fr:telepoche.com program source
*NOT IMPLEMENTED importing records from another channelMap database.
--import-channel-map=foreigndb.dat
New programsources and new satellite channels
are copied from foreigndb.dat to channelMap.dat.
Also, if satellite channels have Content_ID=NULL in channelMap.dat
and a non-zero Content_ID in foreigndb.dat, the Content_ID is updated.
Note that importing will never correct erroneous mappings in
channelMap.dat. It only inserts new data
Database updates
-----------------
If you update dbChannelMap.dat, then please share your updates with
the community. At a later stage, the channelMap.php program will allow
importing data from one dbChannelMap.dat database to another and perhaps
even logging of changes.
For the time being, please extract new records from the ProgramSourceInfo
and SatChannelInfo tables in dbChannelMap.dat as sql statements (if you
know how to do that), or make your database public and let us know
where to find it on the relookplaza.nl or satellites.co.uk relook forums.