This directory contains Python libraries, mostly related to checkpkg.

== TODO items ==

- populating allpkgs based on an existing catalog tree
- Testing uploads (locally)
- Auth for users on the buildfarm
- Unit tests (unit test data)
  - what about callbacks?

== Private Buildfarm ==

Notes:

- When running a RESTful web app, /tmp is used to store temporary files.
  Therefore, if you're uploading a large file and your system has little space
  in /tmp, the server side process will fail with an error. Make sure that the
  /tmp directory has about 1GB of free space.
  If you increase the swap size, /tmp will grow too.
  References:
  http://docs.oracle.com/cd/E19963-01/html/821-1448/ggrln.html
  http://www.c0t0d0s0.org/archives/5815-Less-known-Solaris-features-The-curious-case-of-the-tmp-in-Solaris.html

== Checkpkg ==

Checks to implement:
 - *dev(el)? -> error, suggest *-dev
 - *-?rt -> error, suggest specific library packages
 - empty package without 'transitional' in the name --> error, suggest
   'transitional'
 - CSWpmfoo --> error, suggest CSWpm-foo
 - Dependency on a transitional package --> error
   ('transitional', 'stub', 'legacy')
 - Dependency on CSWcas-initsmf + rc* files --> error
 - A package must not be incompatible with itself
 - If a pkgmap entry has username or group outside the default list of users,
   the file's class needs to be 'ugfiles'
 - If there is bin/*-config and there is lib/(sparcv9|amd64)/.*, there must
   also be bin/(sparcv9|amd64)/*(-|_)config
 - Binaries from /opt/csw/bin and/or sbin should not have the same names
   as any binaries from /usr/bin and/or sbin.

Checks implemented:
 - foo_bar != CSWfoo-bar -> error
 - outside /opt/csw, /etc/opt/csw, /var/opt/csw -> error

Development plan for checkpkg:
  Primary focus:
  - Allow to create a package database on a machine with 1-1.5GB of RAM
    (e.g. for virtual machines)
  - Add fields to the srv4_file_stats table:
    - source URL (for grouping by software)
    - Description (to search for the word 'transitional')
    - Whether the package is obsolete
  - Don't suggest two packages for the same soname.
  Additional, nice to have:
  - Allow maintainers to opt-out from notifications
  - Display stats from each run
  - Shorten the on-screen output, add commands to display override lines
  - Move the set check stats outside of checking functions, remove the special
    status of dependency checking functions; add a progress bar for it.
  - Sort all list data structures so that it's possible to diff the results of
    pprint.pprint() and see meaningful results.  This will be the new
  - Restructure the error reporting, group them by errors.
    implementation for comparepkg.

Also, see ticket list on trac: http://sourceforge.net/apps/trac/gar/report/1

Items done:
- Move the 'data' field of the srv4_file table to a separate table (should
  speed up checking if stats are already collected)
- Store run history
- Generalize dependency checking by adding NeedFile(file_list, reason) to
  error_mgr.  It's possible to need one of the listed files only, so files are
  given as alternatives, but the reason is common.
- Notify maintainers when their package is available from mirrors
- Add support for the 'overridden' field in the database
- Support for relocatable packages
- Database objects garbage collection

Known problems:
- libmagic fails sometimes when processing the whole catalog

Package dependencies:

It's possible to develop checkpkg on a non-Solaris platform, using unit
tests as means to run various bits of code.  Here's the dependency list
for Ubuntu.

sudo aptitude install \
  python-cheetah \
  python-magic \
  python-mox \
  python-mysql \
  python-progressbar \
  python-pycurl \
  python-sqlobject \
  python-unittest2 \
  python-webpy \
  python-yaml

Seem unpackaged: elftools from
http://eli.thegreenplace.net/2012/01/06/pyelftools-python-library-for-parsing-elf-and-dwarf/

The progressbar package in older Ubuntu versions (12.4) is too old and does not
have the progressbar.widget submodule.

Solaris package dependencies:

CSWap2-modwsgi
CSWapache2
CSWpy-cheetah
CSWpy-curl
CSWpy-libmagic
CSWpy-mox
CSWpy-mysql
CSWpy-progressbar
CSWpy-pyelftools
CSWpy-sqlobject
CSWpy-unittest2
CSWpy-webpy

===Checkpkg database===

Additional database indexes:

# TODO(maciej): Add them to the models.py

CREATE INDEX srv4_mtime_idx ON srv4_file_stats(mtime);
CREATE INDEX srv4_md5_idx ON srv4_file_stats(md5_sum);
CREATE INDEX catalog_idx ON srv4_file_in_catalog (arch_id, osrel_id, catrel_id);

A query showing how many packages there are in each catalog.

SELECT
  a.name as arch, o.short_name as osrel, c.name as catrel,
  s.use_to_generate_catalogs, count(*)
FROM
  srv4_file_in_catalog as sit,
  architecture as a,
  os_release as o,
  catalog_release as c,
  srv4_file_stats as s
WHERE
  sit.srv4file_id = s.id
  and
  sit.arch_id = a.id
  and
  sit.osrel_id = o.id
  and
  sit.catrel_id = c.id
GROUP BY
  a.id, osrel_id, catrel_id, s.use_to_generate_catalogs
ORDER BY
  c.name
;
