 
DAISY Pipeline 2.0 Project Charter [Draft]
Last revised February 5, 2010
Table of Contents
DAISY Consortium members and content producers have to keep up with an ever increasing set of digital content formats, all the more with the new revision of the DAISY standards and the rapid growth of EPUB usage in the consumer market. The demand for economic and efficient automated production tools is still very high.
While the DAISY Pipeline first released in 2008 has so far been a successful answer to the needs of the DAISY community, new standards and technologies have emerged and can be leveraged to reduce the maintenance cost, lower the learning curve for new adopters, and increase the interoperability with the heterogeneous community needs.
The DAISY Pipeline 2 project will introduce a fundamental redesign of the original technical solution, which will enable a more robust development of new functionality. It intends to be a long term investment to achieve an enterprise-grade production system for accessible content.
The outcome will benefit all DAISY Consortium members involved in automated content production, as well as commercial companies wishing to take advantage of the open source and liberally licensed deliverables.
In the long term, it does not make economic sense to maintain our own version of the Pipeline core engine since standards with the same scope are now available. The adoption of existing standards (and off-the-shelf implementations of those standards) at the heart of the Pipeline framework will allow us to:
An emerging standard that will particularly play a central role in the Pipeline 2 redesign is XProc. XProc is a W3C specification for an XML Pipeline Language describing operations to be performed on XML documents. It is developed by the W3C XML Processing Model Working Group, and is expected to become a W3C Recommendation in 2010.
XProc has a certain numbers of key features that are particularly relevant in the Pipeline context:
Because of the huge amount of overlap between their fundamental concepts, it is a natural choice to use XProc as the primary language for describing processing workflows in the DAISY Pipeline.
Based on the Board's decision that new core projects will not be started before the financing is secured, the Board of Directors of the DAISY Consortium decided in Seoul (June 2009) to follow a standardized Project Development Cycle for major projects and which consists of the following steps (see Minutes of the Board Meeting in Seoul, Items 9, 10, 49a, and 49b):
- Outline of a Project submitted by the management and approval by the Board for further development of the Project;
- Call for Participation (the members are invited to participate in the development of the corresponding Project Charter);
- Together with the finalized draft Charter a Call for Support is sent out to the membership; it asks for human and/or financial commitments;
- Submission of the final charter to the Board for final approval; it includes the budget and the agreements between the DAISY Consortium and the organizations regarding their commitments
The Outline for the Pipeline 2 project was approved by the Board at its meeting in Seoul (see Minutes of the Board Meeting in Seoul, Items 20 and 49f).
The Call for Participation was sent out by e-mail to the Board and the membership in general on July 13th 2009.
The present draft Charter was edited by Romain Deltour integrating comments from Markus Gylling, Bernhard Heinser and George Kersher. Furthermore, the following individuals reviewed the present charter or contributed to the overall inception effort: Kathleen Asjes (Dedicon), Jostein Austvik Jacobsen (NLB), Christian Egli (SBS), Ian Greenow (RNIB), Joel Håkansson (TPB), Kenny Johar (Vision Australia), Arne Kyrkjebø (NLB), Peter Osborne (RNIB), Gerald N. Schmidt (The Open University).
This DAISY Pipeline 2 project essentially consists of two main tasks:
- the technical redesign of the Pipeline core framework
- the development of new processing functionality
The fundamental framework redesign coupled with the critical need to provide a reference implementation for the DAISY standard revision are very ambitious objectives that demand a strong development effort. To get the best from the available resources, the project objectives are hereafter separated in two main development phase: the first phase focuses on a limited set of requirements towards a usable release and the preparation of the second phase through two dedicated working groups (mainly for TTS and Braille production), while the second phase consists in more advanced framework changes and processing functionality. Although the second phase is outlined in the present charter, its concretion will highly depend on the outcomes of the work done by the two working groups.
Note: because one of the main objective of the project is to create an open framework for the creation of modular processing functions, the charter cannot obviously list all the future contributions explicitly. This charter only mentions the items of particular strategical importance or developed principally by the DAISY Consortium staff.
The first phase encompass the fundamental objectives towards a first usable release. It focuses on the minimal framework redesign required to achieve a first reference implementation of the DAISY standard revision. It also includes the preparatory work required to plan the second phase, such as the creation of dedicated working groups for the development of Braille or TTS-based production.
The first necessary step in the adoption of XProc as the central language for the description of the processing pipelines is for the core developers team to get familiar with the new concepts and technologies.
- Getting started with XProc: the core developers need to acquire some expertise with the XProc specification. They will have to read some documentation and tutorials, and experiment with existing XProc processors and their built-in samples.
- Development of simple XProc prototypes: the learning of XProc and its standard step library needs to be consolidated by the development of small XProc pipelines (e.g. performing basic XML manipulation).
To reduce the initial development cost, an existing open source XProc processor will most probably be reused at the roots of the core Pipeline 2 engine. However, some adaptation work still needs to be done to fully achieve the project requirements, notably in terms of integration within a larger software system.
- Selection of an XProc engine: the engine to use at the core of the DAISY Pipeline 2 needs to be carefully selected amongst the existing third-party XProc implementations. The choice will depend on feature sets, ease of adaptation and licensing compatibilities.
- Development of a basic engine wrapper: the developers need to get familiar with the API of the selected XProc engine in order to be able embed it in the DAISY Pipeline core framework.
- Adaptation of the message/event system: some programming hooks needs to be developed to receive notifications on the execution status and user-oriented messages.
- Integration of extended parameter information: the XProc languages allow pipelines to be configurable via options or parameters, but some language extensions are required for documentation purposes and to improve the accuracy of the representation of these configuration options in a user interface.
The DAISY Pipeline project is fundamentally a very modular project where all the processing functions are split in dedicated components. Some overarching principles are needed to ensure a cohesive architectural design, which is essential to increase the developers productivity and and reduce the maintenance cost.
- Definition of a modular organization for processing components: the DAISY Pipeline is by essence a very modular project composed of an evolving set of dedicated processing components, some rules are therefore required to keep a well-organized structure that is easy to extend and maintain.
- Definition of guidelines for the handling of user or application settings: configuration files are often used to refine the behavior of Pipeline components, the establishment of common practices for the declaration and manipulation of such settings will help in providing a coherent and consistent user experience.
- Development of guidelines and utilities for localization: guidelines and utilities built in the core framework will help the developers of Pipeline components to solve the common need for localization.
The scope of the XProc language does not entirely cover the needs of the DAISY Pipeline project. Fortunately, it is possible to extend the language and tools to cover a broader functional domain. While the processing functions included in the first phase are mostly based on built-in XProc features, it is essential to plan future extensions early in the development to get a complete overview of the technical challenges. It will help to take the right architectural decisions for the next development phase.
This preparatory work basically consists in a set of prototypes. It can be based on a simplified scenario of TTS-based DTB production, which is an ideal candidate encompassing many of the foreseen technical challenges.
- Prototype adaptation of the XProc engine to a service-oriented architecture: a modular and complex project such as the DAISY Pipeline can only be easy to extend and maintain if special care is given to the underlying software architecture. A service-based framework (such as OSGi for Java) can greatly help us to achieve this goal, but some development work is required to adapt state-of-the-art XProc engines to such a framework.
- Development of prototype XProc processor-specific steps: functionality not included in the standard XProc feature set (such as TTS processing) needs to be implemented as extensions to the XProc processor, the development of prototype extensions will give us the expertise required for the development of the second phase.
- Development of prototype XProc steps calling a Web Service: Web services, based on standards and platform-neutral protocols, and are a key solution to interoperability issues ; early prototypes are a first step towards the integration of external services (TTS, metadata repository, etc) as native components within the Pipeline framework.
- Development of prototype XProc steps for complex file sets manipulations: because DAISY book are composed of a set of files including not only of XML documents but also other data files (images, audio, etc), the Pipeline needs to allow the easy manipulation of such file sets. Prototypes will help to adapt the XML-centric XProc framework to this requirement.
The processing functions targeted in the first phase of the Pipeline 2 project intend to provide a first reference implementation of the forthcoming Z39.86-2010 Authoring and Interchange Framework specification. By supporting the "DAISY 4" revision early in the lifespan of the standard, we lower the entry threshold and speed up the adoption rate of said standard. This work needs to be harmonized with the redesign of the core framework and will mostly involve built-in XProc functionality which do not require the advanced extensions planned in the second development phase.
This first implementation of the Z39.86-2010 Authoring and Interchange (ZedAI) framework consists in a set of XSLT-based transformations allowing the conversion of ZedAI documents to a set of popular XML-based formats (and possibly the other way).
- DTBook to ZedAI: conversion of a DTBook (DAISY XML) document to a Z39.86-2010 Authoring and Interchange (ZedAI) document.
- ZedAI to DTBook: conversion of a Z39.86-2010 Authoring and Interchange (ZedAI) document to a DTBook (DAISY XML) document.
- ZedAI to XHTML: conversion of a Z39.86-2010 Authoring and Interchange (ZedAI) document to an XHTML document.
- XHTML to ZedAI: conversion of an XHTML document to a Z39.86-2010 Authoring and Interchange (ZedAI) document.
- XHTML to XHTML: conversion of a flavor of XHTML to another version, supporting XHTML 1.0, XHTML 1.1, XHTML5.
- ZedAI to EPUB3: conversion of a Z39.86-2010 Authoring and Interchange (ZedAI) document to an EPUB file set.
- possibly ZedAI to DocBook5: conversion of a Z39.86-2010 Authoring and Interchange (ZedAI) document to a DocBook5 document.
A new validator for Z39.86-2010 Authoring and Interchange documents is being developed by the DAISY standard revision working group. This validator tool will be integrated in the DAISY Pipeline 2 framework to allow its inclusion into any document processing workflow.
- Integration of the new ZedAI validator as a Pipeline component: the integration must allow to finely configure the validator from a processing pipeline and to easily retrieve and process the validation results.
The integration of DAISY Pipeline 2 functionality in the heterogeneous infrastructural contexts that constitute the current DAISY community needs to be seamless and easy to implement. A new set of user interfaces must be developed, sometimes based on standard protocols. They can increase the framework uptake in both developed and developing countries and improve the interoperability with other parties such as mainstream publishers.
The development of a command line interface will provide a simple way to call the DAISY Pipeline 2 functionality from any of the major operating systems (Windows, Mac OS X and Linux). A command line tool is easy to use and adopt and can even play a role in a basic integration of the Pipeline in a more complex system.
The development of a Java API (Application Programming Interface) will provide a finely grained access to the entire DAISY Pipeline 2 functionality in a third-party Java application. A clear and well-documented API is also the entry gate to the learning of the framework by new adopters.
The development of a simple Web Service API will provide a standard and platform-neutral way to access the DAISY Pipeline 2 functionality. Based on (such as SOAP/WS-* or REST), it allows the fine integration of the Pipeline independently of the programming language used by the host application, thus achieving a truly interoperable system.
TTS-based DTB production and Braille production are particularly popular requirements in the DAISY community. Because of their strategical importance, it is proposed to create special working groups dedicated to the development of these tasks.
The establishement of a Braille in DAISY working group must ensure a unified approach towards the uncontested objective to have Braille as a result of the production workflow. Despite the delicate political and heterogeneous technical requirements, the collaborative DAISY Pipeline project offers a great opportunity for this group to integrate the driving forces and lead a systematic approach.
During the first phase of the project, the objectives of this group are to:
The TTS-based production of DAISY content is clearly the most widely used feature of the DAISY Pipeline. The DAISY Pipeline 2 project is the right opportunity to develop a redesigned TTS-based production facility to better meet the users needs, integrate with recent standards (SSML, PLS), and support the new DAISY revisions.
The establishement of a TTS-based production working group is proposed during the first phase, in order to:
Note: Because of the almost unanimous use of the feature amongst the existing Pipeline adopters and its central role as a test-bed for the core framework redesign, the development of this work item will be led by the DAISY Consortium.
The second phase of the project consists in the finalization of the framework redesign along with the development of advanced production functionality.
The DAISY Pipeline 2 core framework is the common basis for all the processing functions. It must be both complete and flexible enough to allow the easy implementation of all the planned and future developments.
The modular organization already started in the first phase must lead to a dynamic and extensible component architecture.
In order to ever increase the productivity and maintainability, the service-based approach will be adopted in most part of the framework. The primary objectives is to facilitate the development of processor-specific step extensions.
- Adapt the XProc engine to a service-based OSGi runtime (OSGi is the de facto standard for a dynamic and service-oriented module system for Java).
- Allow the definition of XProc steps as OSGi services
- Finalize the definition of a plugin architecture for the framework
The integration of existing third-party services in the production chain is of utmost importance to achieve a truly interoperable framework. The framework must facilitate the development of processing steps based on web-service, and provide some sample code to lower the learning curve for new adopters.
- Develop the adaptive code required to the use of web services as XProc steps
- Develop sample code for calling a web service from the DAISY Pipeline
Advanced production functionality such as DAISY book production requires support for complex file set manipulation. The framework must define a standard and common way to represent a file set throughout the workflow and provide a comprehensive step library for manipulating it.
- Define a standard representation for file sets in XML
- Develop a comprehensive library of file set manipulation functions
- Develop sample pipelines using the aforementioned library
The processing functions targeted in the second phase can leverage more advanced framework possibilities. They include the objectives of the TTS-based production and Braille in DAISY working groups as well as the progressive migration of some Pipeline 1 functionality to the new Pipeline 2 framework.
The primary purpose of the TTS-based production working group is to deliver a reference implementation of the Z39.86-2010 Part B (Distribution) specification.
- ZedAI to ZedDist Audio Profile
- ZedAI to ZedDist Classic Profile
The details of this work item will be refined by the Braille in DAISY working group during the first phase.
It is relevant to port proven and widely used Pipeline 1 functionality to the new framework, so that it can be more easily integrated in Pipeline 2 workflows. Although it can appear an additional work to achieve an already existing functionality, the long term benefit in terms of reuse and maintenance cost far outweighs the initial development effort.
The scripts that could ultimately be ported are:
Note that the Narrator script is not listed here because it will be entirely redesigned by the dedicated TTS working group.
The development of the lightweight graphical user interface (GUI) will provide a user-friendly way to use the DAISY Pipeline 2 framework. The user interface must be simple and easy to use, consisting in a few configuration dialogs and progress and result reports. Additionally, it should be embeddable in a third-party application ("Save as DAISY"-plugins, DAISY authoring tools, etc.).
In addition of the command line UI and the lightweight GUI, more advanced user interfaces can be developed to drive the Pipeline 2 functionality, including:
The details of this work item depends on the interest shown by the project stakeholders and will need to be refined with a charter renewal.
The Web Service API developed in the first phase will be improved and finalized into a more comprehensive API. The final API must allow finer grain access to the Pipeline functionality, including on-the-fly notification of progress changes and messages (possibly via web service call backs).
In addition of the Java API and Web Service API, other wrapper APIs could be developed to allow native calls of the Pipeline 2 runtime from a foreign environment (such as .Net or Python).
The details of this work item depends on technical viability and the interest shown by the project stakeholders and will need to be refined with a charter renewal.
The proposed project lead is Romain Deltour (DAISY Developer), reporting to Markus Gylling (DAISY CTO).
The project lead will supervise the core framework redesign and take special care of ensuring that the development of each separate processing function is harmonized with the overall architecture choices.
The development of processing functions will be handled by sub-groups with dedicated and optimized skill sets. Each group will have a group leader coordinating the work with the main project lead. The efficient management is based on close cooperation between the group leaders and the main project lead.
The development will follow an iterative plan, with each iteration adding refinements and new features to the software product. The objective is to deliver a usable application early in the development phase, which will the be frequently updated.
Conference calls will be scheduled weekly for the core developer group, and upon request for sub-groups members and external contributors.
The first and second phase of the project will be carried out successively during a 3-year time frame. A side objective is to coordinate the roadmap with the formal release of the ANSI/NISO DAISY Z39.86-2010 revision as a NISO standard in mid 2011.
The project will span over 42 months (3.5 years) and focus on delivering iterative releases on a regular schedule (every 6 months), with additional intermediary milestones for the finalization of each phases.
The following lists the major milestones:
Note that the budget presented in this charter only corresponds to Milestones 1 to 3. A call for support for the second phase is expected to be sent in Spring 2011, with the goal of being officially approved by the board in October 2011.
This present charter describes only the budget for the first phase of the project (17 months). We cannot, at the time of this writing, establish an accurate estimated budget for the second phase as the scope of this latter will mostly depend on the decisions and preparatory work carried out in the first phase. Note that a charter extension and associated call for support are expected in 2011 to further secure the second phase of the project.
The budget hereafter detailed shows requirements for both developer resources (estimated in Man-Month) and direct costs (in USD).
| 1. Developer resources |
Cost in Man-Month |
| A1. Fundamental Framework Redesign |
30 |
| A2. Processing Functions |
25 |
| A3. Deployment Options |
6 |
| TOTAL |
61 MM |
| 2. Direct Costs |
Cost in USD |
| Hardware and Software |
5k |
| Outsourced Work (Technical Writing, Proof Reading, Documentation, Manuals, Testing etc.) |
35k |
| Project Management and Administration (600h - including preparation of phase 2, coordinating TTS WG and Braille in DAISY WG) |
50k |
| F2F (2 meetings with 5 persons during 2 days each) |
20k |
| Communication and Marketing (preparation of presentations, attendance at conferences, other activities) |
15k |
| Training and Technical Support (webinars, production of a training video, etc.) |
15k |
| Miscellaneous |
5k |
| TOTAL |
145k USD |
The DAISY Consortium will contribute a total of 22 Man-Months for the 17 months of the first phase, distributed amongst two DAISY staff developers (Marisa DeMeglio and Romain Deltour).
All the results of this project should be available royalty free and under a liberal licensing scheme that will encourage reuse of software components in other products (either commercial or open source).
As usual with open source developments, it must be noted that licensing decisions can have a strong impact on the the project. The reuse of existing open source libraries, to reduce the overall development cost, can only be viable if the licenses of these dependencies are compatible with the license chosen for the final deliverable. In an extreme scenario, the impossibility to use a library due to licensing incompatibilities can lead to major additional development costs which may jeopardize the budget and timeline defined in the present charter.
To minimize the risk of legal dead-end, the licensing decisions will be taken early in the project development phase and will require the input of both developers and legal professionals.
It is strategically important to underline the widely collaborative aspect of the DAISY Pipeline project. The DAISY Consortium is the umbrella organization that maintains and promote the project, but pieces of functionality are expected to be contributed by third-party organizations. In this context, it must be noted that while the DAISY Consortium will support the core framework it cannot systematically allocate resources to the maintenance and further development of organization-specific contributions. These tasks are rather considered as part of the commitments of the said organization.
Because the Pipeline 2 includes a fundamental redesign of the Pipeline 1 framework, it is worth mentioning that both solution will co-exist for a significant amount of time. External contributions based on the new framework are expected to progressively increase and eventually outnumber the contributions to the Pipeline 1, but this duality will initially put high demand on the DAISY Consortium staff for technical maintenance and support.
The Enabling Technologies Framework is a three year project funded by WIPO and endorsed by the Stakeholders Platform of WIPO. Its general objective is to evolve mainstream publishing processes to enable the production of fully accessible digital content.
The DAISY Pipeline 2 will play a major role as one of the work package is entirely dedicated at extending the DAISY pipeline to support the requirements of publishers and conversion houses. This notably involve the participation of the publishing community in a requirements gathering phase.
The project will be jointly run by both the DAISY Consortium and EDItEUR (the international group coordinating development of the standards infrastructure for electronic commerce in the book and serials industries) over a three year time frame. While the project details are not fully available at the time of this writing, they are expected to smoothly integrate within the objectives and timeline described in this charter.
| Threat |
Probability |
Strategy |
Action |
| The license of a desired dependency is incompatible with the project license |
Medium |
Acceptance |
Either decide to change the project license or integrate some additional development effort in the budget and timeline |
| Resources are insufficient to achieve the project objectives |
Medium |
Mitigate |
As the framework stabilizes during the 2010 project phase, we will have a better measure of what the development costs will be. The level of possible further development will depend on additional funding. |
| Contributor withdrawal leads to discontinuation of certain functionality |
Medium to High |
Acceptance |
Given the nature of the project as a framework/host for multiple transformation types, it is a natural risk that certain functionality is discontinued when and if there is no continued community support for it. Under normal circumstances, DC staff does not take on the task of maintaining functionality originally committed by other contributors. This general rule does not apply to functionality of extra high long term strategic importance, for which DC staff may step in to assure longevity and stability. |
| Uptake in commercial sector does not meet expectations |
Low |
Mitigation |
Uptake in commercial sector can be stimulated through dedicated outreach efforts. |
|