View
215
Download
2
Category
Preview:
Citation preview
The ]po[ Data Model Frank Bergmann, 2006-05-22
This guide contains overview information that is useful if you want to write reports or if you need to extract information from the API.
It is not ment as a developer guide. Please check the OpenACS manual for more detailed information.
SoftwareDevelopment
Templates
Trans-lation
CRMFinance
Controlling
HR
System
Collaboration,Content & KM
OOFrame Security
Calendar
OpenACSPermission
Web Server
Database
FinanceBase
Payroll
Skill Database
Filestorage
OnlineDiscussions
IncidentWorkflow
BasicAuthentication
LDAPAuthentication
WorkflowEngine
Chat
PackageManager
AutomaticSoftwareUpdates
PostgreSQL Oracle 8i, 9i, 10g
PageContracts
SQLTemplates
OO Model
ObjectMetadata
LocalizationFramework
ContactMgmt.
ReportingEngine
Wiki
Oracle Intermedia/TextTSearch2
Linux Solaris BSDWindows+ CygWin
Mac OSOperating
System
SearchEngine
AOLServer PoundRevers Proxy
DynField Object Extensions
PlatformServices
Profiling & Performance
DebuggingSystem
ApplicationModules
FormBuilder
Portal &Components
Mail ServerIntegration
ContentManagement
ApplicationServices
DB-APITCL
Quotes &Invoice
Payments
FinancialReporting
Mail ServerIntegration
CustomerWeb Reg.
MarketingCampaigns
CRMTracking
AutomaticAudits
ISDN TelIntegration
ProjectMgmt.Project &Subprojects
ProjectControlling
TranslationWorkflow
TMIntegration
FreelanceInvoicing
TimesheetInvoicing
AvailableDocumentation
TimesheetMgmt. AutomaticInvoicing
]po[ Overview
FinanceGuide
Unix InstallGuide
OpenACSInst. Guide
Operations& Maint.Guide
OpenACSDeveloperGuide
ConfigurationGuide
FilestorageGuide
ForumGuide
TranslationWorkflow
Guide
Full-TextSearch
AutomaticTesting
BigBrotherSys Mgmt.
OtherRoomReservation
E-Commerce
CMS
WebDAV
SOAP &XML-RPC
Surveys
GlossaryWeb-Mail
Blog
RecruitingWorkflow
CVS
Postfix/Sendmail
Database Replication
MondrianData-
Warehouse
Contents
Other Documentation General Conventions "Categories" ]po[ Objects and Types Main Classes Financial Classes Auxilary Tables
Other Documentation
This is a limited overview document. For more detailed information please see:
]po[ Overview Diagram: http://www.project-open.org/ OpenACS Developer Guide: http://www.openacs.org/doc/ ]po[ Whitepapers: http://www.project-open.com/whitepapers/ ]po[ List of modules:
http://www.project-open.org/product/modules/ ]po[ Developer Documentation: http://www.project-
open.org/doc/
General System Architecture
]po[ consists of three main parts:– A powerful object-relational data model– TCL pages that render the contents of the
database and– TCL libraries that provide auxilary functions
to the TCL pages The Data Model is particularly important
in ]po[ because:– It is object-oriented– The TCL layer does not contain objects. It
merely renders the content of the Data Model The Data Model is the location where you
need to go if you want to extract information from the system
Data Model
TCL Library
TCL Pages
General Conventions
Tables names are always in plural form and written in lower case Tables are held in general in 2nd normal form Denormalized table columns are named with a "_cache" postfix. All main tables have an integer primary key <table_name>_id.
This primary key references the table acs_objects which contains the type of the object.
Most object->object mappings are handleled by the "acs_rels" table and its subclasses (see below). Other mapping tables are called something_something_map
Boolean fields have a "_p" postfix and contain 't' or 'f' char values
All main objects include "object_type_id" and "object_status_id" fields. "Type" is used to create minor subclasses of the objects, such as the distinction of im_companies such as "customer" and "provider". "Status" is used to deal with the object's lifecycle, such as "potential", "active", "closed" and "deleted".
Normal arrow means referential integrity relation-ship between tables.
Big arrow tip means inheritance relation-ship. An inheriting "object" references its parent class with its object_id.
Data Model Overview – Business Objects
Most data in the ]po[ data model are structured as “Business Objects” (BizObj).
A “BizObj” is just a database table following a number of conventions
– A integer primary key called <object>_id representing a unique object ID
– The primary key references the “acs_objects” table. acs_objects includes object meta-information such as object type, creation date, creation user etc.
– Every ]po[ BizObj includes a field “object_status_id” re
object_idobject_type
acs_objects
project_id
im_projects
project_lead_id
project_name
project_status_idproject_type_id
supervisor_id…
project_nrcompany_id
Data Model Overview – Business Objects
The ]po[ data model is based on the notion of “Business Objects”
A “BizObj” is just a database table following a number of conventions
– A primary key representing a unique object identifier
– The primary key references the “acs_objects” table. acs_objects includes object meta-information such as object type, creation date, creation user etc.
– Every ]po[ BizObj includes a field “object_status_id” re
object_idobject_type
acs_objects
project_id
im_projects
project_lead_id
project_name
project_status_idproject_type_id
supervisor_id…
project_nrcompany_id
category_id
name
im_categories
description
category_type
persons
person_idfirst_nameslast_name
rel_idobject_id_one
acs_rels
object_id_two
persons
person_idfirst_nameslast_name
Data Model Overview – Business Objects
The ]po[ data model is based on the notion of “Business Objects”
A “BizObj” is just a database table following a number of conventions
– A primary key representing a unique object identifier– The primary key references the “acs_objects” table.
acs_objects includes object meta-information such as object type, creation date, creation user etc.
– Every ]po[ BizObj includes a field “object_status_id” re
Hierarchical Queries
Main Project
Sub-Project 1
Sub-Project 1.1
Sub-Project 1.2
Sub-Project 2
Sub-Project 2.1
Sub-Project 2.2
"Categories"
Conventional/ traditional database designnormally includes a lot of foreign keytables defining the type and status of an object.
]po[ takes a different approach in order tominimize the number of database tables andmaintenance screens. We use a single"im_categories" table for all types of status andtype information:+ The total number of DB tables is reduced to a
third or fourth+ A single maintenance screen is responsible
for managing categories.+ Built-in features:
+ Localization+ Hierarchical categories+ Common GUI widgets
+ Referential Integrity is enforced- It is possible to assign the wrong Category to
a field+ This has never happened in practice yet.
object_type_idobject_status_id
object
object_type_idname
object_type
description
object_status_idname
object_status
description
namedescription
...
object_type_idobject_status_id
object
namedescription
...
category_id
name
im_categories
description
category_type
The "Classical" DB-Design: Every table has it's own tables for type, status and similar information.
]po[ DB-Design: All type and status information is stored in a single "im_categories" table.
parent_category_id
im_category_hierarchy
child_category_id
Categories
Select Everything About Categoriesselect c.*, im_category_from_id(aux_int1) as aux_int1_cat, im_category_from_id(aux_int2) as aux_int2_cat, h.parent_id, im_category_from_id(h.parent_id) as parent from im_categories c left outer join im_category_hierarchy h on (c.category_id = h.child_id) where c.category_type = 'Intranet Cost Types' order by category_type, category_id;
Sample SQL
Categories Definition-- We use categories as a universal storage for business -- object states and types, instead of a zillion of tables -- like 'im_project_status' and 'im_project_type'.
create sequence im_categories_seq start 100000;create table im_categories (
category_id integerconstraint im_categories_pkprimary key,
category varchar(50) not null,category_description varchar(4000),category_type varchar(50),category_gif varchar(100) default 'category',enabled_p char(1) default 't'
constraint im_enabled_p_ckcheck(enabled_p in ('t','f')),-- used to indicate "abstract"-- super-categorys that are not -- valid values for objects.-- For example: "Translation -- Project" is not a
project_type, -- but a class of project_types.
parent_only_p char(1) default 'f'constraint im_parent_only_p_ckcheck(parent_only_p in
('t','f')));
Category Hierarchy-- Optional system to put categories in a hierarchy.-- This table stores the "transitive closure" of the-- is-a relationship between categories in a kind of matrix.-- Let's asume: B isa A and C isa B. So we'll store-- the tupels (C,A), (C,B) and (B,A).-- This structure is a very fast structure for asking:---- "is category A a subcategory of B?"---- but requires n^2 storage space in the worst case and-- it's a mess retracting settings from the hierarchy.-- We won't have very deep hierarchies, so storage complexity-- is not going to be a problem.
create table im_category_hierarchy ( parent_id integer constraint im_parent_category_fk references im_categories, child_id integer constraint im_child_category_fk references im_categories, constraint category_hierarchy_un unique (parent_id, child_id));
Extract Categories Without Join-- A helper functions to make our queries easier to
readcreate or replace function im_category_from_id
(integer)returns varchar as 'DECLARE p_category_id alias for $1; v_category varchar(50);BEGIN select category into v_category from im_categories where category_id = p_category_id;
return v_category;end;' language 'plpgsql';
-- Example:-- select im_category_from_id(48);Create a New Category Entryinsert into im_categories (
category_id, category, category_type,category_description, enabled_p,aux_int1, aux_int2,aux_string1, aux_string2
) values (:category_id, :category, :category_type,:category_description, :enabled_p,:aux_int1, :aux_int2,:aux_string1, :aux_string2
);
]po[ Objects and Types
All major ]po[ tables are "Objects" Objects are identified by an object_id. Information about object type is stored in the
acs_objects table, together with metadata and creation information.
Advantages of "being" an object: Permission system:
Associates objects with "privileges", including all necessary maintenance screens
SQL metadata system: Allows the administrators to dynamically extend object types with new attributes
Full-Text search:Indexes all objects and produces generic URL for the search results
Configurable workflow:Allows to manage the status of any object. The WF includes a graphical WF editor.
Generic Object Relations:acs_rels allows you to define generic relationships between objects.
object_type_idobject_status_id
object
namedescription...
object_idobject_type
acs_objects
acs_object_types
object_typesupertype
table_nameid_column
name_method
attribute_id
table_name
acs_attributes
column_name
object_typepretty_name
]po[ Objects and Types
acs_attributes:– Each row in acs_attributes table defines an attribute of the specified object
type.– The contents of the table is managed by the "intranet-dynfield" package that
contains a SQL metadata editor. acs_privileges:
– Contains permissions "tokens" (=> privileges). These privileges allow a user to perform a certain operation in the system.
acs_permissions:– Defines a mapping between acs_objects and acs_privileges.– The mapping is per user group ("grantee_id"). Groups can be hierarchical.
Main Classes
object_id
im_biz_objects
rel_idobject_id_one
acs_rels
object_id_two
object_idobject_type
acs_objects
group_idgroup_name
groups
profile_idprofile_gif
im_profiles
users_contact
<Home Address><Work Address>
im_employees
<Payroll Info><Recruiting Info>
im_freelancers
<Freelance Info>
parties
party_idemailurl
persons
person_idfirst_nameslast_name
users
user_idusernamepasswordsaltauth_token
company_id
im_companies
main_office_id
company_namecompany_status_id
company_type_id
primary_contact_idaccounting_contact_id
notevat_number
office_id
<Address>
im_offices
office_status_idoffice_type_id
company_idproject_id
im_projects
project_lead_id
project_name
project_status_idproject_type_id
supervisor_idproject_budget
budget_hoursbudget_currency
project_nrproject_pathcustomer_id
descriptionnote
start_dateend_date
Main Classes
im_biz_objects:– This abstract class defines a number of functions to manage "horizonal
permissions" (=> Please see other ]po[ documentation) for project, companies and offices
– "Horizonal" permissions currently include roles such as "Project Manager" of "Full Member".
parties, persons, users:– These three tables form an inheritance hierarchy. There is a view "cc_users"
that unifies the information from these tables. im_freelancers, im_employees, users_contact:
– These are "extension tables" for "persons" and define additional information for users with particular profiles.
im_profiles:– This table defines the groups that are relevant for ]po[, as opposed to
potentially hundereds of groups of the underlying OpenACS system.
Financial Classes
Cost TypesAll costs have an entry in im_costs, plus:
Financial Docs(im_invoices)
– Customer Invoice– Quote– Provider Bill– Purchase Order– Delivery Note
Simple Costs(only im_cost)
– Timesheet Cost– Employee Salary
Expense Item(im_expenses)
Expense Bundle(im_expense_bundles)
cost_center_idcost_center_name
im_cost_centers
cost_center_codecost_center_type_idcost_center_status_iddepartment_p
im_costs
cost_idcost_name
cost_nr
project_idcustomer_idprovider_id
investment_idtemplate_id
cost_status_idcost_type_id
effective_date_idpayment_days
amountcurrency
paid_amountpaid_currency
taxvat
descriptionnote
cost_center_id
invoice_idcompany_contact_id
im_invoices
invoice_nrinvoice_office_id
item_iditem_name
im_invoice_items
project_idinvoice_id
item_unitsitem_uom_idprice_per_unitcurrencysort_orderitem_type_iditem_status_iddescription
payment_idcost_id
im_payments
company_idprovider_idreceived_datepayment_type_idpayment_status_idamountcurrencynote
object_idobject_type
acs_objects
Inheritance!
expense_idext._company_name
im_expenses
ext._comp._vat_num.billable_preimbursablebundle_id
Financial Classes (2)
The Problem:Cost and Projects can have a N:M relationship: One project may have more then one invoice/cost item. One cost item may “relate” to more then one project (in particular: project + sub-project, but also
the item may be split between two main projects). An invoice may actually not belong to any project at all.
Solution: In general, the relationship between projects and costs is defined as an N:M relationship by acs_rel. im_costs.project_id points to im_projects ONLY in the case that one cost item relates to exactly one
project
Watch out: Cost Items can be related to a main-project or to a sub-project or any level. So in order to calculate
the main project’s total Profit & Loss, you need to sum up the financial items through the full project tree (hierarchical query)
im_costs
cost_idcost_name
cost_nr…
project_id…
im_projects
rel_idobject_id_one
acs_rels
object_id_two
Relationship between Costs and Projects
project_id
select distinct c.cost_name, im_category_from_id(c.cost_type_id) as cost_type, c.amount * im_exchange_rate(c.effective_date::date, c.currency, 'EUR') as amount_convertedfrom im_projects main_p, im_projects sub_p, acs_rels r, im_costs cwhere main_p.project_id = :main_project_id and sub_p.tree_sortkey between main_p.tree_sortkey and tree_right(main_p.tree_sortkey) and r.object_id_one = sub_p.project_id and r.object_id_two = c.cost_id
A sample query to extract all financial items related to a certain main-project.
Financial Classes (3)Example
project_id…
im_projects
project_idparent_id
im_projects
project_idparent_id
im_projects
…
…
im_costs
cost_idcost_name
cost_nr…
rel_idobject_id_one
acs_rels
object_id_two
project_id
Main project
Sub-project with cost item
Sub-project with cost item
Financial Classes
im_costs:– This is the main table for all cost items.– All financial elements in the system are stored in this table.
The table also receives updates from timesheet (cost related to users working on a project) etc.
im_cost_centers:– Is currently not used. The table has been included in the
current datamodel for smooth upgrade in future versions. im_invoices:
– This table contains financial documents such as "Invoices", "Quotes", "Bills" and "Purchase Orders" (the name "im_invoices" is a bit misleading).
im_invoice_items:– These are the individual lines of an invoice.
Relationship Classesobject_idobject_type
acs_objects
rel_id
im_biz_object_members
object_role_id
rel_id
acs_rels
object_id_oneobject_id_two
rel_type
rel_id
membership_rels
member_staterel_id
admin_rels
rel_id
composition_rels
rel_id
group_rels
rel_typegroup_id
Defines the relationships between an im_biz_object and some users. The membership type corresponds to "horizontal permissions" and usually includes "full member", "project manager", "key account" and user-defined roles.
rel_type
acs_rel_type
object_type_tworole_one
object_type_one
role_two
min_n_rels_onemax_n_rels_onemin_n_rels_twomax_n_rels_two
Relationship Classes
acs-rels:
im_biz_object_members: – Defines the member of a im_biz_object and their roles (project
manager of full member). – Such a membership has an impact on the access rights of members to
the business object. membership_rels: Defines group->supergroup relations
The acs_rels table is essentially a generic mapping table for acs_objects. Once we come up with a way to associate attributes with relationship types, we could replace many of the ACS 3.x mapping tables like user_content_map, user_group_map, and user_group_type_modules_map with this one table. Much application logic consists of asking questions like "Does object X have a relationship of type Y to object Z?" where all that differs is X, Y, and Z. Thus, the value of consolidating many mapping tables into one is that we can provide a generic API for defining and querying relationships. In addition, we may need to design a way to enable "type_specific" storage for relationships (i.e., foreign key columns for one-to-many relationships and custom mapping tables for many-to-many relationships), instead of only supporting "generic" storage in the acs_rels table. This would parallel what we do with acs_attributes.
Translation Classesobject_idobject_type
acs_objects
im_costs
im_invoices
im_trans_invoices
price_id
uom_id
im_trans_prices
company_idtask_type_idsource_lang_idtarget_lang_idsubject_area_id
currencypricenote
im_trans_tasks
task_idtask_name
task_filename
project_id
source_language_idtarget_language_id
invoice_idquote_id
task_status_id
task_type_id
task_unitsbillable_units
trans_idedit_id
proof_idother_id
match_xmatch_repmatch_100
match_95
task_uom_id
match_85match_75match_50
match_0
end_date
note
subject_area_id
report_id
allowed_errors
im_trans_quality_reports
task_idreport_datereviewer_idsample_size
total_errorscomments
report_id
im_trans_quality_entries
quality_category_idminor_errorsmajor_errorscritical_errors
object_id
im_trans_trados_matrix
match_xmatch_repmatch_100match_95match_85match_75match_50match_0
Translation Classes
im_trans_tasks:– This tables contains the description of the main activity related to translation
projects.– Includes information about Translation Memory repetitions– Inludes information about the translator, editor, proof-reader and "other"
assigned to perform this task (static workflow) im_trans_invoices:
– This is currently just a flag (binary information) to identify invoices that have been created based on im_trans_tasks. Translation invoices are referenced by im_trans_tasks.invoice_id in order to make sure that every trans_task is included in exactly one invoice. These references need to be removed if a translation invoice is deleted.
im_trados_matrix:– Contains the discounts per customer and translation task type for different
Translation Memory matches.– The trados_matrix associated with the "internal" company contains the site-
wide defaults for customer translation prices.
Timesheet Classesobject_idobject_type
acs_objects
im_costs
im_invoices
im_ts_invoices
material_id
description
im_materials
material_namematerial_nrmaterial_type_idmaterial_status_idmaterial_uom_id
task_id_one
im_ts_task_deps.
task_id_twodependency_type_iddifferencehardness_type_id
project_id
im_projects
project_name
...
customer_id
start_dateend_date
task_id
invoice_id
im_timesheet_tasks
material_iduom_idplanned_unitsbillable_unitscost_center_id
priority
price_id
uom_id
im_timesheet_prices
company_idtask_type_idmaterial_id
currencypricenote
project_type_id
Consulting Classes
im_timesheet_tasks:– This tables contains the description of the main activity related to projects.
im_timesheet_invoices:– This is currently just a flag (binary information) to identify invoices that have
been created based on im_timesheet_tasks. Timesheet invoices are referenced by im_timesheet_tasks.invoice_id in order to make sure that every task is included in exactly one invoice. These references need to be removed if a timesheet invoice is deleted.
im_timesheet_prices:– Contains a price per material and customer. – The timesheet_prices associated with the "internal" company contains the site-
wide default price list.
Workflow
Workflow
]po[ uses a Petri-Net based workflow A workflow consists of “Places”
(states) and “Transitions”, linked by “Arcs” that might carry “Guard” expressions.
WF “Transitions” can be linked to pages acting on object’s fields
Transition
Transition
Place
Start-Place
Place
End-Place
[Guard] [Guard]
WF @ Runtime
During runtime, a similar structure is created representing a specific “Case” (a specific instance of a WF):
– The “case” corresponds to a WF in action.
– Transitions become “tasks”, with specific assignments to users.
– Places may include “tokens” that move through transitions to other places. A single token in a WF place can be thought as “the WF is in status XXX”.
Petri-Nets allow for more then one token and more then one token per place, but this option is rarely used in ]po[.
Case
Task
Task
Place
Start-Place
Place
End-Place
[Guard] [Guard]
StaticAssignments
Workflow Case
WF Tables
place_key
wf_places
workflow_key
transition_key
wf_transitions
direction
workflow_key
wf_workflows
case_id
wf_cases
workflow_keyobject_id
task_id
wf_tasks
case_idtransition_key
transition_key
wf_arcs
place_keydirection
token_id
wf_tokens
case_idplace_key
role_key
wf_roles
workflow_key
state
state
state
workflow_keyrole_key
TransitionStates
WF “Callbacks”
“Callbacks” are Pl/SQL databaseprocedures that determine the behaviorof a transition: A number of “Outgoing Events” allow
to customize the behavior of WF transitions:
– Enable: Is called when the transition is enabled (a token is placed in one of its input places)
– Unassigned: Is called if the transition hasn’t been statically (see below) assigned to a user or a group.
– Fire: Is called once the transition gets fired
A number of transition’s parameters can be determined by external routines:
– Time:– Deadline:– Hold Timeout– Notification
Enable
Start
Finish Cancel
Enabled
Started
Finished Canceled
<autom>
“Enable”
“Fire”
“Fire”
Time
Deadline
Hold Timeout
Notification
“Unassigned”
TransitionStates
WF Assignments
“Static Assignments”:– Specified during the definition of the WF– Specified using WF “roles”. Each role can be
assigned to one or multiple parties (groups or users)
– Specified as “roles” using wf_transition.role_key and then mapped to parties using wf_context_assignments.
– The specific assignments at runtime are held in wf_task_assignments.
– This construction might seem a bit complex, but it is necessary, as you will find out when defining your first real WFs.
“Dynamic Assignments”– Use the “Unassigned” callback to define the
assigned parties.– This option allows you to specify the assignee
as a function of object characteristics etc.– The results of dynamic assignments are
stored in wf_task_assignments.
Enable
Start
Finish Cancel
Enabled
Started
Finished Canceled
<autom>
Timesheet & Controlling
Controlling uses caches per project in order to maintain aggregated Profit & Loss (P&L) information.
The project caches are updated via triggers Reporting on portfolios & customers is handled
by the reporting engine.
CustomerGroup
SubProject
MainProject
Customer“Timesheet”
Task
“Forward” PropagationCosts are cached. Caches are updated via triggers
“Backward” ReportingValues are summarized via Reports
Requirements and Architecture
Timesheet & Controlling
Projects contain a set of “xxx_cache” fields that contain the aggregated cost elements per type.
CUD (create, update, delete) operations on costs (timesheet hours, expenses, …) does NOT update the caches, for performance reasons. Instead, triggers reset the “cost_cache_dirty” to NULL.
There is a “sweeper” process that periodically checks “cost_cache_dirty” and updates the cache. This might be a nightly activity, or every few minutes.
project_id
im_projects
…
cost_quotes_cachecost_invoices_cache
cost_timesheet_planned_cachecost_purchase_orders_cache
cost_bills_cachecost_timesheet_logged_cachecost_expense_planned_cache
cost_expense_logged_cachecost_delivery_notes_cache
reported_hours_cache
cost_cache_dirty
“Sweeper”Process
Controlling cache architecture
Timesheet & ControllingControlling cache architecture
user_id
im_hours
project_idday
hoursbilling_rate
note
cost_idconf_object_id
billing_currrency
im_costs
cost_id…
conf_id
im_ts_conf_objects
conf_project_idconf_user_id
start_dateend_date
conf_type_idconf_status_id
Project reportedhours
cost_tslogged
cache_dirty
Main Project (9)
Sub Project1 (4)
Task1.1 4
Task1.2
SubProject2 (5)
Task2.1 3
Task2.2 2
The hours are aggregated from tasks up to main
projects
between
conf_project_id refers to the main
project. All sub projects and tasks
are included.
Auxilary Tables
GUI Define what "plugins" (these grey boxes) should appear
on what page
Defines the hierarchical menu structure of the system
Define the columns for ListPages in the "core". The ListPages need to be extensible at runtime because they might have to accomodate new columns from add-on modules
The list of all countries in the world
The list of all currencies
View
view
ViewColumn
Currency
Country
im_component_plugins
Menu
Auxilary Tables
GUI Define what "plugins" (these grey boxes) should appear
on what page
Defines the hierarchical menu structure of the system
Define the columns for ListPages in the "core". The ListPages need to be extensible at runtime because they might have to accomodate new columns from add-on modules
The list of all countries in the world
The list of all currencies
View
view
ViewColumn
Currency
Country
im_component_plugins
Menu
Recommended