27
Auler Black Paper Copyright (C) 2015 Redhuan D. Oon THIS IS AN IROSES* SPECIAL PAPER PUBLICATION INTENDED FOR ACADEMIC AND RESEARCH PURPOSES. FREE DISTRIBUTION ALLOWED AS LONG AS ACCREDITATION AND EXPRESSED WRITTEN PERMISSION OBTAINED FROM THE AUTHOR

Auler Black Paper - red1red1.org/BlackPaper.pdf · iROSES Auler Black Paper 5. The live day to day operations of the client’s modified iDempiere 1.0c is still patchy with continuous

Embed Size (px)

Citation preview

!Auler Black Paper!

! !!Copyright (C) 2015 Redhuan D. Oon !

THIS IS AN IROSES* SPECIAL PAPER PUBLICATION INTENDED FOR ACADEMIC AND RESEARCH PURPOSES. FREE DISTRIBUTION ALLOWED AS LONG AS ACCREDITATION AND EXPRESSED WRITTEN PERMISSION OBTAINED FROM THE AUTHOR

iROSES Auler Black Paper

Auler Black Paper*!Study on actual damage done to a live software project

!By:!Redhuan D. Oon!ADempiere founding leader and iDempiere community leader!(Paper mostly prepared while in Tokyo, JAPAN)!!Under the auspices of client user:!A!For suffering through a disaster brought about by:!B!(names removed to protect the innocent, and guilty)!July 15, 2015!version 0.5 (initial release)!*iROSES is a new e.V I been trying to setup and A is now helping me achieve that.!*Auler is a German word for light, something which A has finally seen. Karma bless A. But I cannot speak for the failing B who I pray will repent and return some of the money to A. Then his/her Karma is restored. !This is called a black paper as opposed to a white paper that commercial minds tell lies with. Instead this black paper tells the truth. Do not take my word for it. Read it and see.

Copyright (C) 2015 Redhuan D. OonPage � of �2 27

iROSES Auler Black Paper!!!

Table of Contents!!!!

!!!!

How It All Started! 4! Technical Background! 7!

Modifying Code! 7!

Best Practice! 8!

OSGi Framework! 9!

Changes in Plugins! 9!

The Modified Code! 11! Hard core PO class! 14!

Overriding Core! 16! Event Validators! 16!

Callouts! 16!

Extending Model! 17!

Core Model Changes! 18! MOrder ! 18!

MOrderLine! 19!

Extend More! 20! Window Multi-Tabs! 21!

Important Maxims! 24

Copyright (C) 2015 Redhuan D. OonPage � of �3 27

iROSES Auler Black Paper

How It All Started!Sometime in May this year, I was called upon by the client, A, for the purpose to recover, and analyse work done by their previous IT contractor hereby referred to as B. The client has engaged B around mid 2012, as B presented themselves as experts to implement their requirements onto an ERP software suite named ADempiere (fork of Compiere) which is open source and public domain under GPL law. !

I was previously, separately contacted by B around middle of 2013 for advice due to my active role in iDempiere (fork of ADempiere) as they wanted to know whether to switch to iDempiere and I did so on a friendly basis without been aware who exactly the client is. B did not request me to participate throughout the few months later they communicated with me via follow up text messaging and emails requesting only limited technical advice. It was to my surprise when A finally contacted me about this case. It took me some time to absorb all of it. This opens up an opportunity for me to record on paper once and for all, a documented proof of what I been trying to explain as the biggest issue with ERP implementation, is that it is highly dependent on people and skills which is not free. When wrongly done it can cost more.!

A filled me in on all details of the project, inviting me for an alternative opinion whether they are correct in having eventually concluded that B aren’t actually experts in their work, which fell below par and not of the competency and outcome they expected. A’s relationship with B eventually broke down towards end of last year and after a few more attempts to obtain the services of other advocates of the project, which also failed, they finally got to me. I accepted their offer because I hate reality TV.!

According to the client, their negative perception is due to their claims that:!

1. The software has new bugs introduced into the code-base by B’s work when there were none before, but not admitted by B.!

2. The modified software was not ready on time but very late, with little room for end user preparation. !

3. The final delivery for live installation at client’s location was still full of bugs and have to be lodged by A to B who has no idea what the cause of them are. A also has to spend a lot of their own time to resolve such issues.!

4. B incurred a series of overruns in timed costs of which they billed and collected from A about EURXXX,XXX (six figures) which was surpassing the initial budget estimate. Yet the project was not operating in full, at full with important modules not operational at all.!

Copyright (C) 2015 Redhuan D. OonPage � of �4 27

iROSES Auler Black Paper5. The live day to day operations of the client’s modified iDempiere 1.0c is still patchy

with continuous fire-fighting by the internal software engineer such as memory leaking where the server is restarted automatically every midnight. !

I requested and obtained from the client:!

A. The source code that B is said to have modified !

B. The database changes that B created!

C. Previous discussion, contract and meeting notes with B for me to give a preliminary analysis whether there is a prima facie case for such an opinion to be made.!

My observations are made based directly on such sources given and discussions with the client. The source code has chronologically logged data which allows me to examine exact code changes over the course of their work in relation to the original code-base. The logged information also indicated who has filed those changes and what has changed throughout its life. So here it is easy to also show a real-life impact of a bad job.!

The client is very friendly and cooperative in furnishing me details as to who exactly modified what as the client’s own developer has to step in when things were in crisis and after loss of B’s services, have to manage things on their own. I have at most two cases to compare. The proof is always in evidence and evidence here is simply code.!

During my initial investigation, over the next few days, I was able to draw an early but reasonable conclusion that not only conceded with their position, that their changed software is indeed made bad, but contains numerous glaring ugly practices that can result in further issues of performance, maintenance and even lose its characteristic as an Open Source Software application.!

I then proceeded to coach them how to repair the damage while studying broadly those changes to decide whether to abandon them or make anew to ensure the software begins to be more usable and effective in serving the client’s needs and expectations. After another one month of working on it, I have taken note of more examples of mistakes in the code base and database design. Thus at this juncture, I am able to release a substantial report as to what my findings are both for A’s legal recourse as well as my academic purpose.!

This paper is my professional effort based on my personal experience with this exact software project and may not be applicable to non ERP industry using another type of software.!

The client has also expressed intention to use this to substantiate some legal process to reclaim costs incurred as a result of ill implementation of their case. I do not expect them to be successful as this episode is also partially their fault for been gullible. It is also the project’s fault. B was able to pay the ADempiere e.V society in Berlin a higher

Copyright (C) 2015 Redhuan D. OonPage � of �5 27

iROSES Auler Black Papersponsorship fee to be listed there giving the impression they are capable implementors without any further requirements. How are prospects like A supposed to know? OK it is not A’s fault. We are part of the problem. So as community we collectively help bring about these babies into the world, we have to be part of the solution.!

!

I am keenly interested to provide the technical argument in the most objective complete but simple manner possible for the dual purpose to educate and help prevent such failures to occur again.!

I look at failures as part of the standard norm rather than the exception which many advocates are trying to reverse. My travels around the world of over 35 countries have given me lots of horror stories and references and insight into the good, bad and ugly of ERP implementation on either ADempiere or iDempiere.!

I am publishing this as part of my effort in iROSES as a new global organisation to provide good advice to large ERP users to avoid such costly misadventures. One of my primary interest under iROSES is to provide a clear set of terms of reference and resources for implementing ERP correctly, professionally and successfully. !

This report assumes that the reader is basically familiar with terminology in use by the ICT industry particularly software development and ERP software implementation.!

Any comment or feedback over the content of this material can be emailed to me at [email protected] or [email protected].!

!!!

Copyright (C) 2015 Redhuan D. OonPage � of �6 27

iROSES Auler Black Paper

Technical Background!A successful Open Source Software project has the following importance:!

1. There is a possibility to access and modify the code-base to fit the client’s user requirements.!

2. Modifications must be in accordance to prevalent industry best practice of software engineering to ensure best performance.!

!Modifying Code!There are numerous parts of the software open to modification:!

1. Changing the relevant binary code at source level. The source code is located at http://bitbucket.org/idempiere/idempiere. It contains all source-code needed to operate the applications suite on top of an operating system such as Linux, Unix, Mac OSX, and Microsoft Windows. It is based on Java, SQL scripts and setup files. All of these are open to modification. !

2. Changing the meta-data that defines the application structure. This is stored in the database that is either Oracle or PostgreSQL based. It is also open to changes. A rendition of the meta-data can be accessed here: http://red1.org/DocBook/ !

However when one does changes to the above two types of code-base, they are usually managed as:!

1. Tracked history of diff patches of source files, which are then compiled into jars or plugins.!

2. Migration scripts of SQL language applicable to the database only.!

It is relevant here to note that the software industry is: !

A. Fast changing and that software itself undergoes tremendous upheavals in technical paradigm, contextual or environmental progress, developmental approach as well as coding style. It is liken to trying to comprehend the airline industry since inception 120 years ago in the space of a single decade;!

B. ERP Software is highly complex trying to integrate many core business processes within an organisation and beyond its walls. By comparison, 85% of IT projects fail but 92% of ERP IT projects fail.!

I will only make reference to those relevant practice that I observed to be of outmoded legacy style and why the failure to change to the modern approach have resulted in undesired and drastic drop in performance of the software and even risk of data integrity. In short this is another classic example that made it into the 92% statistic.!

!!

Copyright (C) 2015 Redhuan D. OonPage � of �7 27

iROSES Auler Black Paper

!Best Practice!It is important to note that the character of the software can turn into closed or ineffective source if the following happens:!

1. The code-base is modified not in accordance to best practice resulting in,!

2. Been separated from the main public project i.e.,!

3. Not able to take new improvements occurring in the main project into the changed code-base!

4. Difficult to maintain as the changes are now forked in a private manner or effectively closed!

5. No longer compatible with other software that continues to change rapidly!

This will mean an increase in costs rather than the reduction of costs of ownership as intended by the client and the consultants that implement a project based on Open Source Software.!

This is precisely what was happening with the client and the consultants they have as more overrun of schedules, creeping bugs and time pressure forces a vicious cycle of more mistakes and ill-preparation to carry out even more important tasks such as testing and proper staging before switching to live use.!

To ensure a project survives risks over its life cycle and life usage of about 5 to 8 years, there must be from the onset:!

1. Proper documentation to design and change management!

2. Separation of changes from core modules into plugin modules that can decouple and backward compatible with previous and 3rd party changes.!

3. Unit, integrated and regressive testing.!

Though the work has fulfilled partially item 1, it failed miserably in items 2 and 3.!

The impact of item 2 is also the question of performance and scalability. The excellent properties of enterprise software such as iDempiere are rendered useless when unnecessary and excessive code changes were introduced into the heart of the model design in a haphazard manner instead of handling them according to the best practice of separation from core, under a proper versioning and testing regime.!

In next sections, I shall examine exact code change constructs to impart fully what is referred to here and the correct remedy as the alternative. We also have the chance to demonstrate that the recommended best practice actually works and improved or restored the code base instance of the client to a satisfactory high performing condition.!

But first one more thing about the latest disruption to the technology in use by iDempiere that may explain the difficulty or need of competent experience by its advocates.!

!

Copyright (C) 2015 Redhuan D. OonPage � of �8 27

iROSES Auler Black Paper!OSGi Framework!Though this term, OSGi framework is familiar to contemporary Java developers, it is still relatively new in the discipline of proper practice. OSGi is an acronym for Open Services Gateway Initiative which basically refers to a common but advanced standard interface for code to be in the form of bundles or components that interact together as a common application but with clean and clear separation from dependency entanglement. That is achieved with a further set of rules to govern how each bundle is to expose or take from each other. Industry pundits have described OSGi as the first true component style that the Java programming language has been waiting for.!

OSGi concept is likened to the electrical wiring system in your home. If you merely use wires and no switches, plugs or extension points you may still light up your whole house. But imagine when one bulb blows or when you wish to rearrange some bulbs.!

Thus in the large enterprise, complex, but object oriented programming world, this is a holy grail where developers wish to achieve high agility for robust changes, standardisation, efficiency, and maintainability. In the ERP world it is to maintain sanity. It will be insane to go back to pure electric cables, all jumbled up and constantly rewired as definitions, requirements and environment changes.!

The Compiere project was created around 1999 when Java was still at its adolescence. The ADempiere fork up to 2011 was still using the old single wired stack of jars which the programming world sometimes refer to as Jar Hell. Beginning in 2009 some highly skilled developers within the community began to explore the use of an OSGi framework, resulting in a first beta version in 2011 and eventual first production release in 2013. During the client’s first receipt of the modified code at the end of 2013, the version in use that time was around 1.0c.!

!Changes in Plugins!The underlying practice of using OSGi in iDempiere follows the feature described above which allows any changes to the core code-base to be housed entirely outside the core in the form of plugins that are loosely coupled but tightly cohesive. They can be ‘dropped’ into the software application stack while it is running and stopped without impacting the core stack. !

An uninitiated or fresh developer may still program those changes in the outmoded way which is to introduce changes into the core stack thus bypassing the need of plugins. This will then make the OSGi framework of iDempiere meaningless and redundant. The code base will at that point lose all the goodies thus far described.!

Copyright (C) 2015 Redhuan D. OonPage � of �9 27

iROSES Auler Black PaperAs mentioned before, any further improvements by the open community which is another big boon for using Open Source software is also made tremendously difficult because each outside change may now impact the customised local change within the same core.!

Changes in plugins are easier to read and manage. Thus, the opposite is true - the abandoning of plugins for direct change to core is difficult to trace and manage. !

The plugins has declared services and extension points for what is known as Callouts, Processes, database changes, code overrides, and event validators. As of 2014, the project as compared to its predecessors, Compiere, and ADempiere has tremendous developer improvements and community activity. !

OSGi is not only popular but a standard fare in IBM arsenal of developer tools such as Eclipse IDE (Integrated development environment).!

!!!

Copyright (C) 2015 Redhuan D. OonPage � of �10 27

iROSES Auler Black Paper

The Modified Code!

As preempted earlier, the code we shall examine rests in two forms, mostly Java packages and database SQL scripts.!

We shall give some summarised quantification of the changes made and look at selected important ones to study in depth. What we be looking for will be the lack of use of plugins and the direct introduction of changes into core code and other anti-patterns.!

Above is a screenshot of the full history of the code change according to the versioning tool, Mercurial acting within Eclipse, the IBM free IDE. I blocked names with colour boxes, blue for A, and black for B. Note that at the time of switch over from B to the internal engineer of A. Note also how A seems to be more forthcoming in leaving more professional and narrative and useful remarks than done by B.!

It will thus make my analysis easier as I can better differentiate and adjudge the work of A against the work of the earlier B. I blocked out to avoid been prejudicial to anyone. I am more interested in what is done and not who done it.!

Copyright (C) 2015 Redhuan D. OonPage � of �11 27

iROSES Auler Black PaperNow we make a starting selection to examine closely of the changes done prior.!

!Above and below is a snapshot of three commits in the versioning history tracker within the source code provided by the client. The total number of changes by B has many changed Java class files. Most of them seem to be within the core stack of plugins which is under public domain maintenance of the project team as seen in the online code maintenance engine. http://jenkins.idempiere.com/job/iDempiereDaily/changes!

This is a huge concern and certainly raised instantly alarm bells about horror stories that lies ahead of this investigation. The above dated 20th December, 2013 is at the eleventh hour of which the system is supposed to cut life on 1st January 2014. This has all the tell-tale signs of doom: big bang approach, no testing regime in place, no accompanying release notes and definitely a mine-field, time-bomb ticking away. !

The changes are also committed in toto in singular large commits instead of giving atomic treatment, each single change under a separate commit. Now to reverse any mistake is not possible to be done quickly but the maintainer will have to resort to manual and painful stitching.!

The second commit timestamped 13th February, 2014 has again lumped classes and this time with absolutely no commit remarks. Again note that they belong entirely to a

Copyright (C) 2015 Redhuan D. OonPage � of �12 27

iROSES Auler Black Papermain core plugin, i.e. org.adempiere.base. Earlier assumption is correct. We have fire spreading sporadically and the poor client is in for a lot of fire-fighting. I am not surprised that B gave up and left the crime scene for A to fend alone.!

Third sample is dated 28th February, 2014 below. This time there is a nondescript remark to cover quite a high number of classes and there seems to be a single large plugin. It is not clear if they are of the same intent but there indeed is no stated intent.!

More complete explanation is now accorded in the following detail analysis of a set of core classes.!!

Copyright (C) 2015 Redhuan D. OonPage � of �13 27

iROSES Auler Black Paper

Hard core PO class!Perhaps the most deepest of core class is the Persistent Object class which all other data objects depend on and reuse its core definitions. Below is the class header info.!

!The full class can be googled such as here https://searchcode.com/codesearch/view/3306742/. This class is of the highest order in the object dependency hierarchy of which all ORM (Object Relationship Model) or database tables are represented in the Java code. As can be seen in the above header alone, the amount of comments and their references reflect the extreme complexity in dealing with this class. The manner in which any change is introduced to this class is by strictest community peer review. It is the practice of most developers not to even touch this class but forward recommendation to change only under the most demanding of scenarios. Otherwise if the developer is still to enact any change to it will in effect ‘forked’ or relegated his version of the class away from the main project and s/he has to then maintain it each time any change be it from him/her or the now ‘outside’ community as far as that modified class is concerned. A forked private class will then be obscured from outside peer review and suffer even further decay and poor quality assurance.!

And that is what promptly happened to the client’s source code. Though the change is minute (just 2 line remark and a hanging unused variable), an informed end-user or

Copyright (C) 2015 Redhuan D. OonPage � of �14 27

iROSES Auler Black Paperinternal developer will have unnecessary cause of alarm or concern each time he has to deal with this class in future as any pull of changes from core repository will flag this class as having private changes and maybe difficult to merge due to possible conflict. As seen below, compare the remark by B with ‘globalqss’ before it. B’s remark is not helping in explaining any intent nor referring to any peer review.!

� !

Since the minute change is insignificant, making it within core has serious impact. It is as careless as leaving an unused spanner pin in the main engine. One day something can and will go wrong. Such a risk is advised to be restrained from at all costs.!

Such a coding practice gives an impression that the developer is unaware of important matters such as:!

1. Version control within community of peers!

2. Backward compatibility!

3. Stability!

4. Documentation!

5. Project Management!

From here we move on to other core model classes, this time explaining another aspect of software coding style in use for software such as iDempiere.!

!!!

Copyright (C) 2015 Redhuan D. OonPage � of �15 27

iROSES Auler Black Paper

Overriding Core!There are cases where the core code needs to be overwritten due to its limitations to handle certain specific new functionality. There is a best practice way in doing so. Again all changes should be housed in separate and external plugins. The only difference is that they will ‘add-on’ from outside. There are three ways they can do that:!

!Event Validators!This feature has been present in the original Compiere and inherited very well up to today in iDempiere. However during pre-OSGi days, they are defined in another customised jar that needs to be recompiled into the core. Now in OSGi framework, they can now be in an independent plugin jar and thus recompilation of core stack is not needed. It can even be hot deployed, i.e. without even stopping the core application stack. This is a big quantum leap in efficiency and manageability of a large code cauldron that undergoes continuous changes.!

EventValidators basically get triggered during events occurring in documents such as a Sales Order, Invoice, Payment and Inventory movement. Whenever a document is created, changed, saved, processed or deleted, a custom code can interrupt it to give added control and data manipulation.!

In iDempiere, the EventValidator is entirely sitting in an outside plugin, quite literally because even the metadata activation of the Model Validator that houses the EventValidator is also not made in a core configuration file but in external declaration of services. This is also a modern software technique by itself called the Hollywood Principle, “Don’t call me, I will call you.” This makes the core quiet, and lighter allowing faster responsiveness and scalability at the same time.!

In client’s code, such modification is handled in such a manner and we shall examine further into it later.!

!Callouts!Another Compiere inherited feature is the use of Column Callouts that kick in during change of a Window Tab’s Field. In OSGi again metadata definition within the core database of these are removed and rest solely in a factory service extension schema that follows similarly the Hollywood Principle.!

However as we shall soon see, the changed code made no use of defining the Callouts in plugins but remain the same within the base metadata. Worse, the extra custom code is merged into standard base Callouts and will be difficult to separate when the core changes over time.!

Copyright (C) 2015 Redhuan D. OonPage � of �16 27

iROSES Auler Black PaperFrom the summary above there are about 4 core Callouts involved. This add on further to the pain of maintenance that already discovered thus far. That notwithstanding, I will still look into those Callouts themselves to further uncover any further complications from the coding style and design.!

!Extending Model!The core tables or models can be extended for example in a Product table, there may be need to give a product more attributes or character that is not available in the present base design. Again to do so during Compiere and ADempiere times is to define them in a customised.jar and recompile them with the core stack. The definition also repeat back the base model plus the added properties in a lumped mass. This is done in an overloading or overriding manner which is prohibitive to future maintenance.!

In iDempiere, not only are such model extensions defined in external plugins, they have a clear separation and non impact to the core. Say for example the core has a new change and the customsed model did not specify such a change. That new change can then be lost. This is considered as impact or backward incompatibility or unstable when referring to the future tense.!

iDempiere best practice is to make use of a code construct called a POWrapper which wraps the core model design by aptly adding onto it without overriding it. It is simply A + B, instead of B over A. Any further changes to either A or B has no or minimal impact to each other. Whereas an object taking over another object are entangled or blocking exclusively each other’s intent. Sadly such an advancement in coding construction that is already made public in the project online wiki was not practised in B‘s work. !

(More advanced developers may contest the assumption that iDempiere OSGi framework is 100% decoupled from core and that it has no further complication. So to be precise, there is no such thing as a real floating object in a virtual world, but referential space with pointers. Suffice for our purpose, iDempiere is a quantum leap from the way ADempiere is extended. This important fact was made clear to B during my initial full evening and morning meeting with them in 2013 which they arranged to hear directly from me why the main coders in the project desire to switch to this new technology.)!

!!

Copyright (C) 2015 Redhuan D. OonPage � of �17 27

iROSES Auler Black Paper

Core Model Changes!We examine model classes because they are referenced later by the ModelValidator during EventValidation. The first change below is another clear example of recklessness.!MOrder

� !There seem to be no peer review accorded nor concern given to the possible impact it has over a core model class. The remark reason given seems to be partial and summarily executed. The removal of an exception that is meant for a clearly defined method, i.e. copyLinesFrom where it will now copy header if lines are absent. This will worry me of what such a developer is capable of doing throughout the whole ERP that s/he gets to rumble through. There should be a different method to handle this and housed in a separate process within a separate plugin. But wait, there is already a copyHeader function in every normal window tab! B is a deaf bat. Because bats are not blind.!

Anyway, for argument’s sake, to solve this we can also use new MOrder class extends this org.compiere.model.MOrder put together with other new fields in a POWrapper and this method overridden. Clearly the developer here is oblivious to the repercussions or the meaning of the word ‘impact’ in changing code. Below is the other MOrder change:

� !

By now there is no more doubt in my mind that such code is not meant to be upgradable progressively following iDempiere future versions. The developer who did this has no idea at all about handling code design and framework at a professional level. It is very safe to say that the whole code base is meant to be stuck at version 1.0c forever. (iDempiere at this time of writing is moving from 2.1 into 3.0. with more advanced ZK UI and Jetty application server features.)!

Next, I like to kill two birds with a single stone. I want to show a change that B did and then later A reverted.!

!Copyright (C) 2015 Redhuan D. OonPage � of �18 27

iROSES Auler Black Paper

MOrderLine!

You can see that A reverted a piece in the MOrderLine class done by B. What A did is more acceptable, which is to produce a ticket for it (Ticket 116) and offer exhaustive explanation in German which is pasted for Translate.Google output below.!

Ticket [116] The sale discount will be fully rounded after saving. Berichrigt. As we rounded to 2 decimal places and iDempiere also must calculate the discount again when you save the current position, there will be deviations. For this reason I charge even after entering the discount the only possible sale discount, of up to 2 decimal fits exactly to the UK list price and the VK-net price and carry it on. This is in my opinion the only correct value if we want to round spots at the amounts to 2 decimal places.! !

Under what conditions then did B did that change? Let’s trace back this history and we arrive at a remark which is just a hyphen -.!

No remarks offered whatsoever. And to add fuel to fire, again lumping with unrelated class of separate intents. So here we get to learn another important lesson. Document your work and track its changes atomically so that another peer such as me can review it with pleasure or disdain. The choice is yours.!

!Copyright (C) 2015 Redhuan D. OonPage � of �19 27

iROSES Auler Black Paper

Extend More!The client A has a requirement that is classic and can be common to many users. They wish to extend the Order Line to manage a Product item’s many other new properties such as a product rate charge that carries a different set of calculation of the product chosen. A product X may have a charge due to the spot market of X or the user may want to chose another different market rate for it. Different products have different treatment of rates.!

Finally the charge derived from a rate formula is added to the Order Line cost of the product. Now how should one do that? What happened in our poor ugly system is that it was added onto the same OrderLine detail table. It now overloads a very important and busy table line with more information that has many actions going around it as part of the busy Sales Order. OrderLine already has Callouts and event validators working against it to do common tasks such as discount, price calculation and line totalling.!

Rightly so, when I was at the client’s place to try to use that Sales Order in a freshly migrated to latest iDempiere 2.1 (Yes, even ugly ducklings get to be saved to be a swan.), everything was much faster except trying to save the OrderLine. After some checking around we found out that the overweight OrderLine has too many recursive activity going inside it.!

I then pointed out to the present metadata design already prevalent since Compiere on how to handle different activity within a same tab.!

!!

Copyright (C) 2015 Redhuan D. OonPage � of �20 27

iROSES Auler Black Paper

Window Multi-Tabs!This design has been around since Compiere days which is to have simply multiple tabs or tables to be related to a respective master tabs. Below is an example of a case I handled 11 years ago during Compiere days.!

� !

You can see that it is for a Car Sales business. So the Order Header will have the customer key ID which more information such as his registration with the driving school for example is needed and that can be handled in another tab which has respective C_BPartner_ID linked together. And further to that, in the OrderLine is the car key ID which another tab can register the car and so on, its Loan Financing, and so on. Each tab is associated with its own table model and not borrowing from an earlier one. This is proper separation and a more normalised database design, allowing control over many to many relationships.!

Compiere’s Application Dictionary already caters for such behaviour allowing a myriad of many tabs that are sophisticatedly but efficiently linked to, without the user to do redundant boiler plate settings or code. !

This is a classic case of B eager to jump into the lucrative ERP consulting business without paying attention to competency and experience needed. And I have met a lot of Bs in my travels around the world.!

!Copyright (C) 2015 Redhuan D. OonPage � of �21 27

iROSES Auler Black PaperIn our client’s example we want the OrderLine’s Product_ID to have another further detail tab that expounds on the product rate and link its M_Product_ID. This separate into a new model just for the ProductRateCalculation. It can do whatever it wants in its own atomic capsule and not be entangled with another legacy model which has been around since ancient times and Dharma knows what else is going on in there. !

Also remember our principle of separation from core legacy where anything that is done by the peer community to improve it will certainly block your changes or vice versa. But then the user says, ‘What about the totalling that we need to show on the same Order Line?’!

There can then be a necessary evil of adding a single field onto the OrderLine ID to say, ‘RateChageAmount’ and a Callout can take that and add to the PriceList field to show that the product price is added with the extra charge. Below is an image A sent to me after of the requirements to have extra tabs for the OrderLine. Originally B designed it as shown but the underlying tab structure is reusing the same OrderLine table instead of breaking it up into individual ‘MetalCharge’ and ‘WEEE’ tables.!

This is a snapshot of the screen on A’s PC.!

!The improved performance was remarkable, from being hung, to working normally. And the maintenance of it will be easier. Note in the window how there is already a lot of additional new fields in the main header starting with ‘as1’. Yes iDempiere is open to modification but changing without a good foundation is to jump into fire.!

Copyright (C) 2015 Redhuan D. OonPage � of �22 27

iROSES Auler Black PaperWhen I looked into the way how B lumped everything together including other requirements such as ‘Gruppi’ which is examined in next chapter, I find a bloated OrderLine model up to 111 fields. On the right panel you can see the other extra fields.!

� !

Yes, I understand the client user wants many of those fields but there is such an expertise called system design where group of fields can be clustered in separate models for logical handling as well as avoid adverse impact. This same OrderLine was showing up as 3 separate tabs already as shown in later pages with its ‘Metal Charge’ and ‘WEEE’ tabs.!

This is not dissimilar from OrderTax tab which handles tax for the whole order and may handle tax on individual OrderLine. It does not come into the OrderLine tab and in fact remains outside and only inject calculations in many ways but later and out of the way.!

If we need let’s say runtime information i.e. Tax result for a certain Product, it is possible to use a simple direct Callout to look it up and populate back. !

If more information is needed to be visible close by, we can use a virtual column to display a derived value on the fly. If more, then use an extra tab as explained above for a sub-master/detail view or an InfoWindow design which is more hard work but elegant and maintainable in a long run.

Copyright (C) 2015 Redhuan D. OonPage � of �23 27

iROSES Auler Black Paper

Important Maxims!Another important requirement of A is to group the product items under certain packaging for certain purposes:!

1. To hide detail items and to show only a group package price!

2. To still show detail prices but only to internal staff!

3. To access a long history of previous sales prices which its customers still want.!

Now this will be a big headache if the consultant developer just shove such new fields onto a single table. And that is what exactly B did. I explained to A that iDempiere already inherited from Compiere and ADempiere over the years and one important maxim I always teach is:!

Ask yourself if someone else in the world has the same problem. If so, then the solution might already be there. !

I then explained about more models already present in iDempiere such as BOM (build of materials) which can help solve the group packaging. A special EventValidator can pick that group and rewrite the OrderLines just showing or hiding what is wanted or better still a special print format that only does that during printout of the Sales Quotation. !

I also taught another maxim:!

Wouldn’t an advanced and global and public ERP software such as this has what you need?!

However due to the mess created by B, the repair work is more than the original work. Because the system has gone live with all these mistakes, a migration effort to convert the old table structure (overridden with extra tabs) to be more normalised in design with new tables representing the new related models. Good news is that A learns quickly and the new table structure is done and working. Now it is just to fix dependent processes and printouts.!

There is much more to cover such as the ModelValidator done by B is actually 4000 lines long, trying to be everything to everybody. And many of its data model access are not following the ORM template of setters and getters making long unwieldy and risky SQL strings abound inside it. Of course this is not just a backdoor for SQL injection from hackers but also a source of memory leak causing the system to crash everyday.!

!!!

Copyright (C) 2015 Redhuan D. OonPage � of �24 27

iROSES Auler Black Paper

� !

Above we can see how it has passed the 4,000th line. It should have been a more segregated series of events tied appropriately to each individual table model event for easier debugging. One bug here and that is it, the whole validator may fail. Note also the use of the open ORM method allowing hand-written string values, instead of the statically typed one.!

!

Copyright (C) 2015 Redhuan D. OonPage � of �25 27

iROSES Auler Black PaperI paste one more part of the long EventValidator below.!

Note the ‘gruppi’ attempt of packaging product which I would suggest to explore the ‘Phantom’ BOMType already existing in Libero Manufacturing plugin. Of course the wanton use of harmful SQL strings is alarming when there already exist an ORM model that just need a GenerateModel execution to create them. Laziness is expensive. Which shouldn’t be when B has been paid very well to do a good job.!

At this moment, the old iDempiere instance is tested with a major upgrade:!

1. Migration scripts are applied to bring it to latest 2.1!

2. Separation of processes and info-windows into separate plugins!

3. Core overriding of process made into plugin fragment!

4. Separation of multi-tab into normalised tables!

5. Publishing to share all discovery in public: http://wiki.idempiere.org/en/AulerSipel !

6. FItNesse testing suite is still pending and we hope to complete that by end 2015.

Copyright (C) 2015 Redhuan D. OonPage � of �26 27

iROSES Auler Black Paper!As I go through the whole mess, I keep finding more and more of the same. It can drive one insane.!

!All this is averted by simply following the simple but golden principle of ‘separation of concern’ illustrated which happens to be a good old Java maxim on its own. Of course it takes a sharp learning curve to change an old lazy habit to write well and readable work. And the abundance of Open Source Software dictates this habit even more so, now that users can get to touch code. It is like opening up a nuclear reactor to tourists to play around with.!

Ironically, I did commend B to publish a white paper of their pursuit as that time they gave me some hint that they managed to prove that iDempiere is better than MSDynamics in a client case without saying further. I called upon them again last year, cooking them my famous chicken rice, again reminding them that they owe me a white paper which is all I am interested in and not their business. I now do not need to wait for that since I already seen the source and a good story, that lets me cook up something better than reality TV instead. This is what gave me the idea of a ‘Black Paper’.!

In my little passion of free ERP, which clearly is for very expensive use, going against alternative proprietary solutions such as SAP that can run into millions of Euros, it is a difficult challenge to become part of the 8% success ERP stories. Indeed I myself have my share of failures and I constantly learn and write about them. This story is not the first nor last. Publication of such black papers seek to change attitude for a better world.!

Remember, the source may be with you. That makes it is easy to fall onto the dark side. If you do, just admit it, learn from it, publish it, move on. !

Then stay calm, and come into the light. :)!

Karma & Peace!

Copyright (C) 2015 Redhuan D. OonPage � of �27 27