Embed Size (px)
Citation preview
Welcome
Remember NSFDB2? Finally use a true SQL RDBMS with your Notes databases!
Grails4Notes.com Realize the power of your Notes data
Speaker: Justin Hill CTO Prominic.NET -‐ IBM technology cloud hosting since 1998 @JustinProminic linkedin.com/in/justinprominic
Is this session for you?
• This presentation is for you if:
• You have custom Domino applications with no design documentation
• You are being pushed to migrate from Domino to some other technology
• You have NSF performance problems / users complain your application is sluggish
• You want to make the most of XPages new RDBMS integration
• You need data reporting of a complex nature and are tired of writing custom agents to extract it
• You have used tools like DomSQL, BIRT, Notrix, Replic-‐Action but need more
• You need to build a native app on Windows, Mac, iOS, or Android -‐ but how?!
Start with Why
• Prominic’s focus is on IBM technology cloud hosting — Notes, Domino, DB2, WebSphere, Digital Experience Suite, etc.
• Why is Prominic investing in a large and complex platform level initiative?
• Customers with large NSFs (1 million+ documents) complain of performance and lack adequate reporting, especially across many Domino servers and databases
• The market never got true relational DB support (the original implied promise of NSFDB2). It was needed badly in 2007 and is still needed today. http://www-‐10.lotus.com/ldd/nd85forum.nsf/0/0dd34efd50e4f33d8525754d004a0b8e?OpenDocument
• Because we saw unique opportunities emerging to solve long-‐standing Notes/Domino market problems by combining existing proven technologies:
• Grails4Notes.com = Groovy/Grails + Java + Notes/Domino + PostgreSQL + Apache Flex + GraniteDS + Moonshine IDE
Domino4Wine.com
• Sidenote: though not related to this talk directly, many Notes/Domino developers in the room primarily use Mac or Linux running Windows Virtual Machines
• Because Mac and Linux users have complained about a lack of Domino Designer and Administrator running natively, we also took action:
• Domino4Wine.com -‐ run Domino Designer and Administrator natively on Mac and Linux WITHOUT a VM using CrossOver (commercial WINE)
• It isn’t perfect, but it is much better than it was even 8 months ago when first announced at Lotusphere 2015. We continue to improve it with Codeweavers (the makers of CrossOver and largest group of WINE contributors.)
• Your testing and support would help -‐ please spread the word! The ideal goal would be to get IBM to support it, which would allow feature parity across ALL WINE platforms with ONE Win32 code base. The Mac Notes client is too buggy! IBM’s support would improve performance further on WINE…
Domino4Wine.com
• For more information on Domino4Wine.com please attend Rob Novak’s session on Friday at 3pm:
• SA104: Free Beer 12… Southern Style
Grails4Notes Mission
• Grails4Notes Mission: Augment core Notes/Domino technology to “fill the gaps”
• Increase satisfaction of large-‐scale NSF apps to avoid the “why don’t we rewrite this with technology combo [XYZ]?”
• Which gaps is Grails4Notes trying to fill?
• Encourage formal field and form design documentation so new hires have a clue!
• Clearly define relationships between document types (forms, fields, subforms)
• Generate business objects (aka “classes” in Object Oriented Programming parlance) for forms and subforms
• Provide true SQL support for advanced query control. THIS IS NOT A LAYER OF SOME KIND. Your Domino data will ALSO exist in a real RDBMS.
Grails4Notes Mission
• Generate a cross platform native application using Apache Flex (Win, Mac, browser, iOS, Android)
• Utilize GraniteDS: Advanced data binding for real-‐time communication between clients — no “refresh button” and even faster than AJAX
• Because Apache Flex is foreign territory for Notes/Domino shops, provide an integrated IDE, compiler, and ANT build scripts
• Integrated Notes Client access — so you can use your existing Notes ID to get started (or a “demo ID” initially) with full transport encryption
• Social network login support for Facebook, Google+, Twitter, LinkedIn, and SMS
• Easy to use web interface — all delivered “in the cloud”, so it is easy to get started
Grails4Notes Mission
• Use Open Source technology so there is no vendor lock-‐in • Leverage existing Java software investments you already have built • Take advantage of recent market popularity of other technologies like Ruby on Rails, while maintaining compatibility with Java
• Utilize modern Model-‐View-‐Controller best practices • Maintain ALL Grails4Notes design annotations added for comments, relationships, etc. within the EXISTING Domino Designer structure of NSF / NTF
Market Options for Extending Domino
• Several interesting developments in alternative Notes/Domino database storage and application modernization recently
• DomSQL Project on OpenNTF, and before that the NotesSQL driver from IBM
• Red Pill Now -‐ rapidly mobile enable apps using Graph databases
• LDC Via -‐ convert Domino to “MEAN” stack (Mongo, ExpressJS, AngularJS, Node)
• Older Notes-‐>SQL products — like Lotus Enterprise Integrator (LEI) and:
• Axceler’s Notrix
• Casahl’s Replic-‐Action
True SQL -‐ why?
• Most widely used Domino-‐>SQL alternative at the moment is probably DomSQL
• Internet search for “OpenNTF JDBC access for Domino” to find it
• Good for use with the BIRT free reporting tool
• Uses SQLite read-‐only driver
• Second most widely used item is probably the NotesSQL read/write ODBC driver
• Internet search for “IBM Lotus NotesSQL” to find it
• The above options are good but still are not full SQL running on a real RDBMS
• What does true SQL mean? How does it compare to NSF? Why do we need it?
Domino Pain Point #1: PERFORMANCE
Domino Pain Point #1: PERFORMANCE
Million+ document query in Domino vs. SQL
SoluUon: Grails4Notes Importer
NSF Performance Tricks
• There are several tricks to improve Domino performance -‐ an entre talk could easily be on that topic.
• For example, at Prominic, we routnely put in servers with large amounts of RAM to allow for extensive buffering (256+ GB RAM -‐ Again NOT SSD… RAM)
• Domino 9.0.2 (as I recall) is going to allow view indexes to be specified separately from the NSF which can be a big performance boost
• This may be a somewhat unfair comparison because we are startng with an SQL DB and portng it to Domino
• The #1 technical feature RDBMS has over Domino is the ability to optmize specific queries by carefully examining the manner a result set is generated
• To “fix” the performance of many Domino apps, just a FEW sectons of the app need to have that ability. Don’t re-‐write everything —> just fix the slow parts!
• Use our analyzer to find the slow parts of your app with passive HTTP log analysis
HTTP log performance analyzer
NSF Pros and Cons
While NSF databases stll contains market leading features like: • built-‐in Full-‐Text Indexing of file ayachments (with support for different formats) • security model (Readers & Authors) • easy integraton with Web framework – IBM XPages
… other NoSQL alternatves are out there now and well funded by Silicon Valley — including MongoDB and (ironically) Cloudant (IBM bought them)
• Biggest disadvantage of NSF specifically is performance at scale — especially on databases with more than 1 Million documents.
• Also NSF DB lookups can not (easily) be dynamically built based upon end-‐user interacton with the UI of your smart app:
• Notes & XPages developers have to prepare View elements in advance, which makes our applicatons far too statc!
XPages and RDBMS already work
• Grails4Notes enables true SQL functonality for your existng NSF DBs.
What does it gives you? • Boost performance of queries against your data; • Enables dynamic queries based on end-‐user preferences:
Sortng in tables – out of the box (comes with RDBMS ExtLib component) Filtering of big data sets by multple conditons
Remember:
• RDBMS support is already provided out-‐of-‐the-‐box with latest versions of XPages Extension Library from OpenNTF
• This allows you to connect your XPages app directly to your data just a}er you mirror it in SQL DB with Grails4Notes!
Performance comparison: NSF vs. SQL
4 Millions documents NSF DB vs. 4 Millions records SQL DB
MySQL’s Employees sample database
hyp://dev.mysql.com/doc/employee/en/employees-‐introducton.html
… provides a combinaton of a large base of data spread over six separate tables and consistng of 4 million records in total. We imported it into an NSF, created corresponding Forms and Fields, and built a liyle XPages app.
NSF DB: -‐ 3,919,015 documents total -‐ 1,950 MB file size
SQL DB: -‐ 3,919,015 records total -‐ 160 MB file size
Sample SQL database schema layout
Corresponding Sample Notes Forms
SQL table row sizes
• 300K employees, 331K departments, 443K titles, and 2.844 million salary data points
Sample data: Employees and Salaries
• Very simple model: Employees and their annual salary data. Copied the data into NSF and SQL.
• Each employee has exactly one Employee record
• Multple Salary records exist for each Employee and represent the salary level for a given year. The Employee record is the primary key.
• We created a simple XPages applicaton which allows the user to search for employees within specific salary ranges using both NSF and SQL copies of the database.
• To resolve this search query, Notes/Domino must iterate over each record in Salaries each tme and return the Employee results. This is slow.
• It could be optmized by using View Navigators and looking at column values, but that stll takes tme to iterate over and many data sets do not get exposed in columns but do need to be queried.
Timed Query using Domino NSF
Time measurement results: NSF DB – 712,157 ms (~11 min)
Timed Query using SQL
Time measurement results: SQL DB – 1,981 ms (< 2 seconds)
SQL Query Close-‐up
Domino was ~330 Umes slower than this SQL query:
SELECT * FROM employees.employees INNER JOIN employees.salaries ON employees.employees.emp_no = employees.salaries.emp_no WHERE year(employees.salaries.to_date) = 9999 AND employees.salaries.salary >= 54423 AND employees.salaries.salary <= 60598 ;
Processed primary records (Employee records): 300024Processed secondary records (Salary records): 2844047Time taken for processing SQL query: 1981 ms
What about your precious dev time?
NSF DB: • ~4 hours were spent in order to prepare the code for making the query
• Most efforts were spent on preparing code for iterating the documents and lookup related documents based on defined relations
• There is still room for improvement of the query performance by using ViewNavigator, Column values, etc. But this will increase time needed to adjust code to such level.
SQL DB: • Less than 1 hour was spent in order to prepare application logic for making the query
• Mostly time was spent on adjusting UI and fine-‐tuning rather than debugging query itself
Efforts spent for preparing app: NSF DB vs SQL DB
NSF Query Close-‐up
Sub getEmployeeWithSalaryRange(salaryFromParam, salaryToParam)
// Resulting Document collectionDocumentCollection resultEmployeesCol
// Iterate over all Employee documentsFor employeeDoc in allEmployeeView
employeeNo = employeeDoc.getEmployeeNo()
DocumentCollection employeeSalaries = Helper.getEmployeeSalariesByEmpNo(employeeNo)
employeeSalary = 0
// Iterate over Employee's Salaries documents - get employee current salaryFor salaryDoc in employeeSalaries
If Helper.isCurrentSalary(salaryDoc)employeeSalary = salaryDoc.getSalary()
End IfNext salaryDoc
// Filling in resulting document collectionIf employeeSalary >= salaryFromParam And employeeSalary <= salaryToParam
resultEmployeesCol.addDocument(employeeDoc)End If
Next employeeDoc
End Sub
Time measurement: NSF DB query pseudocode
Ouch -‐ Domino lost the battle big time!
• Lets think about that again: Domino was 330 tmes slower than SQL on an RDBMS in this test.
• Using the kind of queries all of us tend to write: Document iterators with custom selector logic we create in Lotuscript or Java agents / XPages.
• The problem only gets worse with more views and documents vying for the NSF engine’s ayenton.
• What can we do about this? Import our Domino data to SQL!
• Caveat: Importng takes tme and must currently be repeated (which can be scheduled / ‘headless’)
VIDEO DEMO #1
VIDEO DEMO #1: Import Domino data to PostgreSQL
• Not using the aforementoned 4 million document DB for tme constraint reasons…
Domino Pain Point #2: Lack of Design Docs
Domino Pain Point #2: DESIGN DOCUMENTATION FOR NOTES APPS
• Comment your Domino code! Fields count as code!
• How many tmes have your new hires asked for documentaton?
• How many tmes have you personally had to start deciphering someone else’s messy Notes/Domino applicaton?
• Formalize the process of commentng your Notes/Domino fields and forms.
• To illustrate, we built a simple app consistng of Authors, Books, and Fans.
• The model: Authors write the Books. Books and Authors both have Fans.
G4N -‐ Notes Client
Author Form
Book Form
Fan Form
Author/Book/Fan
G4N Importer
Imported data viewer
Navicat to PostgreSQL
VIDEO DEMO #2
VIDEO DEMO #2: Add comments to forms and fields
Domino Pain Point #3: CONSTRAINTS
Domino Pain Point #3: CONSTRAINTS
Constraints include ‘field can’t be null’, ‘field is a primary key’, etc.
Domino doesn’t care what goes in fields, which is great most of the Ume. But RDBMS need rules in the form of constraints. We need to annotate Notes fields with constraints.
SoluUon: Grails4Notes Designer
Constraints
VIDEO DEMO #3
VIDEO DEMO #3: Add constraints to fields
Domino Pain Point #4: RELATIONSHIPS
Domino Pain Point #4: RELATIONSHIPS
Almost all Domino DBs are at least mostly (though loosely) relaUonal, but they are not ‘normalized’. SQL needs to know the real relaUonship between Forms and Subforms.
SoluUon: Grails4Notes Designer
VIDEO DEMO #4
VIDEO DEMO #4: Add relaUonships to fields and show the use of Navicat Data Modeler to auto-‐generate a full design schema
Domino Pain Point #5: Real time data
Domino Pain Point #5: REAL TIME DATA
Modern apps don’t need a ‘refresh’ key to see updated data. AJAX can be painful to debug. Grails4Notes supports real Ume updates.
SoluUon: Grails4Notes Broker
VIDEO DEMO #5
VIDEO DEMO #5: Show two automaUcally generated Grails4Notes user interfaces dynamically update content with one another
Sneak Peaks
• Quickly turn Notes databases into Flex applications compiled into NATIVE Windows and Mac applications!
• Utilize social network logins
Domino Pain Point #6: Native applications
Domino Pain Point #6: NATIVE APPLICATIONS
The current trend is for naUve iOS and Android apps. There is also a need for naUve Mac and Windows more focused on your soluUon for which the full “rich Notes client” may not be ideal.
SoluUon: Grails4Notes Broker + Apache Flex + Moonshine IDE
Moonshine IDE
VIDEO DEMO #6
VIDEO DEMO #6: Show the process of auto-‐generaUng an Apache Flex user interface and ediUng it with the Moonshine IDE. Show the resulUng updated applicaUon run.
Domino Pain Point #7: Social networking
Domino Pain Point #7: SOCIAL NETWORKING
Every major site these days allows users to sign-‐in with Facebook, Google+, LinkedIn, Twirer, and can do detailed security analysis of logins based upon cookies, IP addresses, SMS two-‐factor authenUcaUon, etc.
SoluUon: Grails4Notes Broker
Social Media / Broker
VIDEO DEMO #7
VIDEO DEMO #7: Show a user logging into the Grails4Notes Broker’s social network profile using Facebook Connect login one Ume, and their normal Notes e-‐mail and password another.
Feature Recap
• Keep your Notes data in the NSF while also having it available using true SQL served by a true RDBMS
• Use true SQL language queries to report on your data
• Stop writing custom Java or Lotuscript agents to query Notes data in a more complex fashion than can be done with view selection formula syntax
• Develop a true Business Object Model for the data in your Notes databases
• Easily create formal documentation in the industry-‐standard JavaDoc-‐like format for all of your fields on forms and subforms in Notes
Feature Recap
• Eliminate unnecessary views from your Notes databases because they clutter your design and slow down performance for large databases
• Performance optimize the necessary sections of your Domino classic or newer XPages web applications
• Gain insight into the slowest URLs of your Domino classic or newer XPages web applications so you can improve your customer’s web experience.
• Mix the mature security model, reliable replication technology for clustered / off-‐line use, and rapid application NoSQL-‐style development of Notes with cutting-‐edge dynamic language technology from coders so good at Java they invented something even better: Groovy/Grails.
Availability
• Available now for existing Prominic.NET customers (in beta starting now)
• Available “in the cloud” to everyone by end of Q3 2015 as 1.0 release
• This is NOT just for corporations — we actively want individual developer use
• Flexible licensing model (specifics to be determined) — to support continued development while making it accessible to a wide variety of use cases
Future Directions
• Incremental import updates (much faster to keep SQL updated with Domino)
• Possible near real-‐time SQL updates based upon Domino document C API change event hooks
• Pending customer demand, possibly bi-‐directional support for SQL-‐>Domino updates (this is unfortunately technically complex)
• Purpose-‐built applications to demonstrate the power of Grails4Notes
• Catalog of pre-‐made Grails4Notes Designs for existing Domino DBs, such as names.nsf, domlog.nsf, catalog.nsf, etc
• Advanced querying of systemwide Domino e-‐mail (for compliance and CRM purposes) — easily see all correspondence between any employees and customers
Q&A
• Thank you for listening! Please ask questions!
• Join the mailing list for release updates -‐ Grails4Notes.com
• Mac/Linux users -‐ check out Domino4Wine.com
• Spread the word on Twitter: #Grails4Notes and #Domino4Wine
• My contact info:
Justin Hill Prominic.NET @JustinProminic linkedin.com/in/justinprominic
