phax/phoss-smp phoss-smp-parent-pom-8.4.0

v8.4.0 (Server side pagination)

on GitHub

• Updated to the MongoDB driver 5.11.0
• Updated to ph-oton 10.5.0, ph-oton-bootstrap5 0.9.4, peppol-shared-ui 0.10.1 and phoss-directory 0.17.1
• (SQL, MongoDB) Added server side pagination, sorting and searching for the lists of Service Groups, Endpoints, Redirects, Business Cards and Participant Migrations, so that a large SMP no longer sends all rows to the browser and no longer loads all entities into memory to render a single page.
  Based on #526 - thx @vrbyjimmy - with a reworked API.
  See #525 and #535.
  The paged manager methods take a single IPagingSpec parameter (start index, maximum count and the fields to sort by; new in ph-commons 12.4.0) plus the search text, instead of positional index and count parameters, so that the API does not need to be extended for every additional paging or sorting aspect.
• Added the new interface ISMPTableColumn and one implementing enum per domain object - ESMPServiceGroupColumn, ESMPServiceInformationColumn, ESMPRedirectColumn, ESMPBusinessCardColumn and ESMPParticipantMigrationColumn.
  Such a column ties the column shown in the UI to the SQL columns, to the MongoDB fields and to an in-memory Comparator, so that the four cannot drift apart, and it declares whether the column may be sorted and whether the global search covers it.
  The column ID is the field name used in a sort field, and it is the only field name a client may refer to: unknown or non-sortable field names are ignored instead of being passed on, so that a forged sort field can never reach the SQL statement. If no sort field remains, the first column of the enum is used, so that consecutive page requests return disjunct results.
  SMPTableColumnHelper contains the backend independent resolution and the in-memory implementation, which is the reference behaviour all native backend implementations comply to.
• The four paged list pages - the secure Service Groups, Endpoints, Redirects and Business Cards lists - now use the DataTables server side mode ON_DEMAND of ph-oton 10.4.0, so that the paging, the sorting and the search field of DataTables itself drive the server side query.
  Previously every DataTables kept a rendered copy of the whole table in the session (mode PRERENDERED), and the client side sorting therefore only sorted the rows of the currently displayed page. Now every AJAX request queries only the rows of the requested page from the respective manager, and nothing is kept in the session.
  Each page registers its own AJAX function - via the new class SMPDataTablesOnDemand - that renders the rows of one page. The name of a sortable table column is the ID of the respective ISMPTableColumn, so that the column the user clicks is resolved onto the SQL column respectively the MongoDB field. Columns that are derived or that live in child entities - like the number of endpoints of a Service Group or the process ID of an endpoint - are marked as not orderable.
  Note that on the Endpoints and the Business Cards page the paging happens on Service Information respectively Business Card level, whereas a single row represents a single endpoint respectively a single business entity - so a page may contain more rows than the selected page size.
• Incompatible change: the public start page no longer lists the participants managed by this SMP. It contains a static description of this SMP instead - what an SMP is, that the data is served via the SMP REST API and that phoss SMP is an Open Source solution, including the links to the MPL 2.0 and the Apache 2.0 license - so that it is rendered without a single backend query, no matter how many participants the SMP manages. The link to the source code is only shown if webapp.public.showsource is enabled, as in the footer.
  The configuration properties webapp.startpage.dynamictable, webapp.startpage.participants.none and webapp.startpage.extensions.show were removed accordingly, as were the methods SMPWebAppConfiguration.isStartPageDynamicTable (), isStartPageParticipantsNone () and isStartPageExtensionsShow ().
• The Endpoint Tree page is now paged as well and shows 20 service groups per page by default. Previously it loaded every service group and every service information into memory to render the page, which is why the "Tree view" button was hidden for SMPs with more than 1000 service groups - that limit is gone again.
  A tree is not a DataTables, so the paging is driven by the server side pagination bar of the new class SMPPagination instead. It mirrors the layout of a DataTables - the page size selector and the search field above the content, the information on the shown entries and the page links below it - but it is built from plain links and a GET form only, because the Content Security Policy of the application does not allow inline event handlers.
  The page index, the page size and the search text are request parameters, so that a single page of the tree can be bookmarked. The service information is queried per service group of the current page, instead of in one full scan.
• The details view of a Service Group now also shows all Document Types, Processes and Endpoints as well as all Redirects assigned to it - previously it only showed the participant ID, the owner, the extension, the custom properties and the Business Card, so that the assigned endpoints had to be looked up on the Endpoint Tree page.
  The endpoints are shown as the same tree as on the Endpoint Tree page, including the transport profile, the validity and the endpoint reference of every endpoint. Every Document Type carries the link to perform an SMP query on it, and every endpoint and every redirect links to its own details page.
  Only the service information and the redirects of the shown service group are queried, so that the effort of the page does not depend on the size of the SMP.
• (SQL) Fixed that the SQL backend created a paged query without any ORDER BY clause, if the request did not contain an explicit sort order - as it is the case for the initial request of a page. The rows of a page were therefore returned in an undefined order, so that consecutive pages could overlap or lose rows.
• Added the new method ISMPTableColumn.getDefaultSortOrder (), so that the default order of a domain object is declared explicitly instead of being implied by the first column of the enum.
  All columns with a non-null sort order form the default order together, in their declaration order, which is how a composite key is expressed - Redirects and Endpoints e.g. sort by service group and then by document type, exactly as the hand written SQL did before the server side paging was introduced.
  The default order is used by all three backends whenever a client requests no order, or only unknown respectively non-sortable ones - see SMPTableColumnHelper.getAllDefaultSortColumns (...) and getAllDefaultSortFields (...); for the SQL backend it is passed on to DBPagingHelper of ph-db 8.4.3.
  It is also the initial order of the DataTables of the respective page, which is derived from the very same declaration by SMPDataTablesOnDemand.createDataTables (...), so that the order shown in the UI and the order the backend falls back to cannot drift apart.
• (SQl, MongoDB) The SQL and the MongoDB backend now also apply the search natively, instead of falling back to loading all entities into memory. The search text is always passed as a prepared statement parameter respectively as a quoted regular expression, and LIKE wildcards (% and _) contained in the search text are escaped, so that they are matched literally.
  Added SMPJDBCQueryHelper (SQL) and SMPMongoQueryHelper (MongoDB) for that; the database specific ORDER BY and LIMIT/OFFSET rendering comes from ph-db 8.4.2.
  The XML backend uses the new getAllPaged (...) of the ph-commons map based DAO.
• Incompatible change: the manager methods getAllSMPServiceGroups (int, int), getAllSMPServiceGroups (String, int, int) and their counterparts for Service Information, Redirects and Business Cards were replaced by a single getAll... (IPagingSpec, String). The static isMatchingSearchText (...) methods of the four manager interfaces and the classes SMPPagingHelper and SMPJDBCPagingHelper were removed - their behaviour now lives in ISMPTableColumn respectively in ph-commons and ph-db.
• The name of a Service Group export file now contains a random UUID next to the creation date and time, so that two exports created within the same second no longer overwrite each other. The created name is additionally passed through FilenameHelper.getAsSecureValidASCIIFilename (String), so that it is a valid filename on all platforms.
  Based on #536 - thx @vinit-thummar
• Added the new backend aware readiness endpoint /smp-ready next to the existing liveness endpoint /ping. It answers HTTP 200 with {"ready":true} if the configured backend can currently serve requests, and HTTP 503 with {"ready":false} if it cannot, so that a Kubernetes readinessProbe stops routing traffic to a pod whose database is unavailable, without the restart loop a livenessProbe would cause. The endpoint deliberately returns nothing but the boolean - the diagnostic details stay in /smp-status.
  Based on #537 - thx @vinit-thummar. See #529.
  Added the new classes SMPReadyServlet, SMPReadyXServletHandler and SMPReadyProvider as well as the new SPI interface ISMPReadyProviderExtensionSPI. Every backend provides exactly one implementation of it - SMPXMLReadyProviderExtensionSPI (always ready), SMPSQLReadyProviderExtensionSPI and SMPMongoDBReadyProviderExtensionSPI. If no implementation is found at all, the SMP reports itself as not ready, so that a backend missing its readiness check cannot be mistaken for a healthy one.
  The readiness checks run with a time limit, configurable via the new configuration property smp.ready.timeout (default 2 seconds), because a readiness request must never block an HTTP thread indefinitely. Without it a silently unreachable database - one that keeps the TCP connection open but answers nothing - blocks DataSource.getConnection() forever, well beyond jdbc.pooling.max-wait.
  Added the new method SMPServerConfiguration.getReadyTimeout ().
• Added the new methods SMPDataSourceSingleton.isDBConnectionPossible () and SMPDataSourceSingleton.isDBConnectionPossible (int) and the new constant SMPDataSourceSingleton.DEFAULT_CONNECTION_VALIDATION_TIMEOUT_SECONDS. They replace the internal database check of SMPSQLStatusProviderExtensionSPI and additionally validate the received connection, because the connection pool hands out pooled connections unvalidated as long as jdbc.pooling.test-on-borrow is disabled - which is the default. The status item smp.sql.db.connection-possible therefore now also detects a database that died after the connection was pooled.
• The automatic update of the Directory is now performed asynchronously in the PhotonWorkerPool, so that a slow or an unreachable Directory no longer blocks the SMP request that triggered the update. That affects creating and updating a Business Card, deleting a Business Card - including the deletion caused by deleting a Service Group - as well as creating, updating and deleting Service Information of a participant that has a Business Card.
  See #481 - thx @wartab.
  The SMP settings and the Directory client are still resolved in the calling thread, so that the background task requires no open request scope, and the "add" and the "delete" call of a Business Card deletion are performed in a single task, so that their order is maintained. The outcome of the Directory call is therefore no longer part of the HTTP response respectively of the UI feedback - as before it is written to the log file.
  The explicit push actions stay synchronous, because reporting the outcome of the push is their sole purpose: the REST API PUT /businesscard/{ServiceGroupId}/push and the "Update Business Card in " action of the Business Card page. The "Update all participants in Directory" action was already running in the background before.
  Additionally, if the automatic update is enabled but no Directory host name is configured, a warning is logged instead of failing with a NullPointerException.
• Added the new database migration V36 for the SQL backend. It changes the columns creationuserid, lastmoduserid and deleteuserid of the tables smp_secrole, smp_secuser, smp_secusergroup and smp_secusertoken as well as the column userid of the table smp_audit from varchar(20) to varchar(45), for MySQL, PostgreSQL, Oracle, DB2 and SQL Server.
  All user IDs are limited to 45 characters now, so that a user ID created by GlobalIDFactory - which may be up to GlobalIDFactory.STRING_ID_MAX_LENGTH (40) characters long - fits into every column referring to a user. Previously only the id columns of the security tables were varchar(45), whereas the columns referring to the creating, the modifying and the deleting user as well as the executing user of an audit item were varchar(20).
  Only the length of the columns is changed - no column is added, removed or renamed, and the nullability stays as it was, so existing data is preserved. See the Migrations page for details.
• REST API requests that authenticate via HTTP Basic Auth or a Bearer token are now recorded in the audit log with the ID of the authenticated user. Previously they were recorded as $GUEST$, because a REST request is deliberately not logged into a UI session and the audit therefore saw no current user at all.
  Based on #538 - thx @vinit-thummar. See #238.
  After the credentials were validated successfully, SMPUserManagerPhoton.validateUserCredentials (...) remembers the user ID in the new ph-oton 10.5.0 class RequestUserIDProvider, which keeps it in the request scope only. A REST request therefore still creates no HTTP session, and the remembered user ID can neither leak into a subsequent request nor replace the session user of an existing UI login.
  The MongoDB audit manager now resolves the current user via GlobalUserIDProvider.getCurrentUserID (), which prefers the request user over the session user of the LoggedInUserManager. The XML and the SQL backend use the ph-oton default provider, which behaves the same way.
  Added the tests SMPAPIUserIDTest - covering Basic Auth, Bearer token, invalid credentials, that no session is created and that the identity does not leak into the next request - and AuditUserInterfaceTest, which performs real authenticated HTTP requests against the SQL backend and verifies the persisted audit rows.

What's Changed

• Prevent Service Group export filename collisions by @vinit-thummar in #536
• Add backend-aware readiness endpoint by @vinit-thummar in #537
• Server side pagination by @phax in #534
• Record authenticated REST users in audit entries by @vinit-thummar in #538

Full Changelog: phoss-smp-parent-pom-8.3.0...phoss-smp-parent-pom-8.4.0