Backend bug-fix cluster, mail XLSX export, and select-all sync reach QA
Executive summary
A batch promotion of the backend Dev branch into the QA environment (25 files, +131/-667). It bundles a cluster of stability fixes for roles, document types, contact/department deletion, comments, and password-reset emails, adds a new "export selected mails to Excel" capability and select-all support for payment sync, and corrects image-upload handling plus the Dev image-build pipeline.
Why this was needed
Several backend defects surfaced during QA: role updates failed when an Auth0/IAM record was out of sync, document-type counts and pagination behaved inconsistently, bulk contact/department deletes returned unhelpful errors, comment lookups missed singular module names, and password-reset links could fall back to a bare token. Image uploads were also being treated as PDFs, and the bulk payment-sync flow lacked a select-all path while capping batches at 100.
Client / user impact
- Users can export a selected set of mails directly to an Excel (.xlsx) file using the same columns and permissions as the mail grid.
- Editing or deleting a role no longer breaks when the IAM record is missing; the local record is updated and fully cleaned up on delete.
- Document-type listings support an option to return all results (up to 1000) and a clearer count of active, not-yet-completed documents.
- Bulk delete of contacts/departments now returns a meaningful rejection message instead of a generic 500 error.
- Non-PDF uploads (e.g. images) import correctly without spurious PDF-parsing errors.
- Payment sync can target an entire filtered selection and is no longer capped at 100 items.
Technical scope
Grounded in the diff (DSM_Backend, env=QA). Notable changes:
- New endpoint
POST /download_files/mails/bulk/xlsx— exports selected/select-all mails via the canonical grid export service (download_files.py). - Payment sync (
documents_sync.py): typedBulkPaymentsSyncRequest, addedselect_all/filters/excluded_ids viaBulkSelectionService, de-duplicates IDs, and removes the hard 100-document cap; newSYNCbulk-operation type. - Document types (
document_type.py): addedenable_paginationandexclude_completedquery flags; refactored count/pagination into helpers. - Roles (
admin_roles.py): tolerate IAM 404 by applying a local projection update; on delete, also purge role-permission, user-role, and group-role-mapping rows. - Document upload (
document_service.py,documents.py): default content-type toapplication/octet-streamand skip pypdf page-counting for non-PDFs. - Tenant/department/contact deletes (
tenant_management.py): surfaceHTTPExceptiondetail as a clean error response. - Mail update (
mails.py): preflight that blocks select-all transition to "Processing" for unassigned mails (422 with details). - Password reset (
auth_service.py,iam_password_reset.py): consolidate link building, prefer env template / FRONTEND_BASE_URL, and fail loudly if unconfigured. - Comments (
comments_services.py): shared module-key normalization with singular/plural aliases. - CI (
docker-ecr.yml): drop GovCloud ECR push, build Dev image aslinux/arm64. - Pydantic decorator-order fixes in
mail_schemas.py; new regression unit tests and lint baseline updates.
Risk & mitigation
Medium. The IAM-404 fallback and expanded role-delete cascade change deletion semantics, and removing the 100-item sync cap allows much larger sync batches (mitigated by the existing Celery background-job threshold). The password-reset change now raises an error when no URL is configured, so the relevant env vars must be set in QA. The Dev image switched to arm64, so the Dev server architecture must match. Most changes are additive or guarded by try/except, limiting blast radius.
QA validation focus
- Export selected mails and select-all to XLSX; confirm columns, filters, and tenant scoping match the mail grid.
- Update and delete a role whose IAM record is missing; verify local update succeeds and delete removes role-permission/user-role/group-mapping rows.
- List document types with
enable_pagination=false(all results) and withstatus_id+exclude_completed; verify counts. - Bulk-delete contacts/departments that should be rejected; confirm a clear error message and status code.
- Upload an image (non-PDF); confirm it imports with one page and no PDF-parsing error.
- Run payment sync with select-all over a large filtered set; confirm de-duplication and background-job enqueue.
- Attempt select-all mail transition to Processing with unassigned mails; expect a 422 listing affected mails.
- Trigger a password reset; confirm the link is correct and that a missing config fails loudly.