Engineering ISO-Compliant Audit Trails for a Private Equity Data Platform

Background

The challenge

• Two junior manual testers were already on the project, but what they had inherited was not a test suite in any professional sense - cases were spread across multiple Excel spreadsheets in several conflicting versions.
• The cases themselves were closer to observation notes: do a few things, see what changed. No defined steps, no preconditions, no expected results.
• Product knowledge had never been formally documented, so without it the inherited files were close to unworkable. This was the baseline Digital Archer was handed - and held accountable for.
• No automation existed, no reporting cadence, and no shared definition of what full coverage would even look like.
• Continuous delivery pace across three distinct deployment types - infrastructure, data science, and core functionality - which had until then been verified as if they were equivalent.
• ISO certification requirements applied from day one, including how test evidence was logged and stored.

Our approach

• Migrated 2,500+ Excel observation notes into TestRail as properly structured test cases with defined steps, preconditions, and expected results - across core functionality, smoke, security, and load categories.
• Ran that migration alongside live regression cycles rather than pausing for it: the team executed the old cases, cleaned them, and filed coverage reports at the same time. There was no clean handoff between phases.
• Introduced a risk-based testing strategy, ranking cases by business impact and failure likelihood so every release cycle covered the highest-risk areas first - regardless of how much of the suite was fully defined at that point.
• Split smoke testing by deployment type. Infrastructure changes, data science deployments, and core functionality updates each got a tailored suite, because treating them as equivalent had been a source of missed issues.
• Built an end-to-end automation framework from scratch in Java, JUnit, and Playwright, integrated into GitHub Actions with scheduled runs across internal and production environments and parallel execution designed in from the start.
• Automated structured daily reporting through TestRail, so coverage and results stayed visible without anyone assembling them by hand.
• Implemented ISO-compliant logging and results storage with the DevOps team - test outputs transferred to an S3 bucket with controls in place to prevent sensitive-data leakage.
• Built a second, standalone Java and JUnit application to solve a separate problem: documents were being lost during acquisition from a third-party API portal. Containerized and deployed to AWS Lambda, triggered by SQS on each run of the acquisition cronjob, it compared production database records against documents processed in real time and surfaced anything missing.
• Compiled an application feature map in parallel, linking platform features to the specific test cases that cover them.
• Held the reporting rhythm steady for the full engagement: daily coverage reports, weekly QA meetings, and monthly QA newsletters.

Working against the grain

• Test environment integrity was low throughout. Failures appeared in the test environment that could not be reproduced in production, and vice versa, which made it structurally difficult to draw reliable conclusions from a test run.
• The ML-based document extraction added another layer: because the parsing models did not always return identical outputs across runs, any test case relying on an exact data match was inherently unstable. Expected results could not always be fixed in advance. That is a property of the system, not a testing failure.
• The QA function was nonetheless held accountable for defects that reached production, including in areas where no expected result had ever been defined.
• Digital Archer's response was to close that gap directly - demos and alignment sessions were organized two to three times a month to bring product and engineering together, surface the structural issues, and agree on realistic expectations.
• Those sessions were declined or redirected; the feedback was that the detail was not needed, only that everything should work. The process was kept running anyway, and the reporting cadence never lapsed.

Results

• A fully functional E2E automation framework (Java, JUnit, Playwright) with GitHub Actions integration, parallel execution, and ISO-compliant S3 logging - which the client kept running after the engagement ended.
• A revamped TestRail suite of more than 2,500 structured test cases, replacing the inherited Excel files entirely.
• A deployed document acquisition validation service (Java, AWS Lambda, SQS) that resolved the missing-document problem and improved the reliability of the acquisition pipeline.
• Tailored smoke test suites for all three deployment types, developed and maintained across the full engagement.
• An application feature map linking platform features to test coverage across a significant portion of the platform.
• A consistent QA reporting rhythm - daily coverage reports, weekly QA meetings, monthly newsletters - sustained for 15 months.

Key technologies and deliverables

Test management

TestRail - 2,500+ cases migrated and structured

Automation framework

Java, JUnit, Playwright

CI/CD integration

GitHub Actions - scheduled and parallel runs

Cloud infrastructure

AWS Lambda, SQS, S3

Containerization

Docker

Reporting

Automated daily reports, monthly QA newsletters

Compliance

ISO-aligned logging and results storage