[PR #6091] (WIP) Optimize test workflows #6307

New Issue

2025-11-27T00:22:06-06:00

GiteaMirror commented

2025-11-27 00:22:06 -06:00

Original Pull Request: https://github.com/bitwarden/android/pull/6091

State: open
Merged: No

🎟️ Tracking

📔 Objective

This PR restructures the test workflow to execute tests in parallel across multiple jobs, with the goal of significantly reducing CI runtime. The workflow now splits testing into 5 concurrent jobs instead of a single sequential job.

Workflow Structure

Before: Single job running all tasks sequentially
After: 5 parallel jobs with specialized responsibilities

┌─────────────────────────────────────────────────────────────────┐
│ OLD: Single Job (Sequential Execution)                          │
├─────────────────────────────────────────────────────────────────┤
│ detekt → lint → test all modules → coverage → upload            │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ NEW: 5 Parallel Jobs (Concurrent Execution)                     │
├─────────────────────────────────────────────────────────────────┤
│ ┌───────────────────┐  ┌─────────────────┐                      │
│ │ Lint & Static     │  │ Test Libraries  │                      │
│ │ Analysis          │  │ (core, data,    │                      │
│ │                   │  │  network, ui,   │                      │
│ └───────────────────┘  │  etc.)          │                      │
│                        │                 │                      │
│ ┌───────────────────┐  └─────────────────┘                      │
│ │ Test App          │                                           │
│ │ (5,447 tests)     │  ┌─────────────────┐                      │
│ │                   │  │ Test            │                      │
│ └───────────────────┘  │ Authenticator   │                      │
│                        │                 │                      │
│                        └─────────────────┘                      │
│                                                                 │
│                        ┌─────────────────┐                      │
│                        │ Aggregate       │                      │
│                        │ Coverage        │                      │
│                        │                 │                      │
│                        └─────────────────┘                      │
└──────────────────────────────────────────────────────────────���──┘

Jobs Breakdown

1. `lint-and-static-analysis`

Runs detekt for code quality checks
Executes lintStandardDebug and lintDebug
Provides early failure detection (typically fastest to fail)

2. `test-libraries`

Tests all library modules: :core, :data, :network, :ui, :authenticatorbridge, :cxf
Generates XML coverage reports for each module
Represents ~10% of total test suite (423 tests)

3. `test-app`

Tests :app module (5,447 tests - 88.5% of total)
Generates XML coverage report
Critical path bottleneck (longest-running job)

4. `test-authenticator`

Tests :authenticator module (281 tests - 4.6% of total)
Generates XML coverage report

5. `aggregate-coverage`

Downloads coverage reports from all test jobs
Uploads merged coverage to codecov.io
Depends on: test-libraries, test-app, test-authenticator

Technical Changes

1. Enable Gradle Configuration Cache

File: gradle.properties

 org.gradle.caching=true
+org.gradle.configuration-cache=true
+org.gradle.configuration-cache.problems=warn
 org.gradle.jvmargs=-Xmx4g -XX:+UseParallelGC -Dfile.encoding=UTF-8
 org.gradle.parallel=true

Benefits:

Faster Gradle configuration phase through caching
Reduced overhead across all jobs
Cumulative time savings with repeated builds

2. Restructure Test Workflow

File: .github/workflows/test.yml

Key Changes:

Split single test job into 5 specialized jobs
Each job has dedicated test and coverage generation
Coverage reports generated immediately after tests (same Gradle session)
Codecov automatically merges multiple XML reports

Coverage Strategy:

Each test job generates its own Kover XML report
Reports uploaded as artifacts (coverage-libraries, coverage-app, coverage-authenticator)
Aggregation job downloads all reports and uploads to codecov
No dependency on Fastlane for coverage merging (codecov handles it)

3. Combined Test + Coverage Commands

Tests and coverage generation run in a single Gradle invocation to ensure Kover has access to test execution data:

# App module
./gradlew :app:testStandardDebug :app:koverXmlReportStandardDebug

# Authenticator module
./gradlew :authenticator:testDebug :authenticator:koverXmlReportDebug

# Library modules (all in one command)
./gradlew \
  :core:testDebug :core:koverXmlReportDebug \
  :data:testDebug :data:koverXmlReportDebug \
  :network:testDebug :network:koverXmlReportDebug \
  :ui:testDebug :ui:koverXmlReportDebug \
  :authenticatorbridge:testDebug :authenticatorbridge:koverXmlReportDebug \
  :cxf:testDebug :cxf:koverXmlReportDebug

Running both tasks together in the same Gradle daemon session ensures the coverage report generation can access the test execution data produced by the test task.

Performance Impact

Expected Improvements

Metric	Before	After
Critical Path	All tasks sequential	Longest job (test-app) - better utilization
Failure Detection	Late (after all tests)	Early (lint fails first) - faster feedback
Resource Usage	1 runner	5 concurrent runners - better parallelization

Test Distribution

6,151 total tests across all modules
5,447 tests (88.5%) in :app module (still bottleneck)
423 tests (6.9%) in library modules (completes quickly)
281 tests (4.6%) in :authenticator module

Testing & Validation

Local Testing

✅ Configuration cache enabled and tested locally
✅ Test execution with coverage generation validated
✅ All modules generate XML reports correctly

CI Validation Checklist

All 5 jobs run in parallel
Test failures reported correctly per-job
Coverage artifacts upload successfully
Codecov receives and merges coverage data
Total coverage % matches baseline
Performance improvement achieved

Future Optimization Opportunities

This PR implements Tier 1 (Quick Wins) optimizations. Additional improvements are possible:

Tier 2: App Module Sharding

Split :app module's 5,447 tests across 3-4 parallel jobs
Package-based sharding (auth, vault, platform, UI)
Further reduce critical path bottleneck

Tier 3: Strategic Investments

Remote Gradle build cache (Gradle Enterprise or self-hosted)
Predictive test selection (run only affected tests on PRs)
Feature module extraction (break up monolithic :app)

Risks & Mitigations

Risk: Coverage Reporting Changes

Mitigation: Codecov's XML merging is battle-tested and widely used. Coverage % should match previous implementation.

Risk: Workflow Complexity Increase

Mitigation: Clear job naming, comprehensive documentation, each job remains simple and focused.

Risk: More CI Runner Usage

Impact: Uses 5 concurrent runners instead of 1. Total runner-minutes may be similar or slightly higher, but wall-clock time is significantly reduced.

Risk: Configuration Cache Issues

Mitigation: Enabled with problems=warn to identify compatibility issues without failing builds. Can be disabled if problems arise.

Rollback Plan

If issues arise:

Revert this PR to return to single-job workflow
Disable configuration cache by reverting gradle.properties changes
All changes are isolated to workflow and properties files

📸 Screenshots

Coming soon!

⏰ Reminders before review

Contributor guidelines followed
All formatters and local linters executed and passed
Written new unit and / or integration tests where applicable
Protected functional changes with optionality (feature flags)
Used internationalization (i18n) for all UI strings
CI builds passed
Communicated to DevOps any deployment requirements
Updated any necessary documentation (Confluence, contributing docs) or informed the documentation team

🦮 Reviewer guidelines

👍 (:+1:) or similar for great changes
📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
❓ (:question:) for questions
🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion
🎨 (:art:) for suggestions / improvements
❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
⛏ (:pick:) for minor or nitpick changes

**Original Pull Request:** https://github.com/bitwarden/android/pull/6091 **State:** open **Merged:** No --- ## 🎟️ Tracking  ## 📔 Objective This PR restructures the test workflow to execute tests in parallel across multiple jobs, with the goal of significantly reducing CI runtime. The workflow now splits testing into 5 concurrent jobs instead of a single sequential job. ### Workflow Structure **Before:** Single job running all tasks sequentially **After:** 5 parallel jobs with specialized responsibilities ``` ┌─────────────────────────────────────────────────────────────────┐ │ OLD: Single Job (Sequential Execution) │ ├─────────────────────────────────────────────────────────────────┤ │ detekt → lint → test all modules → coverage → upload │ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ NEW: 5 Parallel Jobs (Concurrent Execution) │ ├─────────────────────────────────────────────────────────────────┤ │ ┌───────────────────┐ ┌─────────────────┐ │ │ │ Lint & Static │ │ Test Libraries │ │ │ │ Analysis │ │ (core, data, │ │ │ │ │ │ network, ui, │ │ │ └───────────────────┘ │ etc.) │ │ │ │ │ │ │ ┌───────────────────┐ └─────────────────┘ │ │ │ Test App │ │ │ │ (5,447 tests) │ ┌─────────────────┐ │ │ │ │ │ Test │ │ │ └───────────────────┘ │ Authenticator │ │ │ │ │ │ │ └─────────────────┘ │ │ │ │ ┌─────────────────┐ │ │ │ Aggregate │ │ │ │ Coverage │ │ │ │ │ │ │ └─────────────────┘ │ └──────────────────────────────────────────────────────────────��──┘ ``` ### Jobs Breakdown #### 1. `lint-and-static-analysis` - Runs `detekt` for code quality checks - Executes `lintStandardDebug` and `lintDebug` - Provides early failure detection (typically fastest to fail) #### 2. `test-libraries` - Tests all library modules: `:core`, `:data`, `:network`, `:ui`, `:authenticatorbridge`, `:cxf` - Generates XML coverage reports for each module - Represents ~10% of total test suite (423 tests) #### 3. `test-app` - Tests `:app` module (5,447 tests - 88.5% of total) - Generates XML coverage report - Critical path bottleneck (longest-running job) #### 4. `test-authenticator` - Tests `:authenticator` module (281 tests - 4.6% of total) - Generates XML coverage report #### 5. `aggregate-coverage` - Downloads coverage reports from all test jobs - Uploads merged coverage to codecov.io - Depends on: `test-libraries`, `test-app`, `test-authenticator` ## Technical Changes ### 1. Enable Gradle Configuration Cache **File:** `gradle.properties` ```diff org.gradle.caching=true +org.gradle.configuration-cache=true +org.gradle.configuration-cache.problems=warn org.gradle.jvmargs=-Xmx4g -XX:+UseParallelGC -Dfile.encoding=UTF-8 org.gradle.parallel=true ``` **Benefits:** - Faster Gradle configuration phase through caching - Reduced overhead across all jobs - Cumulative time savings with repeated builds ### 2. Restructure Test Workflow **File:** `.github/workflows/test.yml` **Key Changes:** - Split single `test` job into 5 specialized jobs - Each job has dedicated test and coverage generation - Coverage reports generated immediately after tests (same Gradle session) - Codecov automatically merges multiple XML reports **Coverage Strategy:** - Each test job generates its own Kover XML report - Reports uploaded as artifacts (`coverage-libraries`, `coverage-app`, `coverage-authenticator`) - Aggregation job downloads all reports and uploads to codecov - No dependency on Fastlane for coverage merging (codecov handles it) ### 3. Combined Test + Coverage Commands Tests and coverage generation run in a single Gradle invocation to ensure Kover has access to test execution data: ```yaml # App module ./gradlew :app:testStandardDebug :app:koverXmlReportStandardDebug # Authenticator module ./gradlew :authenticator:testDebug :authenticator:koverXmlReportDebug # Library modules (all in one command) ./gradlew \ :core:testDebug :core:koverXmlReportDebug \ :data:testDebug :data:koverXmlReportDebug \ :network:testDebug :network:koverXmlReportDebug \ :ui:testDebug :ui:koverXmlReportDebug \ :authenticatorbridge:testDebug :authenticatorbridge:koverXmlReportDebug \ :cxf:testDebug :cxf:koverXmlReportDebug ``` Running both tasks together in the same Gradle daemon session ensures the coverage report generation can access the test execution data produced by the test task. ## Performance Impact ### Expected Improvements | Metric | Before | After | |--------|--------|-------| | **Critical Path** | All tasks sequential | Longest job (test-app) - better utilization | | **Failure Detection** | Late (after all tests) | Early (lint fails first) - faster feedback | | **Resource Usage** | 1 runner | 5 concurrent runners - better parallelization | ### Test Distribution - **6,151 total tests** across all modules - **5,447 tests (88.5%)** in `:app` module (still bottleneck) - **423 tests (6.9%)** in library modules (completes quickly) - **281 tests (4.6%)** in `:authenticator` module ## Testing & Validation ### Local Testing - ✅ Configuration cache enabled and tested locally - ✅ Test execution with coverage generation validated - ✅ All modules generate XML reports correctly ### CI Validation Checklist - [x] All 5 jobs run in parallel - [x] Test failures reported correctly per-job - [x] Coverage artifacts upload successfully - [ ] Codecov receives and merges coverage data - [ ] Total coverage % matches baseline - [ ] Performance improvement achieved ## Future Optimization Opportunities This PR implements **Tier 1 (Quick Wins)** optimizations. Additional improvements are possible: ### Tier 2: App Module Sharding - Split `:app` module's 5,447 tests across 3-4 parallel jobs - Package-based sharding (auth, vault, platform, UI) - Further reduce critical path bottleneck ### Tier 3: Strategic Investments - Remote Gradle build cache (Gradle Enterprise or self-hosted) - Predictive test selection (run only affected tests on PRs) - Feature module extraction (break up monolithic `:app`) ## Risks & Mitigations ### Risk: Coverage Reporting Changes **Mitigation:** Codecov's XML merging is battle-tested and widely used. Coverage % should match previous implementation. ### Risk: Workflow Complexity Increase **Mitigation:** Clear job naming, comprehensive documentation, each job remains simple and focused. ### Risk: More CI Runner Usage **Impact:** Uses 5 concurrent runners instead of 1. Total runner-minutes may be similar or slightly higher, but wall-clock time is significantly reduced. ### Risk: Configuration Cache Issues **Mitigation:** Enabled with `problems=warn` to identify compatibility issues without failing builds. Can be disabled if problems arise. ## Rollback Plan If issues arise: 1. Revert this PR to return to single-job workflow 2. Disable configuration cache by reverting `gradle.properties` changes 3. All changes are isolated to workflow and properties files ## 📸 Screenshots Coming soon! ## ⏰ Reminders before review - Contributor guidelines followed - All formatters and local linters executed and passed - Written new unit and / or integration tests where applicable - Protected functional changes with optionality (feature flags) - Used internationalization (i18n) for all UI strings - CI builds passed - Communicated to DevOps any deployment requirements - Updated any necessary documentation (Confluence, contributing docs) or informed the documentation team ## 🦮 Reviewer guidelines  - 👍 (`:+1:`) or similar for great changes - 📝 (`:memo:`) or ℹ️ (`:information_source:`) for notes or general info - ❓ (`:question:`) for questions - 🤔 (`:thinking:`) or 💭 (`:thought_balloon:`) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion - 🎨 (`:art:`) for suggestions / improvements - ❌ (`:x:`) or ⚠️ (`:warning:`) for more significant problems or concerns needing attention - 🌱 (`:seedling:`) or ♻️ (`:recycle:`) for future improvements or indications of technical debt - ⛏ (`:pick:`) for minor or nitpick changes

GiteaMirror added the pull-request label 2025-11-27 00:22:06 -06:00

[PR #6091] (WIP) Optimize test workflows #6307