AsaiYusuke / 6502_test_executor

A cross-platform unit testing tool for MOS 6502 assembly

6502 Unit Test executor

This tool enables unit testing for MOS Technology 6502 assembly programs on a cross-platform basis.

Table of Contents

• Basic Design
• Supported Testing Features
  • Testable Program
  • Conditions
  • Assertions
  • Coverage
  • Test Classification
  • Additional Useful Features
• Getting Started
  • Prerequisites
  • Install
  • Running Examples
    • Unit Testing
    • Coverage File
• Usage
  • Build CA65 Project with Debug Option
  • Create Unit Test
  • Run Test
• Test Scenario Examples
• Dependencies
• License

Basic Design

Execute Tests on Built-In Emulator

• The built-in 6502 emulation by Gianluca Ghettini enables stand-alone testing on platforms such as Linux.
• There's no need to add hooks to the product code to invoke tests.

Write Tests with JSON Schema

• The tool provides JSON Schema document that simplifies test creation.

Evaluate Tests like Modern Frameworks

• It includes many useful evaluation methods found in the modern UNIT testing frameworks.

Supported Testing Features

Testable Program

• Test targets
  • JSR procedure
  • NMI procedure
  • IRQ procedure
  • Address range
• Type of interrupts that can occur during testing
  • NMI
  • IRQ
• Mockable call instructions that set the results in registers and memory
  • JSR
  • JMP

Conditions

• Setting up device conditions before testing

  	Register	Memory	Stack
  Set value	✔️	✔️	✔️
  Mock read value history		✔️

• Evaluating device conditions after testing

  	Register	Memory	Stack
  Check value	✔️	✔️	✔️
  Check read/write count	✔️	✔️
  Check write value history		✔️

• Evaluating processor information
  • Cycle count

Assertions

• Assertion operators that determine how tests are evaluated
  • "eq" (Equal to)
  • "ne" (Not equal to)
  • "gt" (Greater than)
  • "ge" (Greater than or equal to)
  • "lt" (Less than)
  • "le" (Less than or equal to)
  • "anyOf" (Any of)
    Representing logical OR operator
  • Composite operators
    Same as logical AND operator. e.g. "gt"+"lt" for range selection
• Error handling
  • Write access to readonly memory
    Able to detect write operations to readonly memory.
  • Read access from uninitialized memory
    Able to detect read operations from memory that has never had a write operation.
  • Access to undefined memory
    Able to detect memory operations using addresses outside the segment defined at build time.
  • Access to unauthorized memory
    Able to detect access outside the memory for which authorized area has been defined.
  • Illegal instruction
    Able to detect illegal instruction code execution.

Coverage

• Testable coverage
  • LCOV format

Test Classification

• Aggregating tests for the entire project
• Common configuration settings for the entire project
• Grouping of individual tests

Additional Useful Features

• When a test terminates with an error, its callback trace can be output.
• Test evaluations can be executed at any memory address.
• Test ROM images are detected from debug information file.

Getting Started

Prerequisites

This tool is intended for the projects based on [CC65](https://cc65.github.io/).

The easiest way to install CC65 on [Ubuntu](https://ubuntu.com/) linux is by running:

# sudo apt-get install cc65

Verified CA65 version:

# ca65 --version
ca65 V2.18 - Ubuntu 2.19-1

Install

Since this tool repository includes submodules, you must specify the --recurse-submodules option when cloning:

# git clone --recurse-submodules https://github.com/AsaiYusuke/6502_test_executor.git
# cd 6502_test_executor
# make
# ./6502_tester --help

Running Examples

Unit Testing

The simple example project includes many test cases that demonstrate the features:

# cd example
# make
mkdir -p build
ca65 --cpu 6502 --target nes --debug-info -o build/example.o src/example.asm
mkdir -p dist
ld65  -o dist/example.nes --dbgfile dist/example.dbg --config cfg/nes.cfg --obj build/example.o
rm coverage/lcov.info
../6502_tester --debug=dist/example.dbg --coverage=coverage/lcov.info --segment="CODE" --quiet-summary --quiet-ok -t test/ok/customize.configurations.test.json
../6502_tester --debug=dist/example.dbg --coverage=coverage/lcov.info --segment="CODE" --quiet-summary --quiet-ok -t test/ok/definitions.test.json
../6502_tester --debug=dist/example.dbg --coverage=coverage/lcov.info --segment="CODE" --quiet-summary --quiet-ok -t test/ok/error.timeout.test.json
:
All tests passed.

Coverage File

When the project is built, the coverage file is saved in the example/coverage/lcov.info in the case of the example project.

The coverage file can be used to integrate with tools like Coveralls GitHub Action, and more.

The results of the example project can be seen on Coveralls.

Usage

flowchart LR;
  A(Assembly program);
  C[[CA65]];
  B(Binary);
  D(Debug information);
  V[[Visual Studio Code]];
  S(Test scenario);
  U[[6502 Unit Test executor]];
  R[(Test result)];
  L[(Test coverage)];

  subgraph Build CA65 project
    A --> C --> B & D;
  end

  subgraph Create unit test
    V -- Json schema --> S;
  end

  subgraph Run test
    B & D & S --> U --> R & L;
  end

Build CA65 Project with Debug Option

Build your 6502 project using the CA65 assembler and LD65 linker with debug information generation enabled.

Create Unit Test

Create test scenario files containing three key items in JSON format:

• Test target
  The starting address of the test procedure
• Setup condition
  The settings of the registers and memory before the test
• Expected condition
  The expected responses of the registers and memory after the test

JSON Schema File

The tool also provides a JSON Schema document that simplifies creating test scenario files.

If you use Visual Studio Code, it will provide formatting error notifications and auto-completion based on JSON Schema.

Run Test

Run the tool with the prepared debug information file and test scenario file:

6502_tester -d <debug information> -t <test scenario>

Test coverage can also be measured. Both the coverage file and the segment names used in the debug information file must be specified to enable coverage reporting.

6502_tester -d <debug information> -t <test scenario> -c <coverage> -s <segment names>

You can find all command-line arguments in the help:

# ./6502_tester --help
  ./6502_tester {OPTIONS}

    6502 test executor

  OPTIONS:

      -h, --help                        Show this help menu.
      -d[DEBUG], --debug=[DEBUG]        Specify the path of the debug
                                        information file used for testing.
      -t[TEST], --test=[TEST]           (REQUIRED)
                                        Specify the path of the test scenario
                                        file.
      -c[COVERAGE],
      --coverage=[COVERAGE]             Specify the path of the coverage file.
      -s[SEGMENT], --segment=[SEGMENT]  Specify the segment names for coverage.
      -i[ID], --id=[ID]                 Specify the test ID.
      --timeout=[TIMEOUT]               Specify the timeout period before the
                                        test becomes an error.
      -q, --quiet                       Do not show any output.
      --quiet-ok                        Do not show the successful output.
      --quiet-fail                      Do not show the failed output.
      --quiet-summary                   Do not show the test summary output.

Note:

Some options can be specified either as command-line arguments or test scenario file (See example).
If both are specified, the values in the test scenario file take precedence.

Test Scenario Examples

Register Conditions

• Check value of A/X/Y registers
• Check read/write count of A/X/Y registers
• Check value of Processor status register flags
• Check read/write count of Processor status register flags
• Check machine when PC register arrives at a specific address

Memory Conditions

• Memory addressing
• Check value of memory
• Check read/write count of memory
• Batch selection of continuous memory area
• Sequential change value

Stack Conditions

• Check value of stack
• rts to caller

Processor Conditions

• Cycle count information
• Interrupt hooks
• Call mocked proc

Testable Procedures

• JSR proc
• NMI proc
• IRQ proc
• Address range

Error Handling

• Write access to readonly memory
• Read access from uninitialized memory
• Access to undefined memory
• Access to unauthorized memory
• Illegal instruction

Test Classification

• Aggregated testing
• Common configuration settings
  • Override from project-wide configuration settings
• Grouping of individual tests

Misc

• Customize configurations
• Skip test (ignore)
• Complex evaluation
• Definitions

Dependencies

This project uses the following libraries:

• Gianluca Ghettini's 6502 emulator
  • Erik Lothe's 6502 Unit test executor
• Taylor C. Richberger's args
• Niels Lohmann's JSON for Modern C++

License

This project is available under the MIT license. See the LICENSE file for more information.