Skip to content

Conversation

@rcooke-warwick
Copy link
Contributor

@rcooke-warwick rcooke-warwick commented Oct 11, 2023

  • there's been interest in running tests on a phsyical DUT, but no access to autokit hardware
  • this limits some functionality - mainly concerned with flashing, but also hdmi, bluetooth, and some networking manipulation, and GPIO
  • however many of the OS tests actually are fine - so this PR will have a collection of changes to allow for running tests on a physical DUT, from your laptop

How to use:
set your workspace/config.js to:

module.exports = [{
    deviceType: '<DEVICE_TYPE_SLUG',
    suite: `${__dirname}/../suites/os`,
    config: {
        networkWired: false,
        networkWireless: false,
        downloadVersion: 'latest',
        balenaApiKey: '<API_KEY>',
        balenaApiUrl: 'balena-cloud.com',
        organization: '<YOUR_BALENA_CLOUD_USERNAME',
        dutIp: '<LOCAL_IP_OF_YOUR_DUT',
        dutUuid: '<UUID_OF_YOUR_DUT',
	manual: true
    },
    debug: {
        unstable: ["SSH authentication test"],
    },
    image: false, 
    workers: ['http://worker'],
}]

Test suite specific caveats:
OS suite: Must put device in local mode first (via dashboard). Also manually enable persistent logging in the dashboard too. If your device has wifi, provision it to use your wifi (same network as laptop). Enable persistent logging via the dashboard device configuration tab
Cloud suite: Must not be in local mode, "SSH authentication test" must be set as unstable
Hup suite: Annoying in general - make sure you provision your DUT into the OLD OS version you want to hup from. Then in the image property put the path to the <hostapp>.docker hostapp image, instead of false

You must run this branch of the tests from meta balena: https://github.com/balena-os/meta-balena/tree/ryan/manual_tests2

You must provision your DUT manually to a balena cloud fleet, with no release, and your organization must have access to it. The DUT must be on the same network as your laptop. Ensure the DUT is in development mode first. Then run:

make local-test

If there are ssh authentication errors, then the organization you put in config.js doesn't have access to the DUT you provisioned.

Other considerations:

 sshConfig: {
                host: 'ssh.balena-staging-devices.com',
                port: '22',
    }

should be added in the above config.js, right after manual:true, only if testing in staging.

The device should have both local mode enabled in the dashboard, and developmentMode:true in it's config.json, for the OS suite. Bluetooth and HDMI is not tested

If the OS test suite need to be re-run, /mnt/boot/system-proxy should be deleted, the device should then be re-booted and local-mode should be toggled to off and back on prior to restarting the test suite. Depending on the leftovers from a failed or interrupted test suite, re-flashing may be necessary.

On particular devices that also have a p2p-dev-wlP1p1s0 interface for example, the connectivity test should be modified to

 let connection = adaptor === 'wireless' ? 'wifi' : 'Wired';
                                                return this.worker.executeCommandInHostOS(
                                                        `nmcli d  | grep '${connection}' | grep connected | awk '{print $1}' | head -n1`,

For the cloud suite, the following line should be changed to use "that" instead of "this": https://github.com/balena-os/meta-balena/blob/ryan/manual_tests/tests/suites/cloud/suite.js#L248

Change-type: patch
Signed-off-by: Ryan Cooke <ryan@balena.io>
Change-type: patch
Signed-off-by: Ryan Cooke <ryan@balena.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants