The lcc command line tool¶
lcc commands¶
lcc run¶
Like most of the lcc sub-commands, lcc run benefits from the filtering arguments but also
from extra arguments that allow a finer control over tests execution and reporting:
- --project¶
A project path (see Suites discovery for more details).
- --threads¶
The number of threads used to execute tests; default is one thread (meaning tests are run sequentially). The number of threads can also be set through the
$LCC_THREADSenvironment variable (the CLI argument will have precedence over this variable).
- --reporting¶
The list of reporting backends to use, default are: “console”, “html” and “json”. The backends passed as argument will override this list. It can also be customized through the
$LCC_REPORTINGenvironment variable (the CLI argument will have precedence over this variable).Changed in version 1.1.0: It’s possible to add or remove reporting backends to/from the default backends using
+and^prefixes. For instance,--reporting console json html junitto add “junit” can be simplified into--reporting +junit.Other examples:
$ lcc run --reporting ^console # turn off console reporting $ lcc run --reporting ^console +junit # turn off console reporting and turn on junit reporting $ lcc run --reporting html json # only generate the HTML report
Please note that:
turn on/off directives cannot be mixed with a fixed backend list, example:
--reporting console +junitis invalidthe “html” backend generates the static HTML/JS/CSS files required to read the reporting data generated by the “json” backend, meaning that both reporting backends must be specified to have a readable HTML report
- --report-dir¶
The directory where the files generated by the reporting backends will be stored. The default directory is the
reportsub-directory within the project directory, in the case the former report directory will be archived in thereports(with an “s”) sub-directory. The directory can be also be set through the$LCC_REPORT_DIRenvironment variable (the CLI argument will have precedence over this variable).
- --save-report¶
Set the frequency at which reporting backends such as “json” and “xml” will save the reporting data on disk. Possibles values for this option are:
at_end_of_tests,at_each_suite,at_each_test,at_each_failed_test,at_each_log,every_${N}s(the reporting data will be saved every “N” seconds). The default value for this option isat_each_failed_test. This behavior can also be configured through the$LCC_SAVE_REPORTenvironment variable (the CLI argument will have precedence over this variable).
- --exit-error-on-failure¶
lcc runexits with a “0” exit code when the tests have been successfully executed (no matter their status), with this option it will exit with a “1” exit code if any failure (error log of failed check) has been encountered
- --stop-on-failure¶
Stops the execution of the tests on the first non-passed test, remaining tests will be marked as skipped
- --force-disabled¶
Force the execution of disabled tests
lcc check¶
Check if the test project is valid, which means:
the
project.pycan be properly loadedthe suite modules can be properly loaded
fixtures are ok (fixtures are known, scope is ok, no circular dependency, etc…)
metadata policy is respected
- Example:
$ lcc check Everything is ok
lcc show¶
Shows the project tests hierarchy.
$ lcc show * suite_1 - suite_1.test_1 (slow, priority:low) - suite_1.test_2 (priority:low) - suite_1.test_3 (priority:medium, #1235) - suite_1.test_4 (priority:low) - suite_1.test_5 (priority:high) - suite_1.test_6 (slow, priority:high) - suite_1.test_7 (priority:high) - suite_1.test_8 (priority:medium) - suite_1.test_9 (priority:medium) * suite_2 - suite_2.test_1 (priority:low) - suite_2.test_2 (priority:low) - suite_2.test_3 (priority:high) - suite_2.test_4 (priority:medium) - suite_2.test_5 (priority:low) - suite_2.test_6 (priority:low) - suite_2.test_7 (priority:medium) - suite_2.test_8 (slow, priority:low, #1234) - suite_2.test_9 (slow, priority:medium)
lcc diff¶
Compares two reports.
$ lcc diff reports/report-1/ report/ Added tests (1): - suite_3.test_1 (passed) Removed tests (1): - suite_1.test_9 (failed) Status changed (2): - suite_2.test_3 (failed => passed) - suite_2.test_4 (passed => failed)
lcc fixtures¶
Shows available project fixtures.
$ lcc fixtures Fixture with scope pre_run: +---------+--------------+------------------+---------------+ | Fixture | Dependencies | Used by fixtures | Used by tests | +---------+--------------+------------------+---------------+ | fixt_1 | - | 1 | 1 | +---------+--------------+------------------+---------------+ Fixture with scope session: +---------+--------------+------------------+---------------+ | Fixture | Dependencies | Used by fixtures | Used by tests | +---------+--------------+------------------+---------------+ | fixt_2 | fixt_1 | 1 | 2 | | fixt_3 | - | 2 | 1 | +---------+--------------+------------------+---------------+ Fixture with scope suite: +---------+--------------+------------------+---------------+ | Fixture | Dependencies | Used by fixtures | Used by tests | +---------+--------------+------------------+---------------+ | fixt_4 | fixt_3 | 0 | 2 | | fixt_6 | fixt_3 | 1 | 1 | | fixt_5 | - | 0 | 0 | +---------+--------------+------------------+---------------+ Fixture with scope test: +---------+----------------+------------------+---------------+ | Fixture | Dependencies | Used by fixtures | Used by tests | +---------+----------------+------------------+---------------+ | fixt_7 | fixt_6, fixt_2 | 0 | 2 | | fixt_8 | - | 0 | 1 | | fixt_9 | - | 0 | 1 | +---------+----------------+------------------+---------------+
lcc stats¶
Shows project statistics.
$ lcc stats Tags: +------+-------+------+ | Tag | Tests | In % | +------+-------+------+ | slow | 4 | 22% | +------+-------+------+ Properties: +----------+--------+-------+------+ | Property | Value | Tests | In % | +----------+--------+-------+------+ | priority | low | 8 | 44% | | priority | medium | 6 | 33% | | priority | high | 4 | 22% | +----------+--------+-------+------+ Links: +-------+-------------------------+-------+------+ | Name | URL | Tests | In % | +-------+-------------------------+-------+------+ | #1234 | http://example.com/1234 | 1 | 5% | | #1235 | http://example.com/1235 | 1 | 5% | +-------+-------------------------+-------+------+ Total: 18 tests in 2 suites
lcc report¶
Shows a generated report on the console, passing the --short argument will print it the same way as
lcc run does.
$ lcc report Test Organization end-point (github.organization) +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | | Get lemoncheesecake organization information | | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | INFO | GET https://api.github.com/orgs/lemoncheesecake | | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect HTTP code to be equal to 200 | Got 200 | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | INFO | Response | | | | { | | | | "login": "lemoncheesecake", | | | | "id": 28742541, | | | | "node_id": "MDEyOk9yZ2FuaXphdGlvbjI4NzQyNTQx", | | | | "url": "https://api.github.com/orgs/lemoncheesecake", | | | | "repos_url": "https://api.github.com/orgs/lemoncheesecake/repos", | | | | "events_url": "https://api.github.com/orgs/lemoncheesecake/events", | | | | "hooks_url": "https://api.github.com/orgs/lemoncheesecake/hooks", | | | | "issues_url": "https://api.github.com/orgs/lemoncheesecake/issues", | | | | "members_url": "https://api.github.com/orgs/lemoncheesecake/members{/member}", | | | | "public_members_url": "https://api.github.com/orgs/lemoncheesecake/public_members{/member}", | | | | "avatar_url": "https://avatars3.githubusercontent.com/u/28742541?v=4", | | | | "description": "Python framework for functional/QA testing", | | | | "name": "lemoncheesecake", | | | | "company": null, | | | | "blog": "https://github.com/lemoncheesecake/lemoncheesecake", | | | | "location": null, | | | | "email": "", | | | | "is_verified": false, | | | | "has_organization_projects": true, | | | | "has_repository_projects": true, | | | | "public_repos": 1, | | | | "public_gists": 0, | | | | "followers": 0, | | | | "following": 0, | | | | "html_url": "https://github.com/lemoncheesecake", | | | | "created_at": "2017-05-16T22:03:10Z", | | | | "updated_at": "2017-05-25T09:58:35Z", | | | | "type": "Organization" | | | | } | | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | | Check API response | | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "type" to be equal to "Organization" | Got "Organization" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "id" to be an integer | Got 28742541 | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "description" to be not equal to null | Got "Python framework for functional/QA testing" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "login" to be present | Got "lemoncheesecake" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "created_at" to match pattern "^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$" | Got "2017-05-16T22:03:10Z" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "has_organization_projects" to be a boolean that is equal to true | Got true | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "followers" to be greater than or equal to 0 | Got 0 | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "following" to be greater than or equal to 0 | Got 0 | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "repos_url" to end with "/repos" | Got "https://api.github.com/orgs/lemoncheesecake/repos" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "issues_url" to end with "/issues" | Got "https://api.github.com/orgs/lemoncheesecake/issues" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "events_url" to end with "/events" | Got "https://api.github.com/orgs/lemoncheesecake/events" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "hooks_url" to end with "/hooks" | Got "https://api.github.com/orgs/lemoncheesecake/hooks" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "members_url" to end with "/members{/member}" | Got "https://api.github.com/orgs/lemoncheesecake/members{/me | | | | mber}" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+ | CHECK | Expect "public_members_url" to end with "/public_members{/member}" | Got "https://api.github.com/orgs/lemoncheesecake/public_memb | | | | ers{/member}" | +-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
lcc top-suites¶
Show suites ordered by their duration.
$ lcc top-suites Suites, ordered by duration: +---------+----------+------+ | Suite | Duration | In % | +---------+----------+------+ | suite_2 | 2.000s | 66% | | suite_1 | 1.000s | 33% | +---------+----------+------+
lcc top-tests¶
Shows tests ordered by their duration.
$ lcc top-tests Tests, ordered by duration: +--------------+----------+------+ | Suite | Duration | In % | +--------------+----------+------+ | suite_2.test | 2.000s | 66% | | suite_1.test | 1.000s | 33% | +--------------+----------+------+
lcc top-steps¶
Shows steps aggregated, ordered by their duration.
$ lcc top-steps Steps, aggregated and ordered by duration: +--------------------+------+--------+--------+--------+--------+------+ | Step | Occ. | Min. | Max | Avg. | Total | In % | +--------------------+------+--------+--------+--------+--------+------+ | Do something | 2 | 1.000s | 2.000s | 1.500s | 3.000s | 75% | | Do something else | 1 | 1.000s | 1.000s | 1.000s | 1.000s | 25% | +--------------------+------+--------+--------+--------+--------+------+
lcc filtering arguments¶
lcc sub commands run, show, stats, report, top-suites, top-tests, top-steps
and diff take advantage of a powerful set of filtering arguments.
- path¶
Filter on specified test/suite path. The wildcard character
*is supported. A path can also be negated using using the^prefix. Examples:$ lcc run suite.test1 # run suite.test1 $ lcc run suite.test1 suite.test2 # run both suite.test1 and suite.test2 $ lcc.run suite.test* # run all tests that match suite.test* $ lcc run ^suite.test1 # run all tests that are NOT suite.test1 $ lcc report --path suite.test1 # see report results for suite.test1
Please note that
lcc reporttakes the path value with--pathinstead of a positioning argument.
- --desc¶
Filter on specified test/suite description.
- --tag, -a¶
Filter on test/suite tags.
- --property, -m¶
Filter on specified test/suite properties. The value syntax is “key:value”. Example:
$ lcc run --property priority:high
- --link, -l¶
Filter on specified test/suite associated links, both on name and url.
$ lcc run --link https://bugtracker.example.com/myproject/issues/1234
- --passed¶
Filter on passed tests.
- --failed¶
Filter on failed tests.
- --skipped¶
Filter on failed tests.
- --non-passed¶
Alias for
--failed --skipped.
- --disabled¶
Filter on disabled tests.
- --enabled¶
Filter on enabled tests.
- --grep, -g¶
Filter on report’s content (either tests, test session setup, test session teardown, suite setups suite teardowns). Every text content within steps is searched. The argument is a pattern that is case insensitive and multi-line.
New in version 1.2.0.
- --from-report¶
Filter on specified argument. When this option is used, all the filtering option will be applied on the report.
A typical use-case of this functionality is to re-run failed tests from a previous report:
$ lcc run --failed --from-report reports/report-2
Or simply:
$ lcc run --failed
if you want to re-run the failed tests from the latest run.
More about path (or --path), --desc, --tag, --property, --link arguments:
they accept the wilcard character
*they accept a leading
^character before the value (examples:--tag ^slow,--properties priority:^low) standing for an exclusion instead of an inclusionthey can take multiple values and be specified multiple times, acting as (respectively) logical OR and AND, examples:
--tag slow fast: will match tests/suites that have a “slow” OR “fast” associated tags--tag slow --tag critical: will match tests/suites that have a “slow” AND “critical” tagsboth forms can be combined:
--tag slow fast --tag criticalpath/--pathonly provides a OR
when the filter is applied, the suite hierarchy is taken into account. For instance, if a suite has a specified tag, then all (direct or indirect) sub-suites and tests will also match that given tag
More about --passed, --failed, --skipped, --non-passed, --grep:
these arguments work directly with the
lcc reportcommand.things are more subtle with other
lccsub-commands where these arguments are used in combination with the--from-report. When using this later option, the filter will be based on the report.
Please note that:
some options might not be available to specific commands (because they would not relevant)
some options might not be used together (because it’s ambiguous or because it would not make sense)
