The lcc command line tool

lcc commands

lcc run

Like most of the lcc sub-commands, lcc run benefits from the filtering arguments but also from extra arguments that allow a finer control over tests execution and reporting:

--project

A project path (see Suites discovery for more details).

--threads

The number of threads used to execute tests; default is one thread (meaning tests are run sequentially). The number of threads can also be set through the $LCC_THREADS environment variable (the CLI argument will have precedence over this variable).

--reporting

The list of reporting backends to use, default are: “console”, “html” and “json”. The backends passed as argument will override this list. It can also be customized through the $LCC_REPORTING environment variable (the CLI argument will have precedence over this variable).

Changed in version 1.1.0: It’s possible to add or remove reporting backends to/from the default backends using + and ^ prefixes. For instance, --reporting console json html junit to add “junit” can be simplified into --reporting +junit.

Other examples:

$ lcc run --reporting ^console  # turn off console reporting
$ lcc run --reporting ^console +junit  # turn off console reporting and turn on junit reporting
$ lcc run --reporting html json  # only generate the HTML report

Please note that:

  • turn on/off directives cannot be mixed with a fixed backend list, example: --reporting console +junit is invalid
  • the “html” backend generates the static HTML/JS/CSS files required to read the reporting data generated by the “json” backend, meaning that both reporting backends must be specified to have a readable HTML report
--report-dir

The directory where the files generated by the reporting backends will be stored. The default directory is the report sub-directory within the project directory, in the case the former report directory will be archived in the reports (with an “s”) sub-directory. The directory can be also be set through the $LCC_REPORT_DIR environment variable (the CLI argument will have precedence over this variable).

--save-report

Set the frequency at which reporting backends such as “json” and “xml” will save the reporting data on disk. Possibles values for this option are: at_end_of_tests, at_each_suite, at_each_test, at_each_failed_test, at_each_event, every_${N}s (the reporting data will be saved every “N” seconds). The default value for this option is at_each_failed_test. This behavior can also be configured through the $LCC_SAVE_REPORT environment variable (the CLI argument will have precedence over this variable).

--exit-error-on-failure

lcc run exits with a “0” exit code when the tests have been successfully executed (no matter their status), with this option it will exit with a “1” exit code if any failure (error log of failed check) has been encountered

--stop-on-failure

Stops the execution of the tests on the first non-passed test, remaining tests will be marked as skipped

--force-disabled

Force the execution of disabled tests

lcc show

Shows the project tests hierarchy.

$ lcc show
* suite_1
    - suite_1.test_1 (slow, priority:low)
    - suite_1.test_2 (priority:low)
    - suite_1.test_3 (priority:medium, #1235)
    - suite_1.test_4 (priority:low)
    - suite_1.test_5 (priority:high)
    - suite_1.test_6 (slow, priority:high)
    - suite_1.test_7 (priority:high)
    - suite_1.test_8 (priority:medium)
    - suite_1.test_9 (priority:medium)
* suite_2
    - suite_2.test_1 (priority:low)
    - suite_2.test_2 (priority:low)
    - suite_2.test_3 (priority:high)
    - suite_2.test_4 (priority:medium)
    - suite_2.test_5 (priority:low)
    - suite_2.test_6 (priority:low)
    - suite_2.test_7 (priority:medium)
    - suite_2.test_8 (slow, priority:low, #1234)
    - suite_2.test_9 (slow, priority:medium)

lcc diff

Compares two reports.

$ lcc diff reports/report-1/ report/
Added tests (1):
- suite_3.test_1 (passed)

Removed tests (1):
- suite_1.test_9 (failed)

Status changed (2):
- suite_2.test_3 (failed => passed)
- suite_2.test_4 (passed => failed)

lcc fixtures

Shows available project fixtures.

$ lcc fixtures

Fixture with scope pre_run:
+---------+--------------+------------------+---------------+
| Fixture | Dependencies | Used by fixtures | Used by tests |
+---------+--------------+------------------+---------------+
| fixt_1  | -            | 1                | 1             |
+---------+--------------+------------------+---------------+


Fixture with scope session:
+---------+--------------+------------------+---------------+
| Fixture | Dependencies | Used by fixtures | Used by tests |
+---------+--------------+------------------+---------------+
| fixt_2  | fixt_1       | 1                | 2             |
| fixt_3  | -            | 2                | 1             |
+---------+--------------+------------------+---------------+


Fixture with scope suite:
+---------+--------------+------------------+---------------+
| Fixture | Dependencies | Used by fixtures | Used by tests |
+---------+--------------+------------------+---------------+
| fixt_4  | fixt_3       | 0                | 2             |
| fixt_6  | fixt_3       | 1                | 1             |
| fixt_5  | -            | 0                | 0             |
+---------+--------------+------------------+---------------+


Fixture with scope test:
+---------+----------------+------------------+---------------+
| Fixture | Dependencies   | Used by fixtures | Used by tests |
+---------+----------------+------------------+---------------+
| fixt_7  | fixt_6, fixt_2 | 0                | 2             |
| fixt_8  | -              | 0                | 1             |
| fixt_9  | -              | 0                | 1             |
+---------+----------------+------------------+---------------+

lcc stats

Shows project statistics.

$ lcc stats
Tags:
+------+-------+------+
| Tag  | Tests | In % |
+------+-------+------+
| slow | 4     | 22%  |
+------+-------+------+

Properties:
+----------+--------+-------+------+
| Property | Value  | Tests | In % |
+----------+--------+-------+------+
| priority | low    | 8     | 44%  |
| priority | medium | 6     | 33%  |
| priority | high   | 4     | 22%  |
+----------+--------+-------+------+

Links:
+-------+-------------------------+-------+------+
| Name  | URL                     | Tests | In % |
+-------+-------------------------+-------+------+
| #1234 | http://example.com/1234 | 1     |  5%  |
| #1235 | http://example.com/1235 | 1     |  5%  |
+-------+-------------------------+-------+------+

Total: 18 tests in 2 suites

lcc report

Shows a generated report on the console, passing the --short argument will print it the same way as lcc run does.

Test Organization end-point
(github.organization)
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
|       | Get lemoncheesecake organization information                                                     |                                                              |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| INFO  | GET https://api.github.com/orgs/lemoncheesecake                                                  |                                                              |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect HTTP code to be equal to 200                                                              | Got 200                                                      |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| INFO  | Response                                                                                         |                                                              |
|       | {                                                                                                |                                                              |
|       |     "login": "lemoncheesecake",                                                                  |                                                              |
|       |     "id": 28742541,                                                                              |                                                              |
|       |     "node_id": "MDEyOk9yZ2FuaXphdGlvbjI4NzQyNTQx",                                               |                                                              |
|       |     "url": "https://api.github.com/orgs/lemoncheesecake",                                        |                                                              |
|       |     "repos_url": "https://api.github.com/orgs/lemoncheesecake/repos",                            |                                                              |
|       |     "events_url": "https://api.github.com/orgs/lemoncheesecake/events",                          |                                                              |
|       |     "hooks_url": "https://api.github.com/orgs/lemoncheesecake/hooks",                            |                                                              |
|       |     "issues_url": "https://api.github.com/orgs/lemoncheesecake/issues",                          |                                                              |
|       |     "members_url": "https://api.github.com/orgs/lemoncheesecake/members{/member}",               |                                                              |
|       |     "public_members_url": "https://api.github.com/orgs/lemoncheesecake/public_members{/member}", |                                                              |
|       |     "avatar_url": "https://avatars3.githubusercontent.com/u/28742541?v=4",                       |                                                              |
|       |     "description": "Python framework for functional/QA testing",                                 |                                                              |
|       |     "name": "lemoncheesecake",                                                                   |                                                              |
|       |     "company": null,                                                                             |                                                              |
|       |     "blog": "https://github.com/lemoncheesecake/lemoncheesecake",                                |                                                              |
|       |     "location": null,                                                                            |                                                              |
|       |     "email": "",                                                                                 |                                                              |
|       |     "is_verified": false,                                                                        |                                                              |
|       |     "has_organization_projects": true,                                                           |                                                              |
|       |     "has_repository_projects": true,                                                             |                                                              |
|       |     "public_repos": 1,                                                                           |                                                              |
|       |     "public_gists": 0,                                                                           |                                                              |
|       |     "followers": 0,                                                                              |                                                              |
|       |     "following": 0,                                                                              |                                                              |
|       |     "html_url": "https://github.com/lemoncheesecake",                                            |                                                              |
|       |     "created_at": "2017-05-16T22:03:10Z",                                                        |                                                              |
|       |     "updated_at": "2017-05-25T09:58:35Z",                                                        |                                                              |
|       |     "type": "Organization"                                                                       |                                                              |
|       | }                                                                                                |                                                              |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
|       | Check API response                                                                               |                                                              |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "type" to be equal to "Organization"                                                      | Got "Organization"                                           |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "id" to be an integer                                                                     | Got 28742541                                                 |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "description" to be not equal to null                                                     | Got "Python framework for functional/QA testing"             |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "login" to be present                                                                     | Got "lemoncheesecake"                                        |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "created_at" to match pattern "^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$"                    | Got "2017-05-16T22:03:10Z"                                   |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "has_organization_projects" to be a boolean that is equal to true                         | Got true                                                     |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "followers" to be greater than or equal to 0                                              | Got 0                                                        |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "following" to be greater than or equal to 0                                              | Got 0                                                        |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "repos_url" to end with "/repos"                                                          | Got "https://api.github.com/orgs/lemoncheesecake/repos"      |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "issues_url" to end with "/issues"                                                        | Got "https://api.github.com/orgs/lemoncheesecake/issues"     |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "events_url" to end with "/events"                                                        | Got "https://api.github.com/orgs/lemoncheesecake/events"     |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "hooks_url" to end with "/hooks"                                                          | Got "https://api.github.com/orgs/lemoncheesecake/hooks"      |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "members_url" to end with "/members{/member}"                                             | Got "https://api.github.com/orgs/lemoncheesecake/members{/me |
|       |                                                                                                  | mber}"                                                       |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| CHECK | Expect "public_members_url" to end with "/public_members{/member}"                               | Got "https://api.github.com/orgs/lemoncheesecake/public_memb |
|       |                                                                                                  | ers{/member}"                                                |
+-------+--------------------------------------------------------------------------------------------------+--------------------------------------------------------------+

lcc top-suites

Show suites ordered by their duration.

$ lcc top-suites
Suites, ordered by duration:
+---------+----------+------+
| Suite   | Duration | In % |
+---------+----------+------+
| suite_2 | 2.000s   | 66%  |
| suite_1 | 1.000s   | 33%  |
+---------+----------+------+

lcc top-tests

Shows tests ordered by their duration.

$ lcc top-tests
Tests, ordered by duration:
+--------------+----------+------+
| Suite        | Duration | In % |
+--------------+----------+------+
| suite_2.test | 2.000s   | 66%  |
| suite_1.test | 1.000s   | 33%  |
+--------------+----------+------+

lcc top-steps

Shows steps aggregated, ordered by their duration.

$ lcc top-steps
Steps, aggregated and ordered by duration:
+--------------------+------+--------+--------+--------+--------+------+
| Step               | Occ. | Min.   | Max    | Avg.   | Total  | In % |
+--------------------+------+--------+--------+--------+--------+------+
| Do something       | 2    | 1.000s | 2.000s | 1.500s | 3.000s | 75%  |
| Do something else  | 1    | 1.000s | 1.000s | 1.000s | 1.000s | 25%  |
+--------------------+------+--------+--------+--------+--------+------+

lcc filtering arguments

lcc sub commands run, show, stats, report, top-suites, top-tests, top-steps and diff take advantage of a powerful set of filtering arguments.

path

Filter on specified test/suite path. The wildcard character * is supported. A path can also be negated using using the ^ prefix. Examples:

$ lcc run suite.test1  # run suite.test1
$ lcc run suite.test1 suite.test2  # run both suite.test1 and suite.test2
$ lcc.run suite.test*  # run all tests that match suite.test*
$ lcc run ^suite.test1  # run all tests that are NOT suite.test1
$ lcc report --path suite.test1  # see report results for suite.test1

Please note that lcc report takes the path value with --path instead of a positioning argument.

--desc

Filter on specified test/suite description.

--tag, -a

Filter on test/suite tags.

--property, -m

Filter on specified test/suite properties. The value syntax is “key:value”. Example:

$ lcc run --property priority:high

Filter on specified test/suite associated links, both on name and url.

$ lcc run --link https://bugtracker.example.com/myproject/issues/1234
--passed

Filter on passed tests.

--failed

Filter on failed tests.

--skipped

Filter on failed tests.

--non-passed

Alias for --failed --skipped.

--disabled

Filter on disabled tests.

--enabled

Filter on enabled tests.

--grep, -g

Filter on report’s content (either tests, test session setup, test session teardown, suite setups suite teardowns). Every text content within steps is searched. The argument is a pattern that is case insensitive and multi-line.

New in version 1.2.0.

--from-report

Filter on specified argument. When this option is used, all the filtering option will be applied on the report.

A typical use-case of this functionality is to re-run failed tests from a previous report:

$ lcc run --failed --from-report reports/report-2

Or simply:

$ lcc run --failed

if you want to re-run the failed tests from the latest run.

More about path (or --path), --desc, --tag, --property, --link arguments:

  • they accept the wilcard character *

  • they accept a leading ^ character before the value (examples: --tag ^slow, --properties priority:^low) standing for an exclusion instead of an inclusion

  • they can take multiple values and be specified multiple times, acting as (respectively) logical OR and AND, examples:

    • --tag slow fast: will match tests/suites that have a “slow” OR “fast” associated tags
    • --tag slow --tag critical: will match tests/suites that have a “slow” AND “critical” tags
    • both forms can be combined: --tag slow fast --tag critical
    • path/--path only provides a OR
  • when the filter is applied, the suite hierarchy is taken into account. For instance, if a suite has a specified tag, then all (direct or indirect) sub-suites and tests will also match that given tag

More about --passed, --failed, --skipped, --non-passed, --grep:

  • these arguments work directly with the lcc report command.
  • things are more subtle with other lcc sub-commands where these arguments are used in combination with the --from-report. When using this later option, the filter will be based on the report.

Please note that:

  • some options might not be available to specific commands (because they would not relevant)
  • some options might not be used together (because it’s ambiguous or because it would not make sense)