API reference

API compatibility / stability

lemoncheesecake follows the well know Semantic Versioning for it’s public API. What is considered as “public” is everything which is documented on http://docs.lemoncheesecake.io/. Everything else is internal and is subject to change at anytime.

Tests and suites declaration

lemoncheesecake.api.suite(description=None, name=None, rank=None)

Decorator, mark a class as a suite class.

Parameters:
  • description – suite’s description (by default, the suite’s description is built from the name)
  • name – suite’s name (by default, the suite’s name is taken from the class’s name)
  • rank – this value is used to order suites of the same hierarchy level
lemoncheesecake.api.test(description=None, name=None)

Decorator, mark a function/method as a test.

Parameters:
  • description – test’s description
  • name – test’s name (by default, the suite’s name is taken from the class’s name)
lemoncheesecake.api.tags(*tag_names)

Decorator, add tags to a test or a suite.

lemoncheesecake.api.prop(key, value)

Decorator, add a property (key/value) to a test or a suite.

Decorator, set a link (with an optional friendly name) to a test or a suite.

lemoncheesecake.api.disabled(reason=None)

Decorator, mark a test or a suite as disabled, meaning it won’t be executed but will be visible in report. An optional reason can be passed to the decorator (new in version 1.1.0).

lemoncheesecake.api.visible_if(condition)

Decorator, the test or suite will only appear if the given callable return a true value.

Parameters:condition – a callable that will take the test object if applied to a test or the suite class instance if applied to a suite.
lemoncheesecake.api.hidden()

Decorator, the test or suite won’t be visible in the resulting test tree.

lemoncheesecake.api.depends_on(*deps)

Decorator, only applicable to a test. Add dependencies to a test.

Parameters:deps – the test paths that the decorated test is depending on.
lemoncheesecake.api.parametrized(parameter_source, naming_scheme=<function _default_naming_scheme>)

Decorator, make the test parametrized.

Parameters:
  • parameter_source

    it can be either:

    • an iterable of dicts, each dict representing the arguments passed to the test
    • a CSV-like mode, where the first element of the list represents the argument names as an str or a sequence (example: "i,j" or ("i", "j")) and the remaining elements are sequences of the arguments to be passed to the test.
  • naming_scheme

    optional, it can be either:

    • a optional function that takes as parameters the base test name, description, parameters, index and must return the expanded test name and description as a two elements list
    • a tuple/list of two (name + description) format strings, example: ("test_{i}_plus_{j}", "Test {i} plus {j}")
lemoncheesecake.api.inject_fixture(fixture_name=None)

Inject a fixture into a suite. If no fixture name is specified then the name of the variable holding the injected fixture will be used.

lemoncheesecake.api.add_test_into_suite(test, suite)

Add test into suite

Parameters:
  • test – a Test instance
  • suite – a suite decorated class instance (in that case the function must be called when the class is instantiated) or a module marked as a suite (in that case the function must be called when the module is loaded)
class lemoncheesecake.api.Test(name, description, callback)

Internal representation of a test.

path

The complete path of the test node (example: if used on a test named “my_test” and a suite named “my_suite”, then the path is “my_suite.my_test”).

Logging

lemoncheesecake.api.set_step(*args, **kwargs)

Set a new step.

Parameters:
  • description – the step description
  • detached – argument deprecated since 1.4.5, does nothing.
lemoncheesecake.api.detached_step(*args, **kwds)

Context manager deprecated since version 1.4.5, it only does a set_step.

lemoncheesecake.api.log_info(*args, **kwargs)

Log a info level message.

lemoncheesecake.api.log_warning(*args, **kwargs)

Log a warning level message.

lemoncheesecake.api.log_error(*args, **kwargs)

Log an error level message.

lemoncheesecake.api.log_url(*args, **kwargs)

Log an URL.

lemoncheesecake.api.log_check(*args, **kwargs)
lemoncheesecake.api.prepare_attachment(*args, **kwargs)

Context manager. Prepare a attachment using a pseudo filename and an optional description. It returns the real filename on disk that will be used by the caller to write the attachment content.

lemoncheesecake.api.prepare_image_attachment(*args, **kwargs)

Context manager. Prepare an image attachment using a pseudo filename and an optional description. The function returns the real filename on disk that will be used by the caller to write the attachment content.

lemoncheesecake.api.save_attachment_content(*args, **kwargs)

Save a given content as attachment using pseudo filename and optional description.

lemoncheesecake.api.save_attachment_file(*args, **kwargs)

Save an attachment using an existing file (identified by filename) and an optional description. The given file will be copied.

lemoncheesecake.api.save_image_content(*args, **kwargs)

Save a given image content as attachment using pseudo filename and optional description.

lemoncheesecake.api.save_image_file(*args, **kwargs)

Save an image using an existing file (identified by filename) and an optional description. The given file will be copied.

Threading

class lemoncheesecake.api.Thread(*args, **kwargs)

Acts exactly as the standard threading.Thread class and must be used instead when running threads within a test.

Matching

Operations

lemoncheesecake.matching.check_that(hint, actual, matcher, quiet=False)

Check that actual matches given matcher.

A check log is added to the report.

If quiet is set to True, the check details won’t appear in the check log.

lemoncheesecake.matching.require_that(hint, actual, matcher, quiet=False)

Require that actual matches given matcher.

A check log is added to the report. An AbortTest exception is raised if the check does not succeed.

If quiet is set to True, the check details won’t appear in the check log.

lemoncheesecake.matching.assert_that(hint, actual, matcher, quiet=False)

Assert that actual matches given matcher.

If assertion fail, a check log is added to the report and an AbortTest exception is raised.

If quiet is set to True, the check details won’t appear in the check log.

lemoncheesecake.matching.check_that_in(actual, *args, **kwargs)

Equivalent of check_that() over items of a dict.

Example of usage:

check_that_in(
    {"foo": 1, "bar": 2},
    "foo", equal_to(1),
    "bar", equal_to(2)
)

The key can also be a tuple when checking for a nested item:

check_that_in(
    {"foo": {"bar": 2}},
    ("foo", "bar"), equal_to(2)
)

The function can take a base_key keyword-arg to pass repeating nested-keys as a tuple.

If an extra quiet keyword-arg is set to True, the check details won’t appear in the check log.

The function returns a list of MatchResult.

lemoncheesecake.matching.require_that_in(actual, *args, **kwargs)

Does the same thing as check_that_in() except it performs a require_that() on each key-value pair.

lemoncheesecake.matching.assert_that_in(actual, *args, **kwargs)

Does the same thing as check_that_in() except it performs a assert_that() on each key-value pair.

Matchers

lemoncheesecake.matching.equal_to(expected)

Test if value is equal to expected

lemoncheesecake.matching.not_equal_to(expected)

Test if value is not equal to expected

lemoncheesecake.matching.greater_than(expected)

Test if value is greater than expected

lemoncheesecake.matching.greater_than_or_equal_to(expected)

Test if value is greater than or equal to expected

lemoncheesecake.matching.less_than(expected)

Test if value is less than expected

lemoncheesecake.matching.less_than_or_equal_to(expected)

Test if value is greater than or equal to expected

lemoncheesecake.matching.is_between(min, max)

Test if value is between min and max

lemoncheesecake.matching.is_none()

Test if value is None

lemoncheesecake.matching.is_not_none()

Test if value is not None

lemoncheesecake.matching.has_length(length)

Test if value has a length of

lemoncheesecake.matching.is_true()

Test if value is true (boolean type)

lemoncheesecake.matching.is_false()

Test if value is false (boolean type)

lemoncheesecake.matching.is_json(expected)

Test if the two data structures (that can be represented as JSON) match.

If the two values do not match, the match result description will be a textual diff between the two JSON representations.

lemoncheesecake.matching.starts_with(expected)

Test if string begins with given prefix

lemoncheesecake.matching.ends_with(expected)

Test if string ends with given suffix

lemoncheesecake.matching.contains_string(expected)

Test if string contains sub string

lemoncheesecake.matching.match_pattern(pattern, description=None, mention_regexp=False)

Test if string matches given pattern (using the search method of the re module)

lemoncheesecake.matching.is_text(expected, linesep='\n')

Test if the two multi-lines texts match.

If the two values do not match, the match result description will be a diff between the two texts.

lemoncheesecake.matching.is_integer(value_matcher=None)

Test if value is of type an integer

lemoncheesecake.matching.is_float(value_matcher=None)

Test if value is of type a float

lemoncheesecake.matching.is_bool(value_matcher=None)

Test if value is of type a boolean

lemoncheesecake.matching.is_str(value_matcher=None)

Test if value is of type a string

lemoncheesecake.matching.is_list(value_matcher=None)

Test if value is of type a list

lemoncheesecake.matching.is_dict(value_matcher=None)

Test if value is of type a collection

lemoncheesecake.matching.has_item(expected)

Test if the sequence has item matching expected

lemoncheesecake.matching.has_items(values)

Test if the sequence contains at least the given values

lemoncheesecake.matching.has_only_items(expected)

Test if the sequence only contains the given values

lemoncheesecake.matching.has_all_items(expected)

Test if all the items of the sequence match expected

lemoncheesecake.matching.is_in(expected)

Test if the sequence contains the expected item

lemoncheesecake.matching.has_entry(key_matcher, value_matcher=None)

Test if dict has a <key> entry whose value matches (optional) value_matcher. Key entry can a standard dict key or a list of key where each element represent a level of depth of the dict (when dict are imbricated)

lemoncheesecake.matching.is_(matcher)

If the function argument is not an instance of Matcher, wrap it into a matcher using equal_to, otherwise return the matcher argument as-is

lemoncheesecake.matching.not_(matcher)

Negates the matcher in argument

lemoncheesecake.matching.is_not(matcher)

Negates the matcher in argument

lemoncheesecake.matching.all_of(*matchers)

Test if all matchers match (logical AND between matchers).

lemoncheesecake.matching.any_of(*matchers)

Test if at least one of the matcher match (logical OR between matchers)

lemoncheesecake.matching.anything()

Matches anything (always succeed, whatever the actual value)

lemoncheesecake.matching.something()

Same thing as the ‘anything’ matcher but use ‘to be something’ in the matcher description

lemoncheesecake.matching.existing()

Same thing as the ‘anything’ matcher but use ‘to exist’ in the matcher description

lemoncheesecake.matching.present()

Same thing as the ‘anything’ matcher but use ‘to be present’ in the matcher description

Matcher

class lemoncheesecake.matching.matcher.Matcher
build_description(transformation)

Build a description for the matcher given the instance of MatcherDescriptionTransformer passed as argument.

matches(actual)

Test if the passed argument matches.

Parameters:actual – the actual value to match
Returns:an instance of MatchResult
class lemoncheesecake.matching.matcher.MatchResult(is_successful, description=None)
is_successful = None

whether or not the match did succeed

description = None

optional description

classmethod success(description=None)

Shortcut used to create a “successful” MatchResult.

classmethod failure(description=None)

Shortcut used to create a “failed” MatchResult.

__bool__()

Returns whether or not the match is successful.

class lemoncheesecake.matching.matcher.MatcherDescriptionTransformer(conjugate=False, negative=False)

This class is used as a callable and passed to Matcher.build_description() to transform the leading verb in description according to the transformer settings.

conjugate = None

indicate whether or not the verb in the description will be conjugated

negative = None

indicate whether or not the description will be turned into the negative form

__call__(description)

Transform the description according transformer settings.

Project

This is the class that must be used / inherited in your project.py file

class lemoncheesecake.project.Project(project_dir=None)
dir = None

The project’s directory path (optional, defaults to the caller dir)

metadata_policy = None

The project’s metadata policy

threaded = None

Indicates whether or not the project supports parallel execution of tests

show_command_line_in_report = None

Indicated whether or not the command line (“lcc run…”) will be displayed in the report

reporting_backends = None

The reporting backends of the project as a dict (whose key is the reporting backend name)

default_reporting_backend_names = None

The list of default reporting backend (indicated by their name) that will be used by “lcc run”

add_cli_args(cli_parser)

Overridable. This method can be used to add extra CLI arguments to “lcc run”.

create_report_dir()

Overridable. Create the report directory when no report directory is specified to “lcc run”.

load_suites()

Overridable. Load the project’s suites.

load_fixtures()

Overridable. Load the project’s fixtures.

pre_run(cli_args, report_dir)

Overridable. This hook is called before running the tests.

post_run(cli_args, report_dir)

Overridable. This hook is called after running the tests.

build_report_title()

Overridable. Build a custom report title as a string.

build_report_info()

Overridable. Build a list key/value pairs (expressed as a two items tuple) that will be available in the report.

Example:

[
    ("key1", "value1"),
    ("key2", "value2")
]

Loading suites

lemoncheesecake.suite.load_suites_from_directory(dir, recursive=True)

Load a list of suites from a directory.

If the recursive argument is set to True, sub suites will be searched in a directory named from the suite module: if the suite module is “foo.py” then the sub suites directory must be “foo”.

Raise SuiteLoadingError if one or more suite could not be loaded.

lemoncheesecake.suite.load_suites_from_files(patterns, excluding=())

Load a list of suites from a list of files.

Parameters:
  • patterns – a mandatory list (a simple string can also be used instead of a single element list) of files to import; the wildcard ‘*’ character can be used
  • excluding – an optional list (a simple string can also be used instead of a single element list) of elements to exclude from the expanded list of files to import

Example:

load_suites_from_files("test_*.py")
lemoncheesecake.suite.load_suite_from_file(filename)

Load a suite from a Python module indicated by a filename.

Raise SuiteLoadingError if the file cannot be loaded as a suite.

lemoncheesecake.suite.load_suite_from_module(mod)

Load a suite from a module instance.

lemoncheesecake.suite.load_suites_from_classes(classes)

Load a list of suites from a list of classes.

lemoncheesecake.suite.load_suite_from_class(class_)

Load a suite from a class.

Loading fixtures

lemoncheesecake.fixture.load_fixtures_from_directory(dir)

Load fixtures from a given directory (not recursive).

lemoncheesecake.fixture.load_fixtures_from_files(patterns, excluding=[])

Load fixtures from files.

Parameters:
  • patterns – a mandatory list (a simple string can also be used instead of a single element list) of files to import; the wildcard ‘*’ character can be used
  • excluding – an optional list (a simple string can also be used instead of a single element list) of elements to exclude from the expanded list of files to import

Example:

load_fixtures_from_files("test_*.py")
lemoncheesecake.fixture.load_fixtures_from_file(filename)

Load fixtures from a given file.

lemoncheesecake.fixture.load_fixtures_from_module(mod)

Load fixtures from a module instance.

New in version 1.5.1.

lemoncheesecake.fixture.load_fixtures_from_func(func)

Load a fixture from a function.

Metadata Policy

class lemoncheesecake.metadatapolicy.MetadataPolicy
add_property_rule(prop_name, accepted_values=None, on_test=None, on_suite=None, required=False)

Declare a property rule.

Parameters:
  • prop_name – the property name
  • accepted_values – an optional list of accepted values
  • on_test – whether or not the property can be used on a test
  • on_suite – whether or not the property can be used on a suite
  • required – whether or not the property is required

If neither on_test or on_suite argument are set, then the property is only available for tests.

disallow_unknown_properties()

Disallow unknown properties for tests and suites.

add_tag_rule(tag_name, on_test=None, on_suite=None)

Declare a tag rule.

Parameters:
  • tag_name – the tag name
  • on_test – whether or not the tag can be used on a test
  • on_suite – whether or not the tag can be used on a suite

If neither on_test or on_suite argument are set, then the tag is only available for tests.

disallow_unknown_tags()

Disallow unknown tags for tests and suites.

Report

New in version 1.6.0.

The structure of a Report object is the following:

Report (Report)
    (0-1) test_session_setup (Result)
        (1-N) steps (Step)
            (1-N) step logs (StepLog)
    (1-N) suites (SuiteResult)
        (0-1) suite_setup (Result)
            (1-N) steps (Step)
                (1-N) step logs (StepLog)
        (N) tests (TestResult)
            (N) steps (Step)
                (1-N) step logs (StepLog)
        (N) sub-suites (SuiteResult)
            [suites can embed other sub-suite hierarchy]
        (0-1) suite_teardown (Result)
            (1-N) steps (Step)
                (1-N) step logs (StepLog)
    (0-1) test_session_teardown (Result)
        (1-N) steps (Step)
            (1-N) step logs (StepLog)

Step logs (whose base class is StepLog) are one of:
    - Log
    - Check
    - Attachment
    - Url
lemoncheesecake.reporting.load_report(path, backends=None)

Load report from a report directory or file.

class lemoncheesecake.reporting.Report
start_time = None

The test run start time.

end_time = None

The test run end time.

saving_time = None

The report saving time.

title = None

The report title.

nb_threads = None

The number of threads used for the test run.

test_session_setup

The session setup result if any.

test_session_teardown

The session teardown result if any.

duration

The test run duration.

nb_tests

The number of tests in the report.

parallelized

Whether or not the tests were parallelized.

add_info(name, value)

Add extra information (name, value) in the report.

add_suite(suite)

Add suite result to the report.

get_suites()

Get suite results.

is_successful()

Return whether or not the test run is considered as successful.

Please note that every result is taken into account, including tests but also setups and teardowns.

all_suites()

An iterator over all suite results contained in the report (recursive).

all_tests()

An iterator over all test results contained in the report.

all_results()

An iterator over all results (tests, setups, teardowns) contained in the report.

all_steps()

An iterator over all steps contained in the report.

build_message(template)

Build a message from a template that contains variable placeholders.

Example: with template “Test results: {passed}/{enabled} passed ({passed_pct})” the method will return for instance “Test results: 1/2 passed (50%)”.

The following variables are available:

  • start_time: the test run start time
  • end_time: the test run end time
  • duration: the test run duration
  • total: the total number of tests (including disabled tests)
  • enabled: the total number of tests (excluding disabled tests)
  • passed: the number of passed tests
  • passed_pct: the number of passed tests in percentage of enabled tests
  • failed: the number of failed tests
  • failed_pct: the number of failed tests in percentage of enabled tests
  • skipped: the number of skipped tests
  • skipped_pct: the number of skipped tests in percentage of enabled tests
  • disabled: the number of disabled tests
  • disabled_pct: the number of disabled tests in percentage of all tests
save()

Save the report.

class lemoncheesecake.reporting.SuiteResult(name, description)

Contains the results of tests and sub-suites within the suite.

Variables:
  • name (str) – test name
  • description (str) – test description
  • tags (list) – test tags
  • properties (dict) – test properties, as a dict
  • links (dict) – test links, as a list of tuples ((url, description) where description can be None)
start_time = None

The suite start time.

end_time = None

The suite end time.

suite_setup

The suite setup result (if any).

suite_teardown

The suite teardown result (if any).

duration

The suite duration, which is the addition of all results contained in the suite and its sub-suite (recursively).

get_tests()

Return the tests contained in the suite.

get_suites()

Return the sub-suites contained in the suite.

add_suite(suite)

Add a sub-suite to the suite.

add_test(test)

Add test to the suite.

path

The complete path of the test node (example: if used on a test named “my_test” and a suite named “my_suite”, then the path is “my_suite.my_test”).

class lemoncheesecake.reporting.Result

Holds the result of setup/teardown phase and it’s also the base class of TestResult.

STATUSES = ('passed', 'failed', 'skipped', 'disabled')

possible status values for a Result instance

parent_suite = None

Parent suite.

type = None

Result type (it is one of the following: “test_session_setup”, “test_session_teardown”, “suite_setup”, “suite_teardown”, “test”).

start_time = None

Result start time.

end_time = None

Result end time.

status = None

Result status (one of Result.STATUSES or None if the result is not yet complete).

status_details = None

Result status details, if any.

add_step(step)

Add step to the result.

get_steps()

Get steps.

is_successful()

Return whether or not the result is successful (even if the result is not yet complete).

duration

Return the duration as a float or None if the result is not yet complete.

class lemoncheesecake.reporting.TestResult(name, description)

Holds the result of a test. Inherits Result.

Variables:
  • name (str) – test name
  • description (str) – test description
  • tags (list) – test tags
  • properties (dict) – test properties, as a dict
  • links (dict) – test links, as a list of tuples ((url, description) where description can be None)
path

The complete path of the test node (example: if used on a test named “my_test” and a suite named “my_suite”, then the path is “my_suite.my_test”).

class lemoncheesecake.reporting.Step(description)

This class holds logs occurring within a step.

description = None

Step description.

start_time = None

Step start time.

end_time = None

Step end time.

add_log(log)

Add a log to the step.

get_logs()

Get step logs.

is_successful()

Return whether or not (as a boolean) the step is successful.

duration

Return the duration as a float or None if the step is not yet complete.

class lemoncheesecake.reporting.StepLog(ts)

Base class for logs contained in a Step instance.

time = None

Log time (float).

parent_step = None

Parent step.

class lemoncheesecake.reporting.Log(level, message, ts)

The log resulting of log_info/log_debug/log_warning/log_error functions. Inherits StepLog.

level = None

Log level.

message = None

Log message.

class lemoncheesecake.reporting.Check(description, is_successful, details, ts)

The log resulting of a check_that/require_that/assert_that functions. Inherits StepLog.

description = None

Check description.

is_successful = None

Whether the check is successful or not (boolean).

details = None

Optional check details.

class lemoncheesecake.reporting.Attachment(description, filename, as_image, ts)

The log resulting of save/prepare_*attachment* functions. Inherits StepLog.

description = None

Attachment description.

filename = None

Attachment filename, this is a path relative to the report directory.

as_image = None

Whether or not the attachment should be interpreted as an image (boolean).

class lemoncheesecake.reporting.Url(description, url, ts)

The log resulting of log_url function. Inherits StepLog.

description = None

Optional description.

url = None

Actual url.

Exceptions

lemoncheesecake.exceptions.AbortTest(*args)

Raising this exception will stop the currently running test.

lemoncheesecake.exceptions.AbortSuite(*args)

Raising this exception will stop the currently running suite.

lemoncheesecake.exceptions.AbortAllTests(*args)

Raising this exception will stop the currently running test and all the tests waiting to be run.

lemoncheesecake.exceptions.UserError(*args)

This exception is intended to be raised in pre-run and post-run phases of the project to indicate that a required state has not been fulfilled.