conftest.py - pytest fixtures for testing¶
To get started on running tests, see tests/README.rst
These fixtures start the web2py server then submit requests to it.
NOTE: Make sure you don’t have another server running, because it will grab the requests instead of letting the test server respond to requests.
The overall testing approach is functional: rather than test a function, this file primarily tests endpoints on the web server. To accomplish this:
This file includes the
web2py_server
fixture to start a web2py server, and a fixture (test_client
) to make requests of it. To make debug easier, thetest_client
class saves the HTML of a failing test to a file, and also saves any web2py tracebacks in the HTML form to a file.The
runestone_db
and related classes provide the ability to access the web2py database directory, in order to set up and tear down test. In order to leave the database unchanged after a test, almost all routines that modify the database are wrapped in a context manager; on exit, then delete any modifications.
Imports¶
These are listed in the order prescribed by PEP 8.
Standard library¶
Third-party imports¶
Import a shared fixture.
from runestone.shared_conftest import _SeleniumUtils, selenium_driver # noqa: F401
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
Required to allow use of this class on a module-scoped fixture.
Local imports¶
Set this to False if you want to turn off all web page validation.
Pytest setup¶
Add command-line options.
Per the API reference
,
options are argparse style.
Output a coverage report when testing is done. See https://docs.pytest.org/en/latest/reference.html#_pytest.hookspec.pytest_terminal_summary.
Capture the output from the report.
Utilities¶
A simple data-struct object.
Create a web2py controller environment. This is taken from pieces of gluon.shell.run
. It returns a dict
containing the environment.
application: The name of the application to run in, as a string.
Fixtures¶
web2py access¶
These fixtures provide access to the web2py Runestone server and its environment.
This fixture starts and shuts down the web2py server.
Execute this fixture once per session.
HINT: make sure that 0.py
has something like the following, that reads this environment variable:
1config = environ.get("WEB2PY_CONFIG","production")
2
3if config == "production":
4 settings.database_uri = environ["DBURL"]
5elif config == "development":
6 settings.database_uri = environ.get("DEV_DBURL")
7elif config == "test":
8 settings.database_uri = environ.get("TEST_DBURL")
9else:
10 raise ValueError("unknown value for WEB2PY_CONFIG")
HINT: make sure that you export TEST_DBURL
in your environment; it is
not set here because it’s specific to the local setup, possibly with a
password, and thus can’t be committed to the repo.
Extract the components of the DBURL. The expected format is postgresql://user:password@netloc/dbname
, a simplified form of the connection URI.
empty1, postgres_ql, pguser, pgpassword, pgnetloc, dbname, empty2 = re.split(
"^postgres(ql)?://(.*):(.*)@(.*)/(.*)$", os.environ["TEST_DBURL"]
)
assert (not empty1) and (not empty2)
os.environ["PGPASSWORD"] = pgpassword
os.environ["PGUSER"] = pguser
os.environ["DBHOST"] = pgnetloc
rs_path = "applications/{}".format(runestone_name)
Assume we are running with working directory in tests.
In the future, to print the output of the init/build process, see pytest #1599 for code to enable/disable output capture inside a test.
Make sure runestone_test is nice and clean – this will remove many tables that web2py will then re-create.
Copy the test book to the books directory.
Sometimes this fails for no good reason on Windows. Retry.
Build the test book to add in db fields needed.
For debug:
Uncomment the next three lines.
Set
WEB2PY_CONFIG
totest
; all the other usual Runestone environment variables must also be set.Run
python -m celery --app=scheduled_builder worker --pool=gevent --concurrency=4 --loglevel=info
fromapplications/runestone/modules
to use the scheduler. I’m assuming the redis server (which the tests needs regardless of debug) is also running.Run a test (in a separate window). When the debugger stops at the lines below:
Run web2py manually to see all debug messages. Use a command line like
python web2py.py -a pass
.After web2py is started, type “c” then enter to continue the debugger and actually run the tests.
Start the web2py server and the web2py scheduler.
Produce text (not binary) output for nice output in echo()
below.
Wait for the webserver to come up.
Wait for the server to come up.
The server is up. We’re done.
Run Celery. Per https://github.com/celery/celery/issues/3422, it sounds like celery doesn’t support coverage, so omit it.
Celery must be run in the modules
directory, where the worker is defined.
Produce text (not binary) output for nice output in echo()
below.
Start a thread to read web2py output and echo it.
def echo(popen_obj, description_str):
stdout, stderr = popen_obj.communicate()
print("\n" "{} stdout\n" "--------------------\n".format(description_str))
print(stdout)
print("\n" "{} stderr\n" "--------------------\n".format(description_str))
print(stderr)
echo_threads = [
Thread(target=echo, args=(web2py_server, "web2py server")),
Thread(target=echo, args=(celery_process, "celery process")),
]
TODO: Redis for Windows.
Save the password used.
Wait for the server to come up. The delay varies; this is a guess.
After this comes the teardown code.
Terminate the server and schedulers to give web2py time to shut down gracefully.
The name of the Runestone controller. It must be module scoped to allow the web2py_server
to use it.
The environment of a web2py controller.
Close the database connection after the test completes.
Create fixture providing a web2py controller environment for a Runestone application.
Database¶
This fixture provides access to a clean instance of the Runestone database.
Provide acess the the Runestone database through a fixture. After a test runs, restore the database to its initial state.
Restore the database state after the test finishes
Rollback changes, which ensures that any errors in the database connection will be cleared.
This list was generated by running the following query, taken from
https://dba.stackexchange.com/a/173117. Note that the query excludes
specific tables, which the runestone build
populates and which
should not be modified otherwise. One method to identify these tables
which should not be truncated is to run pg_dump --data-only
$TEST_DBURL > out.sql
on a clean database, then inspect the output to
see which tables have data. It also excludes all the scheduler tables,
since truncating these tables makes the process take a lot longer.
The query is:
## SELECT input_table_name || ',' AS truncate_query FROM(SELECT table_schema || '.' || table_name AS input_table_name FROM information_schema.tables WHERE table_schema NOT IN ('pg_catalog', 'information_schema') AND table_name NOT IN ('questions', 'source_code', 'chapters', 'sub_chapters', 'scheduler_run', 'scheduler_task', 'scheduler_task_deps', 'scheduler_worker') AND table_schema NOT LIKE 'pg_toast%') AS information order by input_table_name;
db.executesql(
"""TRUNCATE
public.assignment_questions,
public.assignments,
public.auth_cas,
public.auth_event,
public.auth_group,
public.auth_membership,
public.auth_permission,
public.auth_user,
public.clickablearea_answers,
public.code,
public.codelens_answers,
public.course_attributes,
public.course_instructor,
public.course_practice,
public.courses,
public.dragndrop_answers,
public.fitb_answers,
public.grades,
public.lp_answers,
public.invoice_request,
public.lti_keys,
public.mchoice_answers,
public.parsons_answers,
public.payments,
public.practice_grades,
public.question_grades,
public.question_tags,
public.shortanswer_answers,
public.sub_chapter_taught,
public.tags,
public.timed_exam,
public.useinfo,
public.user_biography,
public.user_chapter_progress,
public.user_courses,
public.user_state,
public.user_sub_chapter_progress,
public.user_topic_practice,
public.user_topic_practice_completion,
public.user_topic_practice_feedback,
public.user_topic_practice_log,
public.user_topic_practice_survey,
public.web2py_session_runestone CASCADE;
"""
)
db.commit()
Provide a class for manipulating the Runestone database.
Create a new course. It returns an object with information about the created course.
The name of the course to create, as a string.
The start date of the course, as a string.
The value of the login_required
flag for the course.
The base course for this course. If None
, it will use course_name
.
The student price for this course.
Sanity check: this class shouldn’t exist.
Create the base course if it doesn’t exist.
Store these values in an object for convenient access.
obj = _object()
obj.course_name = course_name
obj.term_start_date = term_start_date
obj.login_required = login_required
obj.base_course = base_course
obj.student_price = student_price
obj.course_id = db.courses.insert(
course_name=course_name,
base_course=obj.base_course,
term_start_date=term_start_date,
login_required=login_required,
student_price=student_price,
courselevel="",
institution="",
)
db.commit()
return obj
def make_instructor(
self,
The ID of the user to make an instructor.
The ID of the course in which the user will be an instructor.
Present _RunestoneDbTools
as a fixture.
HTTP client¶
Provide access to Runestone through HTTP.
Given the test_client.text
, prepare to write it to a file.
Create a client for accessing the Runestone server.
class _TestClient(WebClient):
def __init__(
self,
web2py_server,
web2py_server_address,
runestone_name,
tmp_path,
pytestconfig,
):
self.web2py_server = web2py_server
self.web2py_server_address = web2py_server_address
self.tmp_path = tmp_path
self.pytestconfig = pytestconfig
super(_TestClient, self).__init__(
"{}/{}/".format(self.web2py_server_address, runestone_name), postbacks=True
)
Use the W3C validator to check the HTML at the given URL.
The relative URL to validate.
An optional string that, if provided, must be in the text returned by the server. If this is a sequence of strings, all of the provided strings must be in the text returned by the server.
The number of validation errors expected. If None, no validation is performed.
The expected status code from the request.
All additional keyword arguments are passed to the post
method.
If this was the expected result, return.
Since this is an error of some type, these paramets must be empty, since they can’t be checked.
Assume expected_string
is a sequence of strings.
Redo this section using html5validate command line
Save the HTML to make fixing the errors easier. Note that self.text
is already encoded as utf-8.
Provide special handling for web2py exceptions by saving the resulting traceback.
Create a client to access the admin interface.
Log in.
Get the error.
Save it to a file.
traceback_file = (
"".join(c if c not in r"\/:*?<>|" else "_" for c in url)
+ "_traceback.html"
)
with open(traceback_file, "wb") as f:
f.write(_html_prep(admin_client.text))
print("Traceback saved to {}.".format(traceback_file))
raise
def logout(self):
self.validate("default/user/logout", "Logged out")
Always logout after a test finishes.
Present _TestClient
as a fixure.
User¶
Provide a method to create a user and perform common user operations.
These are fixtures.
The username for this user.
The password for this user.
The course object returned by create_course
this user will register for.
True if the course is free (no payment required); False otherwise.
The first name for this user.
The last name for this user.
Registration doesn’t work unless we’re logged out.
Now, post the registration.
The e-mail address must be unique.
Note that course_id
is (on the form) actually a course name.
Record IDs
db = self.runestone_db_tools.db
self.user_id = (
db(db.auth_user.username == self.username)
.select(db.auth_user.id)
.first()
.id
)
def login(self):
self.test_client.validate(
"default/user/login",
data=dict(
username=self.username, password=self.password, _formname="login"
),
)
def logout(self):
self.test_client.logout()
def make_instructor(self, course_id=None):
If course_id
isn’t specified, use this user’s course_id
.
A context manager to update this user’s profile. If a course was added, it returns that course’s ID; otherwise, it returns None.
This parameter is passed to test_client.validate
.
An updated username, or None
to use self.username
.
An updated first name, or None
to use self.first_name
.
An updated last name, or None
to use self.last_name
.
An updated email, or None
to use self.email
.
An updated last name, or None
to use self.course.course_name
.
A shortcut for specifying the expected_string
, which only applies if expected_string
is not set. Use None
if a course will not be added, True
if the added course is free, or False
if the added course is paid.
The value of the accept_tcp
checkbox; provide an empty string to leave unchecked. The default value leaves it checked.
accept_tcp="on",
):
if expected_string is None:
if is_free is None:
expected_string = "Course Selection"
else:
expected_string = "Support Runestone" if is_free else "Payment Amount"
username = username or self.username
first_name = first_name or self.first_name
last_name = last_name or self.last_name
email = email or self.email
course_name = course_name or self.course.course_name
Perform the update.
Though the field is course_id
, it’s really the course name.
Call this after registering for a new course or adding a new course via update_profile
to pay for the course.
The Stripe test tokens to use for payment.
The course ID of the course to pay for. None specifies self.course.course_id
.
Get the signature from the HTML of the payment page.
self.test_client.validate("default/payment")
match = re.search(
'<input type="hidden" name="signature" value="([^ ]*)" />',
self.test_client.text,
)
signature = match.group(1)
html = self.test_client.validate(
"default/payment", data=dict(stripeToken=stripe_token, signature=signature)
)
assert ("Thank you for your payment" in html) or ("Payment failed" in html)
def hsblog(self, **kwargs):
Get the time, rounded down to a second, before posting to the server.
Post to the server.
Make sure this didn’t send us to the user profile page to add a course we aren’t registered for.
Present _TestUser
as a fixture.
Provide easy access to a test user and course.
Assigmment¶
class _TestAssignment(object):
assignment_count = 0
def __init__(
self,
test_client,
test_user,
runestone_db_tools,
aname,
course,
is_visible=False,
):
self.test_client = test_client
self.runestone_db_tools = runestone_db_tools
self.assignment_name = aname
self.course = course
self.description = "default description"
self.is_visible = is_visible
self.due = datetime.datetime.utcnow() + datetime.timedelta(days=7)
self.assignment_instructor = test_user(
"assign_instructor_{}".format(_TestAssignment.assignment_count),
"password",
course,
)
self.assignment_instructor.make_instructor()
self.assignment_instructor.login()
self.assignment_id = json.loads(
self.test_client.validate(
"admin/createAssignment", data={"name": self.assignment_name}
)
)[self.assignment_name]
assert self.assignment_id
_TestAssignment.assignment_count += 1
def addq_to_assignment(self, **kwargs):
if "points" not in kwargs:
kwargs["points"] = 1
kwargs["assignment"] = self.assignment_id
res = self.test_client.validate(
"admin/add__or_update_assignment_question", data=kwargs
)
res = json.loads(res)
assert res["status"] == "success"
def autograde(self, sid=None):
print("autograding", self.assignment_name)
vars = dict(assignment=self.assignment_name)
if sid:
vars["sid"] = sid
res = json.loads(self.test_client.validate("assignments/autograde", data=vars))
assert res["message"].startswith("autograded")
return res
def questions(self):
Return a list of all (id, name) values for each question in an assignment
db = self.runestone_db_tools.db
a_q_rows = db(
(db.assignment_questions.assignment_id == self.assignment_id)
& (db.assignment_questions.question_id == db.questions.id)
).select(orderby=db.assignment_questions.sorting_priority)
res = []
for row in a_q_rows:
res.append(tuple([row.questions.id, row.questions.name]))
return res
def calculate_totals(self):
assert json.loads(
self.test_client.validate(
"assignments/calculate_totals",
data=dict(assignment=self.assignment_name),
)
)["success"]
def make_visible(self):
self.is_visible = True
self.save_assignment()
def set_duedate(self, newdeadline):
the newdeadline should be a datetime object
self.due = newdeadline
self.save_assignment()
def save_assignment(self):
assert (
json.loads(
self.test_client.validate(
"admin/save_assignment",
data=dict(
assignment_id=self.assignment_id,
visible="T" if self.is_visible else "F",
description=self.description,
due=str(self.due),
),
)
)["status"]
== "success"
)
def release_grades(self):
self.test_client.post(
"admin/releasegrades",
data=dict(assignmentid=self.assignment_id, released="yes"),
)
assert self.test_client.text == "Success"
@pytest.fixture
def test_assignment(test_client, test_user, runestone_db_tools):
return lambda *args, **kwargs: _TestAssignment(
test_client, test_user, runestone_db_tools, *args, **kwargs
)
Selenium¶
Provide access to Runestone through a web browser using Selenium. There’s a lot of shared code between these tests and the Runestone Component tests using Selenium; see shared_conftest.py
for details.
Create an instance of Selenium once per testing session.
Start a virtual display for Linux if there’s no X terminal available.
Start up the Selenium driver.
When run as root, Chrome complains Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
Here’s a crude check for being root.
selenium_logging: Ask Chrome to save the logs from the JavaScript console. Copied from SO.
Shut everything down.
Provide additional server methods for Selenium.
A _TestUser
instance.
test_user,
):
self.get("default/user/login")
self.driver.find_element_by_id("auth_user_username").send_keys(
test_user.username
)
self.driver.find_element_by_id("auth_user_password").send_keys(
test_user.password
)
self.driver.find_element_by_id("login_button").click()
self.user = test_user
def logout(self):
self.get("default/user/logout")
For some strange reason, the server occasionally doesn’t put the “Logged out” message on a logout. ???
Assume that visiting the logout URL then waiting for a timeout will ensure the logout worked, even if the message can’t be found.
Present _SeleniumServerUtils
as a fixture.
A fixture to login to the test_user_1 account using Selenium before testing, then logout when the tests complete.