test_server.py - Tests using the web2py server¶
These tests start the web2py server then submit requests to it. All the fixtures are auto-imported by pytest from conftest.py.
Contents
Imports¶
These are listed in the order prescribed by PEP 8.
Standard library¶
Third-party imports¶
Local imports¶
Debugging notes¶
Invoke the debugger.
Put this in web2py code, then use the web-based debugger.
Tests¶
Use for easy manual testing of the server, by setting up a user and class automatically. Comment out the line below to enable it.
Modify this as desired to create courses, users, etc. for manual testing.
Pause in the debugger until manual testing is done.
This test ensures that we have the routing set up for testing properly. This test will fail if routes.py is set up as follows. routes_onerror = [
(‘runestone/static/404’, ‘/runestone/static/fail.html’), (‘runestone/500’, ‘/runestone/default/reportabug.html’), ]
for testing purposes we don’t want web2py to capture 500 errors.
Validate the HTML produced by various web2py pages. NOTE – this is the start of a really really long decorator for test_1
Admin¶
FIXME: Flashed messages don’t seem to work. (‘admin/index’, False, ‘You must be registered for a course to access this page’, 1), (‘admin/index’, True, ‘You must be an instructor to access this page’, 1),
Default¶
User
The authentication section gives the URLs exposed by web2py. Check these.
One validation error is a result of removing the input field for the e-mail, but web2py still tries to label it, which is an error.
Runestone doesn’t support this.
This doesn’t display a webpage, but instead redirects to courses. (‘default/user/reset_password, False, ‘Reset password’, 1),
FIXME: This produces an exception.
Other pages
TODO: What is this for? (‘default/call’, False, ‘Not found’, 0),
web2py generates invalid labels for the radio buttons in this form.
Should work in both cases.
(‘default/sendreport’, True, ‘Could not create issue’, 1),
TODO: This doesn’t really test much of the body of either of these.
If we choose an invalid course, then we go to the profile to allow the user to add that course. The second validation failure seems to be about the for attribute of the `<label class="readonly" for="auth_user_email" id="auth_user_email__label"> tag, since the id auth_user_email isn’t defined elsewhere.
TODO: Many other views!
Validate the HTML in instructor-only pages. NOTE – this is the start of a really really long decorator for test_2
FIXME: The element <form id="editIndexRST" action=""> in views/admin/admin.html produces the error Bad value \u201c\u201d for attribute \u201caction\u201d on element \u201cform\u201d: Must be non-empty..
Admin¶
This endpoint produces JSON, so don’t check it.
TODO: This produces an exception. (‘admin/practice’, ‘Choose when students should start their practice.’, 1), TODO: This deletes the course, making the test framework raise an exception. Need a separate case to catch this. (‘admin/deletecourse’, ‘Manage Section’, 2), FIXME: these raise an exception. (‘admin/addinstructor’, ‘Trying to add non-user’, 1), – this is an api call (‘admin/add_practice_items’, ‘xxx’, 1), – this is an api call
(‘admin/backup’, ‘xxx’, 1),
(‘admin/removeassign’, ‘Cannot remove assignment with id of’, 1), (‘admin/removeinstructor’, ‘xxx’, 1), (‘admin/removeStudents’, ‘xxx’, 1),
TODO: added to the createAssignment endpoint so far.
Dashboard
————–
TODO: This doesn’t really test anything about either exercisemetrics or questiongrades other than properly handling a call with no information
("dashboard/exercisemetrics", "Instructor Dashboard", 1),
("dashboard/questiongrades", "Instructor Dashboard", 1),
],
)
def test_validate_instructor_pages(
url, expected_string, expected_errors, test_client, test_user, test_user_1
):
test_instructor_1 = test_user("test_instructor_1", "password_1", test_user_1.course)
test_instructor_1.make_instructor()
Make sure that non-instructors are redirected.
Test the instructor results.
Test the ajax/preview_question endpoint.
Passing no parameters should raise an error.
Passing something not JSON-encoded should raise an error.
Passing invalid RST should produce a Sphinx warning.
Passing valid RST with no Runestone component should produce an error.
Passing a string with Unicode should work. Note that 0x0263 == 611; the JSON-encoded result will use this.
Verify that question_1 is not in the database. TODO: This passes even if the DBURL env variable in ajax.py fucntion preview_question isn’t deleted. So, this test doesn’t work.
TODO: Add a test case for when the runestone build produces a non-zero return code.
Test the default/user/profile endpoint.
Test a non-existant course.
Test an invalid e-mail address. TODO: This doesn’t produce an error message.
Change the user’s profile data; add a new course.
Check the values.
The username shouldn’t be changable.
TODO: The e-mail address isn’t updated. assert user.email == email
TODO: I’m not sure where the section is stored. assert user.section == section
Test that the course name is correctly preserved across registrations if other fields are invalid.
Registration doesn’t work unless we’re logged out.
Now, post the registration.
The e-mail address must be unique.
Note that course_id is (on the form) actually a course name.
Check that the pricing system works correctly.
Check the pricing.
These course names rely on defaults in the runestone_db_tools fixture.
It would be nice to use the test_user fixture, but we’re not using the web interface here – it’s direct database access instead. This is an alternative.
Provide a non-null value for these required fields.
First, test on a base course.
Test in a child course as well. Create a matrix of all base course prices by all child course prices.
for expected_price, actual_base_price, actual_child_price in [
(0, None, None),
(0, None, 0),
(0, None, -1),
(2, None, 2),
(0, 0, None),
(0, 0, 0),
(0, 0, -1),
(2, 0, 2),
(0, -2, None),
(0, -2, 0),
(0, -2, -1),
(2, -2, 2),
(3, 3, None),
(0, 3, 0),
(0, 3, -1),
(2, 3, 2),
]:
db(db.courses.id == base_course.course_id).update(
student_price=actual_base_price
)
db(db.courses.id == child_course_1.course_id).update(
student_price=actual_child_price
)
assert (
default_controller._course_price(child_course_1.course_id) == expected_price
)
Make sure the book is free if the student already owns the base course.
First, create another child course and add the current student to it.
Now check the price of a different child course of the same base course.
Check that setting the price causes redirects to the correct location (payment vs. donation) when registering for a course or adding a new course.
Check registering for a free course.
Verify the user was added to the user_courses table.
Check adding a free course.
Same as above.
Check registering for a paid course.
Check registering for a paid course.
Until payment is provided, the user shouldn’t be added to the user_courses table. Ensure that refresh, login/logout, profile changes, adding another class, etc. don’t allow access.
Check adding a paid course.
Verify no access without payment.
Check that payments are handled correctly.
def test_payments(runestone_controller, runestone_db_tools, test_user):
if not runestone_controller.settings.STRIPE_SECRET_KEY:
pytest.skip("No Stripe keys provided.")
db = runestone_db_tools.db
course_1 = runestone_db_tools.create_course(student_price=100)
test_user_1 = test_user("test_user_1", "password_1", course_1, is_free=False)
def did_payment():
return (
db(
(db.user_courses.course_id == course_1.course_id)
& (db.user_courses.user_id == test_user_1.user_id)
)
.select()
.first()
)
Test some failing tokens.
Check that the payment record is correct.
Test dynamic book routing.
Test that a draft is accessible only to instructors.
Test the no-login case.
Test for a book that doesn’t require a login. First, change the book to not require a login.
Test error cases.
A non-existant course.
A non-existant page.
A directory.
Attempt to access files outside a course.
Attempt to access a course we’re not registered for. TODO: Need to create another base course for this to work.
A valid page. Check the book config as well.
Drafts shouldn’t be accessible by students.
Check routing in a base course.
if is_logged_in:
test_user_1.update_profile(
course_name=test_user_1.course.base_course, is_free=True
)
validate(
"books/published/{}/index.html".format(base_course),
[
"The red car drove away.",
"eBookConfig.course = '{}';".format(base_course),
"eBookConfig.basecourse = '{}';".format(base_course),
],
)
Test static content.
validate(
"books/published/{}/_static/basic.css".format(base_course),
"Sphinx stylesheet -- basic theme.",
)
def test_assignments(test_client, runestone_db_tools, test_user):
course_3 = runestone_db_tools.create_course("test_course_3")
test_instructor_1 = test_user("test_instructor_1", "password_1", course_3)
test_instructor_1.make_instructor()
test_instructor_1.login()
db = runestone_db_tools.db
name_1 = "test_assignment_1"
name_2 = "test_assignment_2"
name_3 = "test_assignment_3"
Create an assignment – using createAssignment
Make sure you can’t create two assignments with the same name
Rename assignment
test_client.post("admin/createAssignment", data=dict(name=name_2))
assign2 = (
db(
(db.assignments.name == name_2)
& (db.assignments.course == test_instructor_1.course.course_id)
)
.select()
.first()
)
assert assign2
test_client.post(
"admin/renameAssignment", data=dict(name=name_3, original=assign2.id)
)
assert db(db.assignments.name == name_3).select().first()
assert not db(db.assignments.name == name_2).select().first()
Make sure you can’t rename an assignment to an already used assignment
Delete an assignment – using removeassignment
test_client.post("admin/removeassign", data=dict(assignid=assign1.id))
assert not db(db.assignments.name == name_1).select().first()
test_client.post("admin/removeassign", data=dict(assignid=assign2.id))
assert not db(db.assignments.name == name_3).select().first()
test_client.post("admin/removeassign", data=dict(assignid=9999999))
assert "Error" in test_client.text
def test_instructor_practice_admin(test_client, runestone_db_tools, test_user):
course_4 = runestone_db_tools.create_course("test_course_1")
test_student_1 = test_user("test_student_1", "password_1", course_4)
test_student_1.logout()
test_instructor_1 = test_user("test_instructor_1", "password_1", course_4)
test_instructor_1.make_instructor()
test_instructor_1.login()
db = runestone_db_tools.db
course_start_date = datetime.datetime.strptime(
course_4.term_start_date, "%Y-%m-%d"
).date()
start_date = course_start_date + datetime.timedelta(days=13)
end_date = datetime.datetime.today().date() + datetime.timedelta(days=30)
max_practice_days = 40
max_practice_questions = 400
day_points = 1
question_points = 0.2
questions_to_complete_day = 5
graded = 0
Test the practice tool settings for the course.
flashcard_creation_method = 2
test_client.post(
"admin/practice",
data={
"StartDate": start_date,
"EndDate": end_date,
"graded": graded,
"maxPracticeDays": max_practice_days,
"maxPracticeQuestions": max_practice_questions,
"pointsPerDay": day_points,
"pointsPerQuestion": question_points,
"questionsPerDay": questions_to_complete_day,
"flashcardsCreationType": 2,
"question_points": question_points,
},
)
practice_settings_1 = (
db(
(db.course_practice.auth_user_id == test_instructor_1.user_id)
& (db.course_practice.course_name == course_4.course_name)
& (db.course_practice.start_date == start_date)
& (db.course_practice.end_date == end_date)
& (
db.course_practice.flashcard_creation_method
== flashcard_creation_method
)
& (db.course_practice.graded == graded)
)
.select()
.first()
)
assert practice_settings_1
if practice_settings_1.spacing == 1:
assert practice_settings_1.max_practice_days == max_practice_days
assert practice_settings_1.day_points == day_points
assert (
practice_settings_1.questions_to_complete_day == questions_to_complete_day
)
else:
assert practice_settings_1.max_practice_questions == max_practice_questions
assert practice_settings_1.question_points == question_points
Test instructor adding a subchapter to the practice tool for students.
I need to call set_tz_offset to set timezoneoffset in the session.
The reason I’m manually stringifying the list value is that test_client.post does something strange with compound objects instead of passing them to json.dumps.
test_client.post(
"admin/add_practice_items",
data={"data": '["1. Test chapter 1/1.2 Subchapter B"]'},
)
practice_settings_1 = (
db(
(db.user_topic_practice.user_id == test_student_1.user_id)
& (db.user_topic_practice.course_name == course_4.course_name)
& (db.user_topic_practice.chapter_label == "test_chapter_1")
& (db.user_topic_practice.sub_chapter_label == "subchapter_b")
)
.select()
.first()
)
assert practice_settings_1
@pytest.mark.skip(reason="Requires BookServer for testing -- TODO")
def test_deleteaccount(test_client, runestone_db_tools, test_user):
course_3 = runestone_db_tools.create_course("test_course_3")
the_user = test_user("user_to_delete", "password_1", course_3)
the_user.login()
validate = the_user.test_client.validate
the_user.hsblog(
event="mChoice",
act="answer:1:correct",
answer="1",
correct="T",
div_id="subc_b_1",
course="test_course_3",
)
validate("default/delete", "About Runestone", data=dict(deleteaccount="checked"))
db = runestone_db_tools.db
res = db(db.auth_user.username == "user_to_delete").select().first()
print(res)
time.sleep(2)
assert not db(db.useinfo.sid == "user_to_delete").select().first()
assert not db(db.code.sid == "user_to_delete").select().first()
for t in [
"clickablearea",
"codelens",
"dragndrop",
"fitb",
"lp",
"mchoice",
"parsons",
"shortanswer",
]:
assert (
not db(db["{}_answers".format(t)].sid == "user_to_delete").select().first()
)
Test the grades report. When this test fails it is very very difficult to figure out why. The data structures being compared are very large which makes it very very difficult to pin down what is failing. In addition it seems there is a dictionary in here somewhere where the order of things shifts around. I think it is currenly broken because more components now return a percent correct value.
Create test users.
Create test data¶
Create test users.
Prepare common arguments for each question type.
shortanswer_kwargs = dict(
event="shortanswer", div_id="test_short_answer_1", course=course_name
)
fitb_kwargs = dict(event="fillb", div_id="test_fitb_1", course=course_name)
mchoice_kwargs = dict(event="mChoice", div_id="test_mchoice_1", course=course_name)
lp_kwargs = dict(
event="lp_build",
div_id="test_lp_1",
course=course_name,
builder="unsafe-python",
)
unittest_kwargs = dict(event="unittest", div_id="units2", course=course_name)
User 0: no data supplied
User 1: correct answers
It doesn’t matter which user logs out, since all three users share the same client.
logout = test_user_array[2].test_client.logout
logout()
test_user_array[1].login()
assert_passing(1, act=test_user_array[1].username, **shortanswer_kwargs)
assert_passing(1, answer=json.dumps(["red", "away"]), **fitb_kwargs)
assert_passing(1, answer="0", correct="T", **mchoice_kwargs)
assert_passing(
1, answer=json.dumps({"code_snippets": ["def one(): return 1"]}), **lp_kwargs
)
assert_passing(1, act="percent:100:passed:2:failed:0", **unittest_kwargs)
User 2: incorrect answers
Add three shortanswer answers, to make sure the number of attempts is correctly recorded.
for x in range(3):
assert_passing(2, act=test_user_array[2].username, **shortanswer_kwargs)
assert_passing(2, answer=json.dumps(["xxx", "xxxx"]), **fitb_kwargs)
assert_passing(2, answer="1", correct="F", **mchoice_kwargs)
assert_passing(
2, answer=json.dumps({"code_snippets": ["def one(): return 2"]}), **lp_kwargs
)
assert_passing(2, act="percent:50:passed:1:failed:1", **unittest_kwargs)
User 3: no data supplied, and no longer in course.
Wait until the autograder is run to remove the student, so they will have a grade but not have any submissions.
Test the grades_report endpoint
Test not being an instructor.
Test an invalid assignment.
Create an assignment.
Add questions to the assignment.
Determine the order of the questions and the point values.
Autograde the assignment.
Remove test user 3 from the course. They can’t be removed from the current course, so create a new one then add this user to it.
Test this assignment.¶
Log back in as the instructor.
Now, we can get the report.
Define a regex string comparison.
See if a date in ISO format followed by a “Z” is close to the current time.
Parse the date string. Assume it ends with a Z and discard this.
Per the docs, this function requires Python 3.7+.
Hope for the best on older Python.
These are based on the data input for each user earlier in this test.
expected_grades = {
"colHeaders": [
"userid",
"Family name",
"Given name",
"e-mail",
"avg grade (%)",
"1",
"1",
"1",
"2.1",
"2",
],
"data": [
[
"div_id",
"",
"",
"",
"",
"test_short_answer_1",
"test_fitb_1",
"test_mchoice_1",
"test_lp_1",
"units2",
],
[
"location",
"",
"",
"",
"",
"index - ",
"index - ",
"index - ",
"lp_demo.py - ",
"index - ",
],
[
"type",
"",
"",
"",
"",
"shortanswer",
"fillintheblank",
"mchoice",
"lp_build",
"activecode",
],
See the point values assigned earlier.
["points", "", "", "", "", 0, 1, 2, 3, 4],
["avg grade (%)", "", "", "", ""],
["avg attempts", "", "", "", ""],
["test_user_0", "user_0", "test", "test_user_0@foo.com", 0.0],
["test_user_1", "user_1", "test", "test_user_1@foo.com", 1.0],
["test_user_2", "user_2", "test", "test_user_2@foo.com", 0.2],
["test_user_3", "user_3", "test", "test_user_3@foo.com", 0.0],
],
Correct since the first 3 questions are all on the index page.
User 0: not submitted.
The format is:
[timestamp, score, answer, correct, num_attempts].
User 1: all correct.
User 2: all incorrect.
Use a regex for the file’s path.
User 3: not submitted.
The format is:
Note: on test failure, pytest will report as incorrect all the AlmostNow() and RegexEquals items, even though they may have actually compared as equal.
assert grades == expected_grades
lets break this up a bit.
Test with no login.
Test the teaming report.
Create test users.
Create test data¶
Create test users.
Prepare common arguments for each question type.
Add three shortanswer answers, to make sure the number of attempts is correctly recorded.
with open(
"applications/{}/books/{}/test_course_1.csv".format(
runestone_name, course.base_course
),
"w",
encoding="utf-8",
) as f:
f.write(
"user id,user name,team name\n"
"test_user_0@foo.com,test user_0,team 1\n"
"test_user_1@foo.com,test user_1,team 1\n"
"test_user_2@foo.com,test user_2,team 1\n"
)
TODO: Test not being an instructor.
TODO: Test with no login.
@pytest.mark.skip(
reason="Can't render new BookServer template using old server. TODO: Port to the BookServer."
)
def test_pageprogress(test_client, runestone_db_tools, test_user_1):
test_user_1.login()
test_user_1.hsblog(
event="mChoice",
act="answer:1:correct",
answer="1",
correct="T",
div_id="subc_b_1",
course=test_user_1.course.course_name,
)
Since the user has answered the question the count for subc_b_1 should be 1 cannot test the totals on the client without javascript but that is covered in the selenium tests on the components side.
test_user_1.test_client.validate(
"books/published/{}/test_chapter_1/subchapter_b.html".format(
test_user_1.course.base_course
),
'"subc_b_1": 1',
)
assert '"LearningZone_poll": 0' in test_user_1.test_client.text
assert '"subc_b_fitb": 0' in test_user_1.test_client.text
@pytest.mark.skip(
reason="Can't render new BookServer template using old server. What does this test do? Should it be ported?"
)
def test_lockdown(test_client, test_user_1):
test_user_1.login()
base_course = test_user_1.course.base_course
res = test_client.validate("books/published/{}/index.html".format(base_course))
assert "Runestone in social media:" in res
assert ">Change Course</a></li>" in res
assert 'id="profilelink">Edit' in res
assert '<ul class="dropdown-menu user-menu">' in res
assert "<span id='numuserspan'></span><span class='loggedinuser'></span>" in res
assert '<script async src="https://hypothes.is/embed.js"></script>' in res