Hi Benjamin,

thanks for the summary ... please make this summary available for discussion in MTS#51 (upload it in some form).

I have already some comments .. but I think it is now time for MTS to make the final discussion.

Thanks for looking into this.
Stephan

-----Original Message-----
From: Benjamin Zeiss [mailto:zeiss@CS.UNI-GOETTINGEN.DE] 
Sent: 1. syyskuuta 2010 11:26
To: STF409_MTS_TTCN3_TOOLS@LIST.ETSI.ORG
Subject: Re: Syntax and semantics tests

Hello again,

we have discussed about the issues raised by Stephan in STF 393 and with
Bogdan. It would be good if any comments or feedback would be provided
as soon as possible as I try to clarify the proforma as soon as possible.

Here is a brief summary of our discussion results:

1) Test classification: syntactic and semantic tests.

This was actually a pretty heated and lengthy discussion. The consensus
that we could reach is the following:

 - Syntactic tests are derived from annex A of part 1 (including the
   few natural language expressed static semantic rules in the BNF).
 - Syntactic tests should be correct with respect to the static
   semantics.

The semantic tests refer to the respective clauses that they test. This
kind of separation is pretty clear and not open for interpretation even
though it may not be entirely satisfying for everyone from a technical
perspective. It turned out that it wasn't possible to satisfy everyone
with a single definition anyway. Bogdans definition was also not agreed
on by everyone.

2) Relevance of Part 4 for STF 409 test specification.

 - Yes, we should refer to them where applicable. About the format, a
   few comments in the point about the test purposes.

3) How are the tests executed?

 - We have two different levels of test results:
   the first level is the verdict provided by the TTCN-3 tool that we
   test, i.e., we stimulate the TTCN-3 test tool with samples and the
   TTCN-3 tool provides a result. This can be a test verdict, and error
   message/code, and so on. On this level, it does not really matter
   what the output or verdict is.

   The second level is the actual test evaluation where we take the
   output from the TTCN-3 tool and compare it with our expected output.
   On this level, we expect the verdict to be "pass" if the tool is
   conformant.
 - Due to these two levels, we somehow need to specify the expected
   verdict from the tool in order to evaluate it. For that purpose, we
   can use the @verdict t3doc tag. It must be extended to be useable on
   modules however. Another alterntive would be to "misuse" the @remark
   tag.

4) Incomplete sections and tables in the proforma document.

 - The document is designed to be copied and filled by the STF 409.
   Therefore, tables are empty only if there are no samples available.
   Nevertheless, they are in the right positions and should not be
   moved.

5) No predefined content and format for the test purposes.

 - Actually there is a table-based format in annex B. There is
   just no example.
 - Currently, purposes are at least specified and referred to in the
   @purposes tag of test cases. This is pretty rudimentary and some
   important information is missing, e.g., the expected verdict (see
   above), a reference to the respetive clause in the standard that the
   test refers to.
 - We can imagine to use also t3doc tags to refer to such requirements,
   e.g.,
     @requirement 1:x.y.z (for annex: 1:A.x (x is the rule number)), 4:x.y
   where 1: refers to part-1, x.y.z to the respective clause etc.
   The nice thing about the t3doc approach would be that a tool such as
   t3d could basically generate the test purpose documentation without
   the necessity to maintain multiple document.
 - Nevertheless, I currently take a look at TPLan whether it makes
   sense in our case. Possibly, we can also generate the TPLan
   descriptions from the t3doc tags as well. Our overall
   impression is that we have to be careful with the resources that we
   have and that it would be best if the overhead with the test purpose
   desriptions is not that big.
 - It would be best if the conformance test suite is aligned with the
   respective version numbers of the TTCN-3 standards. That way, it is
   clear to what version the tool conformance test suite refers to.

6) Completeness / coverage criteria.

 - Syntactic testing is measured with respect to the EBNF coverage as
   Nikolay suggested. We think that the "minimal" variant is the only
   reasonable variant. We will have trouble already to create a
   sufficient amount of tests for this anyway. In the long term, some
   automated approach as Nikolay suggested will probably be absolutely
   necessary. I don't see though that we are able to do this for the
   October session though.
 - Static and dynamics semantics are measured in terms of the text
   clauses in the part-1 document of the TTCN-3 standard. So basically,
   the standard is the requirement catalogue that Nikolay mentioned. We
   all agreed here that maintaining a separate requirements catalog is
   probably too much work and has limited use.

BR,
Benjamin