MIATA Version 0.0 This is the very first MIATA version. It was drastically changed during the public consultation process and is only listed here for demonstration purposes on how MIATA evolved. It is outdated and should not be used for MIATA-compliant reporting.

Posted on Onctober 16, 2009

Updated to  Version 1.0 and moved to archive in May, 2010

 The reporting framework modules; Version 0.0
The initial publication guidelines of immune monitoring strategies are based on the premise that reporting of immune monitoring data from clinical studies will benefit by accurately including information for 5 “modules” that critically impact test results. Detailed annotation of published data for these 5 topics will be of additional benefit if accompanied by the appropriate reporting of raw data to allow for full evaluation of the value of published data sets.

 

 

Module 1: The Sample




    • A. Donor: Essential information on donor including age, gender, HLA type, diagnosis, disease status, treatment, prior relevant treatments.


    • B. Source: Collection methodology for specimens (e.g. heparin, EDTA, citrate, other). Time lapse between blood draw and PBMC isolation, and interim storage. Methodology for PBMC isolation. Was shipment of specimen necessary? If yes, under which conditions was shipment performed? [1,2]


    • C. Cryopreservation and Storage: Methodology for cryopreservation (controlled temperature device, Mr. Frosty, other) and storage. Elapsed time between cryopreservation and final storage and final storage temperature. [1,2,3,4,5]


    • D. Description of quality assurance criteria applied for specimen and cut-off values for sample acceptance (e.g. viability, apoptosis, function). [5]




Module 2: The Assay


  • A. Cell counting: Methodology applied to count cells. Were viability and apoptosis assessed? If so report values. [5,6]
  • B. Medium/serum: Source of medium/serum used for the assay. Description of any pretesting performed and performance characteristics of the chosen medium/serum (e.g. background reactivity levels). [5,7,8]
  • C. Assay: An overview of critical assay procedures including source of main reagents and materials used and crucial treatment procedures of cells (e.g. overnight resting, in vitro stimulation). Description of any published harmonization guidelines and recommendations followed. [6,9,10,11]
  • D. Description of internal (e.g. positive control antigens or peptide pools, irrelevant or negative controls, mitogens) and external controls (e.g. PBMC reference samples) used in assay? [12,13,14] Description of quality assurance criteria for the controls including used peptide or peptide mixes. [15]






Module 3: Data acquisition



  • A. Description of equipment and software versions used for data acquisition. Setup of the equipment. Description of any methodologies employed for instrument calibration and validation and operator training. [16]
  • B. List of controls applied during data acquisition (e.g. spot size, form and intensity settings in the case of Elispot, compensation and gating strategies, control beads in the case of flow cytometry analysis). [11,14]
  • C. Description of quality assurance criteria (e.g. were results audited?) for the results? [6,7,17]

 

 

 

Module 4: The (interpretation of) results



 

  • Description of how raw data were processed and interpreted, and how the threshold for “positive reactivity” was defined. [18] Description of how a vaccine-induced induction of a response between two time points was defined. A description of statistical tests or empiric rules applied and the reasoning behind these test choices. [19,20,21,22] A statement on whether any data was excluded from the analysis, and if so the reason for the exclusion.


Module 5: The lab environment



  • A. A statement of how laboratory operations were guided. Does the lab work under GLP or GLP-like conditions? [23,24].
  • B. A description of whether procedures were standardized in the laboratory and if SOPs were established and followed for the relevant procedures and assays. A description of any deviation from the SOP, and if so, how these were handled. [7,25,26].
  • C. A statement of whether SOPs are qualified or validated. [27] If SOP are qualified or validated, a specific description of how the relevant parameters (accuracy, specificity, precision, sensitivity, upper and lower limits of quantification, robustness) were determined, and how the assay specifications were defined and met. For each validated (or qualified assay if the data is available) the quality control elements for the assay (positive and negative reference sample identity and range), and pass/fail criteria for the assay should also be included. [7,28,29,30,31]
  • D. A statement of whether the laboratory regularly participates in external quality assurance programs (proficiency panels) and harmonization efforts. A statement of whether trained personnel performed the testing, and if so, a description of how personnel is monitored for assay performance? [6,10,32]





IDAuthorModule
1A - Information on the donor
Module 1B - Sample sourceModule 1C - CryopreservationModule 1D - QC of cell material
1Gouttefangeas, Cecile What is essential? Too much asked forno commentAdd time
between storage and testing
no comment
2van der Burg, Sjoerd H. This is out of scope of MIATA but rather information on
biological significance
Basically goodBasically goodCould control outcome from Modules B and C, clearly MIATA
3, 4Speiser, Daniel (two comments)no commentno commentno commentIndication of cell recovery at thawing in percent of cell
numbers at freezing
5Navarrete, Marcelo HLA type of donor might be irrelevantno commentno commentno comment
6Straten, Per Thor and Hardrup SineWhat is essential? Only age and gender for MIATAQuality is most important issue and is dealt with in
Module 1D
Quality is most important issue and is dealt with in
Module 1D
Very good and add indication of cell recovery at thawing 
7Pawelec, GrahamHLA type not required for result interpretation, donor
ethnicity needs to be reported as well as CMV serostatus if available
as it impacts on overall constitution of immune system
Specify conditions not only for PBMC but also whole blood
if performed
Details are critical, strong recommendation to standardise
numbers of cells per amule and density at which cells are frozen.
no comment
8 Ottensmeier, ChristianAge and gender are OK. Relevance of other details will be
determined by the context of the reporting.
Method for anticoagulation. Report on whether time labs
and mode of transport is controlled and time from blood storage to
analyiss. Still keep it as simple as possible
Perhaps average storage time with range, it might be more
important which informaiton has been collected then what the
characteristics are! 
1A-C are captured by this point!
9Odunsi, KunleHlA type may not be critical, patient diagnosis seems
adequate, additional details about disease status and prior treatments
could be provided in the context of the specific study.
no commentInclude information on cryopreservation mediumWhat is meant by QA?
10Walter, Steffen and Samorski, ReginaInformation for biological significance and beyond the
scope of MIATA, IN longitudinal trials it is of importance to report
timing of PBMC sampling with respect to treatment schedule and how many
time points were evaluated from evaluable patients.
no commentIndicate media used for PBMC isolation and
cryopreservation. 
Sample acceptance criteria must be carefully validated for
every individual assays which is most often not performed. Therefore B
and C should be reported in additon to D
11Ferrari, GuidoModules 1A-C sufficient if GCLP is not an requirementModules 1A-C sufficient if GCLP is not an requirementModules 1A-C sufficient if GCLP is not an requirementIntimately connected with GCLP guideleines
12Rooke, Ronald and Bain, ChristineAdd ethnic originDuration and temperature during transport, preparation on
site or centralized
Include information on cryopreservation mediumQA should control Modules 1A-C
13Appay, Victorno commentno commentno commentno comment
14Lopatin, UriDetailed information on the donor might allow a trail back
oto the individual patient and might not be allowable.
Add whether a standard or non-standard protocl for PBMC
colleciton was used.
no commentno comment
15Ottenhoff, Tom HM and Joosten, SimoneInclude country/region of trial and for infectious
diseases the incidence and prevalence of infection in the specific
pupulation studied as well as information on historic vaccination (e.g.
TBC) HLA type might not be feasinble or relevant.
Extremely important (Ref Smith Plos One)OKOK
16Schuler Gerold;  Kämpgen
Eckard; Gross, Stefanie
Necessity to provide such information depends on the sutdy.AgreeStorage affects readout of some assays more than others.
Can be captured in D
Absolutely necessary !!
17Plant, Anne L. no commentno commentno commentno comment
18Kern, Florian  and Powell,
Fiona
HLA not always neded. Time of blood draw should be
indicated
Not enough data on influence of time between blood draw
and preparation available so far. Interpretation difficult.
Enough to say if cells were frozen or used freshlyShould control 1B and 1C
19Betts, Michael no commentno commentno commentno comment
20Weber, Jeffrey I am not sure how useful the clinical data on prior
therapy would be, especially to the scientists
no commentno commentno comment
21Gulley, James L no commentno commentno commentno comment
22D'Souza,
Patricia and Lane, Jim
no commentno commentno commentno comment
23Chen,
Weisan; Cebon,  Jonathan, Davis, Ian
When
peptide-based vaccination is performed, reporting HLA type should be
essential. From our experience with clinical trial patients receiving
full-length NY-ESO-1protein vaccine, we have also found it essential to
know HLA type because the T cell responses (specifically
immunodominance patterns) are totally HLA-dependent. That could help
readers to further assess the T cell response identified in a special
region of the known antigen even if no detailed epitope mapping data
provided.
If we know
that the antigen-specific T cell expanded 40~100 times during a 11~13
day culture, many of these details may be superfluous, so long as a
measure of assay performance and variability is provided.
 Although anticoagulation may affect T cell responsiveness, as
does overnight storage, this should not matter so long as protocols are
consistent.  A statement indicating that standardised protocols
for collection have been used should be sufficient.
as aboveBecause the background will vary depending on patient
samples, we generally consider a cut-off value of 0.1% of total CD8+
cells for ICS is achievable. It would be great if someone could come up
with a value that is both achievable and supported by statistics.
24Cox, Josephineno commentno commentMaybe time since collected ie over the period of 1-3 years
4-10 >10. Some trials take a long time! Labs change procedure over
time too, though hopefully not during a trial
Recovery? Hard to quantify but I think important.
25Janet Siebertno commentno commentno commentno comment
26Cristina Muselli and groupWe agreeWe agree, especially if the blood collection, PBMC
isolation and immune-monitoring assays are done by different labs,
guidelines should be set for the above points. Should be specified what
measures are taken if any to standardize and qualify different clinical
site for blood collection/processing (operator training at the site,
uniformity of protocol/SOPs between sites)
We agree, same as aboveWe agree, we actually think this is perhaps the most
critical aspect of the whole module.  The acceptance criteria and
the sample cut-off values should be developed by each lab during the
assay development phase and applied to all patient samples.  In
our experience, application of acceptance criteria does render some
patient samples non-evaluable and helps to determine false positive and
false negative data.
27Schendel, DoloresFor
example, full HLA typing of patients may not be relevant for many
studies. Furthermore, it is not specified whether this should include
class I and class II typing results and at what level the allotypes
should be specified. This can be a very expensive analysis if it must
be done commercially.
no commentno commentno comment
28Knuth, Alexander; van den Broek, Mariesno commentno commentno commentno comment
29Kreiter, Sebastian; Attig, Sebastian; Sahin, UgurThe amount of information on the donor should depend on
the specific setting of every study. Only information directly relevant
to the test result should be mandatory requirements. This might be HLA
type in some studies. Information needed to interpret results from
immunomonitoring assays should be considered of being outside the scope
of MIATA. Minimal information should be age, gender and ethnic
background.  
no commentno commentIt is well established that the quality of a sample that
goes into an assay will determine the results of the test (garbage in
garbage out). Investigators should therefore (1) take measures to make
sure that samples for T cell assays have a high quality and (2)
implement a QC to make sure that the effect of all measures taken to
increase cell quality were effective.
30Maio,
Michele;  Nicolay, Hugues J.M. 
We agree with age, gender, diagnosis, disease status,
treatment and prior relevant treatment. However, we believe that
reporting HLA type is essential only when peptide-based vaccination is
performed. Otherwise HLA type should not be considered as essential.
We believe that time for the blood sample is important as
far as we observed differences in the detection of the T-cell responses
at several time-points after the vaccination. Also, interim storage
should be considered when trials take a long time, in fact when you run
assays sometimes you use old material that are pre-vaccine sample that
represent the base-line of T-cell response. Shipment of blood specimens
(when necessary) is also important, because deeply relevant for PBMC
integrity.
We agree with Josephine Cox, maybe time since collected
i.e. over the period of 1-3 years 4-10 >10. Some trials take a long
time! Labs could change procedure over time too. Of course procedures
will change among labs and it should be discussed in section 5 B, but
at least the freezing medium composition should be mentioned.
In this terms Module 1D is of highest importance as the QC
will control all effects of measures taken in Modules 1A-C. Any
information on sampling, preparation and storage of samples that is of
specific interest to  the investigator and give a better insight
how a high quality of cells was achieved within a study might be
reported but should not be mandatory.   
31Rammensee, Hans-Georgno commentno comment1C:
Indicate storage system: liquid nitrogen tank or electrical freezer
no comment
32Zwierzina, Heinz; Hakansson, LeiffAge,
gender, diagnosis, disease status, treatment, prior relevant treatments
are all fundamental information which has to be included. The need to
determine the  HLA type depends on the intended study. In
particular, disease status and prior relevant treatment has to be
penetrated as these variables are of crucial importance. The systemic
immune status of controls as well as patients show considerable
variation, which will influence T-cell functional tests. Therefore, we
think that some basic “activation information” of the patients immune
status (e.g. serumconcentration of CRP and certain cytokines) at start
of treatment is important to obtain.
This is a
crucial point, the reproducibility of results obtained using different
tubes for blood sampling has to be checked carefully. The temperature
has to be kept under control and the time lapse before preparation of
PBMCs  has to be defined and should be kept as short as possible.
Even if
there are some data on the possibility to use cryopreservation of PBMCs
for T-cell functional tests, this procedure has to be carefully
controlled.
In principal, each T-cell test to be performed has to be
validated in each collaborating lab, that is the results of the test on
cryopreserved and shipped cells has to be compared to the results
obtained on fresh cells.
33Gnjatic, Sachanot necessarynot
necessary
not necessarynot necessary
34Reap, Elizabeth ; Norberg, Pamela; Watsin, AubreyWe agree.
It will depend on what questions you are asking to decide if you want
HLA type.  We don’t think it is always possible or necessary to
provide HLA type and prior relevant  treaments.  We don’t
think that the lack of HLA types should be reason to reject a
publication.  HLA typing is very expensive and may not be feasible
for all studies but it shouldn’t prevent the results from being shared.
Agree.
PBMC separation should begin as quickly after blood collection as
possible, and freezing of PBMC must begin within 8 hours after blood
collection.  No exceptions as it has been shown in the literature
that signal is lost starting after 8 hours.  This time lapse
should be recorded.  The isolation methodology is important. 
Must note gradient used (Lymphoprep or LeucoSep / Accuspin tubes with
Histopaque-1077)  Is the whole blood added directly to the
gradients or diluted, and what is the volume to tube ratio?  
Spin times and speed because spinning at too high of an RPM will cause
cell clumping,  This needs to be recorded.    
Agree.
 VERY  Important to note time range and type of storage from
PBMC isolation .  If the clinical trial site has no liquid
nitrogen and only -80C storage, how long were the PBMCs stored at
-80C?  It is important to know the cryopreservation media used and
if it contains FBS, HSA, or  serum-free.  We have found that
liquid nitrogen storage and shipment is best.
Many people mentioned reporting recovery.  Most note
how this was obtained.  What if the collection site has a
different counting method than the lab (trypan blue before freezing and
Guava after thawing at lab)?  The difference in counting methods
could have a large impact on % recovery.  
35Nishikawa, HiroyoshiAs
comments already posted, reporting HLA type should be critical for
analyses of T cell responses, even in clinical trials employing
full-length protein because there are immuno-dominant HLA pattern for T
cell responses.
no comment Medium used for cryopreserve would be important to
note.
no comment
36Schoenmaekers-Welters, Marij J.P.Disease
history. Nevertheless, this is to my opinion beyond the scope of MIATA.
Time lapse
between blood draw and treatment (before or after) of the patient could
be relevant information since too short after intervention the results
might be differently then waiting a few weeks longer.
Time lapse
between final storage of PBMC vials and thawing for the immunological
assay.  It needs to be established how critical this is.
Percentages
of viability, death cells and/or apoptosis after PBMC isolation (thus
prior to storage), after thawing and after a resting period (if
applicable). This is required to be able to determine the quality of
the sample.
37Marincola, Francescono commentno commentno commentno comment
38Lehmann,
Paul V,
Depends on the study. Countless critical variables exists. Some are known and
some are unknown (original comment has a long list of known variables)
. How can all of these possibly critical variants be recorded, and who
will go through the effort to record it? How can one even define a
serum or other biological, or document “gently resuspend”?  Even if many follow the MIATA initiative
(which is most desirable), and start feeding the database with such
information (it would need to be quite detailed to be meaningful), how
many entries will it take  before
statistical analysis will allow us to identify the factors that are
critical?  Even after that we will only see correlations that need
to be (and can easily be) confirmed experimentally.  We have  successfully shown that through introduction
of a reference sample and a reference protocol variability could be
dcreased to acceptable levels even in the hands of unexperienced
operators. Reference samples could even be stored and shipped along
with clinical samples allowing one to ascertain whether any of the
hard-to-track variables have affected them.  One
would simply test the reference sample upon receipt for performance as
compared to previously established values. Results obtained testing
reference samples alongside clinical samples would provide a
“yardstick” for interpretation. 
39Andjelic,
Sofija
no commentno commentno commentno comment
40Cerundolo, Vincentono commentno commentno commentno comment
41Nielsen, Julieno commentno commentno commentno comment
42Shafer-Weaver, Kimberly I don’t believe this is always necessary.  It really
depends on the questions being answered
Sample handling can greatly vary and in turn can change
the integrity of the immune function of the cells.  Again, a SOP
for how samples should be collected for a particular use would be very
helpful.  
Yes, this is important.  Please see comments
above.  
Criteria for what is the acceptable cell recovery and
viability should be set remembering that most patient samples are more
fragile then healthy donors.  How the cell numbers and viability
were determined should be added.  However, what criteria will be
used for assessing the function of the cells?  Will this be
standard mitogens or antigens (PHA, ConA, CEF) that should elicit a
global immune response in the particular assay utilized?
43Zitvogel, Laurenceno commentno commentno commentno comment
44Rivoltini, LiciaCollecting
all patients information (age, gender, HLA type, diagnosis, disease
status, treatment, prior relevant treatments) is essential to perform
an exhaustive immunological monitoring.
In our experience, whole blood samples must be collected
in EDTA vacutainer tubes in order to avoid monocytes activation. Blood
should be processed within 2-4 hrs, in order to preserve PBMC maximum
vitality and function. If it is not possible, specimens must be kept at
room temperature no longer than overnight. Whole blood shipment must be
performed at room temperature. In our hands, PBMC isolation by using
Accuspin or Ficoll gives the same results, however the former is an
easier and operator-independent approach. Whatever PBMC isolation
procedure is used, it is important to reduce platelets number by a
final slow washing. 
Freezing
process may be performed using a programmable-rate cell freezing
apparatus. Cryovials must be stored in liquid nitrogen avoiding
 storage at -80°C (for longer than 2-3 days) in order to reduce
leukocyte death.
It is
essential that all immune-monitoring laboratories refer to a common
guidelines and  accepted criteria.
45Vandenbark, Arhtur A.no commentno commentno commentno comment
46 Yuan, JiandaAs noted
in the previously posted comments, HLA type should be reported,
especially for broad T cell epitope response monitoring in clinical
trials with entire proteins or multiple peptides, even in clinical
trails with new immunotherapy such as anti-CTLA-4 antibody.  T
cell epitope response will provide additional mechanism leads. 
Prior relevant treatment information such as prior vaccination is
necessary for interpretation of the data.
Several
baseline values were collected for each patient, allowing us to define
a mean and SD for their individual baseline values. The quantity of
time points and thus blood specimens collected varies according to
treatment schedule.  Physiological variations are frequently
observed in monitoring assays.  If possible, we strongly suggest
to collect two baseline blood samples to eliminate the potential
physiological variations.
This part
is probably unnecessary. Cell viability and recovery listed in 1D are
more critical than 1C, especially if the cell viability and recovery
after cell thawing are within the acceptable range.
Agree. 
It is important as indicated above.
47Malyguine, Anatolisee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaver
48Mazorra Herrera, Zaima; Toledo Santamaría, DarienThese specifications could be important depending on the
study. Some of them are not always available. For example, despite HLA
type is important in some assessments as ELISPOT, testing patient’s HLA
may not be feasible in studies with limited financial support. In the
case of blind clinical studies we don’t know treatment until the trial
is finished.
OKVery important issueSerum should be specified and pretested. 
We suggest the counting of cells before and after freezing
49Roep, BartImproving credibility: blinding of data, stimuli,
subjects, and/or clinical outcome have proven to add greatly to the
credibility of the results. Particularly when the data and the
interpretation thereof by the immune assay operators are locked and
sent to third parties prior to exchange of info on clinical background
and outcome, the value and warm reception of the data improve
substantially
50Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok,
Jedd;  Greenberg,  Philip
no comment
51Yee, Cassianno comment
52Mäurer, MarkusAgreed. Although it would be desirable to know the HLA
type, it may not be necessary for all studies. It should be provided if
MHC-presented peptides are analyzed   since  cross-binding may impact on the results. 
Very important, particularly the time lapse between the
blood draw and PBMC isolation ! Of note, this should not lead to
unnecessary censorship and uncritical comments in general, if e.g. the
time lapse  between blood draw and cell
processing is more than 6hrs. This may be vital for some functional
T-cell assays, yet not for other immunological readouts. Thus, the
quality of biological material associated with the ‘time lapse’ as an
entry point for certain immune cell assays has to be linked with the
immune readout. 
Yes, see B.See comment to B above
53Parker, JoanneWe agree particularly regarding information about the
techniques used for obtaining, processing, freezing, transporting and
storing of the samples. 
54Hural, Johnno comment
IDAuthorModule 2A - Cell countingModule 2B - Medium/Serum sourceModule 2C - AssayModule 2D - Assay controls   
1Gouttefangeas, Cecileno commentdo not ask for specific performance charactersitics of serum but simpy if medium/serum was pretestedmention and describe conditions for in vitro stimulation prior testing if appliedUsusally covered in M and M in many cases
2van der Burg, Sjoerd H.cell counting is also an topic for Module 1D and difficult to controlnot relevant to report as long as test is properly controlled and carefully interpretedreport number of replicates done, guideleines that were followed do not have to be explicitly mentionedgood as stated in the current version of MIATA
3, 4Speiser, Daniel (two comments)no commentno commentexplicitly mention and describe conditions for in vitro stimulation prior testing if applied. Results are only semiquantitativeRecommendation to use PBMC reference samples and to indicate their source
5Navarrete, Marcelono commentno commentAPC source and PBMC/APC ration might be of importance. Explicitly mention prior in vitro expansion!no comment
6Straten, Per Thor and Hardrup Sinecell counting is also an topic for Module 1D and difficult to controlData on serum/medium testnig are too elaborate to be reported and should be reflected by well controlled raw datadescription in the MM section as expeced, references to guidelines can be given but should not stand alone.How to do QA of control samples?
7Pawelec, GrahamCounting directly after thawing and resting. Indicate conditions and recovery of viable cells as fraction of imputuse serum fee medium for ex vivo tests and indicate Lot No.Source of all reagents and not just main !Purity of peptides used is of special importance
8Ottensmeier, Christianagree. Importantagree. Important, should be standard in M and M. performance charactersitics of medium/serum will be captured under 2c (?)agree. Important. Number of replicates is a must.important should be reported. Worry that extend of information will go beyond the scope of MIATA
9Odunsi, Kunleno commentno commentInformation of assay procedures is critical and information on following of a published guideleine may be given as an add onHow to do QA the controls?
10Walter, Steffen and Samorski, Reginavalues should be reported only as group assessments (no raw data)pretesting is not only limited to medium/serum but also for other critical variables such as antibodies. Indicate how many batches have been used for the analysis. What type of pretesting or batch bridginng was applied. If not considered this may be a source for variationinter-assay variation can be important. It should ne noted how samples were distributed to different assays. (e.g. all samples from one patient assayed together.)Last sentence not clear?
11Ferrari, GuidoModules 2A-C sufficient if GCLP is not an requirementModules 2A-C sufficient if GCLP is not an requirementModules 2A-C sufficient if GCLP is not an requirementIntimately connected with GCLP guideleines
12Rooke, Ronald and Bain, ChristineOKOKintegrate Module 5B (SOPs use) hereNon responser as a control should be included
13Appay, Victorno commentno commentno commentno comment
14Lopatin, Uriinclude pre-storage counts and thawed counts to provide an indea of handling. Averaged data for large studies!no commentno commentALSO report data on assay variation. It is important to know if, when looking at a given patients PBMCs run twice, the las allow for 10% or 2% variation.
15Ottenhoff, Tom HM and Joosten, SimoneOKserum source is of importance. Serum free media or same serum batch should be used.A template could be suggested by MIATA as a help!Internal controls are mandatory, external reference controls would be extremely helpful but are not always available.
16Schuler Gerold; KŠmpgen Eckard; Gross, Stefaniecombine with 1DSource of medium/serum should bestated. High backrgound values are not automatically a problem as long as they are incorporated in the response definition and as long all samples are tested with the same medium/serum batch.agree and number of replicates should be tested.agree.
17Plant, Anne L.Should provide details on the methotology for determination viability, apoptosis etc.no commentdetailled data on the characterisation of reagents should be provided if available.no comment
18Kern, Florian and Powell, FionaThese should be standard methods. Is this hugely relevant to data?What of the pretesting rwsults should be reported?Here assay procedure should be explained. For reagents there should be an extra pointFor all control reagents descriptions relevant to the reagent (such as purity of peptides). Indicate the exact physical nature of the control.
19Betts, Michaelno commentno commentno commentno comment
20Weber, Jeffreyno commentno commentThe source of reagents and the concentrations/times used would be most important.no comment
21Gulley, James Lno commentno commentno commentno comment
22D'Souza, Patricia and Lane, Jimno commentno commentno commentno comment
23Chen, Weisan; Cebon, Jonathan, Davis, IanWe do know that counting cells and subsequent nurturing of these cells (time to split and to change medium, what density should they be maintained) are the most important, influential and not easily controlled factors. It may relate largely to the cell density, but we cannot exclude the influence from apoptotic cells or cell debrisTesting multiple batches of sera using PBMC with known CD8+ and CD4+ responses and selecting the best.We agree with Dr Daniel Speiser that the assay types should be clearly defined. Labs should select ex vivo assays over in vitro expansion-based assays if T cell frequency is high enough.Minimal controls should be:1. Culture control (a positive PBMC is cultured for the known specificity every time 2. PBMC quality control (a pooled viral peptides covering a wide HLA types to show that indeed the T cell in that PBMC are able to respond to antigens and expand) 3. importantly, an assay control (for example, in case the ICS fails for unforseen reasons, a frozen vial with known responding T cell population can be used to control the assay procedure)
24Cox, Josephineno commentno commentno commentno comment
25Janet Siebertno commentno commentno commentno comment
26Cristina Muselli and groupWe agree. See 1.DThe background reactivity of different lots of medium/sera will vary. It is important to use the same lot of serum for the development of the assay and the execution of the immuno-monitoring screening. The assay should contain all relevant controls to rule out false reactivity due to reactivity to a specific lot of serum/medium.All guidelines followed for the assay should be consistent for the development as well as for the execution of the immuno-monitoring screening. Also we think it would be necessary to include qualification of reagents and material in addition to the source.We agree.Ê A mention should also be made relative to what source of PBMC are used for qualification of reagents (healthy donors, cancer population with the same or different indication, etc.)
27Schendel, Doloresno commentno commentno commentno comment
28Knuth, Alexander; van den Broek, Mariesno commentno commentno commentno comment
29Kreiter, Sebastian; Attig, Sebastian; Sahin, UgurA) Cell counting can be a major source of variation. Reporting how cells were quantified and which measures were taken to reduce variability is of interest to a reader and might facilitate comparability of results across institutions.ÊThe reporting of source of reagents should be regarded as standard of a M. & M. part of a manuscript. Although the serum choice will influence the test result it is difficult to catch the influence or unique properties from pre-tested sera for a given test series. The impact of unique reagent properties will be reflected in the raw data (e.g. increased BG) and the internal controls. It is therefore of limited use to simply report that serum was pre-tested (what exactly is pretested?).C) Reagents and assay procedures should be reported in enough detail to allow a reader to repeat the experiments under similar conditions.D) Quality and quantity of internal assay controls (medium, irrelevant stimulus, positive control) are of high importance and should be reported in detail
30Maio, Michele; Nicolay, Hugues J.M. Important here is to state if cells are fresh or thawed, and if thawed to include pre-storage counts, and thawed counts, to provide an idea of handling. It should also be mentioned the composition of the thawing medium that could be supplemented with particular ÒadditivesÓ to preserve the integrity of the sample e.g., DNAse.We agree, it is important to mention if serum is from bovine, human or autologous origins because serum origin is well known to be a substantial source of variation in assay outcome.We believe that the assay procedures are really critical, but for sure, even harmonized, it will be really difficult for reagents, materials and treatment procedures. Guidelines should harmonize the number of assays for one sample and the number of replicates of the experiment to validate the results.We agree on the importance of the description of the different positive and negative control that could be harmonized and be listed in section 5B.
31Rammensee, Hans-Georgno commentno comment2C: Indicate time window used for the resting time applied (not just ãovernight restingÒ)no comment
32Zwierzina, Heinz; Hakansson, LeiffSee 1DThe choice of medium for in vitro tests is of paramount importance. Reasonably the reproducibility of the tests will improved if the confounding factors sera are avoided. However, as it is well known from the 1960«ties that serum factors can be of major importance in cancer related immunosuppression, it seems reasonable to consider the use of autologous sera in tests where the objective is to predict the clinical outcome of the patients.This is a critical point. For example, preparation of sera included in the culture media is crucial as the proteolytic activity induced during the clotting process might alter the composition and function of proteins present in sera.agree.
33Gnjatic, Sachanot necessarynot necessaryIn my opinion, the only critical points for reporting assays are:- Describe if and how cells were sensitized with antigen (module 2C)- In ELISPOT, show or state number of spots for irrelevant control targets (module 2D)- State the criteria for interpretation of results (module 4)All other information is either unnecessary (HLA type for example may not be critical to data interpretation), or too much irrelevant information (use of "Mr. Frosty"). This comment assumes the labs performing monitoring have tested and validated their methods, but again, filling a checklist will not guarantee better quality of data if the lab is inexperiencedIn my opinion, the only critical points for reporting assays are:- Describe if and how cells were sensitized with antigen (module 2C)- In ELISPOT, show or state number of spots for irrelevant control targets (module 2D)- State the criteria for interpretation of results (module 4)All other information is either unnecessary (HLA type for example may not be critical to data interpretation), or too much irrelevant information (use of "Mr. Frosty"). This comment assumes the labs performing monitoring have tested and validated their methods, but again, filling a checklist will not guarantee better quality of data if the lab is inexperienced
34Reap, Elizabeth ; Norberg, Pamela; Watsin, AubreyMust say how counted cells i.e. Guava versus hemacytometer.Ê How many cryovials are being thawed at a time should be noted.Ê A standard thawing procedure is essential for obtaining maximum viability and recovery of cryopreserved PBMC.ÊSerum should be pre-tested to check for background levels.Ê We agree with other comments that it is not enough to just say serum was pre-tested, as that could mean anything, but will there really be room in a publication to go into great detail about your pre-testing experiment?Agree that it should be standard to list source of main reagents and materials and crucial treatment procedures.Ê Should also include number of replicate samples, number of events collected.Ê Resting versus no resting makes a big difference in the assay, because you have no apoptotic cells interfering with your sample.Ê ELISPOT Kit or standard test method?ÊÊSTRONGLY agree. Each plate of every assay Êshould have qualified controls that should meet acceptance criteria that guarantee a valid assay in your lab.Ê You must know your assay. Each sample at each visit tested should be run with a mitogen control.ÊÊ Results for irrelevant or negative controls are importantÊ Also should be noted how samples were tested .Ê We agree with the comment by W Chen, J Cebon, and I Davis that if a standard or calibrator can be run with each assay then you should report the co-efficient of variation for day-to-day variability of the assay.Ê There may be cases however when a leukapheresis sample is not available to use as a calibrator, and that is acceptable as long as you list this.
35Nishikawa, Hiroyoshino commentSelecting human AB serum for T cell cultures is a critical factor. However, this requires careful selection of new bathes of serum. Appropriate standard sample(s) is necessary.Control samples are, of course, critical. As mentioned above, one of the most important challenges is to establish the balanced minimum, but required controls. In my opinion, the followings are absolutely necessary. 1. Assay control (Positive/Negative standard samples for every assay) 2. Sample control (pooled-positive antigens, such as viral antigens to confirm T cells in the PBMC are indeed able to respond to antigens).
36Schoenmaekers-Welters, Marij J.P.Thawing procedure is important in terms of percentages of viable cells (SOP ?) For example, monocytes tend to be more fragile than other cells but monocytes are very important in presenting the antigen to the T-cells in the ELISPOT. Should the methodology of cell counting not already be included in Module 1? For quality assurance of the samples this is important to use the same (qualified/validated) counting method. Are the PBMC rested for how many hours and under what conditions? If yes, additional cell counts after this resting period could be very useful.Acceptance criteria why this medium/serum combination is chosen.This in normally done in material and methods but it could be an advise to give more information instead of given it briefly. This could then be given as supplemental/web only data so that everybody is able to read it but it not limiting the text of the main paper.What happens when the positive or negative control is out of range? When do you accept it as the correct control?
37Marincola, Francescono commentno commentno commentno comment
38Lehmann, Paul V,Live/dead counting assessed by staining the cells and counting them with an appropriate machine can be more accurate than just counting trypan blue exclusion by eye. Same for ÒapoptoticÓ counting.Ê However, because the apoptotic process goes through many well-defined phases, there is no clearly defined boundary between live/apoptotic and dead.Ê There are many live/dead/apoptotic stains on the market, and since each measures different cellular conditions/pathways they inherently provide different results.Ê Here too, we need to define a standard.Serum is a major variable in T cell assay performance (Zhang et al, J. Immunotoxicology, 2009, 6:227-34).Ê Even a brief exposure of PBMC to such serum during freezing or washing can ruin a T cell assay.Ê If we want to document the unique biological properties of a serum used, how could we define ÒserumÓ for MIATA purposes such that it helps transparency?Ê Testing a serum against a reference standard would be one way to do so, but for this we would first need to accept a reference standard.As to harmonization guidelines, I have good reasons to state that it is premature (or even wrong) to promote the published harmonization guidelines as a quality standard that will help generally improve T cell immune monitoring (for the specifics see below). Actually, I think the greatest danger of MIATA is that it can promote a trend in which such guidelines (based on low power statistical analysis of data) are imposed on the field before they have been experimentally verified. ÊI think such premature generalizations can Òbe an obstacle to innovation and improvement of immune-monitoring technologies.ÓÊYes, but markers of Òcellular wellnessÓ might add a lot to interpreting the positive control itself (see above).
39Andjelic, Sofijano commentno commentno commentno comment
40Cerundolo, Vincentono commentno commentno commentno comment
41Nielsen, Julieno commentIt would be useful to know how many cells are used for experiments. For example, we all report cell number/well in an ELISPOT, percentage positive by flow cytometry, or cells/ml for in vitro stimulations, but I rarely see details regarding the total number of cells analyzed (flow cytometry) or Ð really important Ð the number of input cells in an in vitro stimulation. If you use 10^7 PBMC in an in vitro stimulation with MART-1, youÕll probably be able to expand MART-1-specific T cells, but if you use 10^4 cells, you probably wonÕt. Although this would be obvious to experts in the field, there certainly may be data published using insufficient cell numbers to make the conclusions that are published. This is important when statements are made regarding whether or not patients have a detectable response pre- or post-stim. Along the same lines, I frequently see data (particularly in vaccine trials) reported as X-fold increase over pre-vaccine levels. It would be useful to see absolute numbers. Was there a detectable response pre-vaccine, or is the frequency so low as to be completely undetectable.
42Shafer-Weaver, KimberlyThe particular thawing procedure impacts both the viability and recovery of cryopreserved PBMC.Ê Again, a SOP is needed.Ê General ranges should be provided for the samples (i.e. all samples were > 90% viable by trypan blue with >70% recovery from freeze by cell counts performed on a Z1 coulter counter).Ê Values for all samples are excessive.The source and type of sera should be provided.Ê However, pretesting and performance characteristic is better suited for core or service labs and is unreasonable for basic research or clinical lab.Ê Nor do I think it is necessary to put all the specifics within the published article.Ê One alternative would be for the vendors to prescreen and certify their sera for particular uses (i.e. ELISPOT, Proliferation, cytokine induction, etc.).ÊYes, a brief description of the source and amount of reagents utilized should be included as well as timings and incubation temperatures. The number of replicates, events collected, and gating should also be specified.Ê ÊAdditionally, if samples were rested prior to counting and use in assay should be included.I strongly agree with this statement. For each assay set-up, we have controls for testing the assay reagents, samples (mitogen or control antigen), response (irrelevant or negative controls) and the particular APC.Ê We have several banks of healthy donors and utilize two ( one high and one low responder) in each assay set up to ensure we are within the sensitivity range of the assay.Ê These normal responses are tracked overtime to ensure our assay quality assurance.Ê All patient samples are run within the same assay or if this is not possible, we run assays with overlapping samples.Ê We also perform side-by-side testing of assay reagents prior to switching to a new lot (again more for a core or service lab). If available, the amount variance in the assay is nice to know.
43Zitvogel, Laurenceno commentno commentno commentno comment
44Rivoltini, LiciaWe agree.We use medium +10% FCS to have a low background; anyway it is important to use the same lot of serum during the whole monitoring study.We agree. In particular, we have the same results (in terms of viability, sensitivity of the assays and PBMC functionality) using either fresh (unfrozen) or thawed and overnight-rested lymphocytes. Obviously, recovery is different.We agree. However, we believe that the inclusion in each assay of a standardization control (for instance a fixed number of PBMC from the same defined donor displaying a know reactivity to viral epitopes, or an Ag-specific T cell line or clone) could help monitoring inter-assay variability.
45Vandenbark, Arhtur A.no commentno commentno commentno comment
46Yuan, Jiandano commentno commentCell culture in vitro stimulation conditions are critical, such as APC preparation, cytokine, peptide and the number of stimulations etc.Ê These all have impacts on the expansion of antigen-specific T cells and produce the variation. ÊWe recommend triplicate cell cultures and corresponding cell assays.We agree with most of the posted comments about controls. Control could be used for reagents and/or samples as either positive or negative controls. We recommend the use of small aliquots of T cell clones or lines as sample controls for the assay.
47Malyguine, Anatolisee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaver
48Mazorra Herrera, Zaima; Toledo Santamar’a, DarienThe viability assays should be standardized (the use of PI is recommended)The use of serum-free medium is recommendedOKInternal control very important. Immune dysfunction of cells because of patientÕs disease and/or freezing- thawing procedures should be discarded. It is important a standardized procedure to eliminate this possibility.
49Roep, BartConsider blinding stimuli and/or subjects; Provide stats on assay: coefficient of intra- and inter-assay variation. Description of quality assurance of controls including used peptides and peptide mixes: extremely important; great point. Also report on solvents and medium/control conditions (e.g., did the medium reference get the same diluent / solvent, e.g. DMSO, FMOC)?) (Or, if you test peptides, was the medium spiked with the peptide solvents (incl. DMSO)?) QC of recombinant antigens: we have developed stringent QC on recombinant antigens, as we have demonstrated that many currently used preparations can interfere in responses of T-cell clones reactive to third party antigens (Refs. Roep Ð J Autoimmunity, Peakman Ð Diabetes), even though they may act as stimulus/antigen for their specific T-cells. T-cells to eukaryotically expressed antigens often do not cross-react with antigens produced in E. coli vectors, and v.v., etc.
50Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philipno comment
51Yee, Cassianno comment
52MŠurer, MarkusThese points make sense. I would opt, particularly for clinical trials, to indicate in greater detail what ÔON restingÕ means, this could be all between 6 and 16 hrs. If flow cytometry analysis is performed, then an internal reference may be considered: the same batch of Ab coupled to the same batch of fluorochrome. If not only percentages are reported, but also MFI, then appropriate control beads should be considered. This will allow to control signals between experiments and even between different laboratories. The source and use of serum is particularly important in these assays. Also the note if any cytokines etc had been added. We need to exercise caution that future reviewers of grants or manuscripts will not expect the entire checklist listed above if the data are not from a clinical study addressing immunological readouts. This may have to be addressed and spelled out in a more precise way.
53Parker, JoanneWe agree. However, we do not feel a statement regarding available recommendations is required. We feel reagents are very important and these are often not detailed with flow cytometry.
54Hural, JohnModule 2.C. While the term Òovernight restÓ is commonly used, it is my opinion that any formal document should state this as an Òovernight incubationÓ. The cells are certainly not resting at this time ;-)M2.D. I would suggest specifically mentioning trend analysis of the PBMC reference sample in this section. Is trend analysis conducted? Is it reviewed by management? How long is one sample used? What are the criteria for requiring investigation of trend variations?
IDAuthorModule 3A device - hardware and softwareModule 3B settings and controlsModule 3C quality assurance criteria / auditing
1Gouttefangeas, CecileDo not ask for instrument validation and calibrationNo use to indicate settings of ELISPOT readerAuditing is not automatically a guarantee
2van der Burg, Sjoerd H.no commentWhat do machine settings tell us? What does desciption of gating strategy help? No real value to report.Auditing is not automatically a guarantee
3, 4Speiser, Daniel (two comments)no commentno commentno comment
5Navarrete, MarceloType of reader and software will suffice for ELIPOTno commentno comment
6Straten, Per Thor and Hardrup SineSimply indicate hard nd software.Indication of settings does not help muchAuditing is not automatically a guarantee
7Pawelec, Grahamno commentno commentno comment
8Ottensmeier, ChristianToo much. We should not tie ourselves into knotsNot meaningful as it is not clear that same settings between instruments are the same. It will be very difficult to make gating strategies objective.Simple statement of audit is of limited use. How much and how was audited? Might go too far.
9Odunsi, Kunleno commentSince training and setting are not standardized it cuold be difficult to interpret this paramterIt will be important to be more specific regarding what is meant by quailty assurance.
10Walter, Steffen and Samorski, ReginaPersonell is already included in 5D description of flow cytometer settings should include description of optics. Instrument calibration should include description on how instrument settings were adjusted for each experiment. For flow data it shold be indicated how compensation was performed.rationale for adjusting software cont settings in ELISPOT or gates (flow) druing raw data analysis. Were global count settings or gates used in the study and were they defined before or after data were acquired.Not clear?
11Ferrari, GuidoModules 3A-B sufficient is GCLP is not an requirementModules 3A-B sufficient is GCLP is not an requirementIntimately connected with GCLP guideleines
12Rooke, Ronald and Bain, ChristineOKOKAuditing should include Raw data as well as final results
13Appay, Victorno commentno commentno comment
14Lopatin, Urino commentno commentno comment
15Ottenhoff, Tom HM and Joosten, SimoneCould be part of template provided by MIATACould be part of template provided by MIATAno comment
16Schuler Gerold; KŠmpgen Eckard; Gross, StefanieAgree with information on software and equipment but how does one validate an operator? You cannot therefore the oinformation is irrelevant (technichian vs PHD student x years)Settings depend on macines so info is not helpful.Information that audit happended does not help much. Giving all information on how it was audited is too much information.
17Plant, Anne L.no commentno commentno comment
18Kern, Florian and Powell, FionaOverkill. Good internal controls could ascertain correct instrument setup.Control beads would be used in the set-up rather than in the assayToo (?) ambitious. This depends on definition of audits
19Betts, Michaelno commentDescription of full gating strategy should always be given. If possible dot plots that are truly (!) representative for the whole analysis should be shown as well as raw data from a positive and a negative control.no comment
20Weber, Jeffreyno commentno commentno comment
21Gulley, James Lno commentno commentno comment
22D'Souza, Patricia and Lane, Jimno commentno commentno comment
23Chen, Weisan; Cebon, Jonathan, Davis, Ianno commentShowing some raw data, for example FACS plots or ELISpot photos, should be sufficient indicators for above mentioned parameters/detailsAgree with Daniel Ð this needs to be more clearly defined
24Cox, Josephineno commentno commentno comment
25Janet Siebertno commentno commentno comment
26Cristina Muselli and groupRegardless of the type of equipment/software version, setup of the equipment and other variables, we think it is paramount to use all the same during the developmental phase and the execution of the screening phase.Ê The same operator should handle all the timepoints for any patient sample to maintain consistency.Perhaps, add rational for why the chosen controls were used? Doing so might significantly assist others in determining what the best set up for future assays would be.no comment
27Schendel, Doloresno commentno commentno comment
28Knuth, Alexander; van den Broek, Mariesno commentno commentno comment
29Kreiter, Sebastian; Attig, Sebastian; Sahin, UgurA) Information on type and version of machine and software used is critical. Information on Instrument calibration, qualification or validation or training of the operator should be a topic for module 5 and are difficult to control.The value of reporting details of machine settings is minimal as same settings on different machines may lead to different results. Settings should not be mandatory for MIATA. On the other hand it might be crucial to document instrument settings for internal use as within a study an investigator might want to always work with same settings. Unexpected results could be due to in house change of settings that can only be tracked if settings are documented.ÊC) Simply stating that data was audited could be anything and is of no help. This module could rater belong to Module 5 which capture aspects of lab environment and QA.
30Maio, Michele; Nicolay, Hugues J.M. We agree with other groups since training is not yet standardized/harmonized, it could be difficult to interpret this parameter. It could be useful but to our knowledge not mandatory.We believe that data acquisition with ELISPOT automatic reader is a really good tool, but really varying upon assay procedure. The controls applied during the data acquisition rely greatly on the operator, and so could be a potential source of variation and bias.We agree it is a critical point that must be harmonized with rules taking in consideration the variation between negative and positive controls. This should be integrated in the results interpretation.
31Rammensee, Hans-Georgno commentno comment3C: Quality assurance criteria: this is important but one should try to limit the efforts for their description.
32Zwierzina, Heinz; Hakansson, Leiffno commentno commentno comment
33Gnjatic, Sachanot necessarynot necessarynot necessary
34Reap, Elizabeth ; Norberg, Pamela; Watsin, AubreyAgree you should report equipment and software used and a brief description of how the equipment is setup or calibrated. It will be hard to say anything more than operator was trained on the equipment.Ê We donÕt think adding the statement about an operator being trained adds anything useful to the publication.Ê It would be expected that any operator should be trained on the equipment.Ê What is considered training will vary widely from lab to lab. There will not be enough room in a publicaton to go into specifics about each individualÕs trainingAgreeÊ Flow cytometry gating strategy needs to be included. ÊThis can make a tremendous difference between labs and even between operators in same lab.This would have to be defined very clearly.Ê This will vary from lab to lab.Ê Auditing will be very different between labs.Ê What is being audited, the final data, performing the assay?Ê Will this exclude otherwise good data because a lab does not have a requirement to audit data.ÊÊ Were validated spreadsheets used to eliminate the possibility of errors?
35Nishikawa, HiroyoshiI agree with comments by others. Operator training is a critical issue, but it is challenging to find ways to evaluate operator training.no commentno comment
36Schoenmaekers-Welters, Marij J.P.no commentno commentAudited by who? This is hard to check. Do you mean audited by another person of the lab to prevent mistakes?
37Marincola, Francescono commentno commentno comment
38Lehmann, Paul VA full validation of an instrument is desirable, but this is a scope that is hardly feasible for the majority of laboratories that publish ELISPOT data. Manufacturers of qualified ELISPOT readers have made sure that their instruments function consistently, and offer tools that permit the user to check whether the instrument is calibrated. Checking calibration should be part of any ELISPOT SOP (just as it is for flow cytometry). ÊIf spot counters have warm up /overheating /ambient light problems, such checks should be performed also in between the analysis.Beyond such general calibration, it will be nearly impossible to document the many variables that affect the functions of the different ELISPOT readers.A qualified ELISPOT reader manufacturer will have accounted for these variables and tailored them to the specific application Ð spot counters have not done so.As stated above, it is impossible to document in writing all the parameters and variables that go into ELISPOT analysis, and even if it were possible, a list of such parameters would not provide reproducibility among different instruments. However, I strongly recommend submitting such raw/counted/QC image material with the corresponding electronic counting parameters as Òsupplemental materialÓELISPOT data need auditing. Occasionally, for example, membrane defects and leakage needs to be corrected. However, any change made to the automated counts will be automatically recorded and annotated creating complete transparency about what has been changed, and why, without over-writing the objective automated counts.Ê The QC data set is a part of the record which can be viewed, submitted, and reanalyzed.Ê Instead of attempting to describe such for MIATA in generic terms (that is impossible and cannot create transparency), I suggest that investigators submit the actual QC material.Ê
39Andjelic, Sofijano commentno commentno comment
40Cerundolo, Vincentono commentno commentno comment
41Nielsen, Julieno commentno commentno comment
42Shafer-Weaver, KimberlyYes, a brief description of the equipment and software for data acquisition should be provided.Ê But operator training and validation of that training is not necessary.Ê However, this will be needed if the assay is under GLP or CLIA.Again, this is better suited for supplementary data.Ê I am not sure if the exact settings are necessary.Ê Perhaps general guidelines such as Òspot size threshold was set at > 10 times the size of a WBCÓ at least for ELISPOTs would be helpful and would suffice.Ê For Flow cytomtery, I agree that gating and voltages are important.ÊA brief description of how the data was quality controlled is important, but we need to be careful here.Ê How detailed do we need to be?Ê Again, SOPs will help.
43Zitvogel, Laurenceno commentno commentno comment
44Rivoltini, LiciaWe agreeWe agreeThe quality of audit differ from lab to lab and its depends from specific laboratories guide lines.
45Vandenbark, Arhtur A.no commentno commentno comment
46Yuan, Jiandano commentMost data acquisition conditions should be be accepted as standard. ÊWe do agree with Weisan, that showing some raw data, such as FACS plots, should be sufficient.no comment
47Malyguine, Anatolisee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaver
48Mazorra Herrera, Zaima; Toledo Santamar’a, DarienA) How to define "methodologies for operating training"B) How to define "quality assurance criteria"no comment
49Roep, Bartno comment
50Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philipno comment
51Yee, Cassianno comment
52MŠurer, Markusno comment
53Parker, JoanneOnce again we agree. We feel it is particularly necessary for flow cytometry that the gating strategies be listed and as Dr Betts has already stated that the raw data and plots be shown. It is often very difficult to figure out how some groups generate their data.Ê
54Hural, Johnno comment
IDAuthorMIATA Module 4 - Results
1Gouttefangeas, CecileProcessing of data has to be explained and is of high importance (e.g. background in ELISPOT and multimers, irrelevant controls)
2van der Burg, Sjoerd H.Ask for time point when response definition criteria were defined (before or after generating the data set)! Describe stat test or empirical rules applied.
3, 4Speiser, Daniel (two comments)Indication of background values and wheter the threshold to define positivity was set at a constant level throughout the experimental series or differently between indiviual data sets.
5Navarrete, Marcelono comment
6Straten, Per Thor and Hardrup SineYes. Indication of background of importance and irrelevant controls should be described.
7Pawelec, Grahamno comment
8Ottensmeier, Christianyes. Critical
9Odunsi, Kunleno comment
10Walter, Steffen and Samorski, ReginaImportant to know if criteria to define positivitiy were defined before the study or post-hoc. Specific criteria should be listed.
11Ferrari, Guidono comment
12Rooke, Ronald and Bain, ChristineSpecificity of a response is interpreted as a function of the variation between negative and positive results. This should be integrated in the results interpratation
13Appay, Victorno comment
14Lopatin, Urino comment
15Ottenhoff, Tom HM and Joosten, SimoneShould also include description of how data were collected in diseased individuals in relation to contorls (important for infectious disease field).
16Schuler Gerold; KŠmpgen Eckard; Gross, StefanieAgree. Detailed description of the method is of highest importqnce and could be supported by a short listing of guideleines or recommendations that were respected.
17Plant, Anne L.no comment
18Kern, Florian and Powell, FionaNext to statistics event countsare of major importance therefore data of events acquired for each data file would be mandatory. Results shown as percentages could be misleading.
19Betts, Michaelno comment
20Weber, Jeffreyno comment
21Gulley, James Lno comment
22D'Souza, Patricia and Lane, Jimno comment
23Chen, Weisan; Cebon, Jonathan, Davis, IanWe do agree with others that some kind of raw data, such as FACS gating and ICS pattern should be provided.
24Cox, Josephineno comment
25Janet SiebertInformation about the software used (name, publisher, and version) should be included.Ê Different software versions, can have different implementations of statistical tests or analytical algorithms, which can impact reproducibility.Ê Descriptions of the statistical tests should include all relevant parameters (e.g. assumptions about variance).Ê Essentially, two different parties should be able to analyze the same data using the same test and get the same result.Ê Documentation should be adequate to support this goal.
26Cristina Muselli and groupWe agree
27Schendel, Doloresno comment
28Knuth, Alexander; van den Broek, Mariesno comment
29Kreiter, Sebastian; Attig, Sebastian; Sahin, UgurResponse determination criteria should generally be defined prior to analyzing the data and reported in any report or publication.
30Maio, Michele; Nicolay, Hugues J.M. no comment
31Rammensee, Hans-Georgno comment
32Zwierzina, Heinz; Hakansson, Leiffno comment
33Gnjatic, SachaIn my opinion, the only critical points for reporting assays are:- Describe if and how cells were sensitized with antigen (module 2C)- In ELISPOT, show or state number of spots for irrelevant control targets (module 2D)- State the criteria for interpretation of results (module 4)All other information is either unnecessary (HLA type for example may not be critical to data interpretation), or too much irrelevant information (use of "Mr. Frosty"). This comment assumes the labs performing monitoring have tested and validated their methods, but again, filling a checklist will not guarantee better quality of data if the lab is inexperienced.
34Reap, Elizabeth ; Norberg, Pamela; Watsin, AubreyAgree
35Nishikawa, HiroyoshiThis is very difficult issues and required much thoughts. Too much obscures the focus of the work, and too little limits ability for readers to interpret.
36Schoenmaekers-Welters, Marij J.P.The definition of a positive reactivity as well as of a vaccine-induced response could be pre-defined or deduced from the results. What option is preferred? Maybe it is wise to advise either one option depending on the type of study, research or clinical trials.
37Marincola, Francescono comment
38Lehmann, Paul V,Yes. For the statistics and cut-offs and thresholds after correct numerical information was obtained and documented. For the latter see above.Ê
39Andjelic, Sofijano comment
40Cerundolo, Vincentono comment
41Nielsen, Julieno comment
42Shafer-Weaver, KimberlyHow positive reactivity and differences between time points is determined was already addressed in an early module.Ê Yes, the statistical tests should be provided, but the reasoning behind the choice does not need to be given because if the data does not fit the assumptions of the test, then that test should not be used to analyze the significance of the data.Ê If data is excluded, it should be stated which data and why.
43Zitvogel, Laurenceno comment
44Rivoltini, LiciaWe agree
45Vandenbark, Arhtur A.no comment
46Yuan, JiandaThe definition of Òpositive reactivityÓ is very difficult and depends highly on Êthe selection of the marker and the magnitude of the response for a biological process.Ê In general, Òpositive reactivityÓ has not been associated with clinical outcome.
47Malyguine, AnatoliStatistical treatment should be applied, but even if the difference is statistically significant it may be not biologically relevant. For example, if in ELISPOT assay we have one spot/100K cells in control and two in experimental well it may be statistically significant but probably not biologically. Also I believe that it is not acceptable to show % of fold of increase/decrease without providing raw data. I am not sure if it possible to establish standard Òthreshold for Òpositive reactivity". For example, one of the often used approaches is to consider vaccination response to be positive (for ELISPOT) if we have double background difference. Again, what if we have 1 vs. 2 spots?Ê What about 100 vs. 200?
48Mazorra Herrera, Zaima; Toledo Santamar’a, DarienThe reporting of data should be standardized. Monitoring strategies need to be designed for vaccine in which the Ag is not defined
49Roep, Bart(see also comment above on interpreting the data without knowledge of clinical outcome; i.e., prediction). Offer accessibility to raw data (to track ratio's, delta-values, etc.) and control/reference values. Very often, data are presented after subtraction of background, without providing background values. Reporting mean values of groups of subjects is not very helpful, since this does not allow adjusting the reported values by recalculating number of spots in an ELISPOT, proliferation rates (delta background; stimulation index), and so forth. In terms of statistics, almost as a rule proliferation data are analyzed as linear scales, while we all know that this should actually be in a more logarithmic fashion: one division doubles the cpm, so a decline in response from 20,000 to 5,000cpm, may be two cell divisions less, not Ô75% inhibitionÕ (if all cells were involved). The ideal transformation may be a hyperbolic arc sin transformation, but I appreciate that we have to take things one at the time...
50Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philipno comment
51Yee, Cassianno comment
52MŠurer, MarkusSee comments at the end of the text. In particular here: In the case of flow cytometry: if percentages of the target (test) population are only reported, then the cutoff for the acceptance should be note as the number of absolute cells. Since apoptotic cells are ÔstickyÕ a live/dead cell marker should be included in the actual test along with the Abs.
53Parker, JoanneWe feel it is very important that there be details on how a positive response was determined. Comments of the significance of the results and correlation to treatment etc are usually provided. Correlation to clinical outcome would be nice but there would need to be details about how that correlation was determined.
54Hural, Johnno comment
IDAuthorModule 5AModule 5BModule 5CModule 5D
1Gouttefangeas, Cecileno commentshorten this as many people do not recognize importance of SOPsshorten this as many people do not use qualified and validated assaysshorten this as many people do not recognize importance of proficiency panels
2van der Burg, Sjoerd H.Fully GLP or no GLP remove GLP-likeStatement that an assay was run acocring to SOP does not mean much as SOPs can be long/short or good/bad (How to control?)Validation is strechable term. Labs could also reproducibly report different results from a sample. How to deal with the differecnes accross institutionsparticipation in a proficiency panel does not automatically mean that performance is good. A certificate would be needed. How to control if training of staff was good or bad?
3, 4Speiser, Daniel (two comments)no commentno commentno commentno comment
5Navarrete, Marcelono commentno commentno commentno comment
6Straten, Per Thor and Hardrup SineFully GLP or no GLP remove GLP-likeToo early, not helpful, not advantageous, strechable terms, difficult to controlToo early, not helpful, not advantageous, strechable terms, difficult to controlToo early, not helpful, not advantageous, strechable terms, difficult to control
7Pawelec, Grahamdelete GLP-likeno commentno commentno comment
8Ottensmeier, Christianiportant to raise awareness of these issues. We are uneasy about setting the goalpost too high for module 5. premature ste as long as we do not know which test is an immunological surrogate for clinical effect. It is not needed to deliver data that is of diagnostic quality for treatment decisions. delete GLP-likeis important but to judge data of deviations of SOP the whole SOP has to be published. SOP is likely to mean a wide range of things between different SOPsimportant but likely out of reach for most academis labs. Doubt that journals would allow to give space to publish such kind of informationTest used in proficiency panel must be the same as used in the reported experiment. Repreotinng of participation is only useful if lab has done well otherwise the statement might simply become an excuse. How is training done? Not the purpose of MIATA and gives inappropiate reassurance on the performance of the laboratory
9Odunsi, Kunleno commentno commentValidated SOPs would be desirable but wheter or not to report the details of how SOPs are qualified or validated would depend on the context and type of studyHow do we define trained personnel?
10Walter, Steffen and Samorski, Reginait should be indicated whether SOPs were available for all aspects of the immune monitoring (sample acquisition, assay, data acquisition and raw data analysis an dinterpretationno commentno commentno comment
11Ferrari, GuidoIntimately connected with GCLP guideleinesIntimately connected with GCLP guideleinesIntimately connected with GCLP guideleinesIntimately connected with GCLP guideleines
12Rooke, Ronald and Bain, ChristineDelete GLP-like, Cite any guideleine that had to be folowed as a mandatory requirement (GCLP)Mention source of SOP (external/internal) and deviation/modificartionOKOK
13Appay, Victorno commentno commentno commentno comment
14Lopatin, Urino commentno commentno commentno comment
15Ottenhoff, Tom HM and Joosten, SimoneTemplate should help.Template should help.Only realistic for few labs and testsOnly realistic for few labs and tests
16Schuler Gerold; KŠmpgen Eckard; Gross, Stefaniedelete GLP-likeagree as long as there are certain criteria on what you call a SOPassay and SOPs should be validated but this strongly depends on the test system.participation in proficiency panels does not state anything.
17Plant, Anne L.no commentno commentno commentno comment
18Kern, Florian and Powell, Fionamight be disqualifying for non-GLP labs. Not sure what it means. Might make sense to mention if data was generated by a certified lab (ISO9000).unless the SOP is published he desciption of deviations is of no use to the reader. All that matters is the correct desciption of the technology.not helpful for MIATA, not interested in auditing other labs as a revier. Important to know what exactly the procedures where and how data was interpreted in case raw data is not shown.fine for accreditation but not for MIATA. MIATA should not explore the general credibility of a lab.
19Betts, Michaelno commentno commentvalidation of assays might not be of interest for research labs that would rather need flexibility to incoporate new markers evrey dayno comment
20Weber, Jeffreyno commentno commentIn the final category #5, how one ensures confidentiality is kept might be something worth reporting.no comment
21Gulley, James Lno commentno commentno commentno comment
22D'Souza, Patricia and Lane, Jimno commentno commentThere should be a distinction between Research and development labs, whose main purpose is discovery and GCLP central labs, whose main purpose is assessment of clinical specimens and product advancement decisions.Ê the latter will have to work under GCLP. We do not think R & D labs need a mandate to use validated assays, an SOP and follow total quality management principals.no comment
23Chen, Weisan; Cebon, Jonathan, Davis, Ianno commentno commentno commentno comment
24Cox, Josephineno commentno commentno commentno comment
25Janet Siebertno commentno commentno commentno comment
26Cristina Muselli and groupWe agreeno commentno commentWe feel this module can be followed and integrated in each of the previous sections.
27Schendel, Doloresno commentno commentno commentno comment
28Knuth, Alexander; van den Broek, Mariesno commentno commentno commentno comment
29Kreiter, Sebastian; Attig, Sebastian; Sahin, UgurA) GLP-like does not exist. Work is done either as GxP or non-GxP.B) Without seeing an SOPs it is not possible judge the value of the document. Publishing complete documents should not be requested by MIATA.C)Ê Effects and degree of assay validation in a lab are difficult to capture.Ê Wherever validation efforts were undertaken that led to increased reproducibility of results and trust in data the investigator should be able to report such efforts.Ê Difficult module as results from a lab that did not fully validate an assay might well be of high quality and trustable.D) Participation in proficiency panel does not automatically mean that a lab has a good performance. Participation in a proficiency panel conducted by an independent lab that certifies a good performance (pre-defined pass or fail criteria) might be interesting to report. Here it would be important to know what was certified (high sensitivity, low variation, good reproducibility etc.).
30Maio, Michele; Nicolay, Hugues J.M. no commentno commentno commentno comment
31Rammensee, Hans-Georgno commentno comment5C: statement on SOPs: yes; but again, I am afraid that if too much paperwork is asked for the description of all parameters this will disencourage people to participate.no comment
32Zwierzina, Heinz; Hakansson, LeiffWe do not believe that any statement of audit or participation in quality assurance programs will help very much, each lab has to have validated SOPs and well trained personnel.no commentno commentno comment
33Gnjatic, Sachanot necessarynot necessarynot necessarynot necessary
34Reap, Elizabeth ; Norberg, Pamela; Watsin, AubreyA lab is either GLP or not. State that the assay performed following a STM, controls qualified, calibrated instruments and SOPs used when writing this sectionWe agree with adding a statement that says SOPs were established and followed during the assay as long as this is in addition to a brief description of how you perform your assay with the critical information from the earlier modules being included.Ê We see no reason to describe any deviations from the SOP.Ê If there is a major deviation for one set of samples that could be noted, but in most cases that should lead to the assay being repeated and those data not used so we are not sure it adds to being able to interpret the data presented.Ê We donÕtÕ think that journals will have room for all this information but agree it is necessary. Maybe a universal checklist for each assay can be agreed upon by leaders in the field?This is a lot of information to require for each publication but agree that the time has come due to the nature of these assays. ÊIt seems like it would be a stand alone paper to go into that much detail about your qualification or validation assay each time you write a paper that has T cell data.Ê If the paper has flow and ELISPOT, then you wouldÊ have to include your qualification or validation for each assay which would be very lengthy.Ê Just stating that your assay is qualified or validation is not enough information but we donÕt think that not having a qualified or validated assay should prevent data from being published.Ê The reader should be able to make their own decision about the quality of the data despite the use of a qualified or validated assay.Information from the earlier modules is enough to determine the quality of the data.Ê Participation in proficiency panels is a plus but should not be required to be reported.Ê Participation in a panel does not mean that you performed well in that panel.Ê We hope it would be standard that only trained personnel perform any assay.Ê Again the amount of training will vary from lab to lab but I donÕt think you will have room in a publication to comment on the training of each individual person, nor is it really appropriate.Ê If the assay has acceptance criteria and controls they should give you an indication of a technical problem with the assay.
35Nishikawa, Hiroyoshino commentno commentno commentno comment
36Schoenmaekers-Welters, Marij J.P.no commentno commentno commentShould MIATA provide information how the personnel could be trained for the assay ? External quality assurance programs not necessarily means that the results are good. As already stated by others we know that some labs hardly improve despite that they participate in proficiency panels.
37Marincola, Francescono commentno commentno commentno comment
38Lehmann, Paul V,Recording this is important. However, we must avoid stigmatizing data based on GLP criteria.Reproducible test results obtained by consistently using reference standards (which can be readily done in any laboratory, academic or regulated) are the most stringent Ôquality assurance criteriaÕ. No GLP structure and 100s of pages long SOPs can substitute for this. Such measurements should surpass the infrastructure-based faith in the ÒqualityÓ of data. GLP in the past relied on paper trails. For ELISPOT analysis (and flow cytometry), these are insufficient. For ELISPOT, however, electronic audit trails are fully suited to this purpose, and are readily available from CTL (including CFR part 11 compliant solutions).
39Andjelic, Sofijano commentno commentno commentno comment
40Cerundolo, Vincentono commentno commentno commentno comment
41Nielsen, Julieno commentno commentno commentno comment
42Shafer-Weaver, KimberlyThe particular SOP should be stated and if the lab is GLP or CLIA, these statements should also be providedAgain, it would be great if leaders in the field would agree upon a basic SOP for the different assays. That way individual investigators could refer back to the SOP and add in their particular modifications.If the manuscript is a technique paper, then I agree.Ê If the manuscript is a scientific paper that utilizes immune assays, then this information is too much to add in.Ê Ê Again, a generally accepted SOP would help in this situation.ÊParticipation in proficiency panels would help labs that donÕt already have an internal mechanism for quality control, but should not be required if internal proficiency standards are in place.Ê If a Lab is GLP or CLIA, it is understood that there are internal controls that must be met before a tech can run the assay or before the data can be distributed.
43Zitvogel, Laurenceno commentno commentno commentno comment
44Rivoltini, LiciaWe agreeWe agreeWe agreeWe agree
45Vandenbark, Arhtur A.no commentno commentno commentno comment
46Yuan, JiandaGLP or GLP-like conditions are not required for T cell monitoring assays, however we learned that the efforts to improve the lab environment to GLP or GLP-like levels benefit for the quality of T cell monitoring.Utilizing SOPs is important for both the maintenance of a standard of proficiency for senior lab members, and is equally essential for adequate training of those newly joining the lab.Ê However, this should be accepted as the standard and therefore not required to be reportedIn general, all assays should undergo validations in the listed essential parameters, yet as above, this should be accepted as the standard and should therefore not need to be reported.Ê If reporting was desired, only the most essential information such as the percent coefficient of variation values should be documented.Our experience has shown that external programs are helpful in cross-checking the efficiency of the techniques and procedures. ÊIt plays a key role in our lab member training, and serves to evaluate the continued proficiency of all members of the lab.Ê Furthermore, it serves for the assessment of how a laboratory is performing in relation to others.Ê Finally, conditions learned through these programs are often used to modify and improve SOPs if necessary.
47Malyguine, Anatolisee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaversee Dr. Kimberly Shafer-Weaver
48Mazorra Herrera, Zaima; Toledo Santamar’a, DarienWe think that laboratories for immunomonitoring of clinical trials should be accredited (they should apply well established SOP and validated methodology). Ideally should be GLP, but in this case, there arenÕt many immune monitoring labs within this category.no commentIn our opinion, since my lab is in a learning phase of cellular techniques, our researchers shouldÊreceive training in accredited labs for immunomonitoring or personnel from these labs could train our staff in our Institutions.
49Roep, Bartno commentno commentno commentno comment
50Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philipno commentno commentno commentno comment
51Yee, Cassianno commentno commentno commentno comment
52MŠurer, MarkusThe same comments apply as in module 2. If clinical studies are addressed, it should be noted if an independent statistician was consulted or not. Immunological data acquisitions will be more and more explorative and across different fields. For instance, cytokine receptor analysis on antigen specific T-cells is currently combined with SNP analysis in the autoimmune field. More complex immune signatures will be obtained. I suggest that genetic and RNA expression profiles are mentioned. We need also to be aware that there are at least two levels of data analysis: Clearly, most protocols work with streamlined, QCÕd and SOPÕd procedures, they address preformed hypotheses.This should not prevent the scientific community to explore in greater detail data which would otherwise not be addressed. This is already reality in gene expression profiling, yet it may also be true for flow cytometry-based readouts. Certain immune cell populations, defined in any test system, may be ignored: simply since they are not part of the readout protocol. This second layer of analysis, i.e. pattern recognition, as a ÔdecipheringÕ of the immune system, may be important for biological relevant questions (which were not asked in the first place). I would encourage to use these precious data from clinical trials and strive to develop / enable more objective data analysis (Ôwhat-you-see-is-what-you-get-approach) using advanced pattern recognition programs. This may in the end allow more objective analysis within defined statistical programs and data-mining approaches.no commentno commentno comment
53Parker, JoanneIt might be somewhat difficult to capture all of this information. There is a lot of confusion regarding GLP particularly since it does not apply to human samples and GCLP doesnÕt really exist in the US.Ê It would be great to have a guideline pertaining to qualification and validation of these types of assays and what can be done with these types of assays to follow ICH guidelines.
54Hural, JohnSince sample processing can occur at laboratories that are remote form the assay laboratory, it may be helpful to emphasize that all of the ÒLaboratory EnvironmentÓ (Module 5) information should be provided for both the processing and the assay laboratories.Likewise, Module 5.D. can apply to external proficiency assessments for the PBMC processing as well as the assay. Might want to think of an appropriate place to mention this.

MIATA Version 0.0 Cited

1. Bull M, Lee D, Stucky J, Chiu YL, Rubin A, Horton H, and McElrath MJ (2007) Defining blood processing parameters for optimal detection of cryopreserved antigen-specific responses for HIV vaccine trials. J Immunol Methods 322:57-69

2. Kierstead LS, Dubey S, Meyer B, Tobery TW, Mogg R, Fernandez VR, Long R, Guan L, Gaunt C, Collins K, Sykes KJ, Mehrotra DV, Chirmule N, Shiver JW, and Casimiro DR (2007) Enhanced rates and magnitude of immune responses detected against an HIV vaccine: effect of using an optimized process for isolating PBMC. AIDS Res Hum Retroviruses 23:86-92

3. Kreher CR, Dittrich MT, Guerkov R, Boehm BO, and Tary-Lehmann M (2003) CD4+ and CD8+ cells in cryopreserved human PBMC maintain full functionality in cytokine ELISPOT assays. J Immunol Methods 278:79-93

4. Maecker HT, Moon J, Bhatia S, Ghanekar SA, Maino VC, Payne JK, Kuus-Reichel K, Chang JC, Summers A, Clay TM, Morse MA, Lyerly HK, Delarosa C, Ankerst DP, and Disis ML (2005) Impact of cryopreservation on tetramer, cytokine flow cytometry, and ELISPOT. BMC Immunol 6:17-

5. Smith JG, Joseph HR, Green T, Field JA, Wooters M, Kaufhold RM, Antonello J, and Caulfield MJ (2007) Establishing acceptance criteria for cell-mediated-immunity assays using frozen peripheral blood mononuclear cells stored under optimal and suboptimal conditions. Clin Vaccine Immunol 14:527-537

6. Janetzki S, Panageas KS, Ben-Porat L, Boyer J, Britten CM, Clay TM, Kalos M, Maecker HT, Romero P, Yuan J, Kast WM, and Hoos A (2008) Results and harmonization guidelines from two large-scale international Elispot proficiency panels conducted by the Cancer Vaccine Consortium (CVC/SVI). Cancer Immunol Immunother 57:303-315

7. Janetzki S, Cox JH, Oden N, and Ferrari G (2005) Standardization and validation issues of the ELISPOT assay. Methods Mol Biol 302:51-86

8. Martinuzzi E, Scotto M, Enee E, Brezar V, Ribeil JA, van Endert P, and Mallone R (2008) Serum-free culture medium and IL-7 costimulation increase the sensitivity of ELISpot detection. J Immunol Methods 333:61-70

9. Britten CM, Gouttefangeas C, Welters MJ, Pawelec G, Koch S, Ottensmeier C, Mander A, Walter S, Paschen A, Muller-Berghaus J, Haas I, Mackensen A, Kollgaard T, thor SP, Schmitt M, Giannopoulos K, Maier R, Veelken H, Bertinetti C, Konur A, Huber C, Stevanovic S, Wolfel T, and van der Burg SH (2008) The CIMT-monitoring panel: a two-step approach to harmonize the enumeration of antigen-specific CD8+ T lymphocytes by structural and functional assays. Cancer Immunol Immunother 57:289-302

10. Britten CM, Janetzki S, Ben Porat L, Clay TM, Kalos M, Maecker H, Odunsi K, Pride M, Old L, Hoos A, and Romero P (2009) Harmonization guidelines for HLA-peptide multimer assays derived from results of a large scale international proficiency panel of the Cancer Vaccine Consortium. Cancer Immunol Immunother Epub ahead of print:

11. Maecker HT and Trotter J (2006) Flow cytometry controls, instrument setup, and the determination of positivity. Cytometry A 69:1037-1042

12. Currier JR, Kuta EG, Turk E, Earhart LB, Loomis-Price L, Janetzki S, Ferrari G, Birx DL, and Cox JH (2002) A panel of MHC class I restricted viral peptides for use as a quality control for vaccine trial ELISPOT assays. J Immunol Methods 260:157-172

13. Maecker HT, Dunn HS, Suni MA, Khatamzas E, Pitcher CJ, Bunde T, Persaud N, Trigona W, Fu TM, Sinclair E, Bredt BM, McCune JM, Maino VC, Kern F, and Picker LJ (2001) Use of overlapping peptide mixtures as antigens for cytokine flow cytometry. J Immunol Methods 255:27-40

14. Maecker HT, Frey T, Nomura LE, and Trotter J (2004) Selecting fluorochrome conjugates for maximum sensitivity. Cytometry A 62:169-173

15. Currier JR, Galley LM, Wenschuh H, Morafo V, Ratto-Kim S, Gray CM, Maboko L, Hoelscher M, Marovich MA, and Cox JH (2008) Peptide impurities in commercial synthetic peptides and their implications for vaccine trial assessment. Clin Vaccine Immunol 15:267-276

16. Perfetto SP, Ambrozak D, Nguyen R, Chattopadhyay P, and Roederer M (2006) Quality assurance for polychromatic flow cytometry. Nat Protoc 1:1522-1530

17. Janetzki S, Schaed S, Blachere NE, Ben-Porat L, Houghton AN, and Panageas KS (2004) Evaluation of Elispot assays: influence of method and operator on variability of results. J Immunol Methods 291:175-183

18. Dubey S, Clair J, Fu TM, Guan L, Long R, Mogg R, Anderson K, Collins KB, Gaunt C, Fernandez VR, Zhu L, Kierstead L, Thaler S, Gupta SB, Straus W, Mehrotra D, Tobery TW, Casimiro DR, and Shiver JW (2007) Detection of HIV vaccine-induced cell-mediated immunity in HIV-seronegative clinical trial participants using an optimized and validated enzyme-linked immunospot assay. J Acquir Immune Defic Syndr 45:20-27

19. Cox JH, Ferrari G, Kalams SA, Lopaczynski W, Oden N, and D’souza MP (2005) Results of an ELISPOT proficiency panel conducted in 11 laboratories participating in international human immunodeficiency virus type 1 vaccine trials. AIDS Res Hum Retroviruses 21:68-81

20. Hudgens MG, Self SG, Chiu YL, Russell ND, Horton H, and McElrath MJ (2004) Statistical considerations for the design and analysis of the ELISpot assay in HIV-1 vaccine trials. J Immunol Methods 288:19-34

21. Lathey JL (2003) Preliminary steps toward validating a clinical bioassay – a case study of the ELISpot assay. Biopharm International – The applied Technologies of Biopharmaceutical development 16:42-50

22. Moodie Z, Huang Y, Gu L, Hural J, and Self SG (2006) Statistical positivity criteria for the analysis of ELISpot assay data in HIV-1 vaccine trials. J Immunol Methods 315:121-132

23. Ezzelle J, Rodriguez-Chavez IR, Darden JM, Stirewalt M, Kunwar N, Hitchcock R, Walter T, and D’souza MP (2008) Guidelines on good clinical laboratory practice: bridging operations between research and clinical research laboratories. J Pharm Biomed Anal 46:18-29

24. Kelley M (20008) Considerations while setting up cell-based assays. Validation of cell based assays in the GLP setting A practical guide John Wiley Chapter 1:1-9

25. Maecker HT, Rinfret A, D’Souza P, Darden J, Roig E, Landry C, Hayes P, Birungi J, Anzala O, Garcia M, Harari A, Frank I, Baydo R, Baker M, Holbrook J, Ottinger J, Lamoreaux L, Epling CL, Sinclair E, Suni MA, Punt K, Calarota S, El Bahi S, Alter G, Maila H, Kuta E, Cox J, Gray C, Altfeld M, Nougarede N, Boyer J, Tussey L, Tobery T, Bredt B, Roederer M, Koup R, Maino VC, Weinhold K, Pantaleo G, Gilmour J, Horton H, and Sekaly RP (2005) Standardization of cytokine flow cytometry assays. BMC Immunol 6:1-18

26. NCCLS (2004) Performance of single cell immuine response assays; approved guidelines. NCCLS document I/LA26-A

27. International Conference on Harmonization (ICH) (1997) Guidance for Industry:Q2B Validation of Analytical Procedures Methodology. http://www ich org/LOB/media/MEDIA417 pdf

28. Findlay JW, Smith WC, Lee JW, Nordblom GD, Das I, DeSilva BS, Khan MN, and Bowsher RR (2000) Validation of immunoassays for bioanalysis: a pharmaceutical industry perspective. J Pharm Biomed Anal 21:1249-1273

29. Maecker HT, Hassler J, Payne JK, Summers A, Comatas K, Ghanayem M, Morse MA, Clay TM, Lyerly HK, Bhatia S, Ghanekar SA, Maino VC, Delarosa C, and Disis ML (2008) Precision and linearity targets for validation of an IFNgamma ELISPOT, cytokine flow cytometry, and tetramer assay using CMV peptides. BMC Immunol 9:1-9

30. Pifat D (2006) Assay Validation. www fda gov/cber/summaries/120600bio10 ppt

31. Tuomela M, Stanescu I, and Krohn K (2005) Validation overview of bio-analytical methods. Gene Ther 12 Suppl 1:S131-S138

32. Britten CM, Janetzki S, van der Burg SH, Gouttefangeas C, and Hoos A (2007) Toward the harmonization of immune monitoring in clinical trials: Quo vadis? Cancer Immunol Immunother 57:285-288