Posted on Onctober 16, 2009
Updated to Version 1.0 and moved to archive in May, 2010
The reporting framework modules; Version 0.0
The initial publication guidelines of immune monitoring strategies are based on the premise that reporting of immune monitoring data from clinical studies will benefit by accurately including information for 5 “modules” that critically impact test results. Detailed annotation of published data for these 5 topics will be of additional benefit if accompanied by the appropriate reporting of raw data to allow for full evaluation of the value of published data sets.
| ID | Author | Module 1A - Information on the donor | Module 1B - Sample source | Module 1C - Cryopreservation | Module 1D - QC of cell material |
|---|---|---|---|---|---|
| 1 | Gouttefangeas, Cecile | What is essential? Too much asked for | no comment | Add time between storage and testing | no comment |
| 2 | van der Burg, Sjoerd H. | This is out of scope of MIATA but rather information on biological significance | Basically good | Basically good | Could control outcome from Modules B and C, clearly MIATA |
| 3, 4 | Speiser, Daniel (two comments) | no comment | no comment | no comment | Indication of cell recovery at thawing in percent of cell numbers at freezing |
| 5 | Navarrete, Marcelo | HLA type of donor might be irrelevant | no comment | no comment | no comment |
| 6 | Straten, Per Thor and Hardrup Sine | What is essential? Only age and gender for MIATA | Quality is most important issue and is dealt with in Module 1D | Quality is most important issue and is dealt with in Module 1D | Very good and add indication of cell recovery at thawing |
| 7 | Pawelec, Graham | HLA type not required for result interpretation, donor ethnicity needs to be reported as well as CMV serostatus if available as it impacts on overall constitution of immune system | Specify conditions not only for PBMC but also whole blood if performed | Details are critical, strong recommendation to standardise numbers of cells per amule and density at which cells are frozen. | no comment |
| 8 | Ottensmeier, Christian | Age and gender are OK. Relevance of other details will be determined by the context of the reporting. | Method for anticoagulation. Report on whether time labs and mode of transport is controlled and time from blood storage to analyiss. Still keep it as simple as possible | Perhaps average storage time with range, it might be more important which informaiton has been collected then what the characteristics are! | 1A-C are captured by this point! |
| 9 | Odunsi, Kunle | HlA type may not be critical, patient diagnosis seems adequate, additional details about disease status and prior treatments could be provided in the context of the specific study. | no comment | Include information on cryopreservation medium | What is meant by QA? |
| 10 | Walter, Steffen and Samorski, Regina | Information for biological significance and beyond the scope of MIATA, IN longitudinal trials it is of importance to report timing of PBMC sampling with respect to treatment schedule and how many time points were evaluated from evaluable patients. | no comment | Indicate media used for PBMC isolation and cryopreservation. | Sample acceptance criteria must be carefully validated for every individual assays which is most often not performed. Therefore B and C should be reported in additon to D |
| 11 | Ferrari, Guido | Modules 1A-C sufficient if GCLP is not an requirement | Modules 1A-C sufficient if GCLP is not an requirement | Modules 1A-C sufficient if GCLP is not an requirement | Intimately connected with GCLP guideleines |
| 12 | Rooke, Ronald and Bain, Christine | Add ethnic origin | Duration and temperature during transport, preparation on site or centralized | Include information on cryopreservation medium | QA should control Modules 1A-C |
| 13 | Appay, Victor | no comment | no comment | no comment | no comment |
| 14 | Lopatin, Uri | Detailed information on the donor might allow a trail back oto the individual patient and might not be allowable. | Add whether a standard or non-standard protocl for PBMC colleciton was used. | no comment | no comment |
| 15 | Ottenhoff, Tom HM and Joosten, Simone | Include country/region of trial and for infectious diseases the incidence and prevalence of infection in the specific pupulation studied as well as information on historic vaccination (e.g. TBC) HLA type might not be feasinble or relevant. | Extremely important (Ref Smith Plos One) | OK | OK |
| 16 | Schuler Gerold; Kämpgen Eckard; Gross, Stefanie | Necessity to provide such information depends on the sutdy. | Agree | Storage affects readout of some assays more than others. Can be captured in D | Absolutely necessary !! |
| 17 | Plant, Anne L. | no comment | no comment | no comment | no comment |
| 18 | Kern, Florian and Powell, Fiona | HLA not always neded. Time of blood draw should be indicated | Not enough data on influence of time between blood draw and preparation available so far. Interpretation difficult. | Enough to say if cells were frozen or used freshly | Should control 1B and 1C |
| 19 | Betts, Michael | no comment | no comment | no comment | no comment |
| 20 | Weber, Jeffrey | I am not sure how useful the clinical data on prior therapy would be, especially to the scientists | no comment | no comment | no comment |
| 21 | Gulley, James L | no comment | no comment | no comment | no comment |
| 22 | D'Souza, Patricia and Lane, Jim | no comment | no comment | no comment | no comment |
| 23 | Chen, Weisan; Cebon, Jonathan, Davis, Ian | When peptide-based vaccination is performed, reporting HLA type should be essential. From our experience with clinical trial patients receiving full-length NY-ESO-1protein vaccine, we have also found it essential to know HLA type because the T cell responses (specifically immunodominance patterns) are totally HLA-dependent. That could help readers to further assess the T cell response identified in a special region of the known antigen even if no detailed epitope mapping data provided. | If we know that the antigen-specific T cell expanded 40~100 times during a 11~13 day culture, many of these details may be superfluous, so long as a measure of assay performance and variability is provided. Although anticoagulation may affect T cell responsiveness, as does overnight storage, this should not matter so long as protocols are consistent. A statement indicating that standardised protocols for collection have been used should be sufficient. | as above | Because the background will vary depending on patient samples, we generally consider a cut-off value of 0.1% of total CD8+ cells for ICS is achievable. It would be great if someone could come up with a value that is both achievable and supported by statistics. |
| 24 | Cox, Josephine | no comment | no comment | Maybe time since collected ie over the period of 1-3 years 4-10 >10. Some trials take a long time! Labs change procedure over time too, though hopefully not during a trial | Recovery? Hard to quantify but I think important. |
| 25 | Janet Siebert | no comment | no comment | no comment | no comment |
| 26 | Cristina Muselli and group | We agree | We agree, especially if the blood collection, PBMC isolation and immune-monitoring assays are done by different labs, guidelines should be set for the above points. Should be specified what measures are taken if any to standardize and qualify different clinical site for blood collection/processing (operator training at the site, uniformity of protocol/SOPs between sites) | We agree, same as above | We agree, we actually think this is perhaps the most critical aspect of the whole module. The acceptance criteria and the sample cut-off values should be developed by each lab during the assay development phase and applied to all patient samples. In our experience, application of acceptance criteria does render some patient samples non-evaluable and helps to determine false positive and false negative data. |
| 27 | Schendel, Dolores | For example, full HLA typing of patients may not be relevant for many studies. Furthermore, it is not specified whether this should include class I and class II typing results and at what level the allotypes should be specified. This can be a very expensive analysis if it must be done commercially. | no comment | no comment | no comment |
| 28 | Knuth, Alexander; van den Broek, Maries | no comment | no comment | no comment | no comment |
| 29 | Kreiter, Sebastian; Attig, Sebastian; Sahin, Ugur | The amount of information on the donor should depend on the specific setting of every study. Only information directly relevant to the test result should be mandatory requirements. This might be HLA type in some studies. Information needed to interpret results from immunomonitoring assays should be considered of being outside the scope of MIATA. Minimal information should be age, gender and ethnic background. | no comment | no comment | It is well established that the quality of a sample that goes into an assay will determine the results of the test (garbage in garbage out). Investigators should therefore (1) take measures to make sure that samples for T cell assays have a high quality and (2) implement a QC to make sure that the effect of all measures taken to increase cell quality were effective. |
| 30 | Maio, Michele; Nicolay, Hugues J.M. | We agree with age, gender, diagnosis, disease status, treatment and prior relevant treatment. However, we believe that reporting HLA type is essential only when peptide-based vaccination is performed. Otherwise HLA type should not be considered as essential. | We believe that time for the blood sample is important as far as we observed differences in the detection of the T-cell responses at several time-points after the vaccination. Also, interim storage should be considered when trials take a long time, in fact when you run assays sometimes you use old material that are pre-vaccine sample that represent the base-line of T-cell response. Shipment of blood specimens (when necessary) is also important, because deeply relevant for PBMC integrity. | We agree with Josephine Cox, maybe time since collected i.e. over the period of 1-3 years 4-10 >10. Some trials take a long time! Labs could change procedure over time too. Of course procedures will change among labs and it should be discussed in section 5 B, but at least the freezing medium composition should be mentioned. | In this terms Module 1D is of highest importance as the QC will control all effects of measures taken in Modules 1A-C. Any information on sampling, preparation and storage of samples that is of specific interest to the investigator and give a better insight how a high quality of cells was achieved within a study might be reported but should not be mandatory. |
| 31 | Rammensee, Hans-Georg | no comment | no comment | 1C: Indicate storage system: liquid nitrogen tank or electrical freezer | no comment |
| 32 | Zwierzina, Heinz; Hakansson, Leiff | Age, gender, diagnosis, disease status, treatment, prior relevant treatments are all fundamental information which has to be included. The need to determine the HLA type depends on the intended study. In particular, disease status and prior relevant treatment has to be penetrated as these variables are of crucial importance. The systemic immune status of controls as well as patients show considerable variation, which will influence T-cell functional tests. Therefore, we think that some basic activation information of the patients immune status (e.g. serumconcentration of CRP and certain cytokines) at start of treatment is important to obtain. | This is a crucial point, the reproducibility of results obtained using different tubes for blood sampling has to be checked carefully. The temperature has to be kept under control and the time lapse before preparation of PBMCs has to be defined and should be kept as short as possible. | Even if there are some data on the possibility to use cryopreservation of PBMCs for T-cell functional tests, this procedure has to be carefully controlled. | In principal, each T-cell test to be performed has to be validated in each collaborating lab, that is the results of the test on cryopreserved and shipped cells has to be compared to the results obtained on fresh cells. |
| 33 | Gnjatic, Sacha | not necessary | not necessary | not necessary | not necessary |
| 34 | Reap, Elizabeth ; Norberg, Pamela; Watsin, Aubrey | We agree. It will depend on what questions you are asking to decide if you want HLA type. We dont think it is always possible or necessary to provide HLA type and prior relevant treaments. We dont think that the lack of HLA types should be reason to reject a publication. HLA typing is very expensive and may not be feasible for all studies but it shouldnt prevent the results from being shared. | Agree. PBMC separation should begin as quickly after blood collection as possible, and freezing of PBMC must begin within 8 hours after blood collection. No exceptions as it has been shown in the literature that signal is lost starting after 8 hours. This time lapse should be recorded. The isolation methodology is important. Must note gradient used (Lymphoprep or LeucoSep / Accuspin tubes with Histopaque-1077) Is the whole blood added directly to the gradients or diluted, and what is the volume to tube ratio? Spin times and speed because spinning at too high of an RPM will cause cell clumping, This needs to be recorded. | Agree. VERY Important to note time range and type of storage from PBMC isolation . If the clinical trial site has no liquid nitrogen and only -80C storage, how long were the PBMCs stored at -80C? It is important to know the cryopreservation media used and if it contains FBS, HSA, or serum-free. We have found that liquid nitrogen storage and shipment is best. | Many people mentioned reporting recovery. Most note how this was obtained. What if the collection site has a different counting method than the lab (trypan blue before freezing and Guava after thawing at lab)? The difference in counting methods could have a large impact on % recovery. |
| 35 | Nishikawa, Hiroyoshi | As comments already posted, reporting HLA type should be critical for analyses of T cell responses, even in clinical trials employing full-length protein because there are immuno-dominant HLA pattern for T cell responses. | no comment | Medium used for cryopreserve would be important to note. | no comment |
| 36 | Schoenmaekers-Welters, Marij J.P. | Disease history. Nevertheless, this is to my opinion beyond the scope of MIATA. | Time lapse between blood draw and treatment (before or after) of the patient could be relevant information since too short after intervention the results might be differently then waiting a few weeks longer. | Time lapse between final storage of PBMC vials and thawing for the immunological assay. It needs to be established how critical this is. | Percentages of viability, death cells and/or apoptosis after PBMC isolation (thus prior to storage), after thawing and after a resting period (if applicable). This is required to be able to determine the quality of the sample. |
| 37 | Marincola, Francesco | no comment | no comment | no comment | no comment |
| 38 | Lehmann, Paul V, | Depends on the study. | Countless critical variables exists. Some are known and some are unknown (original comment has a long list of known variables) . How can all of these possibly critical variants be recorded, and who will go through the effort to record it? How can one even define a serum or other biological, or document gently resuspend? Even if many follow the MIATA initiative (which is most desirable), and start feeding the database with such information (it would need to be quite detailed to be meaningful), how many entries will it take before statistical analysis will allow us to identify the factors that are critical? Even after that we will only see correlations that need to be (and can easily be) confirmed experimentally. We have successfully shown that through introduction of a reference sample and a reference protocol variability could be dcreased to acceptable levels even in the hands of unexperienced operators. Reference samples could even be stored and shipped along with clinical samples allowing one to ascertain whether any of the hard-to-track variables have affected them. One would simply test the reference sample upon receipt for performance as compared to previously established values. Results obtained testing reference samples alongside clinical samples would provide a yardstick for interpretation. | ||
| 39 | Andjelic, Sofija | no comment | no comment | no comment | no comment |
| 40 | Cerundolo, Vincento | no comment | no comment | no comment | no comment |
| 41 | Nielsen, Julie | no comment | no comment | no comment | no comment |
| 42 | Shafer-Weaver, Kimberly | I dont believe this is always necessary. It really depends on the questions being answered | Sample handling can greatly vary and in turn can change the integrity of the immune function of the cells. Again, a SOP for how samples should be collected for a particular use would be very helpful. | Yes, this is important. Please see comments above. | Criteria for what is the acceptable cell recovery and viability should be set remembering that most patient samples are more fragile then healthy donors. How the cell numbers and viability were determined should be added. However, what criteria will be used for assessing the function of the cells? Will this be standard mitogens or antigens (PHA, ConA, CEF) that should elicit a global immune response in the particular assay utilized? |
| 43 | Zitvogel, Laurence | no comment | no comment | no comment | no comment |
| 44 | Rivoltini, Licia | Collecting all patients information (age, gender, HLA type, diagnosis, disease status, treatment, prior relevant treatments) is essential to perform an exhaustive immunological monitoring. | In our experience, whole blood samples must be collected in EDTA vacutainer tubes in order to avoid monocytes activation. Blood should be processed within 2-4 hrs, in order to preserve PBMC maximum vitality and function. If it is not possible, specimens must be kept at room temperature no longer than overnight. Whole blood shipment must be performed at room temperature. In our hands, PBMC isolation by using Accuspin or Ficoll gives the same results, however the former is an easier and operator-independent approach. Whatever PBMC isolation procedure is used, it is important to reduce platelets number by a final slow washing. | Freezing process may be performed using a programmable-rate cell freezing apparatus. Cryovials must be stored in liquid nitrogen avoiding storage at -80°C (for longer than 2-3 days) in order to reduce leukocyte death. | It is essential that all immune-monitoring laboratories refer to a common guidelines and accepted criteria. |
| 45 | Vandenbark, Arhtur A. | no comment | no comment | no comment | no comment |
| 46 | Yuan, Jianda | As noted in the previously posted comments, HLA type should be reported, especially for broad T cell epitope response monitoring in clinical trials with entire proteins or multiple peptides, even in clinical trails with new immunotherapy such as anti-CTLA-4 antibody. T cell epitope response will provide additional mechanism leads. Prior relevant treatment information such as prior vaccination is necessary for interpretation of the data. | Several baseline values were collected for each patient, allowing us to define a mean and SD for their individual baseline values. The quantity of time points and thus blood specimens collected varies according to treatment schedule. Physiological variations are frequently observed in monitoring assays. If possible, we strongly suggest to collect two baseline blood samples to eliminate the potential physiological variations. | This part is probably unnecessary. Cell viability and recovery listed in 1D are more critical than 1C, especially if the cell viability and recovery after cell thawing are within the acceptable range. | Agree. It is important as indicated above. |
| 47 | Malyguine, Anatoli | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver |
| 48 | Mazorra Herrera, Zaima; Toledo Santamaría, Darien | These specifications could be important depending on the study. Some of them are not always available. For example, despite HLA type is important in some assessments as ELISPOT, testing patients HLA may not be feasible in studies with limited financial support. In the case of blind clinical studies we dont know treatment until the trial is finished. | OK | Very important issue | Serum should be specified and pretested. We suggest the counting of cells before and after freezing |
| 49 | Roep, Bart | Improving credibility: blinding of data, stimuli, subjects, and/or clinical outcome have proven to add greatly to the credibility of the results. Particularly when the data and the interpretation thereof by the immune assay operators are locked and sent to third parties prior to exchange of info on clinical background and outcome, the value and warm reception of the data improve substantially | |||
| 50 | Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philip | no comment | |||
| 51 | Yee, Cassian | no comment | |||
| 52 | Mäurer, Markus | Agreed. Although it would be desirable to know the HLA type, it may not be necessary for all studies. It should be provided if MHC-presented peptides are analyzed since cross-binding may impact on the results. | Very important, particularly the time lapse between the blood draw and PBMC isolation ! Of note, this should not lead to unnecessary censorship and uncritical comments in general, if e.g. the time lapse between blood draw and cell processing is more than 6hrs. This may be vital for some functional T-cell assays, yet not for other immunological readouts. Thus, the quality of biological material associated with the time lapse as an entry point for certain immune cell assays has to be linked with the immune readout. | Yes, see B. | See comment to B above |
| 53 | Parker, Joanne | We agree particularly regarding information about the techniques used for obtaining, processing, freezing, transporting and storing of the samples. | |||
| 54 | Hural, John | no comment |
| ID | Author | Module 2A - Cell counting | Module 2B - Medium/Serum source | Module 2C - Assay | Module 2D - Assay controls | |||
|---|---|---|---|---|---|---|---|---|
| 1 | Gouttefangeas, Cecile | no comment | do not ask for specific performance charactersitics of serum but simpy if medium/serum was pretested | mention and describe conditions for in vitro stimulation prior testing if applied | Ususally covered in M and M in many cases | |||
| 2 | van der Burg, Sjoerd H. | cell counting is also an topic for Module 1D and difficult to control | not relevant to report as long as test is properly controlled and carefully interpreted | report number of replicates done, guideleines that were followed do not have to be explicitly mentioned | good as stated in the current version of MIATA | |||
| 3, 4 | Speiser, Daniel (two comments) | no comment | no comment | explicitly mention and describe conditions for in vitro stimulation prior testing if applied. Results are only semiquantitative | Recommendation to use PBMC reference samples and to indicate their source | |||
| 5 | Navarrete, Marcelo | no comment | no comment | APC source and PBMC/APC ration might be of importance. Explicitly mention prior in vitro expansion! | no comment | |||
| 6 | Straten, Per Thor and Hardrup Sine | cell counting is also an topic for Module 1D and difficult to control | Data on serum/medium testnig are too elaborate to be reported and should be reflected by well controlled raw data | description in the MM section as expeced, references to guidelines can be given but should not stand alone. | How to do QA of control samples? | |||
| 7 | Pawelec, Graham | Counting directly after thawing and resting. Indicate conditions and recovery of viable cells as fraction of imput | use serum fee medium for ex vivo tests and indicate Lot No. | Source of all reagents and not just main ! | Purity of peptides used is of special importance | |||
| 8 | Ottensmeier, Christian | agree. Important | agree. Important, should be standard in M and M. performance charactersitics of medium/serum will be captured under 2c (?) | agree. Important. Number of replicates is a must. | important should be reported. Worry that extend of information will go beyond the scope of MIATA | |||
| 9 | Odunsi, Kunle | no comment | no comment | Information of assay procedures is critical and information on following of a published guideleine may be given as an add on | How to do QA the controls? | |||
| 10 | Walter, Steffen and Samorski, Regina | values should be reported only as group assessments (no raw data) | pretesting is not only limited to medium/serum but also for other critical variables such as antibodies. Indicate how many batches have been used for the analysis. What type of pretesting or batch bridginng was applied. If not considered this may be a source for variation | inter-assay variation can be important. It should ne noted how samples were distributed to different assays. (e.g. all samples from one patient assayed together.) | Last sentence not clear? | |||
| 11 | Ferrari, Guido | Modules 2A-C sufficient if GCLP is not an requirement | Modules 2A-C sufficient if GCLP is not an requirement | Modules 2A-C sufficient if GCLP is not an requirement | Intimately connected with GCLP guideleines | |||
| 12 | Rooke, Ronald and Bain, Christine | OK | OK | integrate Module 5B (SOPs use) here | Non responser as a control should be included | |||
| 13 | Appay, Victor | no comment | no comment | no comment | no comment | |||
| 14 | Lopatin, Uri | include pre-storage counts and thawed counts to provide an indea of handling. Averaged data for large studies! | no comment | no comment | ALSO report data on assay variation. It is important to know if, when looking at a given patients PBMCs run twice, the las allow for 10% or 2% variation. | |||
| 15 | Ottenhoff, Tom HM and Joosten, Simone | OK | serum source is of importance. Serum free media or same serum batch should be used. | A template could be suggested by MIATA as a help! | Internal controls are mandatory, external reference controls would be extremely helpful but are not always available. | |||
| 16 | Schuler Gerold; Kmpgen Eckard; Gross, Stefanie | combine with 1D | Source of medium/serum should bestated. High backrgound values are not automatically a problem as long as they are incorporated in the response definition and as long all samples are tested with the same medium/serum batch. | agree and number of replicates should be tested. | agree. | |||
| 17 | Plant, Anne L. | Should provide details on the methotology for determination viability, apoptosis etc. | no comment | detailled data on the characterisation of reagents should be provided if available. | no comment | |||
| 18 | Kern, Florian and Powell, Fiona | These should be standard methods. Is this hugely relevant to data? | What of the pretesting rwsults should be reported? | Here assay procedure should be explained. For reagents there should be an extra point | For all control reagents descriptions relevant to the reagent (such as purity of peptides). Indicate the exact physical nature of the control. | |||
| 19 | Betts, Michael | no comment | no comment | no comment | no comment | |||
| 20 | Weber, Jeffrey | no comment | no comment | The source of reagents and the concentrations/times used would be most important. | no comment | |||
| 21 | Gulley, James L | no comment | no comment | no comment | no comment | |||
| 22 | D'Souza, Patricia and Lane, Jim | no comment | no comment | no comment | no comment | |||
| 23 | Chen, Weisan; Cebon, Jonathan, Davis, Ian | We do know that counting cells and subsequent nurturing of these cells (time to split and to change medium, what density should they be maintained) are the most important, influential and not easily controlled factors. It may relate largely to the cell density, but we cannot exclude the influence from apoptotic cells or cell debris | Testing multiple batches of sera using PBMC with known CD8+ and CD4+ responses and selecting the best. | We agree with Dr Daniel Speiser that the assay types should be clearly defined. Labs should select ex vivo assays over in vitro expansion-based assays if T cell frequency is high enough. | Minimal controls should be:1. Culture control (a positive PBMC is cultured for the known specificity every time 2. PBMC quality control (a pooled viral peptides covering a wide HLA types to show that indeed the T cell in that PBMC are able to respond to antigens and expand) 3. importantly, an assay control (for example, in case the ICS fails for unforseen reasons, a frozen vial with known responding T cell population can be used to control the assay procedure) | |||
| 24 | Cox, Josephine | no comment | no comment | no comment | no comment | |||
| 25 | Janet Siebert | no comment | no comment | no comment | no comment | |||
| 26 | Cristina Muselli and group | We agree. See 1.D | The background reactivity of different lots of medium/sera will vary. It is important to use the same lot of serum for the development of the assay and the execution of the immuno-monitoring screening. The assay should contain all relevant controls to rule out false reactivity due to reactivity to a specific lot of serum/medium. | All guidelines followed for the assay should be consistent for the development as well as for the execution of the immuno-monitoring screening. Also we think it would be necessary to include qualification of reagents and material in addition to the source. | We agree.Ê A mention should also be made relative to what source of PBMC are used for qualification of reagents (healthy donors, cancer population with the same or different indication, etc.) | |||
| 27 | Schendel, Dolores | no comment | no comment | no comment | no comment | |||
| 28 | Knuth, Alexander; van den Broek, Maries | no comment | no comment | no comment | no comment | |||
| 29 | Kreiter, Sebastian; Attig, Sebastian; Sahin, Ugur | A) Cell counting can be a major source of variation. Reporting how cells were quantified and which measures were taken to reduce variability is of interest to a reader and might facilitate comparability of results across institutions.Ê | The reporting of source of reagents should be regarded as standard of a M. & M. part of a manuscript. Although the serum choice will influence the test result it is difficult to catch the influence or unique properties from pre-tested sera for a given test series. The impact of unique reagent properties will be reflected in the raw data (e.g. increased BG) and the internal controls. It is therefore of limited use to simply report that serum was pre-tested (what exactly is pretested?). | C) Reagents and assay procedures should be reported in enough detail to allow a reader to repeat the experiments under similar conditions. | D) Quality and quantity of internal assay controls (medium, irrelevant stimulus, positive control) are of high importance and should be reported in detail | |||
| 30 | Maio, Michele; Nicolay, Hugues J.M. | Important here is to state if cells are fresh or thawed, and if thawed to include pre-storage counts, and thawed counts, to provide an idea of handling. It should also be mentioned the composition of the thawing medium that could be supplemented with particular ÒadditivesÓ to preserve the integrity of the sample e.g., DNAse. | We agree, it is important to mention if serum is from bovine, human or autologous origins because serum origin is well known to be a substantial source of variation in assay outcome. | We believe that the assay procedures are really critical, but for sure, even harmonized, it will be really difficult for reagents, materials and treatment procedures. Guidelines should harmonize the number of assays for one sample and the number of replicates of the experiment to validate the results. | We agree on the importance of the description of the different positive and negative control that could be harmonized and be listed in section 5B. | |||
| 31 | Rammensee, Hans-Georg | no comment | no comment | 2C: Indicate time window used for the resting time applied (not just ãovernight restingÒ) | no comment | |||
| 32 | Zwierzina, Heinz; Hakansson, Leiff | See 1D | The choice of medium for in vitro tests is of paramount importance. Reasonably the reproducibility of the tests will improved if the confounding factors sera are avoided. However, as it is well known from the 1960«ties that serum factors can be of major importance in cancer related immunosuppression, it seems reasonable to consider the use of autologous sera in tests where the objective is to predict the clinical outcome of the patients. | This is a critical point. For example, preparation of sera included in the culture media is crucial as the proteolytic activity induced during the clotting process might alter the composition and function of proteins present in sera. | agree. | |||
| 33 | Gnjatic, Sacha | not necessary | not necessary | In my opinion, the only critical points for reporting assays are:- Describe if and how cells were sensitized with antigen (module 2C)- In ELISPOT, show or state number of spots for irrelevant control targets (module 2D)- State the criteria for interpretation of results (module 4)All other information is either unnecessary (HLA type for example may not be critical to data interpretation), or too much irrelevant information (use of "Mr. Frosty"). This comment assumes the labs performing monitoring have tested and validated their methods, but again, filling a checklist will not guarantee better quality of data if the lab is inexperienced | In my opinion, the only critical points for reporting assays are:- Describe if and how cells were sensitized with antigen (module 2C)- In ELISPOT, show or state number of spots for irrelevant control targets (module 2D)- State the criteria for interpretation of results (module 4)All other information is either unnecessary (HLA type for example may not be critical to data interpretation), or too much irrelevant information (use of "Mr. Frosty"). This comment assumes the labs performing monitoring have tested and validated their methods, but again, filling a checklist will not guarantee better quality of data if the lab is inexperienced | |||
| 34 | Reap, Elizabeth ; Norberg, Pamela; Watsin, Aubrey | Must say how counted cells i.e. Guava versus hemacytometer.Ê How many cryovials are being thawed at a time should be noted.Ê A standard thawing procedure is essential for obtaining maximum viability and recovery of cryopreserved PBMC.Ê | Serum should be pre-tested to check for background levels.Ê We agree with other comments that it is not enough to just say serum was pre-tested, as that could mean anything, but will there really be room in a publication to go into great detail about your pre-testing experiment? | Agree that it should be standard to list source of main reagents and materials and crucial treatment procedures.Ê Should also include number of replicate samples, number of events collected.Ê Resting versus no resting makes a big difference in the assay, because you have no apoptotic cells interfering with your sample.Ê ELISPOT Kit or standard test method?ÊÊ | STRONGLY agree. Each plate of every assay Êshould have qualified controls that should meet acceptance criteria that guarantee a valid assay in your lab.Ê You must know your assay. Each sample at each visit tested should be run with a mitogen control.ÊÊ Results for irrelevant or negative controls are importantÊ Also should be noted how samples were tested .Ê We agree with the comment by W Chen, J Cebon, and I Davis that if a standard or calibrator can be run with each assay then you should report the co-efficient of variation for day-to-day variability of the assay.Ê There may be cases however when a leukapheresis sample is not available to use as a calibrator, and that is acceptable as long as you list this. | |||
| 35 | Nishikawa, Hiroyoshi | no comment | Selecting human AB serum for T cell cultures is a critical factor. However, this requires careful selection of new bathes of serum. Appropriate standard sample(s) is necessary. | Control samples are, of course, critical. As mentioned above, one of the most important challenges is to establish the balanced minimum, but required controls. In my opinion, the followings are absolutely necessary. 1. Assay control (Positive/Negative standard samples for every assay) 2. Sample control (pooled-positive antigens, such as viral antigens to confirm T cells in the PBMC are indeed able to respond to antigens). | ||||
| 36 | Schoenmaekers-Welters, Marij J.P. | Thawing procedure is important in terms of percentages of viable cells (SOP ?) For example, monocytes tend to be more fragile than other cells but monocytes are very important in presenting the antigen to the T-cells in the ELISPOT. Should the methodology of cell counting not already be included in Module 1? For quality assurance of the samples this is important to use the same (qualified/validated) counting method. Are the PBMC rested for how many hours and under what conditions? If yes, additional cell counts after this resting period could be very useful. | Acceptance criteria why this medium/serum combination is chosen. | This in normally done in material and methods but it could be an advise to give more information instead of given it briefly. This could then be given as supplemental/web only data so that everybody is able to read it but it not limiting the text of the main paper. | What happens when the positive or negative control is out of range? When do you accept it as the correct control? | |||
| 37 | Marincola, Francesco | no comment | no comment | no comment | no comment | |||
| 38 | Lehmann, Paul V, | Live/dead counting assessed by staining the cells and counting them with an appropriate machine can be more accurate than just counting trypan blue exclusion by eye. Same for ÒapoptoticÓ counting.Ê However, because the apoptotic process goes through many well-defined phases, there is no clearly defined boundary between live/apoptotic and dead.Ê There are many live/dead/apoptotic stains on the market, and since each measures different cellular conditions/pathways they inherently provide different results.Ê Here too, we need to define a standard. | Serum is a major variable in T cell assay performance (Zhang et al, J. Immunotoxicology, 2009, 6:227-34).Ê Even a brief exposure of PBMC to such serum during freezing or washing can ruin a T cell assay.Ê If we want to document the unique biological properties of a serum used, how could we define ÒserumÓ for MIATA purposes such that it helps transparency?Ê Testing a serum against a reference standard would be one way to do so, but for this we would first need to accept a reference standard. | As to harmonization guidelines, I have good reasons to state that it is premature (or even wrong) to promote the published harmonization guidelines as a quality standard that will help generally improve T cell immune monitoring (for the specifics see below). Actually, I think the greatest danger of MIATA is that it can promote a trend in which such guidelines (based on low power statistical analysis of data) are imposed on the field before they have been experimentally verified. ÊI think such premature generalizations can Òbe an obstacle to innovation and improvement of immune-monitoring technologies.ÓÊ | Yes, but markers of Òcellular wellnessÓ might add a lot to interpreting the positive control itself (see above). | |||
| 39 | Andjelic, Sofija | no comment | no comment | no comment | no comment | |||
| 40 | Cerundolo, Vincento | no comment | no comment | no comment | no comment | |||
| 41 | Nielsen, Julie | no comment | It would be useful to know how many cells are used for experiments. For example, we all report cell number/well in an ELISPOT, percentage positive by flow cytometry, or cells/ml for in vitro stimulations, but I rarely see details regarding the total number of cells analyzed (flow cytometry) or Ð really important Ð the number of input cells in an in vitro stimulation. If you use 10^7 PBMC in an in vitro stimulation with MART-1, youÕll probably be able to expand MART-1-specific T cells, but if you use 10^4 cells, you probably wonÕt. Although this would be obvious to experts in the field, there certainly may be data published using insufficient cell numbers to make the conclusions that are published. This is important when statements are made regarding whether or not patients have a detectable response pre- or post-stim. Along the same lines, I frequently see data (particularly in vaccine trials) reported as X-fold increase over pre-vaccine levels. It would be useful to see absolute numbers. Was there a detectable response pre-vaccine, or is the frequency so low as to be completely undetectable. | |||||
| 42 | Shafer-Weaver, Kimberly | The particular thawing procedure impacts both the viability and recovery of cryopreserved PBMC.Ê Again, a SOP is needed.Ê General ranges should be provided for the samples (i.e. all samples were > 90% viable by trypan blue with >70% recovery from freeze by cell counts performed on a Z1 coulter counter).Ê Values for all samples are excessive. | The source and type of sera should be provided.Ê However, pretesting and performance characteristic is better suited for core or service labs and is unreasonable for basic research or clinical lab.Ê Nor do I think it is necessary to put all the specifics within the published article.Ê One alternative would be for the vendors to prescreen and certify their sera for particular uses (i.e. ELISPOT, Proliferation, cytokine induction, etc.).Ê | Yes, a brief description of the source and amount of reagents utilized should be included as well as timings and incubation temperatures. The number of replicates, events collected, and gating should also be specified.Ê ÊAdditionally, if samples were rested prior to counting and use in assay should be included. | I strongly agree with this statement. For each assay set-up, we have controls for testing the assay reagents, samples (mitogen or control antigen), response (irrelevant or negative controls) and the particular APC.Ê We have several banks of healthy donors and utilize two ( one high and one low responder) in each assay set up to ensure we are within the sensitivity range of the assay.Ê These normal responses are tracked overtime to ensure our assay quality assurance.Ê All patient samples are run within the same assay or if this is not possible, we run assays with overlapping samples.Ê We also perform side-by-side testing of assay reagents prior to switching to a new lot (again more for a core or service lab). If available, the amount variance in the assay is nice to know. | |||
| 43 | Zitvogel, Laurence | no comment | no comment | no comment | no comment | |||
| 44 | Rivoltini, Licia | We agree. | We use medium +10% FCS to have a low background; anyway it is important to use the same lot of serum during the whole monitoring study. | We agree. In particular, we have the same results (in terms of viability, sensitivity of the assays and PBMC functionality) using either fresh (unfrozen) or thawed and overnight-rested lymphocytes. Obviously, recovery is different. | We agree. However, we believe that the inclusion in each assay of a standardization control (for instance a fixed number of PBMC from the same defined donor displaying a know reactivity to viral epitopes, or an Ag-specific T cell line or clone) could help monitoring inter-assay variability. | |||
| 45 | Vandenbark, Arhtur A. | no comment | no comment | no comment | no comment | |||
| 46 | Yuan, Jianda | no comment | no comment | Cell culture in vitro stimulation conditions are critical, such as APC preparation, cytokine, peptide and the number of stimulations etc.Ê These all have impacts on the expansion of antigen-specific T cells and produce the variation. ÊWe recommend triplicate cell cultures and corresponding cell assays. | We agree with most of the posted comments about controls. Control could be used for reagents and/or samples as either positive or negative controls. We recommend the use of small aliquots of T cell clones or lines as sample controls for the assay. | |||
| 47 | Malyguine, Anatoli | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | |||
| 48 | Mazorra Herrera, Zaima; Toledo Santamara, Darien | The viability assays should be standardized (the use of PI is recommended) | The use of serum-free medium is recommended | OK | Internal control very important. Immune dysfunction of cells because of patientÕs disease and/or freezing- thawing procedures should be discarded. It is important a standardized procedure to eliminate this possibility. | |||
| 49 | Roep, Bart | Consider blinding stimuli and/or subjects; Provide stats on assay: coefficient of intra- and inter-assay variation. Description of quality assurance of controls including used peptides and peptide mixes: extremely important; great point. Also report on solvents and medium/control conditions (e.g., did the medium reference get the same diluent / solvent, e.g. DMSO, FMOC)?) (Or, if you test peptides, was the medium spiked with the peptide solvents (incl. DMSO)?) QC of recombinant antigens: we have developed stringent QC on recombinant antigens, as we have demonstrated that many currently used preparations can interfere in responses of T-cell clones reactive to third party antigens (Refs. Roep Ð J Autoimmunity, Peakman Ð Diabetes), even though they may act as stimulus/antigen for their specific T-cells. T-cells to eukaryotically expressed antigens often do not cross-react with antigens produced in E. coli vectors, and v.v., etc. | ||||||
| 50 | Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philip | no comment | ||||||
| 51 | Yee, Cassian | no comment | ||||||
| 52 | Murer, Markus | These points make sense. I would opt, particularly for clinical trials, to indicate in greater detail what ÔON restingÕ means, this could be all between 6 and 16 hrs. If flow cytometry analysis is performed, then an internal reference may be considered: the same batch of Ab coupled to the same batch of fluorochrome. If not only percentages are reported, but also MFI, then appropriate control beads should be considered. This will allow to control signals between experiments and even between different laboratories. The source and use of serum is particularly important in these assays. Also the note if any cytokines etc had been added. We need to exercise caution that future reviewers of grants or manuscripts will not expect the entire checklist listed above if the data are not from a clinical study addressing immunological readouts. This may have to be addressed and spelled out in a more precise way. | ||||||
| 53 | Parker, Joanne | We agree. However, we do not feel a statement regarding available recommendations is required. We feel reagents are very important and these are often not detailed with flow cytometry. | ||||||
| 54 | Hural, John | Module 2.C. While the term Òovernight restÓ is commonly used, it is my opinion that any formal document should state this as an Òovernight incubationÓ. The cells are certainly not resting at this time ;-) | M2.D. I would suggest specifically mentioning trend analysis of the PBMC reference sample in this section. Is trend analysis conducted? Is it reviewed by management? How long is one sample used? What are the criteria for requiring investigation of trend variations? |
| ID | Author | Module 3A device - hardware and software | Module 3B settings and controls | Module 3C quality assurance criteria / auditing |
|---|---|---|---|---|
| 1 | Gouttefangeas, Cecile | Do not ask for instrument validation and calibration | No use to indicate settings of ELISPOT reader | Auditing is not automatically a guarantee |
| 2 | van der Burg, Sjoerd H. | no comment | What do machine settings tell us? What does desciption of gating strategy help? No real value to report. | Auditing is not automatically a guarantee |
| 3, 4 | Speiser, Daniel (two comments) | no comment | no comment | no comment |
| 5 | Navarrete, Marcelo | Type of reader and software will suffice for ELIPOT | no comment | no comment |
| 6 | Straten, Per Thor and Hardrup Sine | Simply indicate hard nd software. | Indication of settings does not help much | Auditing is not automatically a guarantee |
| 7 | Pawelec, Graham | no comment | no comment | no comment |
| 8 | Ottensmeier, Christian | Too much. We should not tie ourselves into knots | Not meaningful as it is not clear that same settings between instruments are the same. It will be very difficult to make gating strategies objective. | Simple statement of audit is of limited use. How much and how was audited? Might go too far. |
| 9 | Odunsi, Kunle | no comment | Since training and setting are not standardized it cuold be difficult to interpret this paramter | It will be important to be more specific regarding what is meant by quailty assurance. |
| 10 | Walter, Steffen and Samorski, Regina | Personell is already included in 5D description of flow cytometer settings should include description of optics. Instrument calibration should include description on how instrument settings were adjusted for each experiment. For flow data it shold be indicated how compensation was performed. | rationale for adjusting software cont settings in ELISPOT or gates (flow) druing raw data analysis. Were global count settings or gates used in the study and were they defined before or after data were acquired. | Not clear? |
| 11 | Ferrari, Guido | Modules 3A-B sufficient is GCLP is not an requirement | Modules 3A-B sufficient is GCLP is not an requirement | Intimately connected with GCLP guideleines |
| 12 | Rooke, Ronald and Bain, Christine | OK | OK | Auditing should include Raw data as well as final results |
| 13 | Appay, Victor | no comment | no comment | no comment |
| 14 | Lopatin, Uri | no comment | no comment | no comment |
| 15 | Ottenhoff, Tom HM and Joosten, Simone | Could be part of template provided by MIATA | Could be part of template provided by MIATA | no comment |
| 16 | Schuler Gerold; Kmpgen Eckard; Gross, Stefanie | Agree with information on software and equipment but how does one validate an operator? You cannot therefore the oinformation is irrelevant (technichian vs PHD student x years) | Settings depend on macines so info is not helpful. | Information that audit happended does not help much. Giving all information on how it was audited is too much information. |
| 17 | Plant, Anne L. | no comment | no comment | no comment |
| 18 | Kern, Florian and Powell, Fiona | Overkill. Good internal controls could ascertain correct instrument setup. | Control beads would be used in the set-up rather than in the assay | Too (?) ambitious. This depends on definition of audits |
| 19 | Betts, Michael | no comment | Description of full gating strategy should always be given. If possible dot plots that are truly (!) representative for the whole analysis should be shown as well as raw data from a positive and a negative control. | no comment |
| 20 | Weber, Jeffrey | no comment | no comment | no comment |
| 21 | Gulley, James L | no comment | no comment | no comment |
| 22 | D'Souza, Patricia and Lane, Jim | no comment | no comment | no comment |
| 23 | Chen, Weisan; Cebon, Jonathan, Davis, Ian | no comment | Showing some raw data, for example FACS plots or ELISpot photos, should be sufficient indicators for above mentioned parameters/details | Agree with Daniel Ð this needs to be more clearly defined |
| 24 | Cox, Josephine | no comment | no comment | no comment |
| 25 | Janet Siebert | no comment | no comment | no comment |
| 26 | Cristina Muselli and group | Regardless of the type of equipment/software version, setup of the equipment and other variables, we think it is paramount to use all the same during the developmental phase and the execution of the screening phase.Ê The same operator should handle all the timepoints for any patient sample to maintain consistency. | Perhaps, add rational for why the chosen controls were used? Doing so might significantly assist others in determining what the best set up for future assays would be. | no comment |
| 27 | Schendel, Dolores | no comment | no comment | no comment |
| 28 | Knuth, Alexander; van den Broek, Maries | no comment | no comment | no comment |
| 29 | Kreiter, Sebastian; Attig, Sebastian; Sahin, Ugur | A) Information on type and version of machine and software used is critical. Information on Instrument calibration, qualification or validation or training of the operator should be a topic for module 5 and are difficult to control. | The value of reporting details of machine settings is minimal as same settings on different machines may lead to different results. Settings should not be mandatory for MIATA. On the other hand it might be crucial to document instrument settings for internal use as within a study an investigator might want to always work with same settings. Unexpected results could be due to in house change of settings that can only be tracked if settings are documented.Ê | C) Simply stating that data was audited could be anything and is of no help. This module could rater belong to Module 5 which capture aspects of lab environment and QA. |
| 30 | Maio, Michele; Nicolay, Hugues J.M. | We agree with other groups since training is not yet standardized/harmonized, it could be difficult to interpret this parameter. It could be useful but to our knowledge not mandatory. | We believe that data acquisition with ELISPOT automatic reader is a really good tool, but really varying upon assay procedure. The controls applied during the data acquisition rely greatly on the operator, and so could be a potential source of variation and bias. | We agree it is a critical point that must be harmonized with rules taking in consideration the variation between negative and positive controls. This should be integrated in the results interpretation. |
| 31 | Rammensee, Hans-Georg | no comment | no comment | 3C: Quality assurance criteria: this is important but one should try to limit the efforts for their description. |
| 32 | Zwierzina, Heinz; Hakansson, Leiff | no comment | no comment | no comment |
| 33 | Gnjatic, Sacha | not necessary | not necessary | not necessary |
| 34 | Reap, Elizabeth ; Norberg, Pamela; Watsin, Aubrey | Agree you should report equipment and software used and a brief description of how the equipment is setup or calibrated. It will be hard to say anything more than operator was trained on the equipment.Ê We donÕt think adding the statement about an operator being trained adds anything useful to the publication.Ê It would be expected that any operator should be trained on the equipment.Ê What is considered training will vary widely from lab to lab. There will not be enough room in a publicaton to go into specifics about each individualÕs training | AgreeÊ Flow cytometry gating strategy needs to be included. ÊThis can make a tremendous difference between labs and even between operators in same lab. | This would have to be defined very clearly.Ê This will vary from lab to lab.Ê Auditing will be very different between labs.Ê What is being audited, the final data, performing the assay?Ê Will this exclude otherwise good data because a lab does not have a requirement to audit data.ÊÊ Were validated spreadsheets used to eliminate the possibility of errors? |
| 35 | Nishikawa, Hiroyoshi | I agree with comments by others. Operator training is a critical issue, but it is challenging to find ways to evaluate operator training. | no comment | no comment |
| 36 | Schoenmaekers-Welters, Marij J.P. | no comment | no comment | Audited by who? This is hard to check. Do you mean audited by another person of the lab to prevent mistakes? |
| 37 | Marincola, Francesco | no comment | no comment | no comment |
| 38 | Lehmann, Paul V | A full validation of an instrument is desirable, but this is a scope that is hardly feasible for the majority of laboratories that publish ELISPOT data. Manufacturers of qualified ELISPOT readers have made sure that their instruments function consistently, and offer tools that permit the user to check whether the instrument is calibrated. Checking calibration should be part of any ELISPOT SOP (just as it is for flow cytometry). ÊIf spot counters have warm up /overheating /ambient light problems, such checks should be performed also in between the analysis.Beyond such general calibration, it will be nearly impossible to document the many variables that affect the functions of the different ELISPOT readers.A qualified ELISPOT reader manufacturer will have accounted for these variables and tailored them to the specific application Ð spot counters have not done so. | As stated above, it is impossible to document in writing all the parameters and variables that go into ELISPOT analysis, and even if it were possible, a list of such parameters would not provide reproducibility among different instruments. However, I strongly recommend submitting such raw/counted/QC image material with the corresponding electronic counting parameters as Òsupplemental materialÓ | ELISPOT data need auditing. Occasionally, for example, membrane defects and leakage needs to be corrected. However, any change made to the automated counts will be automatically recorded and annotated creating complete transparency about what has been changed, and why, without over-writing the objective automated counts.Ê The QC data set is a part of the record which can be viewed, submitted, and reanalyzed.Ê Instead of attempting to describe such for MIATA in generic terms (that is impossible and cannot create transparency), I suggest that investigators submit the actual QC material.Ê |
| 39 | Andjelic, Sofija | no comment | no comment | no comment |
| 40 | Cerundolo, Vincento | no comment | no comment | no comment |
| 41 | Nielsen, Julie | no comment | no comment | no comment |
| 42 | Shafer-Weaver, Kimberly | Yes, a brief description of the equipment and software for data acquisition should be provided.Ê But operator training and validation of that training is not necessary.Ê However, this will be needed if the assay is under GLP or CLIA. | Again, this is better suited for supplementary data.Ê I am not sure if the exact settings are necessary.Ê Perhaps general guidelines such as Òspot size threshold was set at > 10 times the size of a WBCÓ at least for ELISPOTs would be helpful and would suffice.Ê For Flow cytomtery, I agree that gating and voltages are important.Ê | A brief description of how the data was quality controlled is important, but we need to be careful here.Ê How detailed do we need to be?Ê Again, SOPs will help. |
| 43 | Zitvogel, Laurence | no comment | no comment | no comment |
| 44 | Rivoltini, Licia | We agree | We agree | The quality of audit differ from lab to lab and its depends from specific laboratories guide lines. |
| 45 | Vandenbark, Arhtur A. | no comment | no comment | no comment |
| 46 | Yuan, Jianda | no comment | Most data acquisition conditions should be be accepted as standard. ÊWe do agree with Weisan, that showing some raw data, such as FACS plots, should be sufficient. | no comment |
| 47 | Malyguine, Anatoli | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver |
| 48 | Mazorra Herrera, Zaima; Toledo Santamara, Darien | A) How to define "methodologies for operating training" | B) How to define "quality assurance criteria" | no comment |
| 49 | Roep, Bart | no comment | ||
| 50 | Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philip | no comment | ||
| 51 | Yee, Cassian | no comment | ||
| 52 | Murer, Markus | no comment | ||
| 53 | Parker, Joanne | Once again we agree. We feel it is particularly necessary for flow cytometry that the gating strategies be listed and as Dr Betts has already stated that the raw data and plots be shown. It is often very difficult to figure out how some groups generate their data.Ê | ||
| 54 | Hural, John | no comment |
| ID | Author | MIATA Module 4 - Results |
|---|---|---|
| 1 | Gouttefangeas, Cecile | Processing of data has to be explained and is of high importance (e.g. background in ELISPOT and multimers, irrelevant controls) |
| 2 | van der Burg, Sjoerd H. | Ask for time point when response definition criteria were defined (before or after generating the data set)! Describe stat test or empirical rules applied. |
| 3, 4 | Speiser, Daniel (two comments) | Indication of background values and wheter the threshold to define positivity was set at a constant level throughout the experimental series or differently between indiviual data sets. |
| 5 | Navarrete, Marcelo | no comment |
| 6 | Straten, Per Thor and Hardrup Sine | Yes. Indication of background of importance and irrelevant controls should be described. |
| 7 | Pawelec, Graham | no comment |
| 8 | Ottensmeier, Christian | yes. Critical |
| 9 | Odunsi, Kunle | no comment |
| 10 | Walter, Steffen and Samorski, Regina | Important to know if criteria to define positivitiy were defined before the study or post-hoc. Specific criteria should be listed. |
| 11 | Ferrari, Guido | no comment |
| 12 | Rooke, Ronald and Bain, Christine | Specificity of a response is interpreted as a function of the variation between negative and positive results. This should be integrated in the results interpratation |
| 13 | Appay, Victor | no comment |
| 14 | Lopatin, Uri | no comment |
| 15 | Ottenhoff, Tom HM and Joosten, Simone | Should also include description of how data were collected in diseased individuals in relation to contorls (important for infectious disease field). |
| 16 | Schuler Gerold; Kmpgen Eckard; Gross, Stefanie | Agree. Detailed description of the method is of highest importqnce and could be supported by a short listing of guideleines or recommendations that were respected. |
| 17 | Plant, Anne L. | no comment |
| 18 | Kern, Florian and Powell, Fiona | Next to statistics event countsare of major importance therefore data of events acquired for each data file would be mandatory. Results shown as percentages could be misleading. |
| 19 | Betts, Michael | no comment |
| 20 | Weber, Jeffrey | no comment |
| 21 | Gulley, James L | no comment |
| 22 | D'Souza, Patricia and Lane, Jim | no comment |
| 23 | Chen, Weisan; Cebon, Jonathan, Davis, Ian | We do agree with others that some kind of raw data, such as FACS gating and ICS pattern should be provided. |
| 24 | Cox, Josephine | no comment |
| 25 | Janet Siebert | Information about the software used (name, publisher, and version) should be included.Ê Different software versions, can have different implementations of statistical tests or analytical algorithms, which can impact reproducibility.Ê Descriptions of the statistical tests should include all relevant parameters (e.g. assumptions about variance).Ê Essentially, two different parties should be able to analyze the same data using the same test and get the same result.Ê Documentation should be adequate to support this goal. |
| 26 | Cristina Muselli and group | We agree |
| 27 | Schendel, Dolores | no comment |
| 28 | Knuth, Alexander; van den Broek, Maries | no comment |
| 29 | Kreiter, Sebastian; Attig, Sebastian; Sahin, Ugur | Response determination criteria should generally be defined prior to analyzing the data and reported in any report or publication. |
| 30 | Maio, Michele; Nicolay, Hugues J.M. | no comment |
| 31 | Rammensee, Hans-Georg | no comment |
| 32 | Zwierzina, Heinz; Hakansson, Leiff | no comment |
| 33 | Gnjatic, Sacha | In my opinion, the only critical points for reporting assays are:- Describe if and how cells were sensitized with antigen (module 2C)- In ELISPOT, show or state number of spots for irrelevant control targets (module 2D)- State the criteria for interpretation of results (module 4)All other information is either unnecessary (HLA type for example may not be critical to data interpretation), or too much irrelevant information (use of "Mr. Frosty"). This comment assumes the labs performing monitoring have tested and validated their methods, but again, filling a checklist will not guarantee better quality of data if the lab is inexperienced. |
| 34 | Reap, Elizabeth ; Norberg, Pamela; Watsin, Aubrey | Agree |
| 35 | Nishikawa, Hiroyoshi | This is very difficult issues and required much thoughts. Too much obscures the focus of the work, and too little limits ability for readers to interpret. |
| 36 | Schoenmaekers-Welters, Marij J.P. | The definition of a positive reactivity as well as of a vaccine-induced response could be pre-defined or deduced from the results. What option is preferred? Maybe it is wise to advise either one option depending on the type of study, research or clinical trials. |
| 37 | Marincola, Francesco | no comment |
| 38 | Lehmann, Paul V, | Yes. For the statistics and cut-offs and thresholds after correct numerical information was obtained and documented. For the latter see above.Ê |
| 39 | Andjelic, Sofija | no comment |
| 40 | Cerundolo, Vincento | no comment |
| 41 | Nielsen, Julie | no comment |
| 42 | Shafer-Weaver, Kimberly | How positive reactivity and differences between time points is determined was already addressed in an early module.Ê Yes, the statistical tests should be provided, but the reasoning behind the choice does not need to be given because if the data does not fit the assumptions of the test, then that test should not be used to analyze the significance of the data.Ê If data is excluded, it should be stated which data and why. |
| 43 | Zitvogel, Laurence | no comment |
| 44 | Rivoltini, Licia | We agree |
| 45 | Vandenbark, Arhtur A. | no comment |
| 46 | Yuan, Jianda | The definition of Òpositive reactivityÓ is very difficult and depends highly on Êthe selection of the marker and the magnitude of the response for a biological process.Ê In general, Òpositive reactivityÓ has not been associated with clinical outcome. |
| 47 | Malyguine, Anatoli | Statistical treatment should be applied, but even if the difference is statistically significant it may be not biologically relevant. For example, if in ELISPOT assay we have one spot/100K cells in control and two in experimental well it may be statistically significant but probably not biologically. Also I believe that it is not acceptable to show % of fold of increase/decrease without providing raw data. I am not sure if it possible to establish standard Òthreshold for Òpositive reactivity". For example, one of the often used approaches is to consider vaccination response to be positive (for ELISPOT) if we have double background difference. Again, what if we have 1 vs. 2 spots?Ê What about 100 vs. 200? |
| 48 | Mazorra Herrera, Zaima; Toledo Santamara, Darien | The reporting of data should be standardized. Monitoring strategies need to be designed for vaccine in which the Ag is not defined |
| 49 | Roep, Bart | (see also comment above on interpreting the data without knowledge of clinical outcome; i.e., prediction). Offer accessibility to raw data (to track ratio's, delta-values, etc.) and control/reference values. Very often, data are presented after subtraction of background, without providing background values. Reporting mean values of groups of subjects is not very helpful, since this does not allow adjusting the reported values by recalculating number of spots in an ELISPOT, proliferation rates (delta background; stimulation index), and so forth. In terms of statistics, almost as a rule proliferation data are analyzed as linear scales, while we all know that this should actually be in a more logarithmic fashion: one division doubles the cpm, so a decline in response from 20,000 to 5,000cpm, may be two cell divisions less, not Ô75% inhibitionÕ (if all cells were involved). The ideal transformation may be a hyperbolic arc sin transformation, but I appreciate that we have to take things one at the time... |
| 50 | Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philip | no comment |
| 51 | Yee, Cassian | no comment |
| 52 | Murer, Markus | See comments at the end of the text. In particular here: In the case of flow cytometry: if percentages of the target (test) population are only reported, then the cutoff for the acceptance should be note as the number of absolute cells. Since apoptotic cells are ÔstickyÕ a live/dead cell marker should be included in the actual test along with the Abs. |
| 53 | Parker, Joanne | We feel it is very important that there be details on how a positive response was determined. Comments of the significance of the results and correlation to treatment etc are usually provided. Correlation to clinical outcome would be nice but there would need to be details about how that correlation was determined. |
| 54 | Hural, John | no comment |
| ID | Author | Module 5A | Module 5B | Module 5C | Module 5D |
|---|---|---|---|---|---|
| 1 | Gouttefangeas, Cecile | no comment | shorten this as many people do not recognize importance of SOPs | shorten this as many people do not use qualified and validated assays | shorten this as many people do not recognize importance of proficiency panels |
| 2 | van der Burg, Sjoerd H. | Fully GLP or no GLP remove GLP-like | Statement that an assay was run acocring to SOP does not mean much as SOPs can be long/short or good/bad (How to control?) | Validation is strechable term. Labs could also reproducibly report different results from a sample. How to deal with the differecnes accross institutions | participation in a proficiency panel does not automatically mean that performance is good. A certificate would be needed. How to control if training of staff was good or bad? |
| 3, 4 | Speiser, Daniel (two comments) | no comment | no comment | no comment | no comment |
| 5 | Navarrete, Marcelo | no comment | no comment | no comment | no comment |
| 6 | Straten, Per Thor and Hardrup Sine | Fully GLP or no GLP remove GLP-like | Too early, not helpful, not advantageous, strechable terms, difficult to control | Too early, not helpful, not advantageous, strechable terms, difficult to control | Too early, not helpful, not advantageous, strechable terms, difficult to control |
| 7 | Pawelec, Graham | delete GLP-like | no comment | no comment | no comment |
| 8 | Ottensmeier, Christian | iportant to raise awareness of these issues. We are uneasy about setting the goalpost too high for module 5. premature ste as long as we do not know which test is an immunological surrogate for clinical effect. It is not needed to deliver data that is of diagnostic quality for treatment decisions. delete GLP-like | is important but to judge data of deviations of SOP the whole SOP has to be published. SOP is likely to mean a wide range of things between different SOPs | important but likely out of reach for most academis labs. Doubt that journals would allow to give space to publish such kind of information | Test used in proficiency panel must be the same as used in the reported experiment. Repreotinng of participation is only useful if lab has done well otherwise the statement might simply become an excuse. How is training done? Not the purpose of MIATA and gives inappropiate reassurance on the performance of the laboratory |
| 9 | Odunsi, Kunle | no comment | no comment | Validated SOPs would be desirable but wheter or not to report the details of how SOPs are qualified or validated would depend on the context and type of study | How do we define trained personnel? |
| 10 | Walter, Steffen and Samorski, Regina | it should be indicated whether SOPs were available for all aspects of the immune monitoring (sample acquisition, assay, data acquisition and raw data analysis an dinterpretation | no comment | no comment | no comment |
| 11 | Ferrari, Guido | Intimately connected with GCLP guideleines | Intimately connected with GCLP guideleines | Intimately connected with GCLP guideleines | Intimately connected with GCLP guideleines |
| 12 | Rooke, Ronald and Bain, Christine | Delete GLP-like, Cite any guideleine that had to be folowed as a mandatory requirement (GCLP) | Mention source of SOP (external/internal) and deviation/modificartion | OK | OK |
| 13 | Appay, Victor | no comment | no comment | no comment | no comment |
| 14 | Lopatin, Uri | no comment | no comment | no comment | no comment |
| 15 | Ottenhoff, Tom HM and Joosten, Simone | Template should help. | Template should help. | Only realistic for few labs and tests | Only realistic for few labs and tests |
| 16 | Schuler Gerold; Kmpgen Eckard; Gross, Stefanie | delete GLP-like | agree as long as there are certain criteria on what you call a SOP | assay and SOPs should be validated but this strongly depends on the test system. | participation in proficiency panels does not state anything. |
| 17 | Plant, Anne L. | no comment | no comment | no comment | no comment |
| 18 | Kern, Florian and Powell, Fiona | might be disqualifying for non-GLP labs. Not sure what it means. Might make sense to mention if data was generated by a certified lab (ISO9000). | unless the SOP is published he desciption of deviations is of no use to the reader. All that matters is the correct desciption of the technology. | not helpful for MIATA, not interested in auditing other labs as a revier. Important to know what exactly the procedures where and how data was interpreted in case raw data is not shown. | fine for accreditation but not for MIATA. MIATA should not explore the general credibility of a lab. |
| 19 | Betts, Michael | no comment | no comment | validation of assays might not be of interest for research labs that would rather need flexibility to incoporate new markers evrey day | no comment |
| 20 | Weber, Jeffrey | no comment | no comment | In the final category #5, how one ensures confidentiality is kept might be something worth reporting. | no comment |
| 21 | Gulley, James L | no comment | no comment | no comment | no comment |
| 22 | D'Souza, Patricia and Lane, Jim | no comment | no comment | There should be a distinction between Research and development labs, whose main purpose is discovery and GCLP central labs, whose main purpose is assessment of clinical specimens and product advancement decisions.Ê the latter will have to work under GCLP. We do not think R & D labs need a mandate to use validated assays, an SOP and follow total quality management principals. | no comment |
| 23 | Chen, Weisan; Cebon, Jonathan, Davis, Ian | no comment | no comment | no comment | no comment |
| 24 | Cox, Josephine | no comment | no comment | no comment | no comment |
| 25 | Janet Siebert | no comment | no comment | no comment | no comment |
| 26 | Cristina Muselli and group | We agree | no comment | no comment | We feel this module can be followed and integrated in each of the previous sections. |
| 27 | Schendel, Dolores | no comment | no comment | no comment | no comment |
| 28 | Knuth, Alexander; van den Broek, Maries | no comment | no comment | no comment | no comment |
| 29 | Kreiter, Sebastian; Attig, Sebastian; Sahin, Ugur | A) GLP-like does not exist. Work is done either as GxP or non-GxP. | B) Without seeing an SOPs it is not possible judge the value of the document. Publishing complete documents should not be requested by MIATA. | C)Ê Effects and degree of assay validation in a lab are difficult to capture.Ê Wherever validation efforts were undertaken that led to increased reproducibility of results and trust in data the investigator should be able to report such efforts.Ê Difficult module as results from a lab that did not fully validate an assay might well be of high quality and trustable. | D) Participation in proficiency panel does not automatically mean that a lab has a good performance. Participation in a proficiency panel conducted by an independent lab that certifies a good performance (pre-defined pass or fail criteria) might be interesting to report. Here it would be important to know what was certified (high sensitivity, low variation, good reproducibility etc.). |
| 30 | Maio, Michele; Nicolay, Hugues J.M. | no comment | no comment | no comment | no comment |
| 31 | Rammensee, Hans-Georg | no comment | no comment | 5C: statement on SOPs: yes; but again, I am afraid that if too much paperwork is asked for the description of all parameters this will disencourage people to participate. | no comment |
| 32 | Zwierzina, Heinz; Hakansson, Leiff | We do not believe that any statement of audit or participation in quality assurance programs will help very much, each lab has to have validated SOPs and well trained personnel. | no comment | no comment | no comment |
| 33 | Gnjatic, Sacha | not necessary | not necessary | not necessary | not necessary |
| 34 | Reap, Elizabeth ; Norberg, Pamela; Watsin, Aubrey | A lab is either GLP or not. State that the assay performed following a STM, controls qualified, calibrated instruments and SOPs used when writing this section | We agree with adding a statement that says SOPs were established and followed during the assay as long as this is in addition to a brief description of how you perform your assay with the critical information from the earlier modules being included.Ê We see no reason to describe any deviations from the SOP.Ê If there is a major deviation for one set of samples that could be noted, but in most cases that should lead to the assay being repeated and those data not used so we are not sure it adds to being able to interpret the data presented.Ê We donÕtÕ think that journals will have room for all this information but agree it is necessary. Maybe a universal checklist for each assay can be agreed upon by leaders in the field? | This is a lot of information to require for each publication but agree that the time has come due to the nature of these assays. ÊIt seems like it would be a stand alone paper to go into that much detail about your qualification or validation assay each time you write a paper that has T cell data.Ê If the paper has flow and ELISPOT, then you wouldÊ have to include your qualification or validation for each assay which would be very lengthy.Ê Just stating that your assay is qualified or validation is not enough information but we donÕt think that not having a qualified or validated assay should prevent data from being published.Ê The reader should be able to make their own decision about the quality of the data despite the use of a qualified or validated assay. | Information from the earlier modules is enough to determine the quality of the data.Ê Participation in proficiency panels is a plus but should not be required to be reported.Ê Participation in a panel does not mean that you performed well in that panel.Ê We hope it would be standard that only trained personnel perform any assay.Ê Again the amount of training will vary from lab to lab but I donÕt think you will have room in a publication to comment on the training of each individual person, nor is it really appropriate.Ê If the assay has acceptance criteria and controls they should give you an indication of a technical problem with the assay. |
| 35 | Nishikawa, Hiroyoshi | no comment | no comment | no comment | no comment |
| 36 | Schoenmaekers-Welters, Marij J.P. | no comment | no comment | no comment | Should MIATA provide information how the personnel could be trained for the assay ? External quality assurance programs not necessarily means that the results are good. As already stated by others we know that some labs hardly improve despite that they participate in proficiency panels. |
| 37 | Marincola, Francesco | no comment | no comment | no comment | no comment |
| 38 | Lehmann, Paul V, | Recording this is important. However, we must avoid stigmatizing data based on GLP criteria.Reproducible test results obtained by consistently using reference standards (which can be readily done in any laboratory, academic or regulated) are the most stringent Ôquality assurance criteriaÕ. No GLP structure and 100s of pages long SOPs can substitute for this. Such measurements should surpass the infrastructure-based faith in the ÒqualityÓ of data. GLP in the past relied on paper trails. For ELISPOT analysis (and flow cytometry), these are insufficient. For ELISPOT, however, electronic audit trails are fully suited to this purpose, and are readily available from CTL (including CFR part 11 compliant solutions). | |||
| 39 | Andjelic, Sofija | no comment | no comment | no comment | no comment |
| 40 | Cerundolo, Vincento | no comment | no comment | no comment | no comment |
| 41 | Nielsen, Julie | no comment | no comment | no comment | no comment |
| 42 | Shafer-Weaver, Kimberly | The particular SOP should be stated and if the lab is GLP or CLIA, these statements should also be provided | Again, it would be great if leaders in the field would agree upon a basic SOP for the different assays. That way individual investigators could refer back to the SOP and add in their particular modifications. | If the manuscript is a technique paper, then I agree.Ê If the manuscript is a scientific paper that utilizes immune assays, then this information is too much to add in.Ê Ê Again, a generally accepted SOP would help in this situation.Ê | Participation in proficiency panels would help labs that donÕt already have an internal mechanism for quality control, but should not be required if internal proficiency standards are in place.Ê If a Lab is GLP or CLIA, it is understood that there are internal controls that must be met before a tech can run the assay or before the data can be distributed. |
| 43 | Zitvogel, Laurence | no comment | no comment | no comment | no comment |
| 44 | Rivoltini, Licia | We agree | We agree | We agree | We agree |
| 45 | Vandenbark, Arhtur A. | no comment | no comment | no comment | no comment |
| 46 | Yuan, Jianda | GLP or GLP-like conditions are not required for T cell monitoring assays, however we learned that the efforts to improve the lab environment to GLP or GLP-like levels benefit for the quality of T cell monitoring. | Utilizing SOPs is important for both the maintenance of a standard of proficiency for senior lab members, and is equally essential for adequate training of those newly joining the lab.Ê However, this should be accepted as the standard and therefore not required to be reported | In general, all assays should undergo validations in the listed essential parameters, yet as above, this should be accepted as the standard and should therefore not need to be reported.Ê If reporting was desired, only the most essential information such as the percent coefficient of variation values should be documented. | Our experience has shown that external programs are helpful in cross-checking the efficiency of the techniques and procedures. ÊIt plays a key role in our lab member training, and serves to evaluate the continued proficiency of all members of the lab.Ê Furthermore, it serves for the assessment of how a laboratory is performing in relation to others.Ê Finally, conditions learned through these programs are often used to modify and improve SOPs if necessary. |
| 47 | Malyguine, Anatoli | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver | see Dr. Kimberly Shafer-Weaver |
| 48 | Mazorra Herrera, Zaima; Toledo Santamara, Darien | We think that laboratories for immunomonitoring of clinical trials should be accredited (they should apply well established SOP and validated methodology). Ideally should be GLP, but in this case, there arenÕt many immune monitoring labs within this category. | no comment | In our opinion, since my lab is in a learning phase of cellular techniques, our researchers shouldÊreceive training in accredited labs for immunomonitoring or personnel from these labs could train our staff in our Institutions. | |
| 49 | Roep, Bart | no comment | no comment | no comment | no comment |
| 50 | Allison, James; Sharama, Padmanee; Yee, Cassian; Wolchok, Jedd; Greenberg, Philip | no comment | no comment | no comment | no comment |
| 51 | Yee, Cassian | no comment | no comment | no comment | no comment |
| 52 | Murer, Markus | The same comments apply as in module 2. If clinical studies are addressed, it should be noted if an independent statistician was consulted or not. Immunological data acquisitions will be more and more explorative and across different fields. For instance, cytokine receptor analysis on antigen specific T-cells is currently combined with SNP analysis in the autoimmune field. More complex immune signatures will be obtained. I suggest that genetic and RNA expression profiles are mentioned. We need also to be aware that there are at least two levels of data analysis: Clearly, most protocols work with streamlined, QCÕd and SOPÕd procedures, they address preformed hypotheses.This should not prevent the scientific community to explore in greater detail data which would otherwise not be addressed. This is already reality in gene expression profiling, yet it may also be true for flow cytometry-based readouts. Certain immune cell populations, defined in any test system, may be ignored: simply since they are not part of the readout protocol. This second layer of analysis, i.e. pattern recognition, as a ÔdecipheringÕ of the immune system, may be important for biological relevant questions (which were not asked in the first place). I would encourage to use these precious data from clinical trials and strive to develop / enable more objective data analysis (Ôwhat-you-see-is-what-you-get-approach) using advanced pattern recognition programs. This may in the end allow more objective analysis within defined statistical programs and data-mining approaches. | no comment | no comment | no comment |
| 53 | Parker, Joanne | It might be somewhat difficult to capture all of this information. There is a lot of confusion regarding GLP particularly since it does not apply to human samples and GCLP doesnÕt really exist in the US.Ê It would be great to have a guideline pertaining to qualification and validation of these types of assays and what can be done with these types of assays to follow ICH guidelines. | |||
| 54 | Hural, John | Since sample processing can occur at laboratories that are remote form the assay laboratory, it may be helpful to emphasize that all of the ÒLaboratory EnvironmentÓ (Module 5) information should be provided for both the processing and the assay laboratories. | Likewise, Module 5.D. can apply to external proficiency assessments for the PBMC processing as well as the assay. Might want to think of an appropriate place to mention this. |
1. Bull M, Lee D, Stucky J, Chiu YL, Rubin A, Horton H, and McElrath MJ (2007) Defining blood processing parameters for optimal detection of cryopreserved antigen-specific responses for HIV vaccine trials. J Immunol Methods 322:57-69
2. Kierstead LS, Dubey S, Meyer B, Tobery TW, Mogg R, Fernandez VR, Long R, Guan L, Gaunt C, Collins K, Sykes KJ, Mehrotra DV, Chirmule N, Shiver JW, and Casimiro DR (2007) Enhanced rates and magnitude of immune responses detected against an HIV vaccine: effect of using an optimized process for isolating PBMC. AIDS Res Hum Retroviruses 23:86-92
3. Kreher CR, Dittrich MT, Guerkov R, Boehm BO, and Tary-Lehmann M (2003) CD4+ and CD8+ cells in cryopreserved human PBMC maintain full functionality in cytokine ELISPOT assays. J Immunol Methods 278:79-93
4. Maecker HT, Moon J, Bhatia S, Ghanekar SA, Maino VC, Payne JK, Kuus-Reichel K, Chang JC, Summers A, Clay TM, Morse MA, Lyerly HK, Delarosa C, Ankerst DP, and Disis ML (2005) Impact of cryopreservation on tetramer, cytokine flow cytometry, and ELISPOT. BMC Immunol 6:17-
5. Smith JG, Joseph HR, Green T, Field JA, Wooters M, Kaufhold RM, Antonello J, and Caulfield MJ (2007) Establishing acceptance criteria for cell-mediated-immunity assays using frozen peripheral blood mononuclear cells stored under optimal and suboptimal conditions. Clin Vaccine Immunol 14:527-537
6. Janetzki S, Panageas KS, Ben-Porat L, Boyer J, Britten CM, Clay TM, Kalos M, Maecker HT, Romero P, Yuan J, Kast WM, and Hoos A (2008) Results and harmonization guidelines from two large-scale international Elispot proficiency panels conducted by the Cancer Vaccine Consortium (CVC/SVI). Cancer Immunol Immunother 57:303-315
7. Janetzki S, Cox JH, Oden N, and Ferrari G (2005) Standardization and validation issues of the ELISPOT assay. Methods Mol Biol 302:51-86
8. Martinuzzi E, Scotto M, Enee E, Brezar V, Ribeil JA, van Endert P, and Mallone R (2008) Serum-free culture medium and IL-7 costimulation increase the sensitivity of ELISpot detection. J Immunol Methods 333:61-70
9. Britten CM, Gouttefangeas C, Welters MJ, Pawelec G, Koch S, Ottensmeier C, Mander A, Walter S, Paschen A, Muller-Berghaus J, Haas I, Mackensen A, Kollgaard T, thor SP, Schmitt M, Giannopoulos K, Maier R, Veelken H, Bertinetti C, Konur A, Huber C, Stevanovic S, Wolfel T, and van der Burg SH (2008) The CIMT-monitoring panel: a two-step approach to harmonize the enumeration of antigen-specific CD8+ T lymphocytes by structural and functional assays. Cancer Immunol Immunother 57:289-302
10. Britten CM, Janetzki S, Ben Porat L, Clay TM, Kalos M, Maecker H, Odunsi K, Pride M, Old L, Hoos A, and Romero P (2009) Harmonization guidelines for HLA-peptide multimer assays derived from results of a large scale international proficiency panel of the Cancer Vaccine Consortium. Cancer Immunol Immunother Epub ahead of print:
11. Maecker HT and Trotter J (2006) Flow cytometry controls, instrument setup, and the determination of positivity. Cytometry A 69:1037-1042
12. Currier JR, Kuta EG, Turk E, Earhart LB, Loomis-Price L, Janetzki S, Ferrari G, Birx DL, and Cox JH (2002) A panel of MHC class I restricted viral peptides for use as a quality control for vaccine trial ELISPOT assays. J Immunol Methods 260:157-172
13. Maecker HT, Dunn HS, Suni MA, Khatamzas E, Pitcher CJ, Bunde T, Persaud N, Trigona W, Fu TM, Sinclair E, Bredt BM, McCune JM, Maino VC, Kern F, and Picker LJ (2001) Use of overlapping peptide mixtures as antigens for cytokine flow cytometry. J Immunol Methods 255:27-40
14. Maecker HT, Frey T, Nomura LE, and Trotter J (2004) Selecting fluorochrome conjugates for maximum sensitivity. Cytometry A 62:169-173
15. Currier JR, Galley LM, Wenschuh H, Morafo V, Ratto-Kim S, Gray CM, Maboko L, Hoelscher M, Marovich MA, and Cox JH (2008) Peptide impurities in commercial synthetic peptides and their implications for vaccine trial assessment. Clin Vaccine Immunol 15:267-276
16. Perfetto SP, Ambrozak D, Nguyen R, Chattopadhyay P, and Roederer M (2006) Quality assurance for polychromatic flow cytometry. Nat Protoc 1:1522-1530
17. Janetzki S, Schaed S, Blachere NE, Ben-Porat L, Houghton AN, and Panageas KS (2004) Evaluation of Elispot assays: influence of method and operator on variability of results. J Immunol Methods 291:175-183
18. Dubey S, Clair J, Fu TM, Guan L, Long R, Mogg R, Anderson K, Collins KB, Gaunt C, Fernandez VR, Zhu L, Kierstead L, Thaler S, Gupta SB, Straus W, Mehrotra D, Tobery TW, Casimiro DR, and Shiver JW (2007) Detection of HIV vaccine-induced cell-mediated immunity in HIV-seronegative clinical trial participants using an optimized and validated enzyme-linked immunospot assay. J Acquir Immune Defic Syndr 45:20-27
19. Cox JH, Ferrari G, Kalams SA, Lopaczynski W, Oden N, and D’souza MP (2005) Results of an ELISPOT proficiency panel conducted in 11 laboratories participating in international human immunodeficiency virus type 1 vaccine trials. AIDS Res Hum Retroviruses 21:68-81
20. Hudgens MG, Self SG, Chiu YL, Russell ND, Horton H, and McElrath MJ (2004) Statistical considerations for the design and analysis of the ELISpot assay in HIV-1 vaccine trials. J Immunol Methods 288:19-34
21. Lathey JL (2003) Preliminary steps toward validating a clinical bioassay – a case study of the ELISpot assay. Biopharm International – The applied Technologies of Biopharmaceutical development 16:42-50
22. Moodie Z, Huang Y, Gu L, Hural J, and Self SG (2006) Statistical positivity criteria for the analysis of ELISpot assay data in HIV-1 vaccine trials. J Immunol Methods 315:121-132
23. Ezzelle J, Rodriguez-Chavez IR, Darden JM, Stirewalt M, Kunwar N, Hitchcock R, Walter T, and D’souza MP (2008) Guidelines on good clinical laboratory practice: bridging operations between research and clinical research laboratories. J Pharm Biomed Anal 46:18-29
24. Kelley M (20008) Considerations while setting up cell-based assays. Validation of cell based assays in the GLP setting A practical guide John Wiley Chapter 1:1-9
25. Maecker HT, Rinfret A, D’Souza P, Darden J, Roig E, Landry C, Hayes P, Birungi J, Anzala O, Garcia M, Harari A, Frank I, Baydo R, Baker M, Holbrook J, Ottinger J, Lamoreaux L, Epling CL, Sinclair E, Suni MA, Punt K, Calarota S, El Bahi S, Alter G, Maila H, Kuta E, Cox J, Gray C, Altfeld M, Nougarede N, Boyer J, Tussey L, Tobery T, Bredt B, Roederer M, Koup R, Maino VC, Weinhold K, Pantaleo G, Gilmour J, Horton H, and Sekaly RP (2005) Standardization of cytokine flow cytometry assays. BMC Immunol 6:1-18
26. NCCLS (2004) Performance of single cell immuine response assays; approved guidelines. NCCLS document I/LA26-A
27. International Conference on Harmonization (ICH) (1997) Guidance for Industry:Q2B Validation of Analytical Procedures Methodology. http://www ich org/LOB/media/MEDIA417 pdf
28. Findlay JW, Smith WC, Lee JW, Nordblom GD, Das I, DeSilva BS, Khan MN, and Bowsher RR (2000) Validation of immunoassays for bioanalysis: a pharmaceutical industry perspective. J Pharm Biomed Anal 21:1249-1273
29. Maecker HT, Hassler J, Payne JK, Summers A, Comatas K, Ghanayem M, Morse MA, Clay TM, Lyerly HK, Bhatia S, Ghanekar SA, Maino VC, Delarosa C, and Disis ML (2008) Precision and linearity targets for validation of an IFNgamma ELISPOT, cytokine flow cytometry, and tetramer assay using CMV peptides. BMC Immunol 9:1-9
30. Pifat D (2006) Assay Validation. www fda gov/cber/summaries/120600bio10 ppt
31. Tuomela M, Stanescu I, and Krohn K (2005) Validation overview of bio-analytical methods. Gene Ther 12 Suppl 1:S131-S138
32. Britten CM, Janetzki S, van der Burg SH, Gouttefangeas C, and Hoos A (2007) Toward the harmonization of immune monitoring in clinical trials: Quo vadis? Cancer Immunol Immunother 57:285-288