Validation

C-SATS started as a research project at the University of Washington and has since validated its methods in many applications.  

 
 
 

C-SATS uses a three-pronged analytic approach to provide customers accurate, unbiased results:

  • We use advanced hierarchical statistical models to generate performance scores that account for multiple levels of variability.  
  • We weight ratings from multiple groups of reviewers according to their historical accuracy and experience with C-SATS.
  • We utilize split-sample cross-validation to develop prediction models that provide unbiased results, even for the very best (or worst) performances.
 

Our Research

 

Published Peer-Reviewed Validation Manuscripts

  1. Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K. Aragon C, Holst D, Lendvay T.  Crowd-Sourced Assessment of Technical Skills (C-SATS): A Novel Method to Evaluate Surgical Performance. J Surgical Research 187:65-71, 2014. 
     
  2. Holst D, Kowalewski TM, White L, Brand T, Harper J, Sorenson M, Kirsch S, Lendvay TS. Crowd Sourced Assessment of Technical Skills (CSATS): An Adjunct to Urology Resident Surgical Simulation Training. J Endourology  29(5):604-9, 2015.
     
  3. Aghdasi N, Bly R, White L, Hannaford B, Lendvay TS. Crowd-Sourced Assessment of Surgical Skills in Cricothyrotomy Procedure. J Surgical Research 196(2):302-6, 2015.
     
  4. Holst D, Kowalewski TM, White LW, Brand T, Harper JD, Sorensen MS, Mirelle T, Simpson K, Tanaka A, Smith R, Lendvay TS. Crowd-Sourced Assessment of Technical Skills (C-SATS): Differentiating Animate Surgical Skill Through the Wisdom of Crowds. J Endourol 29:1183-8, 2015.
     
  5. Kirsch S, Comstock B, Harder L, Henriques A, Lendvay TS. Crowd sourced assessment of technical skills (CSATS): A scalable assessment tool for the nursing workforce. J Invest Med 63:92, 2015.
     
  6. Lendvay TS, White L, Kowalewski T. Crowdsourcing to Assess Surgical Skill. JAMA Surgery 150:1086-7, 2015.
     
  7. Chen SP, Kirsch S, Zlatev DV, Chang T, Comstock B, Lendvay TS, Liao JC. Optical Biopsy of Bladder Cancer Using Crowd-Sourced Assessment. JAMA Surgery doi:10.1001/jamasurg.2015.3121.
     
  8. Deal S, Brand TC, Haque MI, Warren J, Comstock B, Lendvay TS, Alseidi A. Crowd-Sourced Assessment of Technical Skills (C-SATS):  An Opportunity for Improvement in the Assessment of Laparoscopic Surgical Skills Amer J Surg 211:398-404, 2016.
     
  9. Kowalewski T, Sweet R, Lendvay T, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, McDougall E,. Validation of the AUA BLUS Tasks. Journal of Urology Vol 195, Issue 4, Part 1, Pages 998–1005, 2016.
     
  10. Ghani, K.R., Miller, D.C., Linsell, S., Brachulis, A., Lane, B., Sarle, R., Dalela, D., Menon, M., Comstock, B., Lendvay, T.S. and Montie, J., 2016. Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy. European Urology. 69:547-550, 2016.
     
  11. Kowalewski T, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, McDougall E, Lendvay T. Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills (BLUS) Tasks. J Urol [Accepted Jan 8, 2016, JU-15-2047R1].
     
  12. Hesham H, Lendvay T, Salani R, Martino, MA. “Innovative Methods to Teach and Train Minimally Invasive Surgery.” In An Atlas of Gynecologic Oncology, 2016, [in Press].
     
  13. Polin MR, Siddiqui NY, Brown C, Hesham H, Comstock B, Lendvay TS, Martino MA. Crowdsourcing: A Valid Alternative to Expert Evaluation of Robotic Surgery Skills. American Journal of Obstetrics and Gynecology [Accepted with Revisions May, 2016]
     
  14. Kowalewski T, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, McDougall E, Lendvay TS. Correlation of High-Volume Crowdsourced Evaluation and Tool Motion Metrics for Surgical Skills Assessment: The Basic Laparoscopic Urologic Skills (BLUS) Validation Project.  J EndoUrol [in Review, 2016].
     
  15. Tse, Raymond W. M.D.; Oh, Eugene M.D., Ph.D.; Gruss, Joseph S. M.D.; Hopper, Richard A. M.D.; Birgfeld, Craig B. M.D. Crowdsourcing as a Novel Method to Evaluate Aesthetic Outcomes of Treatment for Unilateral Cleft Lip. Plastic & Reconstructive Surgery 138:864-874, 2016
     
  16. Kirsch S, Comstock B, Harder L, Henriques A, Lendvay TS. Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce. Archives of Nursing Practice and Care 2(1): 040-044, 2016.
 

Peer-Reviewed Abstracts

  1. Chen C, Lendvay TS.  Crowd-Sourced Assessment of Technical Skills (C-SATS): Faculty Experts vs. Amazon.com Mechanical Turk Project.vs. Facebook. University of Washington School of Medicine, 2012 Medical Student III Poster Session, November 15, 2012.
     
  2. Chen C, White L, Kowalewski T, Comstock B, Lendvay TS. Crowd-sourcing assessment of technical skills (C-SATS): Faculty experts vs. Amazon.com Mechanical Turk  ProjectTM vs. FacebookTM. Northwest Urological Society Meeting, Podium presentation, Abstract #1204 Dec 8, 2012.
     
  3. Chen C, White L, Kowalewski T, Comstock B, Lendvay TS.  Crowd-Sourced Assessment of Technical Skill (C-SATS): Faculty Experts vs. Amazon.com Mechanical Turk ProjectTM vs. FacebookTM, Medicine Meets Virtual Reality (MMVR), Podium presentation, San Diego, CA, February 22, 2013.
     
  4. White LW, Kowalewski TM, Brand T, Harper J, Comstock B, Hannaford B, Lendvay TS. Validation of a Crowd-Sourced Assessment of Technical Skills (C-SATS™) Measuring Warm-Up Benefit in Robotic Surgery. MMVR/NextMed Meeting, February 20-22, 2014.
     
  5. Holst D, White L, Brand T, Harper J, Lendvay T. Crowd-Sourced Assessment of Technical Skills (C-SATS): An Adjunct to Urology Resident Simulation Training. American Urological Association Annual Meeting, Orlando, FL, May 16-21, 2014 [Moderated Poster #14-5364].
     
  6. White L, Kowalewski T, Brand T, Harper J, Comstock B, Hannaford B, Lendvay T. Crowd-Sourced Assessment of Technical Skills (C-SATS): Fast, Economical, and Accurate assessment of Robotic Surgery. American Urological Association Annual Meeting, Orlando, FL, May 16-21, 2014 [Podium #14-5475].
     
  7. White L, Wright A, Holst, Brobely Y, Bekele A, Lendvay T. Using Crowd-Assessment to Support Training in the Developing World. American Urological Association Annual Meeting, Engineers in Urology Session, Orlando, FL, May 16-21, 2014 [Moderated Poster #76].
     
  8. Lendvay T, Holst D, White L, Kowalewski T, Brand T, Sorenson M, Harper J, Truong M, Simpson K, Smith R. Differentiating Surgical Skill Through the Wisdom of Crowds. American Urological Association Annual Meeting, Engineers in Urology Session, Orlando, FL, May 16-21, 2014 [Moderated Poster #82].
     
  9. Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Knudsen B, Brand T, Ferrandino M, Kaouk J, Landman J, Lee B, McDougall E, Schwartz B, Lendvay TS. Construct Validity of the Basic laparoscopic Urologic Surgery (BLUS) Skill Tasks. World Congress of Endourology Meeting, Taipei, Taiwan, September 6, 2014 [Moderated Poster].
     
  10. Lendvay T, Comstock B, Averch T, Box G, Knudsen B, Brand T, Ferrandino M, Kaouk J, Landman J, Lee B, McDougall E, Menhadji A, Schwartz B, Sweet R, Kowalewski T. Crowd-Sourced Assessment of Technical Skills (C-SATS): Validation Through the Basic Laparoscopic Urologic Surgery (BLUS) Curriculum. World Congress of Endourology Meeting, Taipei, Taiwan, September 6, 2014 [Moderated Poster].
     
  11. Aghdasi N, Bly R, White L, Hannaford B, Moe K, Lendvay T. Crowd-sourced assessment of surgical skills in cricothyroidotomy procedure. IEEE Global Humanitarian Conference, San Jose, CA, October 10-13, 2014.
     
  12. Lendvay TS, White LW, Holst D, Kowalewski T, Harper JD, Sorenson M, Brand TC, Truong M, Simpson K, Smith R. Quantifying Surgical Skill Using the Wisdom of Crowds. American College of Surgeons Clinical Congress, San Francisco, CA, October 26-30, 2014. [Poster #PP2014-51161].
     
  13. White LW, Wright A, Bekele A, Kotisso B, Tadesse A, Oelschlager B, Holst D, Borbely Y, Oyetunji L, Lendvay TS. Using Crowd-Assessment to Support Surgical Training in the Developing World. American College of Surgeons Clinical Congress, San Francisco, CA, October 26-30, 2014. [Podium]
     
  14. Peabody JO, Miller DC, Linsell S, Lendvay T, Comstock B, Lane B, Sarle R, Brachulis A, Kim T-K, Deepansh D, Montie J, Ghani KR.  Wisdom of the crowds:  Use of crowdsourcing to assess surgical skill of robot-assisted radical prostatectomy in a statewide surgical collaborative. European Association of Urology, Madrid, Spain, March, 2015. Abstract ID #AM15-2491 Video review abstract [Poster]
     
  15. Ghani KR, Miller DC, Lane B, Sarle R, Brachulis A, Linsell S, Kim TK, Montie J, Comstock B, Lendvay T, Peabody JO.  Video-based independent peer review of robot-assisted radical prostatectomy in a statewide quality improvement collaborative. European Association of Urology, Madrid, Spain, March, 2015. Abstract ID #AM15-2482 Video review abstract [Poster]
     
  16. Deal S, Warren J, Haque M, Brand T, Comstock B, Lendvay TS, Alseidi A. Crowd Sourced Assessment of Technical Skills (CSATS): Solving the assessment dilemma in surgical skills evaluation. Association for Surgical Education Annual Meeting, Seattle, WA, April 23, 2015. [Podium]
     
  17. Ghani K, Miller D, Lane B, Sarle R, Brachulis A, Linsell S, Kim T-K, Delela D, Monti J, Comstock B, Lendvay T, Peabody J. Video-based peer review of robot-assisted radical prostatectomy in statewide quality improvement collaborative. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium]
     
  18. Peabody J, Miller D, Lane B, Sarle R, Brachulis A, Linsell S, Kim T-K, Delela D, Monti J, Comstock B, Lendvay T, Ghani K. Wisdom of the crowds: Use of crowdsourcing to assess surgical skill of robotic-assisted radical prostatectomy in a statewide surgical collaborative. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium]
     
  19. Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, McDougall E, Lendvay T. Large-Scale Evidence of Construct Validity: The Basic Laparoscopic Urologic Skills (BLUS) Initiative. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium]
     
  20. Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, McDougall E, Lendvay T. Validation of Laparoscopic Training Curriculum: The Basic Laparoscopic Urologic Skills (BLUS) Initiative. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium]
     
  21. Lendvay T, Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, McDougall E. Using the Wisdom of Crowds: Validation Through the Basic Laparoscopic Urologic Surgery (BLUS) Curriculum. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium] 
     
  22. Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, McDougall E, Lendvay T. High-Volume Assessment of Surgical Videos via Crowd-Sourcing: The Basic Laparoscopic Urologic Skills (BLUS) Initiative. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium]
     
  23. Liao J, Kirsch S, Lendvay T, Comstock B, Chen SP, Zlatev DV, Chang T.  “Crowd Sourced Assessment of Confocal Laser Endomicroscopy Imaging of Bladder Cancer.”  Engineers in Urology Session, American Urological Association Meeting, New Orleans, LA, May 15-18, 2015. [Moderated Poster]
     
  24. Polin MR, Siddiqui NY, Brown C, Hesham H, Comstock B, Lendvay TS, Martino MA. Crowdsourcing: A Valid Alternative to Expert Evaluation of Robotic Surgery Skills. AUGS Annual Meeting, Seattle, WA, October 2015 [Podium].
     
  25. Polin MR, Siddiqui NY, Brown C, Hesham H, Comstock B, Lendvay TS, Martino MA. Crowdsourcing: A Valid Alternative to Expert Evaluation of Robotic Surgery Skills. American Association of Gynecologic Laparoscopists Annual Meeting, Las Vegas, Nevada, November 2015 [Podium].
     
  26. Powers M, Lendvay T, Boonjindasup A, Pinsky M, Maddox M, Warren J, Su Li-Ming, Gettman M, Sundaram C, Lee BR. Crowd-sourcing assessment of surgeon dissection of renal artery and vein during robotic partial nephrectomy: A novel approach for quantitative assessment of surgical performance. World Congress of Endourology, London, England, October, 2015. [Abstract]
     
  27. Ghani K, Guru K, Lane B, Sarle R, Linsell S, Brachulis A, Miller D, Comstock B, Lendvay T, Peabody J. Variation in the Technical Skill of Surgeons Performing Robot-Assisted Radical Prostatectomy. AUA Annual Meeting San Diego, CA, May, 2016.  [Moderated Poster].
     
  28. McDougall E, Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, Lendvay T. Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills (BLUS) Tasks. Canadian Urological Association Meeting, Vancouver, British Columbia, June 25th-28th, 2016. [Accepted Moderated Poster]
 

Invited National and International C-SATS Presentations for Academic Centers/Professional Societies

  1. Visiting Professorship, University of Illinois – Chicago, C-SATS™: Crowd-sourced Assessment of Technical Skills using Amazon.com™ and Facebook™. Grand Rounds, Department of Urology, January 23, 2013.
     
  2. Visiting Professorship, University of British Columbia, British Columbia Children’s Hospital, Vancouver, British Columbia, Canada, June 11-12, 2013. [CANADA], The Future of Surgical Robotic Education, Simulation and Technology, Combined Department Grand Rounds.
     
  3. Visiting Professorship, Johns Hopkins University, Computer Science Engineering Department, Baltimore, MD, Host: Greg Hager, February 26, 2014. Assessing Technical Performance in Medicine: Using the Wisdom of Crowds. Johns Hopkins University, Department Seminar.
     
  4. Visiting Professorship, University of Pennsylvania, Assessing Technical Performance in Medicine: Using the Wisdom of Crowds. University of Pennsylvania Urology Grand Rounds, Philadelphia, PA, February 27, 2014.
     
  5. CSATS and BLUS Update. American Urological Association Annual Meeting, Office of Education  BLUS Group, Orlando, FL, May 17, 2014.
     
  6. CSATS Education for AUA. American Urological Association Annual Meeting, Education Council, Orlando, FL, May 18, 2014.
     
  7. University of Michigan Health Services Research Division Meeting, Ann Arbor Michigan, June 3rd, 2014.
     
  8. M2CAI Annual Meeting, Key-Note Lecturer, Open Challenges in Today’s OR: Skill, Evaluation, Error Reduction, Boston, MA, Sept 14, 2014.
     
  9. Human Cyber-Physical Interfaces Annual Conference, What Does the Surgeon Need? Challenges in Today’s OR, Paris, France, Sept 23, 2014. [FRANCE]
     
  10. Michigan Bariatric Surgery Collaborative Annual Meeting, Using the Wisdom of Crowds for Surgical Skills Assessment, Ann Arbor, Michigan, October 3, 2014.
     
  11. North Pacific Surgical Association Annual Meeting, Using the Wisdom of Crowds to Assess Surgical Skills, Seattle, WA, November 17, 2014.
     
  12. Society of Gastroendoscopic Surgeons (SAGES) Annual Meeting, Crowdsourcing to Assess Surgical Skill, TED-Like talk at Surgical Innovation Session, Nashville, TN, April, 2015.
     
  13. Society of Urologic Chairpersons and Program Directors (SUCPD), AUA Annual Meeting, Crowdsourcing to Assess Surgical Skill, New Orleans, LA, May, 2015.
     
  14. American Urological Association Annual Meeting, Plenary Talk, Crowdsourcing to Assess Surgical Skill, New Orleans, LA, May, 2015.
     
  15. Society of Urologic Robotic Surgeons (SURS), Crowdsourcing to Assess Surgical Skill, New Orleans, LA, May, 2015.
     
  16. American College of Surgeons Validation and Verification Committee Conference, Crowdsourcing to Assess Surgical Skill, Chicago, IL, June 19th, 2015.
     
  17. American Board of Urology, Annual Retreat, Crowdsourcing to Assess Surgical Skill, Quebec City, Quebec, August 6th, 2015. [CANADA]
     
  18. Madigan Army Medical Center Surgery Grand Rounds, Crowdsourcing to Assess Surgical Skill, Tacoma, WA, August 21st, 2015.
     
  19. World Congress of Endourology, Annual Meeting, Plenary Session, Crowdsourcing to Assess Surgical Skill, London, UK, October 3rd, 2015. [UNITED KINGDOM]

  20. Fundamentals of Laparoscopic Surgery Executive Committee, American College of Surgeons Annual Congress, Crowdsourcing to Assess Surgical Skill, Chicago, IL, October 5th, 2015.
     
  21. Lehigh Valley Medical Center General Surgery Grand Rounds, Crowdsourcing to Assess Surgical Skill, Allentown, PA, October 6th, 2015.
     
  22. Johns Hopkins University Urology Grand Rounds, Crowdsourcing to Assess Surgical Skill, Baltimore, MD, October 8th, 2015.
     
  23. Japanese Society of Endoscopic Surgeons, Plenary Session, Crowdsourcing to Assess Surgical Skill, Osaka, Japan,  Dec 9th, 2015. [JAPAN]
     
  24. University of California –Irvine, Dept. of Urology Grand Rounds, Breaking Down Barriers: Crowdsourcing to Assess Surgical Skill, Orange County, CA, Feb 29, 2016.
     
  25. Seattle Surgical Society Meeting, Crowdsourcing to Assess Surgical Skill, Seattle, WA, April 25th, 2016.
     
  26. American Urological Association Annual Meeting, Is Crowdsourcing Surgical Skill Assessment Reliable? An Analysis of Robotic Prostatectomies, Boston, MA,  May 14, 2017