C-SATS, Inc. started as a research project at the University of Washington and has since validated its methods in many applications.
2022
Featured
In this initial multicenter study on Video-Based Assessment (VBA) for robotic sleeve gastrectomy, operative time and several GEARS subcomponent scores (efficiency, bimanual dexterity, and depth perception) were associated with surgical experience, suggesting that these variables may be used as surrogate metrics for quality of an operation.
Addison, P, et al. Does Surgeon Experience Correlate with Crowd-Sourced Skill Assessment in Robotic Bariatric Surgery? Am Surg. 2022; doi: 10.1177/00031348221142586.
Based on the current published literature, the C-SATS platform has been shown to efficiently provide crowd-sourced evaluations that correlate favorably with expert evaluation across a range of surgical specialties and approaches. Use of crowdsourcing has uniformly yielded accurate evaluations of surgeons’ technical skills in a markedly shorter time than expert reviews. C-SATS may be a cost-effective complement or alternative to traditional models of evaluating surgical proficiency. Future studies are needed to determine whether the use of C-SATS will lead to improved surgical performance and patient outcomes.
Giovanni, T.A., et al. Value of the Crowd-Sourced Assessment of Technical Skills (C-SATS) Platform in Surgical Procedures: A Systematic Review of Evidence. SciVision Pub. 2022.
2021
Featured
Validity, scalability and discoverability are current but removable barriers to video-based assessment.
Pugh, C., et al. The what? How? And Who? of video-based assessment (VBA). Am J Surg. 2021 Jan;221(1):13-18. doi: 10.1016/j.amjsurg.2020.06.027. Epub 2020 Jun 23.
2020
Featured
The findings of this study suggest that there is wide variation in technical skill among practicing surgeons, accounting for more than 25% of the variation in patient outcomes. Higher colectomy technical skill scores appear to be associated with lower complication rates for colectomy and for all other procedures performed by a surgeon. Efforts to improve surgeon technical skills may result in better patient outcomes.
Stulberg, J., et al. Association Between Surgeon Technical Skills and Patient Outcomes JAMA Surg. Published online August 19, 2020. doi:10.1001/jamasurg.2020.3007
This study demonstrated the feasibility of using a patient-centered health-data-sharing platform, Hugo, to aggregate multiple real-world data sources in near real time, thereby enabling evaluation of medical products, in this case medical devices, for both research and regulatory purposes and facilitating better understanding of patient recovery, function and symptoms.
Dhruva, S., et al. Aggregating multiple real-world data sources using a patient-centered health-data-sharing platform. npj Digital Medicine volume 3, Article number: 60. 2020.
2019
Featured
This article highlights novel strategies that may assist surgeons and hospitals in addressing the challenges associated with incorporating emerging procedures and technology with safe patient care. Peer coaching by experienced surgeons has been proposed to assist with safe surgical skill acquisition in practice. Hospital credentialing committees may better utilize expert surgeons as proctors for less-experienced surgeons who apply for privileges to perform new procedures. Furthermore, opportunities exist in leveraging industry's financial incentives with surgeons’ drive for clinical advancements. The strategies described in this article offer ways to ensure procedural competency to benefit and protect patients when surgical innovations enter practice.
Pradarelli, J.C., et al. Facilitating the Safe Diffusion of Surgical Innovations Annals of Surgery: April 2019 - Volume 269 - Issue 4 - p 610–611. DOI: 10.1097/SLA.0000000000003134.
Use of systematic video review (SVR) in laparoscopic colorectal cancer surgery as an adjunct to the narrative operative report (NR) might be superior in documenting important steps of the operation compared with NR alone, adding to the overall availability of necessary intra-operative information and contributing to quality control and objectivity.
Van de Graaf, F.W., et al. Comparison of Systematic Video Documentation With Narrative Operative Report in Colorectal Cancer Surgery JAMA Surg. Published online January 23, 2019. doi:10.1001/jamasurg.2018.5246
If surgeons want to take the next step at improving the quality of our craft, we need to take advantage of the richest source of data available to us: operative video. Most efforts to improve surgical quality focus on optimizing care before surgery (eg, adhering to evidence-based processes around preventing wound infection and deep venous thrombosis) and the early recognition and treatment of complications to mitigate downstream harm (eg, failure to rescue). While these efforts will improve care, we need to push the frontier to improving what happens in the operating room—the quality of the operation itself.
Dimick, J.B., et al. A Video Is Worth a Thousand Operative Notes JAMA Surg. Published Online: January 23, 2019. doi:10.1001/jamasurg.2018.5247
JAMA Surgery
The surgical workforce is aging. According to the Association of American Medical Colleges Physician Specialty Data Report, in 2017, 44.1% of 103 032 active surgeons in the United States were 55 years or older. The percentage of surgeons 55 years or older varies by surgical specialty, ranging from a low of 40.9% in vascular surgery to a high of 58.1% in thoracic surgery.1 This is an important issue because older surgeons have more experience caring for patients, which needs to be balanced with the potential impairment in their ability to provide high-quality surgical care because of age-related functional limitations.
Mark R. Katlic, MD, JoAnn Coleman, DNP, ACNP-BC, Marcia M. Russell, MD (2019) Assessing the Performance of Aging Surgeons. JAMA Surg. 2019 doi:10.1001/jama.2018.22216
2018
Featured
JAMA Surgery
Artificial intelligence (AI) is gaining high visibility in the realm of health care innovation. Broadly defined, AI is a field of computer science that aims to mimic human intelligence with computer systems.1 This mimicry is accomplished through iterative, complex pattern matching, generally at a speed and scale that exceed human capability. Proponents suggest, often enthusiastically, that AI will revolutionize health care for patients and populations. However, key questions must be answered to translate its promise into action.
Thomas M. Maddox, MD, MsC, John S. Rumsfeld, MD, PhD, Philip R. O. Payne, PhD (2018) Questions for Artificial Intelligence in Health Care. JAMA Surg. 2018 doi:10.1001/jama.2018.18932
JAMA Surgery
Artificial intelligence (AI) and deep learning are entering the mainstream of clinical medicine. For example, in December 2016, Gulshan et al1 reported development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. An accompanying editorial by Wong and Bressler2 pointed out limits of the study, the need for further validation of the algorithm in different populations, and unresolved challenges (eg, incorporating the algorithm into clinical work flows and convincing clinicians and patients to “trust a ‘black box’”).
William W. Stead, MD (2018) Clinical Implications and Challenges of Artificial Intelligence and Deep Learning. JAMA Surg. 2018. doi:10.1001/jama.2018.11029
JAMA Surgery
During video review of 192 robotic pancreaticoduodenectomies, the angle of the gastrojejunal anastomosis, gastrostomy size, and surgical device style of gastrojejunal anastomosis were technical variables that were associated with postoperative delayed gastric emptying.
Jae Pil Jung, MD, Mazen S. Zenati, MD, PhD, Mashaal Dhir, MD, Amer H. Zureikat, MD, Herbert J. Zeh, MD, Richard L. Simmons, MD, Melissa E. Hogg, MD,MS (2018) Use of Video Review to Investigate Technical Factors That May Be Associated With Delayed Gastric Emptying After Pancreaticoduodenectomy. JAMA Surg. 2018;153(10):918-927. doi:10.1001/jamasurg.2018.2089
The Journal of Bone and Joint Surgery, Incorporated
We educate residents to orchestrate the future delivery of medical and surgical care. However, the tenets and principles of this task vary, resulting in a wide distribution of physician skill, knowledge, and judgment by completion of training1. Surgical training is no exception, and the resultant skills of recent orthopaedic surgery residency graduates are comparatively variable2-4. The potential for uneven skill acquisition has pushed the Accreditation Council for Graduate Medical Education (ACGME), and subsequently the American Board of Orthopaedic Surgery (ABOS), to develop some general standards: the ACGME/ABOS Orthopaedic Surgery Residency Training Milestones5,6.
Kofi D. Agyeman, MD, Seth D. Dodds, MD, Jason S. Klein, MD, Michael G. Baraga, MD, Victor H. Hernandez, MD, and Sheila Conway, MD (2018) Innovation in Residence Education J Bone Joint Surg Am. 2018;100:e90(1-5) d http://dx.doi.org/10.2106/JBJS.17.00839
Prolonged Operative Duration is Associated with Complications: A Systematic Review and Meta-Analysis
ScienceDirect
Prolonged operative time is associated with an increase in the risk of complications. Given the adverse consequences of complications, decreased operative times should be a universal goal for surgeons, hospitals, and policy-makers. Future study is recommended on the evaluation of interventions targeted to reducing operating time.
lHang Cheng, MS; Jeffrey W. Clymer, PhD; Brian Po-Han Chen, ScM; Behnam Sadeghirad, PhD; Nicole C. Ferko, MS; Chris G. Cameron, PhD; PietHinoul, MD. (2018) Prolonged Operative Duration is Associated with Complications: A Systematic Review and Meta-Analysis https://doi.org/10.1016/j.jss.2018.03.022
2017
Featured
This study demonstrates comprehensively that prolonged operative time can increase the likelihood of developing SSI across a broad array of surgical procedures and surgical specialties. Overall, pooled analyses demonstrated that patients with extended operative times across a wide range of procedures had approximately twice the likelihood of developing SSI, and on overage, the mean operative time was approximately 30 min longer in patients with SSIs compared with those patients without SSIs. Nevertheless, given the importance of SSIs on patient outcome and health care economics including hospital reimbursement penalties, hospitals should focus efforts to reduce operative time. This may include strategies such as the adoption of new technologies that can help improve operative efficiency, utilization of specialized care teams, ensuring that operating staff are not overworked or fatigued, and better preoperative planning. These strategies should be considered in light of other possible risk factors for SSI.
Cheng, H., et al. Prolonged Operative Duration Increases Risk of Surgical Site Infections: A Systematic Review. Surgical Infections. Aug 2017.722-735. http://doi.org/10.1089/sur.2017.089
Journal of Graduate Medical Education
Jessica C. Dai, Thomas S. Lendvay, and Mathew D. Sorensen (2017) Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education. Journal of Graduate Medical Education, In press.
JAMA Surgery
Aggarwal R. Intraoperative Surgical Performance Measurement and Outcomes Choose Your Tools Carefully. JAMA Surg. Published online August 09, 2017. doi:10.1001/jamasurg.2017.0837
Journal of Urology
Lee, Jason Y., Andonian, Sero, Pace, Kenneth T., Grober, Ethan. Basic Laparoscopic Skills Assessment Study: Validation and Standard Setting among Canadian Urology Trainees, The Journal of Urology, 197 6 (2017): 1539-1544.
American Urological Association Annual Meeting
American Urological Association Annual Meeting, Is Crowdsourcing Surgical Skill Assessment Reliable? An Analysis of Robotic Prostatectomies, Boston, MA, May 14, 2017
Meyers E, Cowan C, Bernier R, Lendvay TS, Comstock B, Stone W. Use of Crowd-Sourcing to Assess Social Communicative Behavior in Toddlers with and without ASD. International Meeting for Autism Research. San Francisco, CA, May 12th, 2017. [Moderated Poster]
Skill Assessment of Surgeons Performing Robot-Assisted Radical Prostatectomy: Relationship Between Crowdsourced Review
Ghani KR, Comstock B, Miller DC, Dunn RL, Kim T, Linsell S, Lane BR, Sarle R, Lendvay T, Montie J, Peabody JO, The Michigan Urological Surgery Improvement Collaborative. Skill Assessment of Surgeons Performing Robot-Assisted Radical Prostatectomy: Relationship Between Crowdsourced Review. American Urological Association National Meeting. Boston MA USA May 2017. [BEST ABSTRACT - PLENARY]
American Association of Genitourinary Surgeons
McDougal E, Ghani K, Peabody J, Lendvay TS. Crowd-Sourced Assessment of Surgical Skills. American Association or Genitourinary Surgeons (AAGUS), April, 2017. [Accepted Abstract]
2016
Featured
Journal of Endourology
Kowalewski T, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, McDougall E, Lendvay TS. Correlation of High-Volume Crowdsourced Evaluation and Tool Motion Metrics for Surgical Skills Assessment: The Basic Laparoscopic Urologic Skills (BLUS) Validation Project. Journal of Endourology [in Review, 2016].
American Journal Of Obstetrics And Gynecology
Polin MR, Siddiqui NY, Comstock BA, Hesham H, Brown C, Lendvay TS, Martino MA. Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. American Journal Of Obstetrics And Gynecology. 2016 Nov 30;215(5):644-e1.
Journal of Endourology
Vernez SL, Huynh V, Osann K, Okhunov Z, Landman J, Clayman RV. Assessing Surgical Skills Among Urology Resident Applicants: Can Crowdsourcing Identify the Next Generation of Surgeons? Journal of Endourology. 2016 Oct 11.
Crowdsourcing as a Novel Method to Evaluate Aesthetic Outcomes of Treatment for Unilateral Cleft Lip
Plastic and Reconstructive Surgery
Raymond WT, Oh E, Gruss JS, Hopper RA, Birgfeld CB. Crowdsourcing as a Novel Method to Evaluate Aesthetic Outcomes of Treatment for Unilateral Cleft Lip. Plastic and Reconstructive Surgery. 2016 Oct 1;138(4):864-74.
Journal of Medical Devices
Schaffhausen CR, Kowalewski TM, Sweet R. Crowdsourcing Unmet Clinical Needs in Minimally Invasive Surgery. Journal of Medical Devices. 2016 Sep 1;10(3):030920.
Surgical Laparoscopy Endoscopy & Percutaneous Techniques
Katz AJ. The Role of Crowdsourcing in Assessing Surgical Skills. Surgical Laparoscopy Endoscopy & Percutaneous Techniques. 2016 Aug 1;26(4):271-7
The Journal of Urology
Kowalewski TM, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Lendvay TS. Crowd-sourced assessment of technical skills for validation of basic laparoscopic urologic skills tasks. The Journal of Urology. 2016 Jun 30;195(6):1859-65.
Canadian Urological Association Meeting
McDougall E, Kowalewski T, Sweet R, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J, Knudsen B, Landman J, Lee B, Schwartz B, Comstock B, Schaffhausen C, Lendvay T. Crowd-Sourced Assessment of Technical Skills for Validation of Basic Laparoscopic Urologic Skills (BLUS) Tasks. Canadian Urological Association Meeting, Vancouver, British Columbia, June 25th-28th, 2016. [Moderated Poster]
AUA Annual Meeting
Lendvay TS, Ghani KR, Peabody JO, Miller DC, Comstock B. Is Crowdsourcing Surgical Skill Assessment Reliable? An Analysis of Robotic Prostatectomies. AUA Meeting, Boston, MA, May, 2016. [PODIUM].
AUA Annual Meeting
Ghani K, Guru K, Lane B, Sarle R, Linsell S, Brachulis A, Miller D, Comstock B, Lendvay T, Peabody J. Variation in the Technical Skill of Surgeons Performing Robot-Assisted Radical Prostatectomy. AUA Annual Meeting San Diego, CA, May, 2016. [Moderated Poster].
2015
Featured
AUA Annual Meeting
Ghani K, Miller D, Lane B, Sarle R, Brachulis A, Linsell S, Kim T-K, Delela D, Monti J, Comstock B, Lendvay T, Peabody J. Video-based peer review of robot-assisted radical prostatectomy in statewide quality improvement collaborative. AUA Annual Meeting, New Orleans, LA, May, 2015. [Podium]
Japanese Society of Endoscopic Surgeons
Plenary Session, Crowdsourcing to Assess Surgical Skill, Osaka, Japan, Dec 9th, 2015. [JAPAN]
American Association of Gynecologic Laparoscopists Annual Meeting
Polin MR, Siddiqui NY, Brown C, Hesham H, Comstock B, Lendvay TS, Martino MA. Crowdsourcing: A Valid Alternative to Expert Evaluation of Robotic Surgery Skills. American Association of Gynecologic Laparoscopists Annual Meeting, Las Vegas, Nevada, November 2015 [Podium].
Polin MR, Siddiqui NY, Brown C, Hesham H, Comstock BA, Lendvay TS, Martino MA. Crowdsourcing: A Valid Alternative to Expert Evaluation of Robotic Surgery Skills. Journal of Minimally Invasive Gynecology. 2015 Nov 1;22(6):S19-20.
JAMA Surgery
Lendvay TS, White L, Kowalewski T. Crowdsourcing to assess surgical skill.
JAMA surgery. 2015 Nov 1;150(11):1086-7.
Journal of Endourology
White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS. Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. Journal of Endourology. 2015 Nov 1;29(11):1295-301.
Polin MR, Siddiqui NY, Brown C, Hesham H, Comstock B, Lendvay TS, Martino MA. Crowdsourcing: A Valid Alternative to Expert Evaluation of Robotic Surgery Skills. AUGS Annual Meeting, Seattle, WA, October 2015 [Podium].
Johns Hopkins University Urology Grand Rounds, Crowdsourcing to Assess Surgical Skill, Baltimore, MD, October 8th, 2015.
Lehigh Valley Medical Center General Surgery Grand Rounds, Crowdsourcing to Assess Surgical Skill, Allentown, PA, October 6th, 2015.
Fundamentals of Laparoscopic Surgery Executive Committee, American College of Surgeons Annual Congress, Crowdsourcing to Assess Surgical Skill, Chicago, IL, October 5th, 2015.
2014
Featured
Surgery already takes lessons from the airline industry and can take similar lessons from professional sport with regards to training. The relationship of a coach and elite sportsman is similar in many respects to that of trainer and trainee. As surgical technology advances, so must the teaching tools and strategies to train the next generation of surgeon.
Chand, M., et al. Evolution in surgical training: what can we learn from professional coaches and elite athletes? J R Soc Med. 2014 Jul; 107(7): 290–292. doi: 10.1177/0141076814532395
North Pacific Surgical Association Annual Meeting, Using the Wisdom of Crowds to Assess Surgical Skills, Seattle, WA, November 17, 2014.
White LW, Wright A, Bekele A, Kotisso B, Tadesse A, Oelschlager B, Holst D, Borbely Y, Oyetunji L, Lendvay TS. Using Crowd-Assessment to Support Surgical Training in the Developing World. American College of Surgeons Clinical Congress, San Francisco, CA, October 26-30, 2014. [Podium]
Lendvay TS, White LW, Holst D, Kowalewski T, Harper JD, Sorenson M, Brand TC, Truong M, Simpson K, Smith R. Quantifying Surgical Skill Using the Wisdom of Crowds. American College of Surgeons Clinical Congress, San Francisco, CA, October 26-30, 2014. [Poster #PP2014-51161].
Aghdasi N, Bly R, White L, Hannaford B, Moe K, Lendvay T. Crowd-sourced assessment of surgical skills in cricothyroidotomy procedure. IEEE Global Humanitarian Conference, San Jose, CA, October 10-13, 2014.
Michigan Bariatric Surgery Collaborative Annual Meeting, Using the Wisdom of Crowds for Surgical Skills Assessment, Ann Arbor, Michigan, October 3, 2014.
Human Cyber-Physical Interfaces Annual Conference, What Does the Surgeon Need? Challenges in Today’s OR, Paris, France, Sept 23, 2014. [FRANCE]
Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, Bodenstedt S. Crowdsourcing for reference correspondence generation in endoscopic images. In MICCAI (2) 2014 Sep 14 (pp. 349-356).
M2CAI Annual Meeting, Key-Note Lecturer, Open Challenges in Today’s OR: Skill, Evaluation, Error Reduction, Boston, MA, Sept 14, 2014.
A properly-sized and qualified crowd can accurately score laparoscopic skill performances on par with faculty experts. Crowd-based ratings may be an efficient method for assessing passing/failing performances, and for measuring change in performance after training.
Lendvay, T., et al.. Crowd-sourced assessment of technical skills (C-SATS): Validation through the basic laparoscopic urologic surgery (BLUS) curriculum - Interview. 32nd World Congress of Endourology & SWL - September 3 - 7, 2014 - Taipei, Taiwan.
2013-2012
Featured
The technical skill of practicing bariatric surgeons varied widely, and greater skill was associated with fewer postoperative complications and lower rates of reoperation, readmission, and visits to the emergency department. Although these findings are preliminary, they suggest that peer rating of operative skill may be an effective strategy for assessing a surgeon's proficiency.
Birkmeyer, J.D., et al. for the Michigan Bariatric Surgery Collaborative, Surgical Skill and Complication Rates after Bariatric Surgery The New England Journal of Medicine. Published online October 10, 2013. N Engl J Med 2013; 369:1434-1442
We discovered that crowd-sourced surgery-naïve graders provided assessments equivalent to trained expert surgeon evaluators, and at a faster rate.
Chen C., et al. Crowd-Sourced Assessment of Technical Skills (C-SATS): Faculty Experts vs. Amazon.com Mechanical Turk Project™ vs. Facebook™. University of Washington School of Medicine, 2012. Medical Student III Poster Session.