{"id":5280,"date":"2024-09-26T10:00:53","date_gmt":"2024-09-26T14:00:53","guid":{"rendered":"https:\/\/cytel.agencyukdev.com\/?post_type=perspectives&p=5280"},"modified":"2024-10-08T16:36:22","modified_gmt":"2024-10-08T20:36:22","slug":"automating-log-checks-a-case-for-innovation-in-statistical-programming","status":"publish","type":"perspectives","link":"https:\/\/cytel.agencyukdev.com\/fr\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming\/","title":{"rendered":"Automating Log Checks: A Case for Innovation in Statistical Programming"},"content":{"rendered":"

Innovation in statistical programming is essential for the evolving clinical development landscape, where the demands of managing and analyzing large datasets is growing exponentially. As clinical trials become more data-intensive and require faster timelines, outdated manual processes and inefficiencies in statistical programming workflows can cause costly delays and even impact the integrity of trial results.<\/p>\n

One of the biggest challenges faced by statistical programmers is managing the large volume of outputs \u2014 tables, graphs, and listings \u2014 generated during trials, particularly in later phases. A few years ago, I faced a similar challenge that many programmers dealt with then and continue to struggle with today \u2014 the tedious and error-prone process of manually checking log files. To address this, I developed an automated solution<\/a> that significantly improved the speed and accuracy of the process.<\/p>\n

In this blog, I share how this simple, yet innovative solution solved the problem of manual log checks.<\/p>\n

<\/p>\n

<\/h2>\n

The Challenge<\/h2>\n

As part of Cytel\u2019s Functional Service Provider engagement with a large biotech<\/a> company, I led a team of statistical programmers working on a Phase 4 immunology trial. The project had a tight deadline, requiring us to deliver 150 to 200 outputs within a couple of days. We used a variety of macros to produce these outputs, from box plots and Kaplan-Meier (KM) plots to efficacy tables.<\/p>\n

Before submitting a task, each log had to be manually checked for 1) errors, 2) warnings, 3) uninitialized variables, 4) MERGE statements with repeated BY values, 5) character to numeric conversions, and 6) numeric to character conversions. If any issues were found in the logs, the code had to be updated and rerun until the logs were clean.<\/p>\n

This manual process quickly became tedious and time-consuming. Moreover, it was prone to error as there was always the risk of missing specific terms in the review cycle. Recognizing the inefficiency of this approach, I began brainstorming a more effective solution, ultimately creating an automated log-checking process.<\/p>\n

 <\/p>\n

The Solution<\/h2>\n

After a couple of weeks of trial and error, I created a code that automated the log-checking process. The code was designed to automatically scan the logs for:<\/p>\n

    \n
  • Errors<\/li>\n
  • Warnings<\/li>\n
  • Uninitialized values<\/li>\n
  • MERGE statement that has more than one data set with repetitions of BY values<\/li>\n
  • Character values that have been converted to numeric values<\/li>\n
  • Numeric values that have been converted to character values<\/li>\n
  • Missing values that were generated as a result of performing an operation on missing values<\/li>\n<\/ul>\n

     <\/p>\n

    The syntax is as follows:<\/p>\n

    options noxwait; <\/strong><\/p>\n

    X \u2018Findstr\/i \u201cerror warning repeats\u201d input-path\\progname.log >output-path\\log_check.txt\u2019; <\/strong><\/p>\n

    X \u2018output-path\\log_check.txt\u2019; <\/strong><\/p>\n

     <\/p>\n

    First Command: <\/strong><\/p>\n

    options noxwait; <\/strong><\/p>\n

    This command makes the Command Prompt window close automatically without your having to type EXIT when the process is finished. <\/em><\/p>\n

    Second Command: <\/strong><\/p>\n

    X \u2018Findstr\/i \u201cerror warning repeats\u201d input-path\\progname.log >output-path\\log_check.txt\u2019; <\/strong><\/p>\n

    This command searches for specific words in double quotes within the log files located on the input-path and records them in the text file on the output-path. <\/em><\/p>\n

    X:<\/strong> Enables you to enter UNIX commands without ending the SAS session<\/p>\n

    FINDSTR <\/strong>(short for \u201cfind string\u201d):<\/strong> Locates files containing a specific string of plain text<\/p>\n

    \/i:<\/strong> Specifies that the search is not case-sensitive<\/p>\n

    \u201cerror warning repeats\u201d:<\/strong> Words that need to be searched in log files<\/p>\n

    input-path\\progname.log:<\/strong> Input path and name of logs. You can specify a regular expression *.log to check all log files or abc_xyz_*.log for similar types of logs with different subtypes.<\/p>\n

    > output-path\\log_check.txt:<\/strong> Redirects the output of the program to output-path\\log_check.txt<\/p>\n

    Third Command: <\/strong><\/p>\n

    X \u2018output-path\\log_check.txt\u2019;\u00a0 <\/strong><\/p>\n

    This command opens the text file on the output-path, which lists the specific words mentioned in double quotes. <\/em><\/p>\n

    If the log check file is empty, it means no issues were detected and no further action is needed. If issues are found, the file lists the program name followed by lines from the log check files that contain any of the words mentioned in the second command. Once the issues are resolved, the batch codes are rerun, and the process is repeated until the log check file is empty. The goal is to reach a stage where no issues are detected, resulting in an empty log check file.<\/p>\n

     <\/p>\n

    Outcome<\/h2>\n

    Despite the effort involved in manually verifying the logs files, we were never sure that we had caught everything. Fortunately, with just three commands, my code eliminated the need to spend time manually checking logs once it was integrated into our batch files. The code, a clean and concise MS-DOS script, is far simpler than the existing complex SAS solutions that achieve similar objectives. Using code to automate the process, we not only saved time but also improved the accuracy and reliability of our outputs. It can be used as a macro or snippet in the program, offering flexibility for different workflows. \u00a0With this automated process in place, we consistently saw clean log check files, boosting our confidence in the final results.<\/p>\n

     <\/p>\n

    Final Takeaways<\/h2>\n

    With the growing emphasis on innovation, it is crucial to implement practical, user-friendly solutions that meet the evolving needs of clinical trial data analysis. While more sophisticated tools and SAS platforms with built-in macros have emerged in recent years, many of them are overly complex. My code offers a simple alternative that can be used within macros to automate log checks. It\u2019s a flexible, adaptable solution that continues to benefit statistical programmers today.<\/p>","protected":false},"featured_media":3722,"parent":0,"template":"","meta":{"_acf_changed":false},"tags":[76],"by-topic":[437,459],"by-year":[74],"class_list":["post-5280","perspectives","type-perspectives","status-publish","has-post-thumbnail","hentry","tag-blog","by-topic-fsp","by-topic-statistical-programming","by-year-74"],"acf":[],"yoast_head":"\nAutomating Log Checks: A Case for Innovation in Statistical Programming - Clinical Trial Software & Data Analysis | Cytel | Contact Us<\/title>\n<meta name=\"description\" content=\"Manual log checking can be inefficient and error-prone. Discover an innovative solution that helps overcome this challenge.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/cytel.agencyukdev.com\/fr\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming\/\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Automating Log Checks: A Case for Innovation in Statistical Programming - Clinical Trial Software & Data Analysis | Cytel | Contact Us\" \/>\n<meta property=\"og:description\" content=\"Manual log checking can be inefficient and error-prone. Discover an innovative solution that helps overcome this challenge.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/cytel.agencyukdev.com\/fr\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming\/\" \/>\n<meta property=\"og:site_name\" content=\"Clinical Trial Software & Data Analysis | Cytel | Contact Us\" \/>\n<meta property=\"article:modified_time\" content=\"2024-10-08T20:36:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cytel.agencyukdev.com\/wp-content\/uploads\/2024\/05\/iStock-1438242735_abstract.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1732\" \/>\n\t<meta property=\"og:image:height\" content=\"1732\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming\\\/\",\"url\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming\",\"name\":\"Automating Log Checks: A Case for Innovation in Statistical Programming - Clinical Trial Software & Data Analysis | Cytel | Contact Us\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/wp-content\\\/uploads\\\/2024\\\/05\\\/iStock-1438242735_abstract.jpg\",\"datePublished\":\"2024-09-26T14:00:53+00:00\",\"dateModified\":\"2024-10-08T20:36:22+00:00\",\"description\":\"Manual log checking can be inefficient and error-prone. Discover an innovative solution that helps overcome this challenge.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming#breadcrumb\"},\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"fr-FR\",\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming#primaryimage\",\"url\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/wp-content\\\/uploads\\\/2024\\\/05\\\/iStock-1438242735_abstract.jpg\",\"contentUrl\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/wp-content\\\/uploads\\\/2024\\\/05\\\/iStock-1438242735_abstract.jpg\",\"width\":1732,\"height\":1732,\"caption\":\"Abstract Modern Background\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/perspectives\\\/automating-log-checks-a-case-for-innovation-in-statistical-programming#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Automating Log Checks: A Case for Innovation in Statistical Programming\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/#website\",\"url\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/\",\"name\":\"Clinical Trial Software & Data Analysis | Cytel | Contact Us\",\"description\":\"Innovative Trial Design. Cytel is the largest provider of statistical software and advanced analytics for clinical trial design and execution.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/cytel.agencyukdev.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"fr-FR\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Automating Log Checks: A Case for Innovation in Statistical Programming - Clinical Trial Software & Data Analysis | Cytel | Contact Us","description":"Manual log checking can be inefficient and error-prone. Discover an innovative solution that helps overcome this challenge.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/cytel.agencyukdev.com\/fr\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming\/","og_locale":"fr_FR","og_type":"article","og_title":"Automating Log Checks: A Case for Innovation in Statistical Programming - Clinical Trial Software & Data Analysis | Cytel | Contact Us","og_description":"Manual log checking can be inefficient and error-prone. Discover an innovative solution that helps overcome this challenge.","og_url":"https:\/\/cytel.agencyukdev.com\/fr\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming\/","og_site_name":"Clinical Trial Software & Data Analysis | Cytel | Contact Us","article_modified_time":"2024-10-08T20:36:22+00:00","og_image":[{"width":1732,"height":1732,"url":"https:\/\/cytel.agencyukdev.com\/wp-content\/uploads\/2024\/05\/iStock-1438242735_abstract.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Dur\u00e9e de lecture estim\u00e9e":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming\/","url":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming","name":"Automating Log Checks: A Case for Innovation in Statistical Programming - Clinical Trial Software & Data Analysis | Cytel | Contact Us","isPartOf":{"@id":"https:\/\/cytel.agencyukdev.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming#primaryimage"},"image":{"@id":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming#primaryimage"},"thumbnailUrl":"https:\/\/cytel.agencyukdev.com\/wp-content\/uploads\/2024\/05\/iStock-1438242735_abstract.jpg","datePublished":"2024-09-26T14:00:53+00:00","dateModified":"2024-10-08T20:36:22+00:00","description":"Manual log checking can be inefficient and error-prone. Discover an innovative solution that helps overcome this challenge.","breadcrumb":{"@id":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming#breadcrumb"},"inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming"]}]},{"@type":"ImageObject","inLanguage":"fr-FR","@id":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming#primaryimage","url":"https:\/\/cytel.agencyukdev.com\/wp-content\/uploads\/2024\/05\/iStock-1438242735_abstract.jpg","contentUrl":"https:\/\/cytel.agencyukdev.com\/wp-content\/uploads\/2024\/05\/iStock-1438242735_abstract.jpg","width":1732,"height":1732,"caption":"Abstract Modern Background"},{"@type":"BreadcrumbList","@id":"https:\/\/cytel.agencyukdev.com\/perspectives\/automating-log-checks-a-case-for-innovation-in-statistical-programming#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/cytel.agencyukdev.com\/"},{"@type":"ListItem","position":2,"name":"Automating Log Checks: A Case for Innovation in Statistical Programming"}]},{"@type":"WebSite","@id":"https:\/\/cytel.agencyukdev.com\/#website","url":"https:\/\/cytel.agencyukdev.com\/","name":"Clinical Trial Software & Data Analysis | Cytel | Contact Us","description":"Innovative Trial Design. Cytel is the largest provider of statistical software and advanced analytics for clinical trial design and execution.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/cytel.agencyukdev.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"fr-FR"}]}},"_links":{"self":[{"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/perspectives\/5280","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/perspectives"}],"about":[{"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/types\/perspectives"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/media\/3722"}],"wp:attachment":[{"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/media?parent=5280"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/tags?post=5280"},{"taxonomy":"by-topic","embeddable":true,"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/by-topic?post=5280"},{"taxonomy":"by-year","embeddable":true,"href":"https:\/\/cytel.agencyukdev.com\/fr\/wp-json\/wp\/v2\/by-year?post=5280"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}