pytranscripts

An Open source👨‍🔧 Python Library for Automated classification of Electronic Medical records

Installation

To install the latest version , simply use

pip install -U pytranscripts

Stages

1. Data Extraction
2. Target Identification
3. Finetuning Annotated Data on Pretrained models (Bert & Electra)
4. Extracting Interviwer/Interviewee records from the specified docx file storage
5. Metrics Evaluation (Accuracy & Cohen Kappa Score)
6. Reordering records as a neatly arranged and flagged spreadsheet, alongside metrics and reports from pretrained models.
7. Running inference on raw documents and Color Coding them

Example Usage

Mount Google Drive (Optional)

If using Google Drive as the data source:

from google.colab import drive
drive.mount('/content/drive')

Automated Data Export

To export and combine all .docx files from a folder into a single file:

from pytranscripts import export_docx_from_folder

# Define paths for document processing
INPUT_FOLDER = "/content/drive/MyDrive/Kalu+Deola/PYTRANSCRIPTS PUBLIC DEMO/synthetic_transcripts"  # Folder containing source DOCX files

OUTPUT_FILE = "FULL_INTERVIEW.csv"  # Output consolidated spreadsheet (either .csv or .excel)

LABELS = [
    "Clinical_Experience",        # Descriptions of personal experience using lung ultrasound
    "Diagnostic_Utility",         # How lung ultrasound helps in diagnosing diseases
    "Comparative_Analysis",       # Comparisons with other imaging modalities like X-ray or CT
    "Implementation_Challenges",  # Barriers to adoption and practical difficulties
    "Training_and_Education",     # Aspects related to learning and teaching lung ultrasound
    "System_Infrastructure",      # How hospital systems, devices, and software support ultrasound use
    "Administrative_Buying",       # Role of hospital leadership and institutional support
    "Workflow_Impact",            # How lung ultrasound affects daily hospital operations
    "Patient_Engagement",         # Ways ultrasound enhances patient understanding and involvement
    "Future_Adoption",            # Predictions about the role of lung ultrasound in hospital practice
]  # Labels to be used for the columns in the output spreadsheet to be filled up with empty 0s



#-------------------------------------        PLEASE NOTE         --------------------------------------

# AS YOU SELECT YOUR PREDEFINED LIST OF   LABELS ABOVE THEY SHOULD BE  SAME ONE YOU WOULD PASS INTO YOUR "TranscriptTrainer"

# Export and combine all DOCX files from the input folder
# This function will:
# 1. Read all .docx files from INPUT_FOLDER
# 2. Combine their contents
# 3. Save to a single OUTPUT_FILE

export_docx_from_folder(
    input_directory=INPUT_FOLDER,
    output_file=OUTPUT_FILE,
    labels = LABELS
)

This will:

• Read all .docx files from INPUT_FOLDER.
• Combine their content into a single file.
• Apply the defined labels to create a structured dataset.

Requirements

Python 3.6 or later GPU access recommended for optimal performance (if using Jupyter Notebook). pytranscripts version 1.2.4 or higher.

Model Training

Now , the detailed class shows how to properly use our transcript trainer in making training and inference easy based on your document

from pytranscripts import TranscriptTrainer


trainer = TranscriptTrainer(
    input_file='/content/drive/MyDrive/Kalu+Deola/PYTRANSCRIPTS PUBLIC DEMO/FULL_INTERVIEW_TAGGED.xlsx',  # Path to the CSV / XLSX file containing the tagged documents. This is the main data source for training and evaluation.

    destination_path='/content/',  # Directory where all the training results, models, and logs will be saved. , We are using colab path to make things seamless

    text_column='Interviewee',  # Specifies the column name in the CSV file that contains the text data to be used for training.

    test_size=0.2,  # Determines the fraction of the data that will be used for testing the model, instead of training it. Here, 20% of data will be used for testing.

    max_length=512, #The maximum number of tokens to include in each input sequence, this helps in managing memory and computational resources. Sequences longer than this will be truncated.

    num_train_epochs=10, # The number of times the model will iterate over the entire training dataset during training. More epochs will mean more training.

    labels=[
    "Clinical_Experience",        # Descriptions of personal experience using lung ultrasound
    "Diagnostic_Utility",         # How lung ultrasound helps in diagnosing diseases
    "Comparative_Analysis",       # Comparisons with other imaging modalities like X-ray or CT
    "Implementation_Challenges",  # Barriers to adoption and practical difficulties
    "Training_and_Education",     # Aspects related to learning and teaching lung ultrasound
    "System_Infrastructure",      # How hospital systems, devices, and software support ultrasound use
    "Administrative_Buying",       # Role of hospital leadership and institutional support
    "Workflow_Impact",            # How lung ultrasound affects daily hospital operations
    "Patient_Engagement",         # Ways ultrasound enhances patient understanding and involvement
    "Future_Adoption",            # Predictions about the role of lung ultrasound in hospital practice
    ],


     # PLEASE MAKE SURE THAT THE LIST YOU ARE GOING TO BE USING HERE MATCHES THE ONE IN YOUR INPUT FILE


    upper_lower_mapping = {
    "multi_level_org_char": [  # High-level category
        "Clinical_Experience",  # Provider Characteristics
        "System_Infrastructure"  # Health System Characteristics
    ],

    "multi_level_org_perspect": [  # High-level category
        "Comparative_Analysis",  # Imaging modalities in general
        "Administrative_Buying",  # Value equation
        "Diagnostic_Utility",  # Clinical utility & efficiency-Provider perspective
        "Patient_Engagement",  # Patient/Physician interaction in LUS
        "Workflow_Impact"  # Workflow related problems
    ],

    "impl_sust_infra": [  # High-level category
        "Training_and_Education",  # Training
        "Implementation_Challenges",  # Credentialing / Quality Assurance Infrastructure
        "Future_Adoption"  # Financial Impact
    ]
}
)

Next, we initialize the training job using a single line of code.

bert_model, electra_model = trainer.train_and_classify()

Inferencing and Automated Document Classification (via Color Coding)

This involves making use of any of the trained models to predict on a folder containing raw EHR transcripts

trainer.inference_documents(
    input_folder = "/content/drive/MyDrive/Kalu+Deola/PYTRANSCRIPTS PUBLIC DEMO/synthetic_transcripts",
    output_folder = "/content/drive/MyDrive/Kalu+Deola/PYTRANSCRIPTS PUBLIC DEMO/inferenced_transcripts",
    threshold  = 0.15, # default value = 0.15
    model_type = 'bert' # defaults to bert ,  options (bert, electra)
)

Contributing

We welcome contributions! Please follow the contributing guidelines.

License

This project is licensed under the MIT License. See the LICENSE file for details.